2024-04-14T23:34:51.954 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-14T23:34:51.954 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/teuthology-2024-04-14_21:24:03-fs-squid-distro-default-smithi/7656585 --name teuthology-2024-04-14_21:24:03-fs-squid-distro-default-smithi --owner scheduled_teuthology@teuthology --verbose --description fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} -- /home/teuthworker/archive/teuthology-2024-04-14_21:24:03-fs-squid-distro-default-smithi/7656585/orig.config.yaml 2024-04-14T23:34:52.003 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-14T23:34:52.098 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-14_21:24:03-fs-squid-distro-default-smithi/7656585 branch: squid description: fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7656585' kernel: client: branch: testing kdb: 1 sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: teuthology-2024-04-14_21:24:03-fs-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: cephfs: session_timeout: 300 conf: client: client mount timeout: 600 client_shutdown_timeout: 120 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds oft prefetch dirfrags: true mds op complaint time: 180 mds verify scatter: true mds_dir_prefetch: false mds_sleep_rank_change: 5000000.0 mon client directed command retry: 5 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: client mds osd ms inject socket failures: 2500 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon client directed command retry: 5 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: client mds ms inject socket failures: 2500 osd blocked scrub grace period: 3600 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - Replacing daemon mds sha1: a9a752df26c63acad72e1b3569fd79a515ca0765 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: a9a752df26c63acad72e1b3569fd79a515ca0765 kclient: syntax: v1 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: a9a752df26c63acad72e1b3569fd79a515ca0765 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.y - mds.a - mds.c - mds.e - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - mds.d - mds.f - osd.4 - osd.5 - osd.6 - osd.7 seed: 840 sha1: a9a752df26c63acad72e1b3569fd79a515ca0765 sleep_before_teardown: 0 subset: 427/512 suite: fs suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_a9a752df26c63acad72e1b3569fd79a515ca0765/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: a9a752df26c63acad72e1b3569fd79a515ca0765 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi060.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFjV/pqyaEiSS62epvH8tgHmy+F/PJpE6WM74CJKJ2mX8YiPrJ7j+BVMPex/LpLCEZB9HKVb/OF6pAfvG9ajCuE= smithi186.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLQ6oA9Y2BUpihFO/opwukGeh9/GE1Y47E4ZKJRZd++cP5yuL+qzcFkgQ6J4vfQSLvhNz3d0ALuw0/VE76drTXE= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - sequential: - mgrmodules - kclient: null - mds_thrash: null - workunit: clients: all: - fs/snaps timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 6c637841c215537a4502385240412f1966e0faab timestamp: 2024-04-14_21:24:03 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-14T23:34:52.098 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_a9a752df26c63acad72e1b3569fd79a515ca0765/qa; will attempt to use it 2024-04-14T23:34:52.098 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_a9a752df26c63acad72e1b3569fd79a515ca0765/qa/tasks 2024-04-14T23:34:52.099 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-14T23:34:52.100 INFO:teuthology.task.internal:Checking packages... 2024-04-14T23:34:52.123 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'a9a752df26c63acad72e1b3569fd79a515ca0765' 2024-04-14T23:34:52.124 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-14T23:34:52.124 INFO:teuthology.packaging:ref: None 2024-04-14T23:34:52.124 INFO:teuthology.packaging:tag: None 2024-04-14T23:34:52.124 INFO:teuthology.packaging:branch: squid 2024-04-14T23:34:52.124 INFO:teuthology.packaging:sha1: a9a752df26c63acad72e1b3569fd79a515ca0765 2024-04-14T23:34:52.124 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-04-14T23:34:52.438 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2114.ga9a752df 2024-04-14T23:34:52.440 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-14T23:34:52.448 INFO:teuthology.task.internal:no buildpackages task found 2024-04-14T23:34:52.448 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-14T23:34:52.490 INFO:teuthology.task.internal:Saving configuration 2024-04-14T23:34:52.506 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-14T23:34:52.514 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-14T23:34:52.538 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi060.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-14_21:24:03-fs-squid-distro-default-smithi/7656585', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-14 23:29:25.731194', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFjV/pqyaEiSS62epvH8tgHmy+F/PJpE6WM74CJKJ2mX8YiPrJ7j+BVMPex/LpLCEZB9HKVb/OF6pAfvG9ajCuE='} 2024-04-14T23:34:52.562 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi186.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-14_21:24:03-fs-squid-distro-default-smithi/7656585', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-14 23:29:25.733090', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLQ6oA9Y2BUpihFO/opwukGeh9/GE1Y47E4ZKJRZd++cP5yuL+qzcFkgQ6J4vfQSLvhNz3d0ALuw0/VE76drTXE='} 2024-04-14T23:34:52.562 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-14T23:34:52.570 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-04-14T23:34:52.570 INFO:teuthology.task.internal:roles: ubuntu@smithi186.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2024-04-14T23:34:52.570 INFO:teuthology.run_tasks:Running task console_log... 2024-04-14T23:34:52.658 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f79ae9700d0>, signals=[15]) 2024-04-14T23:34:52.658 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-14T23:34:52.666 INFO:teuthology.task.internal:Opening connections... 2024-04-14T23:34:52.666 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi060.front.sepia.ceph.com 2024-04-14T23:34:52.668 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-14T23:34:52.772 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi186.front.sepia.ceph.com 2024-04-14T23:34:52.773 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-14T23:34:52.858 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-14T23:34:52.866 DEBUG:teuthology.orchestra.run.smithi060:> uname -m 2024-04-14T23:34:52.884 INFO:teuthology.orchestra.run.smithi060.stdout:x86_64 2024-04-14T23:34:52.884 DEBUG:teuthology.orchestra.run.smithi060:> cat /etc/os-release 2024-04-14T23:34:52.941 INFO:teuthology.orchestra.run.smithi060.stdout:NAME="CentOS Stream" 2024-04-14T23:34:52.941 INFO:teuthology.orchestra.run.smithi060.stdout:VERSION="9" 2024-04-14T23:34:52.941 INFO:teuthology.orchestra.run.smithi060.stdout:ID="centos" 2024-04-14T23:34:52.941 INFO:teuthology.orchestra.run.smithi060.stdout:ID_LIKE="rhel fedora" 2024-04-14T23:34:52.941 INFO:teuthology.orchestra.run.smithi060.stdout:VERSION_ID="9" 2024-04-14T23:34:52.941 INFO:teuthology.orchestra.run.smithi060.stdout:PLATFORM_ID="platform:el9" 2024-04-14T23:34:52.941 INFO:teuthology.orchestra.run.smithi060.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-14T23:34:52.941 INFO:teuthology.orchestra.run.smithi060.stdout:ANSI_COLOR="0;31" 2024-04-14T23:34:52.942 INFO:teuthology.orchestra.run.smithi060.stdout:LOGO="fedora-logo-icon" 2024-04-14T23:34:52.942 INFO:teuthology.orchestra.run.smithi060.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-14T23:34:52.942 INFO:teuthology.orchestra.run.smithi060.stdout:HOME_URL="https://centos.org/" 2024-04-14T23:34:52.942 INFO:teuthology.orchestra.run.smithi060.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-14T23:34:52.942 INFO:teuthology.orchestra.run.smithi060.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-14T23:34:52.942 INFO:teuthology.orchestra.run.smithi060.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-14T23:34:52.942 INFO:teuthology.lock.ops:Updating smithi060.front.sepia.ceph.com on lock server 2024-04-14T23:34:52.961 DEBUG:teuthology.orchestra.run.smithi186:> uname -m 2024-04-14T23:34:52.978 INFO:teuthology.orchestra.run.smithi186.stdout:x86_64 2024-04-14T23:34:52.978 DEBUG:teuthology.orchestra.run.smithi186:> cat /etc/os-release 2024-04-14T23:34:53.038 INFO:teuthology.orchestra.run.smithi186.stdout:NAME="CentOS Stream" 2024-04-14T23:34:53.038 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION="9" 2024-04-14T23:34:53.038 INFO:teuthology.orchestra.run.smithi186.stdout:ID="centos" 2024-04-14T23:34:53.039 INFO:teuthology.orchestra.run.smithi186.stdout:ID_LIKE="rhel fedora" 2024-04-14T23:34:53.039 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION_ID="9" 2024-04-14T23:34:53.039 INFO:teuthology.orchestra.run.smithi186.stdout:PLATFORM_ID="platform:el9" 2024-04-14T23:34:53.039 INFO:teuthology.orchestra.run.smithi186.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-14T23:34:53.039 INFO:teuthology.orchestra.run.smithi186.stdout:ANSI_COLOR="0;31" 2024-04-14T23:34:53.039 INFO:teuthology.orchestra.run.smithi186.stdout:LOGO="fedora-logo-icon" 2024-04-14T23:34:53.039 INFO:teuthology.orchestra.run.smithi186.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-14T23:34:53.039 INFO:teuthology.orchestra.run.smithi186.stdout:HOME_URL="https://centos.org/" 2024-04-14T23:34:53.039 INFO:teuthology.orchestra.run.smithi186.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-14T23:34:53.039 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-14T23:34:53.039 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-14T23:34:53.039 INFO:teuthology.lock.ops:Updating smithi186.front.sepia.ceph.com on lock server 2024-04-14T23:34:53.061 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-14T23:34:53.071 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-14T23:34:53.114 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-14T23:34:53.114 DEBUG:teuthology.orchestra.run.smithi060:> test '!' -e /home/ubuntu/cephtest 2024-04-14T23:34:53.116 DEBUG:teuthology.orchestra.run.smithi186:> test '!' -e /home/ubuntu/cephtest 2024-04-14T23:34:53.131 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-14T23:34:53.147 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-14T23:34:53.147 DEBUG:teuthology.orchestra.run.smithi060:> test -z $(ls -A /var/lib/ceph) 2024-04-14T23:34:53.173 DEBUG:teuthology.orchestra.run.smithi186:> test -z $(ls -A /var/lib/ceph) 2024-04-14T23:34:53.277 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-14T23:34:53.333 INFO:teuthology.run_tasks:Running task kernel... 2024-04-14T23:34:53.348 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': 1, 'sha1': 'distro'} 2024-04-14T23:34:53.348 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2024-04-14T23:34:53.348 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2024-04-14T23:34:53.348 DEBUG:teuthology.orchestra.run.smithi060:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-14T23:34:53.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T23:34:53.366 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-04-14T23:34:53.633 DEBUG:teuthology.task.kernel:sha1 for client.0 is 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 2024-04-14T23:34:53.634 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8"... 2024-04-14T23:34:53.634 DEBUG:teuthology.orchestra.run.smithi060:> uname -r 2024-04-14T23:34:53.651 INFO:teuthology.orchestra.run.smithi060.stdout:5.14.0-361.el9.x86_64 2024-04-14T23:34:53.651 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 2024-04-14T23:34:53.652 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-04-14T23:34:53.652 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum upgrade -y linux-firmware 2024-04-14T23:34:55.960 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS-9 - AppStream 13 MB/s | 8.1 MB 00:00 2024-04-14T23:34:57.653 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS-9 - BaseOS 8.8 MB/s | 2.5 MB 00:00 2024-04-14T23:34:58.669 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-04-14T23:35:00.445 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - AppStream 30 MB/s | 19 MB 00:00 2024-04-14T23:35:05.105 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - CRB 7.0 MB/s | 6.2 MB 00:00 2024-04-14T23:35:06.789 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - Extras packages 36 kB/s | 16 kB 00:00 2024-04-14T23:35:09.724 INFO:teuthology.orchestra.run.smithi060.stdout:Extra Packages for Enterprise Linux 7.5 MB/s | 21 MB 00:02 2024-04-14T23:35:15.057 INFO:teuthology.orchestra.run.smithi060.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-04-14T23:35:17.402 INFO:teuthology.orchestra.run.smithi060.stderr:Modular dependency problems: 2024-04-14T23:35:17.402 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-04-14T23:35:17.402 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 1: conflicting requests 2024-04-14T23:35:17.402 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.402 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 2: conflicting requests 2024-04-14T23:35:17.402 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.403 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 3: conflicting requests 2024-04-14T23:35:17.403 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.403 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 4: conflicting requests 2024-04-14T23:35:17.403 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.403 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 5: conflicting requests 2024-04-14T23:35:17.403 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.403 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 6: conflicting requests 2024-04-14T23:35:17.403 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.403 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 7: conflicting requests 2024-04-14T23:35:17.403 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.403 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 8: conflicting requests 2024-04-14T23:35:17.403 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.403 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 9: conflicting requests 2024-04-14T23:35:17.403 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.403 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 10: conflicting requests 2024-04-14T23:35:17.404 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.404 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 11: conflicting requests 2024-04-14T23:35:17.404 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.404 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 12: conflicting requests 2024-04-14T23:35:17.404 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.404 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 13: conflicting requests 2024-04-14T23:35:17.404 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.404 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 14: conflicting requests 2024-04-14T23:35:17.404 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.404 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 15: conflicting requests 2024-04-14T23:35:17.404 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.404 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 16: conflicting requests 2024-04-14T23:35:17.405 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.405 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 17: conflicting requests 2024-04-14T23:35:17.405 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.405 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 18: conflicting requests 2024-04-14T23:35:17.405 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.405 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 19: conflicting requests 2024-04-14T23:35:17.405 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.405 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 20: conflicting requests 2024-04-14T23:35:17.405 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.405 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 21: conflicting requests 2024-04-14T23:35:17.405 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.405 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 22: conflicting requests 2024-04-14T23:35:17.405 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.405 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 23: conflicting requests 2024-04-14T23:35:17.406 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.406 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 24: conflicting requests 2024-04-14T23:35:17.406 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.406 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 25: conflicting requests 2024-04-14T23:35:17.406 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.406 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 26: conflicting requests 2024-04-14T23:35:17.406 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.406 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 27: conflicting requests 2024-04-14T23:35:17.406 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.406 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 28: conflicting requests 2024-04-14T23:35:17.406 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.406 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 29: conflicting requests 2024-04-14T23:35:17.406 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.407 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 30: conflicting requests 2024-04-14T23:35:17.407 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.407 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 31: conflicting requests 2024-04-14T23:35:17.407 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.407 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 32: conflicting requests 2024-04-14T23:35:17.407 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.407 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 33: conflicting requests 2024-04-14T23:35:17.407 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.407 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 34: conflicting requests 2024-04-14T23:35:17.407 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.407 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 35: conflicting requests 2024-04-14T23:35:17.407 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.407 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 36: conflicting requests 2024-04-14T23:35:17.407 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.408 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 37: conflicting requests 2024-04-14T23:35:17.408 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.408 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.408 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.408 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 38: cannot install the best candidate for the job 2024-04-14T23:35:17.408 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.408 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.408 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.408 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.408 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 39: conflicting requests 2024-04-14T23:35:17.408 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.408 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.408 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.408 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.408 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.409 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.409 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 40: cannot install the best candidate for the job 2024-04-14T23:35:17.409 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.409 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.409 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.409 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.409 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 41: cannot install the best candidate for the job 2024-04-14T23:35:17.409 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.409 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.409 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.409 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.409 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 42: cannot install the best candidate for the job 2024-04-14T23:35:17.410 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.410 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.410 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.410 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.410 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 43: cannot install the best candidate for the job 2024-04-14T23:35:17.410 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.410 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.410 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.410 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.410 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 44: cannot install the best candidate for the job 2024-04-14T23:35:17.410 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.410 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.410 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.411 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.411 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 45: cannot install the best candidate for the job 2024-04-14T23:35:17.411 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.411 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.411 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.411 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-14T23:35:17.974 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-04-14T23:35:17.985 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-04-14T23:35:17.985 INFO:teuthology.orchestra.run.smithi060.stdout: Package Architecture Version Repository Size 2024-04-14T23:35:17.985 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-04-14T23:35:17.985 INFO:teuthology.orchestra.run.smithi060.stdout:Upgrading: 2024-04-14T23:35:17.985 INFO:teuthology.orchestra.run.smithi060.stdout: linux-firmware noarch 20240219-143.el9 baseos 376 M 2024-04-14T23:35:17.985 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:35:17.986 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-04-14T23:35:17.986 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-04-14T23:35:17.986 INFO:teuthology.orchestra.run.smithi060.stdout:Upgrade 1 Package 2024-04-14T23:35:17.986 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:35:17.986 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 376 M 2024-04-14T23:35:17.986 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-04-14T23:35:27.625 INFO:teuthology.orchestra.run.smithi060.stdout:linux-firmware-20240219-143.el9.noarch.rpm 40 MB/s | 376 MB 00:09 2024-04-14T23:35:27.625 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-04-14T23:35:27.625 INFO:teuthology.orchestra.run.smithi060.stdout:Total 39 MB/s | 376 MB 00:09 2024-04-14T23:35:30.815 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-04-14T23:35:30.908 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-04-14T23:35:30.909 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-04-14T23:35:34.458 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-04-14T23:35:34.458 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-04-14T23:35:41.925 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-04-14T23:35:42.268 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : linux-firmware-20240219-143.el9.noarch 1/2 2024-04-14T23:35:42.364 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-04-14T23:35:43.622 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-04-14T23:35:43.622 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 1/2 2024-04-14T23:35:44.086 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-04-14T23:35:44.086 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:35:44.086 INFO:teuthology.orchestra.run.smithi060.stdout:Upgraded: 2024-04-14T23:35:44.086 INFO:teuthology.orchestra.run.smithi060.stdout: linux-firmware-20240219-143.el9.noarch 2024-04-14T23:35:44.087 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:35:44.087 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-04-14T23:35:44.388 INFO:teuthology.task.kernel:Downloading kernel 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 on client.0... 2024-04-14T23:35:44.388 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 2024-04-14T23:35:44.538 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8/centos/9/flavors/default/x86_64/ 2024-04-14T23:35:44.538 DEBUG:teuthology.orchestra.run.smithi060:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.9.0_rc3_g3f60ed14f7c1-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8/centos/9/flavors/default/x86_64/ --input-file=- 2024-04-14T23:35:44.541 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi060.front.sepia.ceph.com to complete... 2024-04-14T23:35:56.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14 23:35:56 URL:https://3.chacra.ceph.com/r/kernel/testing/3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8/centos/9/flavors/default/x86_64/kernel-6.9.0_rc3_g3f60ed14f7c1-1.x86_64.rpm [357344123/357344123] -> "/tmp/kernel.x86_64.rpm" [1] 2024-04-14T23:35:56.707 INFO:teuthology.orchestra.run.smithi060.stderr:FINISHED --2024-04-14 23:35:56-- 2024-04-14T23:35:56.707 INFO:teuthology.orchestra.run.smithi060.stderr:Total wall clock time: 12s 2024-04-14T23:35:56.707 INFO:teuthology.orchestra.run.smithi060.stderr:Downloaded: 1 files, 341M in 12s (28.7 MB/s) 2024-04-14T23:35:56.710 INFO:teuthology.task.kernel:Installing kernel 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 on client.0... 2024-04-14T23:35:56.710 DEBUG:teuthology.orchestra.run.smithi060:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-04-14T23:35:58.869 INFO:teuthology.orchestra.run.smithi060.stdout:Verifying... ######################################## 2024-04-14T23:35:58.933 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing... ######################################## 2024-04-14T23:35:59.017 INFO:teuthology.orchestra.run.smithi060.stdout:Updating / installing... 2024-04-14T23:36:07.309 INFO:teuthology.orchestra.run.smithi060.stdout:kernel-6.9.0_rc3_g3f60ed14f7c1-1 ######################################## 2024-04-14T23:36:24.454 INFO:teuthology.orchestra.run.smithi060.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-14T23:36:48.210 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi060.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-04-14T23:36:48.211 DEBUG:teuthology.orchestra.run.smithi060:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-04-14T23:36:48.289 INFO:teuthology.orchestra.run.smithi060.stdout:/boot/System.map-6.9.0-rc3-g3f60ed14f7c1 2024-04-14T23:36:48.289 INFO:teuthology.orchestra.run.smithi060.stdout:/boot/config-6.9.0-rc3-g3f60ed14f7c1 2024-04-14T23:36:48.289 INFO:teuthology.orchestra.run.smithi060.stdout:/boot/vmlinuz-6.9.0-rc3-g3f60ed14f7c1 2024-04-14T23:36:48.289 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/System.map 2024-04-14T23:36:48.290 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/config 2024-04-14T23:36:48.290 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel 2024-04-14T23:36:48.290 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch 2024-04-14T23:36:48.290 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86 2024-04-14T23:36:48.290 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto 2024-04-14T23:36:48.290 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-04-14T23:36:48.290 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-04-14T23:36:48.290 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-04-14T23:36:48.290 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-04-14T23:36:48.290 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-04-14T23:36:48.290 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-04-14T23:36:48.290 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-04-14T23:36:48.290 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/crc32c-intel.ko 2024-04-14T23:36:48.291 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-04-14T23:36:48.291 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-04-14T23:36:48.291 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-04-14T23:36:48.291 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-04-14T23:36:48.291 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/serpent-avx2.ko 2024-04-14T23:36:48.291 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-04-14T23:36:48.291 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-04-14T23:36:48.291 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-04-14T23:36:48.291 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-04-14T23:36:48.291 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-04-14T23:36:48.291 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-04-14T23:36:48.291 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-04-14T23:36:48.291 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel 2024-04-14T23:36:48.291 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel/cpu 2024-04-14T23:36:48.291 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel/cpu/mce 2024-04-14T23:36:48.292 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-04-14T23:36:48.292 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kvm 2024-04-14T23:36:48.292 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kvm/kvm-amd.ko 2024-04-14T23:36:48.292 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kvm/kvm-intel.ko 2024-04-14T23:36:48.292 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kvm/kvm.ko 2024-04-14T23:36:48.292 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto 2024-04-14T23:36:48.292 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/ansi_cprng.ko 2024-04-14T23:36:48.292 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/anubis.ko 2024-04-14T23:36:48.292 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/arc4.ko 2024-04-14T23:36:48.292 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx 2024-04-14T23:36:48.292 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_memcpy.ko 2024-04-14T23:36:48.292 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_pq.ko 2024-04-14T23:36:48.292 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_raid6_recov.ko 2024-04-14T23:36:48.293 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_tx.ko 2024-04-14T23:36:48.293 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_xor.ko 2024-04-14T23:36:48.293 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/raid6test.ko 2024-04-14T23:36:48.293 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/authenc.ko 2024-04-14T23:36:48.293 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/authencesn.ko 2024-04-14T23:36:48.293 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/blake2b_generic.ko 2024-04-14T23:36:48.293 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/blowfish_common.ko 2024-04-14T23:36:48.293 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/blowfish_generic.ko 2024-04-14T23:36:48.293 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/camellia_generic.ko 2024-04-14T23:36:48.293 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cast5_generic.ko 2024-04-14T23:36:48.293 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cast6_generic.ko 2024-04-14T23:36:48.293 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cast_common.ko 2024-04-14T23:36:48.293 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/ccm.ko 2024-04-14T23:36:48.293 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cmac.ko 2024-04-14T23:36:48.294 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/crc32_generic.ko 2024-04-14T23:36:48.294 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/crypto_engine.ko 2024-04-14T23:36:48.294 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/crypto_user.ko 2024-04-14T23:36:48.294 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cts.ko 2024-04-14T23:36:48.294 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/des_generic.ko 2024-04-14T23:36:48.294 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/dh_generic.ko 2024-04-14T23:36:48.294 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/echainiv.ko 2024-04-14T23:36:48.294 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/essiv.ko 2024-04-14T23:36:48.294 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/fcrypt.ko 2024-04-14T23:36:48.294 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/khazad.ko 2024-04-14T23:36:48.294 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/lz4.ko 2024-04-14T23:36:48.294 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/lz4hc.ko 2024-04-14T23:36:48.294 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/md4.ko 2024-04-14T23:36:48.294 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/michael_mic.ko 2024-04-14T23:36:48.295 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/pcbc.ko 2024-04-14T23:36:48.295 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/pcrypt.ko 2024-04-14T23:36:48.295 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/rmd160.ko 2024-04-14T23:36:48.295 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/seed.ko 2024-04-14T23:36:48.295 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/serpent_generic.ko 2024-04-14T23:36:48.295 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/tcrypt.ko 2024-04-14T23:36:48.295 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/tea.ko 2024-04-14T23:36:48.295 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/twofish_common.ko 2024-04-14T23:36:48.295 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/twofish_generic.ko 2024-04-14T23:36:48.295 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/vmac.ko 2024-04-14T23:36:48.295 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/wp512.ko 2024-04-14T23:36:48.295 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/xcbc.ko 2024-04-14T23:36:48.295 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/xor.ko 2024-04-14T23:36:48.295 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/xxhash_generic.ko 2024-04-14T23:36:48.295 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers 2024-04-14T23:36:48.296 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi 2024-04-14T23:36:48.296 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/acpi_ipmi.ko 2024-04-14T23:36:48.296 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/acpi_pad.ko 2024-04-14T23:36:48.296 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/ec_sys.ko 2024-04-14T23:36:48.296 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/platform_profile.ko 2024-04-14T23:36:48.296 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/sbs.ko 2024-04-14T23:36:48.296 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/sbshc.ko 2024-04-14T23:36:48.296 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/video.ko 2024-04-14T23:36:48.296 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata 2024-04-14T23:36:48.296 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/acard-ahci.ko 2024-04-14T23:36:48.296 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/ahci_platform.ko 2024-04-14T23:36:48.296 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/ata_generic.ko 2024-04-14T23:36:48.296 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/libahci_platform.ko 2024-04-14T23:36:48.296 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_acpi.ko 2024-04-14T23:36:48.296 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_ali.ko 2024-04-14T23:36:48.296 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_amd.ko 2024-04-14T23:36:48.297 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_artop.ko 2024-04-14T23:36:48.297 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_atiixp.ko 2024-04-14T23:36:48.297 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_atp867x.ko 2024-04-14T23:36:48.297 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_cmd640.ko 2024-04-14T23:36:48.297 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_cmd64x.ko 2024-04-14T23:36:48.297 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_cypress.ko 2024-04-14T23:36:48.297 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_efar.ko 2024-04-14T23:36:48.297 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_hpt366.ko 2024-04-14T23:36:48.297 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_hpt37x.ko 2024-04-14T23:36:48.297 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_hpt3x2n.ko 2024-04-14T23:36:48.297 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_hpt3x3.ko 2024-04-14T23:36:48.297 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_it8213.ko 2024-04-14T23:36:48.297 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_it821x.ko 2024-04-14T23:36:48.297 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_jmicron.ko 2024-04-14T23:36:48.297 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_marvell.ko 2024-04-14T23:36:48.297 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_mpiix.ko 2024-04-14T23:36:48.298 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_netcell.ko 2024-04-14T23:36:48.298 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_ninja32.ko 2024-04-14T23:36:48.298 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_ns87410.ko 2024-04-14T23:36:48.298 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_ns87415.ko 2024-04-14T23:36:48.298 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_oldpiix.ko 2024-04-14T23:36:48.298 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_opti.ko 2024-04-14T23:36:48.298 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_optidma.ko 2024-04-14T23:36:48.298 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_pcmcia.ko 2024-04-14T23:36:48.298 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_pdc2027x.ko 2024-04-14T23:36:48.298 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-04-14T23:36:48.298 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_piccolo.ko 2024-04-14T23:36:48.298 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_rdc.ko 2024-04-14T23:36:48.298 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_sch.ko 2024-04-14T23:36:48.298 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_serverworks.ko 2024-04-14T23:36:48.299 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_sil680.ko 2024-04-14T23:36:48.299 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_sis.ko 2024-04-14T23:36:48.299 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_sl82c105.ko 2024-04-14T23:36:48.299 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_triflex.ko 2024-04-14T23:36:48.299 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_via.ko 2024-04-14T23:36:48.299 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pdc_adma.ko 2024-04-14T23:36:48.299 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_inic162x.ko 2024-04-14T23:36:48.299 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_mv.ko 2024-04-14T23:36:48.299 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_nv.ko 2024-04-14T23:36:48.299 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_promise.ko 2024-04-14T23:36:48.299 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_qstor.ko 2024-04-14T23:36:48.299 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_sil.ko 2024-04-14T23:36:48.299 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_sil24.ko 2024-04-14T23:36:48.299 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_sis.ko 2024-04-14T23:36:48.299 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_svw.ko 2024-04-14T23:36:48.299 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_sx4.ko 2024-04-14T23:36:48.300 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_uli.ko 2024-04-14T23:36:48.300 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_via.ko 2024-04-14T23:36:48.300 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_vsc.ko 2024-04-14T23:36:48.300 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/auxdisplay 2024-04-14T23:36:48.300 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/auxdisplay/ks0108.ko 2024-04-14T23:36:48.300 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/base 2024-04-14T23:36:48.300 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/base/regmap 2024-04-14T23:36:48.300 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/base/regmap/regmap-i2c.ko 2024-04-14T23:36:48.300 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/base/regmap/regmap-mmio.ko 2024-04-14T23:36:48.300 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/bcma 2024-04-14T23:36:48.300 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/bcma/bcma.ko 2024-04-14T23:36:48.300 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block 2024-04-14T23:36:48.300 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/aoe 2024-04-14T23:36:48.300 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/aoe/aoe.ko 2024-04-14T23:36:48.300 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/brd.ko 2024-04-14T23:36:48.301 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/drbd 2024-04-14T23:36:48.301 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/drbd/drbd.ko 2024-04-14T23:36:48.301 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/floppy.ko 2024-04-14T23:36:48.301 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/loop.ko 2024-04-14T23:36:48.301 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/mtip32xx 2024-04-14T23:36:48.301 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-04-14T23:36:48.301 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/nbd.ko 2024-04-14T23:36:48.301 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/null_blk 2024-04-14T23:36:48.301 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/null_blk/null_blk.ko 2024-04-14T23:36:48.301 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/pktcdvd.ko 2024-04-14T23:36:48.301 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/rbd.ko 2024-04-14T23:36:48.301 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/virtio_blk.ko 2024-04-14T23:36:48.301 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/xen-blkback 2024-04-14T23:36:48.301 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-04-14T23:36:48.302 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/xen-blkfront.ko 2024-04-14T23:36:48.302 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/zram 2024-04-14T23:36:48.302 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/zram/zram.ko 2024-04-14T23:36:48.302 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char 2024-04-14T23:36:48.302 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hangcheck-timer.ko 2024-04-14T23:36:48.302 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random 2024-04-14T23:36:48.302 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/amd-rng.ko 2024-04-14T23:36:48.302 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/intel-rng.ko 2024-04-14T23:36:48.302 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-04-14T23:36:48.302 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/via-rng.ko 2024-04-14T23:36:48.302 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/virtio-rng.ko 2024-04-14T23:36:48.302 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi 2024-04-14T23:36:48.302 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-04-14T23:36:48.302 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-04-14T23:36:48.303 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-04-14T23:36:48.303 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_si.ko 2024-04-14T23:36:48.303 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-04-14T23:36:48.303 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-04-14T23:36:48.303 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/lp.ko 2024-04-14T23:36:48.303 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/mwave 2024-04-14T23:36:48.303 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/mwave/mwave.ko 2024-04-14T23:36:48.303 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ppdev.ko 2024-04-14T23:36:48.303 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tlclk.ko 2024-04-14T23:36:48.303 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm 2024-04-14T23:36:48.303 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm.ko 2024-04-14T23:36:48.303 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_atmel.ko 2024-04-14T23:36:48.303 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_infineon.ko 2024-04-14T23:36:48.303 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_nsc.ko 2024-04-14T23:36:48.303 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_tis.ko 2024-04-14T23:36:48.304 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-04-14T23:36:48.304 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/uv_mmtimer.ko 2024-04-14T23:36:48.304 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/virtio_console.ko 2024-04-14T23:36:48.304 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq 2024-04-14T23:36:48.304 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-04-14T23:36:48.304 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-04-14T23:36:48.304 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/p4-clockmod.ko 2024-04-14T23:36:48.304 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-04-14T23:36:48.304 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/powernow-k8.ko 2024-04-14T23:36:48.304 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/speedstep-lib.ko 2024-04-14T23:36:48.304 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto 2024-04-14T23:36:48.304 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/ccp 2024-04-14T23:36:48.304 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-04-14T23:36:48.304 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/ccp/ccp.ko 2024-04-14T23:36:48.304 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel 2024-04-14T23:36:48.305 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat 2024-04-14T23:36:48.305 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/qat_common 2024-04-14T23:36:48.305 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-04-14T23:36:48.305 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-04-14T23:36:48.305 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-04-14T23:36:48.305 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/padlock-aes.ko 2024-04-14T23:36:48.305 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/padlock-sha.ko 2024-04-14T23:36:48.305 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/virtio 2024-04-14T23:36:48.305 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-04-14T23:36:48.305 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dca 2024-04-14T23:36:48.305 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dca/dca.ko 2024-04-14T23:36:48.305 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/devfreq 2024-04-14T23:36:48.305 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-04-14T23:36:48.305 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma 2024-04-14T23:36:48.305 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/dw 2024-04-14T23:36:48.306 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/dw/dw_dmac.ko 2024-04-14T23:36:48.306 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/ioat 2024-04-14T23:36:48.306 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/ioat/ioatdma.ko 2024-04-14T23:36:48.306 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac 2024-04-14T23:36:48.306 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/amd64_edac.ko 2024-04-14T23:36:48.306 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/e752x_edac.ko 2024-04-14T23:36:48.306 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/edac_mce_amd.ko 2024-04-14T23:36:48.306 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i3000_edac.ko 2024-04-14T23:36:48.306 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i3200_edac.ko 2024-04-14T23:36:48.306 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i5100_edac.ko 2024-04-14T23:36:48.306 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i5400_edac.ko 2024-04-14T23:36:48.306 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i7300_edac.ko 2024-04-14T23:36:48.306 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i7core_edac.ko 2024-04-14T23:36:48.307 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i82975x_edac.ko 2024-04-14T23:36:48.307 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/ie31200_edac.ko 2024-04-14T23:36:48.307 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/sb_edac.ko 2024-04-14T23:36:48.307 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/x38_edac.ko 2024-04-14T23:36:48.307 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/firmware 2024-04-14T23:36:48.307 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/firmware/edd.ko 2024-04-14T23:36:48.307 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/firmware/iscsi_ibft.ko 2024-04-14T23:36:48.307 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio 2024-04-14T23:36:48.307 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-ich.ko 2024-04-14T23:36:48.307 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-viperboard.ko 2024-04-14T23:36:48.307 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid 2024-04-14T23:36:48.307 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-appleir.ko 2024-04-14T23:36:48.307 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-aureal.ko 2024-04-14T23:36:48.307 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-axff.ko 2024-04-14T23:36:48.307 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-dr.ko 2024-04-14T23:36:48.308 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-elecom.ko 2024-04-14T23:36:48.308 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-elo.ko 2024-04-14T23:36:48.308 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-emsff.ko 2024-04-14T23:36:48.308 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-gaff.ko 2024-04-14T23:36:48.308 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-gt683r.ko 2024-04-14T23:36:48.308 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-gyration.ko 2024-04-14T23:36:48.308 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-holtek-kbd.ko 2024-04-14T23:36:48.308 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-holtek-mouse.ko 2024-04-14T23:36:48.308 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-holtekff.ko 2024-04-14T23:36:48.308 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-hyperv.ko 2024-04-14T23:36:48.308 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-icade.ko 2024-04-14T23:36:48.308 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-keytouch.ko 2024-04-14T23:36:48.308 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-kye.ko 2024-04-14T23:36:48.308 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-lcpower.ko 2024-04-14T23:36:48.308 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-led.ko 2024-04-14T23:36:48.309 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-lenovo.ko 2024-04-14T23:36:48.309 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-logitech-dj.ko 2024-04-14T23:36:48.309 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-04-14T23:36:48.309 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-multitouch.ko 2024-04-14T23:36:48.309 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-ortek.ko 2024-04-14T23:36:48.309 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-penmount.ko 2024-04-14T23:36:48.309 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-petalynx.ko 2024-04-14T23:36:48.309 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-picolcd.ko 2024-04-14T23:36:48.309 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-pl.ko 2024-04-14T23:36:48.309 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-plantronics.ko 2024-04-14T23:36:48.309 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-primax.ko 2024-04-14T23:36:48.309 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-rmi.ko 2024-04-14T23:36:48.309 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-arvo.ko 2024-04-14T23:36:48.309 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-common.ko 2024-04-14T23:36:48.309 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-isku.ko 2024-04-14T23:36:48.310 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-kone.ko 2024-04-14T23:36:48.310 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-04-14T23:36:48.310 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-konepure.ko 2024-04-14T23:36:48.310 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-04-14T23:36:48.310 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-lua.ko 2024-04-14T23:36:48.310 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-pyra.ko 2024-04-14T23:36:48.310 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-ryos.ko 2024-04-14T23:36:48.310 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-savu.ko 2024-04-14T23:36:48.310 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat.ko 2024-04-14T23:36:48.310 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-saitek.ko 2024-04-14T23:36:48.310 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-samsung.ko 2024-04-14T23:36:48.310 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-sensor-hub.ko 2024-04-14T23:36:48.310 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-sjoy.ko 2024-04-14T23:36:48.310 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-sony.ko 2024-04-14T23:36:48.311 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-speedlink.ko 2024-04-14T23:36:48.311 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-steelseries.ko 2024-04-14T23:36:48.311 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-sunplus.ko 2024-04-14T23:36:48.311 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-thrustmaster.ko 2024-04-14T23:36:48.311 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-tivo.ko 2024-04-14T23:36:48.311 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-tmff.ko 2024-04-14T23:36:48.311 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-topseed.ko 2024-04-14T23:36:48.311 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-twinhan.ko 2024-04-14T23:36:48.311 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-uclogic.ko 2024-04-14T23:36:48.311 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-waltop.ko 2024-04-14T23:36:48.311 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-wiimote.ko 2024-04-14T23:36:48.311 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-xinmo.ko 2024-04-14T23:36:48.311 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-zpff.ko 2024-04-14T23:36:48.312 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-zydacron.ko 2024-04-14T23:36:48.312 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/uhid.ko 2024-04-14T23:36:48.312 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/wacom.ko 2024-04-14T23:36:48.312 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hv 2024-04-14T23:36:48.312 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hv/hv_balloon.ko 2024-04-14T23:36:48.312 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hv/hv_utils.ko 2024-04-14T23:36:48.312 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hv/hv_vmbus.ko 2024-04-14T23:36:48.312 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon 2024-04-14T23:36:48.312 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/abituguru.ko 2024-04-14T23:36:48.312 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/abituguru3.ko 2024-04-14T23:36:48.312 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/acpi_power_meter.ko 2024-04-14T23:36:48.312 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ad7414.ko 2024-04-14T23:36:48.312 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ad7418.ko 2024-04-14T23:36:48.312 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adc128d818.ko 2024-04-14T23:36:48.312 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm1025.ko 2024-04-14T23:36:48.313 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm1026.ko 2024-04-14T23:36:48.313 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm1029.ko 2024-04-14T23:36:48.313 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm1031.ko 2024-04-14T23:36:48.313 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm9240.ko 2024-04-14T23:36:48.313 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ads7828.ko 2024-04-14T23:36:48.313 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7410.ko 2024-04-14T23:36:48.313 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7411.ko 2024-04-14T23:36:48.313 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7462.ko 2024-04-14T23:36:48.313 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7470.ko 2024-04-14T23:36:48.313 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7475.ko 2024-04-14T23:36:48.313 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7x10.ko 2024-04-14T23:36:48.313 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/amc6821.ko 2024-04-14T23:36:48.313 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/applesmc.ko 2024-04-14T23:36:48.313 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/asb100.ko 2024-04-14T23:36:48.313 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/asc7621.ko 2024-04-14T23:36:48.314 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/asus_atk0110.ko 2024-04-14T23:36:48.314 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/atxp1.ko 2024-04-14T23:36:48.314 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/coretemp.ko 2024-04-14T23:36:48.314 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-04-14T23:36:48.314 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/dme1737.ko 2024-04-14T23:36:48.314 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ds1621.ko 2024-04-14T23:36:48.314 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ds620.ko 2024-04-14T23:36:48.314 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/emc1403.ko 2024-04-14T23:36:48.314 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/emc6w201.ko 2024-04-14T23:36:48.314 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/f71805f.ko 2024-04-14T23:36:48.314 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/f71882fg.ko 2024-04-14T23:36:48.314 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/f75375s.ko 2024-04-14T23:36:48.315 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/fam15h_power.ko 2024-04-14T23:36:48.315 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/fschmd.ko 2024-04-14T23:36:48.315 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/g760a.ko 2024-04-14T23:36:48.315 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/g762.ko 2024-04-14T23:36:48.315 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/gl518sm.ko 2024-04-14T23:36:48.315 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/gl520sm.ko 2024-04-14T23:36:48.315 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/hwmon-vid.ko 2024-04-14T23:36:48.315 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/i5500_temp.ko 2024-04-14T23:36:48.315 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/i5k_amb.ko 2024-04-14T23:36:48.315 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ibmaem.ko 2024-04-14T23:36:48.315 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ibmpex.ko 2024-04-14T23:36:48.315 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ina209.ko 2024-04-14T23:36:48.315 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ina2xx.ko 2024-04-14T23:36:48.315 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/it87.ko 2024-04-14T23:36:48.315 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/k10temp.ko 2024-04-14T23:36:48.316 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/k8temp.ko 2024-04-14T23:36:48.316 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lineage-pem.ko 2024-04-14T23:36:48.316 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm63.ko 2024-04-14T23:36:48.316 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm73.ko 2024-04-14T23:36:48.316 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm75.ko 2024-04-14T23:36:48.316 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm77.ko 2024-04-14T23:36:48.316 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm78.ko 2024-04-14T23:36:48.316 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm80.ko 2024-04-14T23:36:48.316 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm83.ko 2024-04-14T23:36:48.316 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm85.ko 2024-04-14T23:36:48.316 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm87.ko 2024-04-14T23:36:48.316 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm90.ko 2024-04-14T23:36:48.316 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm92.ko 2024-04-14T23:36:48.317 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm93.ko 2024-04-14T23:36:48.317 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm95234.ko 2024-04-14T23:36:48.317 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm95241.ko 2024-04-14T23:36:48.317 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm95245.ko 2024-04-14T23:36:48.317 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc2945.ko 2024-04-14T23:36:48.317 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4151.ko 2024-04-14T23:36:48.317 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4215.ko 2024-04-14T23:36:48.317 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4222.ko 2024-04-14T23:36:48.317 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4245.ko 2024-04-14T23:36:48.317 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4260.ko 2024-04-14T23:36:48.317 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4261.ko 2024-04-14T23:36:48.317 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max16065.ko 2024-04-14T23:36:48.317 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max1619.ko 2024-04-14T23:36:48.317 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max1668.ko 2024-04-14T23:36:48.317 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max197.ko 2024-04-14T23:36:48.317 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max6639.ko 2024-04-14T23:36:48.318 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max6650.ko 2024-04-14T23:36:48.318 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max6697.ko 2024-04-14T23:36:48.318 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/mcp3021.ko 2024-04-14T23:36:48.318 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/nct6683.ko 2024-04-14T23:36:48.318 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/nct6775-core.ko 2024-04-14T23:36:48.318 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/nct6775.ko 2024-04-14T23:36:48.318 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/nct7802.ko 2024-04-14T23:36:48.318 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pc87360.ko 2024-04-14T23:36:48.318 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pc87427.ko 2024-04-14T23:36:48.318 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pcf8591.ko 2024-04-14T23:36:48.318 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus 2024-04-14T23:36:48.318 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-04-14T23:36:48.318 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-04-14T23:36:48.319 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-04-14T23:36:48.319 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/max16064.ko 2024-04-14T23:36:48.319 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/max34440.ko 2024-04-14T23:36:48.319 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/max8688.ko 2024-04-14T23:36:48.319 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-04-14T23:36:48.319 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-04-14T23:36:48.319 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-04-14T23:36:48.319 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-04-14T23:36:48.319 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-04-14T23:36:48.319 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-04-14T23:36:48.319 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/powr1220.ko 2024-04-14T23:36:48.319 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sch5627.ko 2024-04-14T23:36:48.319 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sch5636.ko 2024-04-14T23:36:48.319 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sch56xx-common.ko 2024-04-14T23:36:48.319 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sht15.ko 2024-04-14T23:36:48.319 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sht21.ko 2024-04-14T23:36:48.320 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/shtc1.ko 2024-04-14T23:36:48.320 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sis5595.ko 2024-04-14T23:36:48.320 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/smsc47b397.ko 2024-04-14T23:36:48.320 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/smsc47m1.ko 2024-04-14T23:36:48.320 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/smsc47m192.ko 2024-04-14T23:36:48.320 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/thmc50.ko 2024-04-14T23:36:48.320 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/tmp102.ko 2024-04-14T23:36:48.320 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/tmp103.ko 2024-04-14T23:36:48.320 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/tmp401.ko 2024-04-14T23:36:48.320 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/tmp421.ko 2024-04-14T23:36:48.320 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/via-cputemp.ko 2024-04-14T23:36:48.320 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/via686a.ko 2024-04-14T23:36:48.320 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/vt1211.ko 2024-04-14T23:36:48.320 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/vt8231.ko 2024-04-14T23:36:48.320 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83627ehf.ko 2024-04-14T23:36:48.320 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83627hf.ko 2024-04-14T23:36:48.321 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83781d.ko 2024-04-14T23:36:48.321 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83791d.ko 2024-04-14T23:36:48.321 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83792d.ko 2024-04-14T23:36:48.321 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83793.ko 2024-04-14T23:36:48.321 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83795.ko 2024-04-14T23:36:48.321 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83l785ts.ko 2024-04-14T23:36:48.321 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83l786ng.ko 2024-04-14T23:36:48.321 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c 2024-04-14T23:36:48.321 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/algos 2024-04-14T23:36:48.321 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-04-14T23:36:48.321 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-04-14T23:36:48.321 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses 2024-04-14T23:36:48.321 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-04-14T23:36:48.321 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-04-14T23:36:48.322 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-04-14T23:36:48.322 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-04-14T23:36:48.322 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-04-14T23:36:48.322 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-04-14T23:36:48.322 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-04-14T23:36:48.322 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-04-14T23:36:48.322 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-i801.ko 2024-04-14T23:36:48.322 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-isch.ko 2024-04-14T23:36:48.322 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-04-14T23:36:48.322 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-04-14T23:36:48.322 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-04-14T23:36:48.322 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-parport.ko 2024-04-14T23:36:48.322 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-04-14T23:36:48.322 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-04-14T23:36:48.323 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-04-14T23:36:48.323 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-04-14T23:36:48.323 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-04-14T23:36:48.323 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-04-14T23:36:48.323 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-via.ko 2024-04-14T23:36:48.323 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-04-14T23:36:48.323 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-04-14T23:36:48.323 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-dev.ko 2024-04-14T23:36:48.323 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-mux.ko 2024-04-14T23:36:48.323 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-04-14T23:36:48.323 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-smbus.ko 2024-04-14T23:36:48.323 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-stub.ko 2024-04-14T23:36:48.323 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband 2024-04-14T23:36:48.323 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core 2024-04-14T23:36:48.323 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ib_cm.ko 2024-04-14T23:36:48.324 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ib_core.ko 2024-04-14T23:36:48.324 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ib_umad.ko 2024-04-14T23:36:48.324 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-04-14T23:36:48.324 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/iw_cm.ko 2024-04-14T23:36:48.324 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/rdma_cm.ko 2024-04-14T23:36:48.324 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-04-14T23:36:48.324 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw 2024-04-14T23:36:48.324 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/cxgb4 2024-04-14T23:36:48.324 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-04-14T23:36:48.324 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mlx4 2024-04-14T23:36:48.324 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-04-14T23:36:48.324 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mlx5 2024-04-14T23:36:48.324 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-04-14T23:36:48.324 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mthca 2024-04-14T23:36:48.324 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-04-14T23:36:48.325 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp 2024-04-14T23:36:48.325 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/ipoib 2024-04-14T23:36:48.325 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-04-14T23:36:48.325 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/iser 2024-04-14T23:36:48.325 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-04-14T23:36:48.325 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/isert 2024-04-14T23:36:48.325 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-04-14T23:36:48.325 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/srp 2024-04-14T23:36:48.325 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-04-14T23:36:48.325 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/srpt 2024-04-14T23:36:48.325 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-04-14T23:36:48.325 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input 2024-04-14T23:36:48.325 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/gameport 2024-04-14T23:36:48.325 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-04-14T23:36:48.325 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/gameport/fm801-gp.ko 2024-04-14T23:36:48.326 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/gameport/gameport.ko 2024-04-14T23:36:48.326 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/joydev.ko 2024-04-14T23:36:48.326 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard 2024-04-14T23:36:48.326 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/gpio_keys.ko 2024-04-14T23:36:48.326 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse 2024-04-14T23:36:48.326 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/appletouch.ko 2024-04-14T23:36:48.326 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/bcm5974.ko 2024-04-14T23:36:48.326 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/cyapatp.ko 2024-04-14T23:36:48.326 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/elan_i2c.ko 2024-04-14T23:36:48.326 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/sermouse.ko 2024-04-14T23:36:48.326 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-04-14T23:36:48.326 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/synaptics_usb.ko 2024-04-14T23:36:48.326 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/vsxxxaa.ko 2024-04-14T23:36:48.326 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/rmi4 2024-04-14T23:36:48.327 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/rmi4/rmi_core.ko 2024-04-14T23:36:48.327 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio 2024-04-14T23:36:48.327 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/altera_ps2.ko 2024-04-14T23:36:48.327 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/arc_ps2.ko 2024-04-14T23:36:48.327 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-04-14T23:36:48.327 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/serio_raw.ko 2024-04-14T23:36:48.327 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/sparse-keymap.ko 2024-04-14T23:36:48.327 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds 2024-04-14T23:36:48.327 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-blinkm.ko 2024-04-14T23:36:48.327 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-lm3530.ko 2024-04-14T23:36:48.327 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-lp3944.ko 2024-04-14T23:36:48.327 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-ss4200.ko 2024-04-14T23:36:48.327 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger 2024-04-14T23:36:48.327 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-04-14T23:36:48.327 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-04-14T23:36:48.327 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-04-14T23:36:48.328 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-04-14T23:36:48.328 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-04-14T23:36:48.328 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-04-14T23:36:48.328 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-04-14T23:36:48.328 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-04-14T23:36:48.328 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-04-14T23:36:48.328 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md 2024-04-14T23:36:48.328 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/bcache 2024-04-14T23:36:48.328 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/bcache/bcache.ko 2024-04-14T23:36:48.328 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-bio-prison.ko 2024-04-14T23:36:48.328 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-cache-smq.ko 2024-04-14T23:36:48.328 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-cache.ko 2024-04-14T23:36:48.328 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-crypt.ko 2024-04-14T23:36:48.328 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-delay.ko 2024-04-14T23:36:48.328 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-flakey.ko 2024-04-14T23:36:48.329 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-log-userspace.ko 2024-04-14T23:36:48.329 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-multipath.ko 2024-04-14T23:36:48.329 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-queue-length.ko 2024-04-14T23:36:48.329 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-raid.ko 2024-04-14T23:36:48.329 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-round-robin.ko 2024-04-14T23:36:48.329 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-service-time.ko 2024-04-14T23:36:48.329 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-switch.ko 2024-04-14T23:36:48.329 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-thin-pool.ko 2024-04-14T23:36:48.329 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-verity.ko 2024-04-14T23:36:48.329 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/persistent-data 2024-04-14T23:36:48.329 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-04-14T23:36:48.329 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/raid0.ko 2024-04-14T23:36:48.329 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/raid1.ko 2024-04-14T23:36:48.329 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/raid10.ko 2024-04-14T23:36:48.330 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/raid456.ko 2024-04-14T23:36:48.330 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message 2024-04-14T23:36:48.330 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion 2024-04-14T23:36:48.330 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptbase.ko 2024-04-14T23:36:48.330 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptctl.ko 2024-04-14T23:36:48.330 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptfc.ko 2024-04-14T23:36:48.330 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptlan.ko 2024-04-14T23:36:48.330 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptsas.ko 2024-04-14T23:36:48.330 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptscsih.ko 2024-04-14T23:36:48.330 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptspi.ko 2024-04-14T23:36:48.330 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd 2024-04-14T23:36:48.330 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/lpc_ich.ko 2024-04-14T23:36:48.330 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/lpc_sch.ko 2024-04-14T23:36:48.330 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/mfd-core.ko 2024-04-14T23:36:48.331 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/sm501.ko 2024-04-14T23:36:48.331 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/viperboard.ko 2024-04-14T23:36:48.331 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/vx855.ko 2024-04-14T23:36:48.331 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/wl1273-core.ko 2024-04-14T23:36:48.331 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc 2024-04-14T23:36:48.331 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/altera-stapl 2024-04-14T23:36:48.331 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-04-14T23:36:48.331 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/apds9802als.ko 2024-04-14T23:36:48.331 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/apds990x.ko 2024-04-14T23:36:48.331 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/bh1770glc.ko 2024-04-14T23:36:48.331 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/cb710 2024-04-14T23:36:48.331 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/cb710/cb710.ko 2024-04-14T23:36:48.331 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/eeprom 2024-04-14T23:36:48.331 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/eeprom/at24.ko 2024-04-14T23:36:48.332 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-04-14T23:36:48.332 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/eeprom/max6875.ko 2024-04-14T23:36:48.332 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/enclosure.ko 2024-04-14T23:36:48.332 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/hpilo.ko 2024-04-14T23:36:48.332 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/isl29003.ko 2024-04-14T23:36:48.332 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/isl29020.ko 2024-04-14T23:36:48.332 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/lis3lv02d 2024-04-14T23:36:48.332 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-04-14T23:36:48.332 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-04-14T23:36:48.332 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/mei 2024-04-14T23:36:48.332 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/mei/mei-me.ko 2024-04-14T23:36:48.332 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/mei/mei-txe.ko 2024-04-14T23:36:48.332 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/mei/mei.ko 2024-04-14T23:36:48.332 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-gru 2024-04-14T23:36:48.332 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-gru/gru.ko 2024-04-14T23:36:48.333 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-xp 2024-04-14T23:36:48.333 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-xp/xp.ko 2024-04-14T23:36:48.333 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-xp/xpc.ko 2024-04-14T23:36:48.333 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-04-14T23:36:48.333 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/tifm_7xx1.ko 2024-04-14T23:36:48.333 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/tifm_core.ko 2024-04-14T23:36:48.333 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/tsl2550.ko 2024-04-14T23:36:48.333 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/vmw_balloon.ko 2024-04-14T23:36:48.333 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/vmw_vmci 2024-04-14T23:36:48.333 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-04-14T23:36:48.333 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net 2024-04-14T23:36:48.333 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/bonding 2024-04-14T23:36:48.333 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/bonding/bonding.ko 2024-04-14T23:36:48.333 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa 2024-04-14T23:36:48.333 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/b53 2024-04-14T23:36:48.334 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/b53/b53_common.ko 2024-04-14T23:36:48.334 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/bcm-sf2.ko 2024-04-14T23:36:48.334 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/mv88e6060.ko 2024-04-14T23:36:48.334 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/mv88e6xxx 2024-04-14T23:36:48.334 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-04-14T23:36:48.334 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dummy.ko 2024-04-14T23:36:48.334 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/eql.ko 2024-04-14T23:36:48.334 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet 2024-04-14T23:36:48.334 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com 2024-04-14T23:36:48.334 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-04-14T23:36:48.334 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-04-14T23:36:48.334 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-04-14T23:36:48.334 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-04-14T23:36:48.334 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390 2024-04-14T23:36:48.335 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/8390.ko 2024-04-14T23:36:48.335 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-04-14T23:36:48.335 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-04-14T23:36:48.335 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-04-14T23:36:48.335 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/adaptec 2024-04-14T23:36:48.335 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-04-14T23:36:48.335 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/agere 2024-04-14T23:36:48.335 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/agere/et131x.ko 2024-04-14T23:36:48.335 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/alteon 2024-04-14T23:36:48.335 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-04-14T23:36:48.335 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/altera 2024-04-14T23:36:48.335 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-04-14T23:36:48.335 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/amd 2024-04-14T23:36:48.335 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-04-14T23:36:48.335 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-04-14T23:36:48.336 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-04-14T23:36:48.336 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros 2024-04-14T23:36:48.336 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/alx 2024-04-14T23:36:48.336 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-04-14T23:36:48.336 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atl1c 2024-04-14T23:36:48.336 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-04-14T23:36:48.336 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atl1e 2024-04-14T23:36:48.336 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-04-14T23:36:48.336 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atlx 2024-04-14T23:36:48.336 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-04-14T23:36:48.336 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-04-14T23:36:48.336 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom 2024-04-14T23:36:48.336 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-04-14T23:36:48.336 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-04-14T23:36:48.336 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-04-14T23:36:48.337 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-04-14T23:36:48.337 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-04-14T23:36:48.337 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/genet 2024-04-14T23:36:48.337 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-04-14T23:36:48.337 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-04-14T23:36:48.337 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/brocade 2024-04-14T23:36:48.337 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/brocade/bna 2024-04-14T23:36:48.337 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-04-14T23:36:48.337 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio 2024-04-14T23:36:48.337 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb 2024-04-14T23:36:48.337 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-04-14T23:36:48.337 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-04-14T23:36:48.337 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-04-14T23:36:48.337 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-04-14T23:36:48.338 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-04-14T23:36:48.338 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-04-14T23:36:48.338 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-04-14T23:36:48.338 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-04-14T23:36:48.338 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-04-14T23:36:48.338 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/cisco 2024-04-14T23:36:48.338 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/cisco/enic 2024-04-14T23:36:48.338 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-04-14T23:36:48.338 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec 2024-04-14T23:36:48.338 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip 2024-04-14T23:36:48.338 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-04-14T23:36:48.338 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-04-14T23:36:48.338 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-04-14T23:36:48.338 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-04-14T23:36:48.338 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-04-14T23:36:48.339 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-04-14T23:36:48.339 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dlink 2024-04-14T23:36:48.339 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-04-14T23:36:48.339 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-04-14T23:36:48.339 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dnet.ko 2024-04-14T23:36:48.339 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/emulex 2024-04-14T23:36:48.339 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/emulex/benet 2024-04-14T23:36:48.339 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-04-14T23:36:48.339 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/ethoc.ko 2024-04-14T23:36:48.341 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/fealnx.ko 2024-04-14T23:36:48.341 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel 2024-04-14T23:36:48.341 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e100.ko 2024-04-14T23:36:48.341 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e1000 2024-04-14T23:36:48.341 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-04-14T23:36:48.341 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e1000e 2024-04-14T23:36:48.341 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-04-14T23:36:48.341 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/fm10k 2024-04-14T23:36:48.341 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-04-14T23:36:48.341 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/i40e 2024-04-14T23:36:48.342 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-04-14T23:36:48.342 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/iavf 2024-04-14T23:36:48.342 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-04-14T23:36:48.342 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/igb 2024-04-14T23:36:48.342 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-04-14T23:36:48.342 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/igbvf 2024-04-14T23:36:48.342 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-04-14T23:36:48.342 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ixgbe 2024-04-14T23:36:48.342 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-04-14T23:36:48.342 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ixgbevf 2024-04-14T23:36:48.342 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-04-14T23:36:48.342 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/jme.ko 2024-04-14T23:36:48.342 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/marvell 2024-04-14T23:36:48.342 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-04-14T23:36:48.343 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/marvell/skge.ko 2024-04-14T23:36:48.343 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-04-14T23:36:48.343 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox 2024-04-14T23:36:48.343 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx4 2024-04-14T23:36:48.343 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-04-14T23:36:48.343 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-04-14T23:36:48.343 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx5 2024-04-14T23:36:48.343 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-04-14T23:36:48.343 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-04-14T23:36:48.343 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/micrel 2024-04-14T23:36:48.343 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-04-14T23:36:48.343 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/myricom 2024-04-14T23:36:48.343 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/myricom/myri10ge 2024-04-14T23:36:48.343 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-04-14T23:36:48.344 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/natsemi 2024-04-14T23:36:48.344 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-04-14T23:36:48.344 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-04-14T23:36:48.344 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/neterion 2024-04-14T23:36:48.344 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-04-14T23:36:48.344 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/nvidia 2024-04-14T23:36:48.344 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-04-14T23:36:48.344 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/packetengines 2024-04-14T23:36:48.344 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-04-14T23:36:48.344 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-04-14T23:36:48.344 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic 2024-04-14T23:36:48.344 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/netxen 2024-04-14T23:36:48.344 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-04-14T23:36:48.344 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-04-14T23:36:48.344 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-04-14T23:36:48.345 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-04-14T23:36:48.345 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/rdc 2024-04-14T23:36:48.345 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-04-14T23:36:48.345 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek 2024-04-14T23:36:48.345 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-04-14T23:36:48.345 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-04-14T23:36:48.345 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/atp.ko 2024-04-14T23:36:48.345 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-04-14T23:36:48.345 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/rocker 2024-04-14T23:36:48.345 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-04-14T23:36:48.345 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sfc 2024-04-14T23:36:48.345 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-04-14T23:36:48.345 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/silan 2024-04-14T23:36:48.345 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-04-14T23:36:48.346 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sis 2024-04-14T23:36:48.346 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sis/sis190.ko 2024-04-14T23:36:48.346 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sis/sis900.ko 2024-04-14T23:36:48.346 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc 2024-04-14T23:36:48.346 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-04-14T23:36:48.346 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-04-14T23:36:48.346 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-04-14T23:36:48.346 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-04-14T23:36:48.346 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro 2024-04-14T23:36:48.346 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro/stmmac 2024-04-14T23:36:48.346 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-04-14T23:36:48.346 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-04-14T23:36:48.346 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun 2024-04-14T23:36:48.346 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/cassini.ko 2024-04-14T23:36:48.347 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/niu.ko 2024-04-14T23:36:48.347 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/sungem.ko 2024-04-14T23:36:48.347 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-04-14T23:36:48.347 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/tehuti 2024-04-14T23:36:48.347 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-04-14T23:36:48.347 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/ti 2024-04-14T23:36:48.347 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/ti/tlan.ko 2024-04-14T23:36:48.347 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/via 2024-04-14T23:36:48.347 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-04-14T23:36:48.347 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-04-14T23:36:48.347 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/wiznet 2024-04-14T23:36:48.347 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-04-14T23:36:48.347 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-04-14T23:36:48.347 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/xircom 2024-04-14T23:36:48.347 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-04-14T23:36:48.348 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/geneve.ko 2024-04-14T23:36:48.348 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/hyperv 2024-04-14T23:36:48.348 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-04-14T23:36:48.348 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ieee802154 2024-04-14T23:36:48.348 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ieee802154/fakelb.ko 2024-04-14T23:36:48.348 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ifb.ko 2024-04-14T23:36:48.348 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ipvlan 2024-04-14T23:36:48.348 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ipvlan/ipvlan.ko 2024-04-14T23:36:48.348 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/macvlan.ko 2024-04-14T23:36:48.348 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/macvtap.ko 2024-04-14T23:36:48.348 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio 2024-04-14T23:36:48.348 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio.ko 2024-04-14T23:36:48.348 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-04-14T23:36:48.348 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-04-14T23:36:48.348 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio/mdio-regmap.ko 2024-04-14T23:36:48.348 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mii.ko 2024-04-14T23:36:48.349 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/net_failover.ko 2024-04-14T23:36:48.349 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/netconsole.ko 2024-04-14T23:36:48.349 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/nlmon.ko 2024-04-14T23:36:48.349 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/pcs 2024-04-14T23:36:48.349 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/pcs/pcs-lynx.ko 2024-04-14T23:36:48.349 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-04-14T23:36:48.349 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy 2024-04-14T23:36:48.349 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/amd.ko 2024-04-14T23:36:48.349 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-04-14T23:36:48.349 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-04-14T23:36:48.349 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/bcm7xxx.ko 2024-04-14T23:36:48.349 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/bcm87xx.ko 2024-04-14T23:36:48.349 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/broadcom.ko 2024-04-14T23:36:48.349 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/cicada.ko 2024-04-14T23:36:48.350 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/davicom.ko 2024-04-14T23:36:48.350 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/dp83640.ko 2024-04-14T23:36:48.350 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/et1011c.ko 2024-04-14T23:36:48.350 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/icplus.ko 2024-04-14T23:36:48.350 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/lxt.ko 2024-04-14T23:36:48.350 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/marvell.ko 2024-04-14T23:36:48.350 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/micrel.ko 2024-04-14T23:36:48.350 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/national.ko 2024-04-14T23:36:48.350 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/phylink.ko 2024-04-14T23:36:48.350 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/qsemi.ko 2024-04-14T23:36:48.350 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/realtek.ko 2024-04-14T23:36:48.350 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/smsc.ko 2024-04-14T23:36:48.350 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/ste10Xp.ko 2024-04-14T23:36:48.350 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/vitesse.ko 2024-04-14T23:36:48.351 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp 2024-04-14T23:36:48.351 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/bsd_comp.ko 2024-04-14T23:36:48.351 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_async.ko 2024-04-14T23:36:48.351 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_deflate.ko 2024-04-14T23:36:48.351 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_generic.ko 2024-04-14T23:36:48.351 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_mppe.ko 2024-04-14T23:36:48.351 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_synctty.ko 2024-04-14T23:36:48.351 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/pppoe.ko 2024-04-14T23:36:48.351 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/pppox.ko 2024-04-14T23:36:48.351 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/pptp.ko 2024-04-14T23:36:48.351 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/slip 2024-04-14T23:36:48.351 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/slip/slhc.ko 2024-04-14T23:36:48.351 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/slip/slip.ko 2024-04-14T23:36:48.351 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/sungem_phy.ko 2024-04-14T23:36:48.351 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/tap.ko 2024-04-14T23:36:48.352 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team 2024-04-14T23:36:48.352 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team.ko 2024-04-14T23:36:48.352 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_activebackup.ko 2024-04-14T23:36:48.352 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_broadcast.ko 2024-04-14T23:36:48.352 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-04-14T23:36:48.352 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_random.ko 2024-04-14T23:36:48.352 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-04-14T23:36:48.352 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/tun.ko 2024-04-14T23:36:48.352 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/veth.ko 2024-04-14T23:36:48.352 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/virtio_net.ko 2024-04-14T23:36:48.352 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/vmxnet3 2024-04-14T23:36:48.352 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-04-14T23:36:48.352 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/vxlan 2024-04-14T23:36:48.352 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/vxlan/vxlan.ko 2024-04-14T23:36:48.352 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/xen-netback 2024-04-14T23:36:48.353 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/xen-netback/xen-netback.ko 2024-04-14T23:36:48.353 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/xen-netfront.ko 2024-04-14T23:36:48.353 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ntb 2024-04-14T23:36:48.353 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ntb/ntb.ko 2024-04-14T23:36:48.353 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/nvme 2024-04-14T23:36:48.353 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/nvme/host 2024-04-14T23:36:48.353 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/nvme/host/nvme-core.ko 2024-04-14T23:36:48.353 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/nvme/host/nvme.ko 2024-04-14T23:36:48.353 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport 2024-04-14T23:36:48.353 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/parport.ko 2024-04-14T23:36:48.353 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/parport_cs.ko 2024-04-14T23:36:48.353 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/parport_pc.ko 2024-04-14T23:36:48.353 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/parport_serial.ko 2024-04-14T23:36:48.353 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci 2024-04-14T23:36:48.354 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/hotplug 2024-04-14T23:36:48.354 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-04-14T23:36:48.354 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/pcie 2024-04-14T23:36:48.354 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/pcie/aer_inject.ko 2024-04-14T23:36:48.354 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/xen-pcifront.ko 2024-04-14T23:36:48.354 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia 2024-04-14T23:36:48.354 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia/i82092.ko 2024-04-14T23:36:48.354 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia/pd6729.ko 2024-04-14T23:36:48.354 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia/yenta_socket.ko 2024-04-14T23:36:48.354 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pinctrl 2024-04-14T23:36:48.354 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pinctrl/intel 2024-04-14T23:36:48.354 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-04-14T23:36:48.354 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform 2024-04-14T23:36:48.354 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86 2024-04-14T23:36:48.354 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/acer-wmi.ko 2024-04-14T23:36:48.355 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/acerhdf.ko 2024-04-14T23:36:48.355 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/apple-gmux.ko 2024-04-14T23:36:48.355 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/asus-laptop.ko 2024-04-14T23:36:48.355 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-04-14T23:36:48.355 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/asus-wmi.ko 2024-04-14T23:36:48.355 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/classmate-laptop.ko 2024-04-14T23:36:48.355 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-04-14T23:36:48.355 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-04-14T23:36:48.355 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-04-14T23:36:48.355 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-04-14T23:36:48.355 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/hdaps.ko 2024-04-14T23:36:48.355 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/intel 2024-04-14T23:36:48.355 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-04-14T23:36:48.355 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/intel_ips.ko 2024-04-14T23:36:48.355 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/msi-wmi.ko 2024-04-14T23:36:48.356 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/mxm-wmi.ko 2024-04-14T23:36:48.356 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-04-14T23:36:48.356 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/samsung-laptop.ko 2024-04-14T23:36:48.356 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/samsung-q10.ko 2024-04-14T23:36:48.356 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/topstar-laptop.ko 2024-04-14T23:36:48.356 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-04-14T23:36:48.356 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/toshiba_haps.ko 2024-04-14T23:36:48.356 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/wmi-bmof.ko 2024-04-14T23:36:48.356 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/wmi.ko 2024-04-14T23:36:48.356 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/powercap 2024-04-14T23:36:48.356 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/powercap/intel_rapl_common.ko 2024-04-14T23:36:48.356 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/powercap/intel_rapl_msr.ko 2024-04-14T23:36:48.356 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps 2024-04-14T23:36:48.356 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/clients 2024-04-14T23:36:48.357 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/clients/pps-gpio.ko 2024-04-14T23:36:48.357 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/clients/pps-ldisc.ko 2024-04-14T23:36:48.357 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/clients/pps_parport.ko 2024-04-14T23:36:48.357 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/pps_core.ko 2024-04-14T23:36:48.357 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ptp 2024-04-14T23:36:48.357 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ptp/ptp.ko 2024-04-14T23:36:48.357 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ptp/ptp_kvm.ko 2024-04-14T23:36:48.357 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras 2024-04-14T23:36:48.357 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/amd 2024-04-14T23:36:48.357 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/amd/atl 2024-04-14T23:36:48.357 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-04-14T23:36:48.357 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/amd/fmpm.ko 2024-04-14T23:36:48.357 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc 2024-04-14T23:36:48.357 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-bq32k.ko 2024-04-14T23:36:48.357 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1286.ko 2024-04-14T23:36:48.358 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1307.ko 2024-04-14T23:36:48.358 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1374.ko 2024-04-14T23:36:48.358 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1511.ko 2024-04-14T23:36:48.358 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1553.ko 2024-04-14T23:36:48.358 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1672.ko 2024-04-14T23:36:48.358 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1742.ko 2024-04-14T23:36:48.358 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds2404.ko 2024-04-14T23:36:48.358 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds3232.ko 2024-04-14T23:36:48.358 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-em3027.ko 2024-04-14T23:36:48.358 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-fm3130.ko 2024-04-14T23:36:48.358 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-isl12022.ko 2024-04-14T23:36:48.358 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-isl1208.ko 2024-04-14T23:36:48.358 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-m41t80.ko 2024-04-14T23:36:48.358 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-m48t35.ko 2024-04-14T23:36:48.359 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-m48t59.ko 2024-04-14T23:36:48.359 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-max6900.ko 2024-04-14T23:36:48.359 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-msm6242.ko 2024-04-14T23:36:48.359 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf2127.ko 2024-04-14T23:36:48.359 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf85063.ko 2024-04-14T23:36:48.359 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf8523.ko 2024-04-14T23:36:48.359 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf8563.ko 2024-04-14T23:36:48.359 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf8583.ko 2024-04-14T23:36:48.359 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rp5c01.ko 2024-04-14T23:36:48.359 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rs5c372.ko 2024-04-14T23:36:48.359 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-04-14T23:36:48.359 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rx8025.ko 2024-04-14T23:36:48.359 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rx8581.ko 2024-04-14T23:36:48.359 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-04-14T23:36:48.359 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-x1205.ko 2024-04-14T23:36:48.360 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi 2024-04-14T23:36:48.360 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/3w-9xxx.ko 2024-04-14T23:36:48.360 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/3w-sas.ko 2024-04-14T23:36:48.360 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/3w-xxxx.ko 2024-04-14T23:36:48.360 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/BusLogic.ko 2024-04-14T23:36:48.360 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/a100u2w.ko 2024-04-14T23:36:48.360 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aacraid 2024-04-14T23:36:48.360 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aacraid/aacraid.ko 2024-04-14T23:36:48.360 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/advansys.ko 2024-04-14T23:36:48.360 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aic7xxx 2024-04-14T23:36:48.360 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-04-14T23:36:48.360 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-04-14T23:36:48.360 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/am53c974.ko 2024-04-14T23:36:48.360 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/arcmsr 2024-04-14T23:36:48.361 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-04-14T23:36:48.361 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/atp870u.ko 2024-04-14T23:36:48.361 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/be2iscsi 2024-04-14T23:36:48.361 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-04-14T23:36:48.361 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bfa 2024-04-14T23:36:48.361 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bfa/bfa.ko 2024-04-14T23:36:48.361 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bnx2fc 2024-04-14T23:36:48.361 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-04-14T23:36:48.361 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bnx2i 2024-04-14T23:36:48.361 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-04-14T23:36:48.361 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ch.ko 2024-04-14T23:36:48.361 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/csiostor 2024-04-14T23:36:48.361 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/csiostor/csiostor.ko 2024-04-14T23:36:48.361 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi 2024-04-14T23:36:48.362 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/cxgb3i 2024-04-14T23:36:48.362 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-04-14T23:36:48.362 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/cxgb4i 2024-04-14T23:36:48.362 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-04-14T23:36:48.362 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-04-14T23:36:48.362 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/dc395x.ko 2024-04-14T23:36:48.362 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler 2024-04-14T23:36:48.362 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-04-14T23:36:48.362 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-04-14T23:36:48.362 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-04-14T23:36:48.362 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-04-14T23:36:48.362 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/dmx3191d.ko 2024-04-14T23:36:48.362 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/esas2r 2024-04-14T23:36:48.362 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/esas2r/esas2r.ko 2024-04-14T23:36:48.363 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/esp_scsi.ko 2024-04-14T23:36:48.363 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fcoe 2024-04-14T23:36:48.363 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fcoe/fcoe.ko 2024-04-14T23:36:48.363 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-04-14T23:36:48.363 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fnic 2024-04-14T23:36:48.363 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fnic/fnic.ko 2024-04-14T23:36:48.363 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/hpsa.ko 2024-04-14T23:36:48.363 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/hptiop.ko 2024-04-14T23:36:48.363 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/hv_storvsc.ko 2024-04-14T23:36:48.363 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/initio.ko 2024-04-14T23:36:48.363 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ipr.ko 2024-04-14T23:36:48.363 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ips.ko 2024-04-14T23:36:48.363 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/isci 2024-04-14T23:36:48.363 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/isci/isci.ko 2024-04-14T23:36:48.363 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-04-14T23:36:48.364 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/iscsi_tcp.ko 2024-04-14T23:36:48.364 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libfc 2024-04-14T23:36:48.364 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libfc/libfc.ko 2024-04-14T23:36:48.364 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libiscsi.ko 2024-04-14T23:36:48.364 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libiscsi_tcp.ko 2024-04-14T23:36:48.364 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libsas 2024-04-14T23:36:48.364 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libsas/libsas.ko 2024-04-14T23:36:48.364 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/lpfc 2024-04-14T23:36:48.364 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/lpfc/lpfc.ko 2024-04-14T23:36:48.364 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid 2024-04-14T23:36:48.364 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid.ko 2024-04-14T23:36:48.364 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-04-14T23:36:48.364 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-04-14T23:36:48.364 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-04-14T23:36:48.364 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mpt3sas 2024-04-14T23:36:48.365 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-04-14T23:36:48.365 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mvsas 2024-04-14T23:36:48.365 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mvsas/mvsas.ko 2024-04-14T23:36:48.365 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mvumi.ko 2024-04-14T23:36:48.365 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/pm8001 2024-04-14T23:36:48.365 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-04-14T23:36:48.365 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/pmcraid.ko 2024-04-14T23:36:48.365 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla1280.ko 2024-04-14T23:36:48.365 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla2xxx 2024-04-14T23:36:48.365 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-04-14T23:36:48.365 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-04-14T23:36:48.365 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla4xxx 2024-04-14T23:36:48.365 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-04-14T23:36:48.365 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/raid_class.ko 2024-04-14T23:36:48.365 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_debug.ko 2024-04-14T23:36:48.365 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_fc.ko 2024-04-14T23:36:48.366 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-04-14T23:36:48.366 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_sas.ko 2024-04-14T23:36:48.366 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_spi.ko 2024-04-14T23:36:48.366 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_srp.ko 2024-04-14T23:36:48.366 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ses.ko 2024-04-14T23:36:48.366 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/st.ko 2024-04-14T23:36:48.366 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/stex.ko 2024-04-14T23:36:48.366 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/sym53c8xx_2 2024-04-14T23:36:48.366 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-04-14T23:36:48.366 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/virtio_scsi.ko 2024-04-14T23:36:48.366 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/vmw_pvscsi.ko 2024-04-14T23:36:48.366 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/wd719x.ko 2024-04-14T23:36:48.366 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ssb 2024-04-14T23:36:48.366 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ssb/ssb.ko 2024-04-14T23:36:48.366 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target 2024-04-14T23:36:48.367 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/iscsi 2024-04-14T23:36:48.367 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-04-14T23:36:48.367 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/loopback 2024-04-14T23:36:48.367 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/loopback/tcm_loop.ko 2024-04-14T23:36:48.367 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_file.ko 2024-04-14T23:36:48.367 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_iblock.ko 2024-04-14T23:36:48.367 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_mod.ko 2024-04-14T23:36:48.367 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_pscsi.ko 2024-04-14T23:36:48.367 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_user.ko 2024-04-14T23:36:48.367 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/tcm_fc 2024-04-14T23:36:48.367 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-04-14T23:36:48.367 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal 2024-04-14T23:36:48.367 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel 2024-04-14T23:36:48.367 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal 2024-04-14T23:36:48.367 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-04-14T23:36:48.368 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-04-14T23:36:48.368 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-04-14T23:36:48.368 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-04-14T23:36:48.368 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-04-14T23:36:48.368 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-04-14T23:36:48.368 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-04-14T23:36:48.368 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-04-14T23:36:48.368 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-04-14T23:36:48.368 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-04-14T23:36:48.368 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-04-14T23:36:48.368 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-04-14T23:36:48.368 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-04-14T23:36:48.368 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-04-14T23:36:48.368 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-04-14T23:36:48.368 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-04-14T23:36:48.369 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-04-14T23:36:48.369 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-04-14T23:36:48.369 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty 2024-04-14T23:36:48.369 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/ipwireless 2024-04-14T23:36:48.369 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-04-14T23:36:48.369 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/n_gsm.ko 2024-04-14T23:36:48.369 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/n_hdlc.ko 2024-04-14T23:36:48.369 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/nozomi.ko 2024-04-14T23:36:48.369 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial 2024-04-14T23:36:48.369 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/8250 2024-04-14T23:36:48.369 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-04-14T23:36:48.369 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/arc_uart.ko 2024-04-14T23:36:48.369 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/jsm 2024-04-14T23:36:48.369 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/jsm/jsm.ko 2024-04-14T23:36:48.369 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs 2024-04-14T23:36:48.370 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/core 2024-04-14T23:36:48.370 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/core/ufshcd-core.ko 2024-04-14T23:36:48.370 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/host 2024-04-14T23:36:48.370 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-04-14T23:36:48.370 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio 2024-04-14T23:36:48.370 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio.ko 2024-04-14T23:36:48.370 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_aec.ko 2024-04-14T23:36:48.370 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_cif.ko 2024-04-14T23:36:48.370 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_pci_generic.ko 2024-04-14T23:36:48.370 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_sercos3.ko 2024-04-14T23:36:48.370 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb 2024-04-14T23:36:48.370 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class 2024-04-14T23:36:48.370 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/cdc-acm.ko 2024-04-14T23:36:48.370 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/cdc-wdm.ko 2024-04-14T23:36:48.370 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/usblp.ko 2024-04-14T23:36:48.371 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/usbtmc.ko 2024-04-14T23:36:48.371 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/host 2024-04-14T23:36:48.371 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/host/sl811-hcd.ko 2024-04-14T23:36:48.371 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/image 2024-04-14T23:36:48.371 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/image/mdc800.ko 2024-04-14T23:36:48.371 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/image/microtek.ko 2024-04-14T23:36:48.371 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc 2024-04-14T23:36:48.371 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/adutux.ko 2024-04-14T23:36:48.371 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/appledisplay.ko 2024-04-14T23:36:48.371 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/emi26.ko 2024-04-14T23:36:48.371 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/emi62.ko 2024-04-14T23:36:48.371 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/ezusb.ko 2024-04-14T23:36:48.371 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/idmouse.ko 2024-04-14T23:36:48.371 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/iowarrior.ko 2024-04-14T23:36:48.372 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/isight_firmware.ko 2024-04-14T23:36:48.372 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/ldusb.ko 2024-04-14T23:36:48.372 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/legousbtower.ko 2024-04-14T23:36:48.372 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/sisusbvga 2024-04-14T23:36:48.372 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-04-14T23:36:48.372 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/trancevibrator.ko 2024-04-14T23:36:48.372 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/usb3503.ko 2024-04-14T23:36:48.372 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/usblcd.ko 2024-04-14T23:36:48.372 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/usbsevseg.ko 2024-04-14T23:36:48.372 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/uss720.ko 2024-04-14T23:36:48.372 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/yurex.ko 2024-04-14T23:36:48.372 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/phy 2024-04-14T23:36:48.372 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/phy/phy-generic.ko 2024-04-14T23:36:48.372 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial 2024-04-14T23:36:48.372 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/aircable.ko 2024-04-14T23:36:48.372 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ark3116.ko 2024-04-14T23:36:48.373 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/belkin_sa.ko 2024-04-14T23:36:48.373 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ch341.ko 2024-04-14T23:36:48.373 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/cp210x.ko 2024-04-14T23:36:48.373 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/cyberjack.ko 2024-04-14T23:36:48.373 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/cypress_m8.ko 2024-04-14T23:36:48.373 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/digi_acceleport.ko 2024-04-14T23:36:48.373 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/empeg.ko 2024-04-14T23:36:48.373 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ftdi_sio.ko 2024-04-14T23:36:48.373 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/garmin_gps.ko 2024-04-14T23:36:48.373 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/io_edgeport.ko 2024-04-14T23:36:48.373 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/io_ti.ko 2024-04-14T23:36:48.373 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ipaq.ko 2024-04-14T23:36:48.373 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ipw.ko 2024-04-14T23:36:48.373 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ir-usb.ko 2024-04-14T23:36:48.373 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-04-14T23:36:48.373 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/keyspan.ko 2024-04-14T23:36:48.374 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/keyspan_pda.ko 2024-04-14T23:36:48.374 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/kl5kusb105.ko 2024-04-14T23:36:48.374 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/kobil_sct.ko 2024-04-14T23:36:48.374 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/mct_u232.ko 2024-04-14T23:36:48.374 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/mos7720.ko 2024-04-14T23:36:48.374 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/mos7840.ko 2024-04-14T23:36:48.374 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/navman.ko 2024-04-14T23:36:48.374 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/omninet.ko 2024-04-14T23:36:48.374 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/opticon.ko 2024-04-14T23:36:48.374 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/option.ko 2024-04-14T23:36:48.374 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/oti6858.ko 2024-04-14T23:36:48.374 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/pl2303.ko 2024-04-14T23:36:48.374 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/qcaux.ko 2024-04-14T23:36:48.374 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/qcserial.ko 2024-04-14T23:36:48.375 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/quatech2.ko 2024-04-14T23:36:48.375 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/safe_serial.ko 2024-04-14T23:36:48.375 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/sierra.ko 2024-04-14T23:36:48.375 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/spcp8x5.ko 2024-04-14T23:36:48.375 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ssu100.ko 2024-04-14T23:36:48.375 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/symbolserial.ko 2024-04-14T23:36:48.375 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-04-14T23:36:48.375 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-04-14T23:36:48.375 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/usb_debug.ko 2024-04-14T23:36:48.375 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/usb_wwan.ko 2024-04-14T23:36:48.375 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/visor.ko 2024-04-14T23:36:48.375 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/whiteheat.ko 2024-04-14T23:36:48.375 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/xsens_mt.ko 2024-04-14T23:36:48.375 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage 2024-04-14T23:36:48.375 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/uas.ko 2024-04-14T23:36:48.375 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-alauda.ko 2024-04-14T23:36:48.376 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-cypress.ko 2024-04-14T23:36:48.376 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-datafab.ko 2024-04-14T23:36:48.376 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-04-14T23:36:48.376 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-freecom.ko 2024-04-14T23:36:48.376 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-isd200.ko 2024-04-14T23:36:48.376 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-04-14T23:36:48.376 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-karma.ko 2024-04-14T23:36:48.376 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-onetouch.ko 2024-04-14T23:36:48.376 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-realtek.ko 2024-04-14T23:36:48.376 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-sddr09.ko 2024-04-14T23:36:48.376 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-sddr55.ko 2024-04-14T23:36:48.376 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-usbat.ko 2024-04-14T23:36:48.376 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/usb-storage.ko 2024-04-14T23:36:48.376 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/usbip 2024-04-14T23:36:48.376 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/usbip/usbip-core.ko 2024-04-14T23:36:48.377 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/usbip/usbip-host.ko 2024-04-14T23:36:48.377 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-04-14T23:36:48.377 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio 2024-04-14T23:36:48.377 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/pci 2024-04-14T23:36:48.377 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-04-14T23:36:48.377 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/pci/vfio-pci.ko 2024-04-14T23:36:48.377 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/vfio.ko 2024-04-14T23:36:48.377 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-04-14T23:36:48.377 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost 2024-04-14T23:36:48.377 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/vhost.ko 2024-04-14T23:36:48.377 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/vhost_iotlb.ko 2024-04-14T23:36:48.377 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/vhost_net.ko 2024-04-14T23:36:48.377 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/vhost_scsi.ko 2024-04-14T23:36:48.377 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video 2024-04-14T23:36:48.377 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight 2024-04-14T23:36:48.378 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/apple_bl.ko 2024-04-14T23:36:48.378 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/lcd.ko 2024-04-14T23:36:48.378 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/platform_lcd.ko 2024-04-14T23:36:48.378 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio 2024-04-14T23:36:48.378 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio.ko 2024-04-14T23:36:48.378 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_balloon.ko 2024-04-14T23:36:48.378 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_mmio.ko 2024-04-14T23:36:48.378 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_pci.ko 2024-04-14T23:36:48.378 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-04-14T23:36:48.378 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-04-14T23:36:48.378 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_ring.ko 2024-04-14T23:36:48.378 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1 2024-04-14T23:36:48.378 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/masters 2024-04-14T23:36:48.378 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/masters/ds2482.ko 2024-04-14T23:36:48.378 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/masters/ds2490.ko 2024-04-14T23:36:48.379 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves 2024-04-14T23:36:48.379 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-04-14T23:36:48.379 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-04-14T23:36:48.379 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-04-14T23:36:48.379 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-04-14T23:36:48.379 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-04-14T23:36:48.379 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-04-14T23:36:48.379 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-04-14T23:36:48.379 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-04-14T23:36:48.379 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-04-14T23:36:48.379 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_smem.ko 2024-04-14T23:36:48.379 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_therm.ko 2024-04-14T23:36:48.379 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/wire.ko 2024-04-14T23:36:48.380 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog 2024-04-14T23:36:48.380 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/alim1535_wdt.ko 2024-04-14T23:36:48.380 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/alim7101_wdt.ko 2024-04-14T23:36:48.380 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/f71808e_wdt.ko 2024-04-14T23:36:48.380 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/hpwdt.ko 2024-04-14T23:36:48.380 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/i6300esb.ko 2024-04-14T23:36:48.380 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-04-14T23:36:48.380 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/iTCO_wdt.ko 2024-04-14T23:36:48.380 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/ib700wdt.ko 2024-04-14T23:36:48.380 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/ibmasr.ko 2024-04-14T23:36:48.380 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-04-14T23:36:48.380 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/it8712f_wdt.ko 2024-04-14T23:36:48.380 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/it87_wdt.ko 2024-04-14T23:36:48.380 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/machzwd.ko 2024-04-14T23:36:48.380 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/nv_tco.ko 2024-04-14T23:36:48.381 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/pcwd_pci.ko 2024-04-14T23:36:48.381 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/pcwd_usb.ko 2024-04-14T23:36:48.381 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-04-14T23:36:48.381 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/sch311x_wdt.ko 2024-04-14T23:36:48.381 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/softdog.ko 2024-04-14T23:36:48.381 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/sp5100_tco.ko 2024-04-14T23:36:48.381 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/via_wdt.ko 2024-04-14T23:36:48.381 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-04-14T23:36:48.381 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/w83877f_wdt.ko 2024-04-14T23:36:48.381 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/w83977f_wdt.ko 2024-04-14T23:36:48.381 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/wdt_pci.ko 2024-04-14T23:36:48.381 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/xen_wdt.ko 2024-04-14T23:36:48.381 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen 2024-04-14T23:36:48.381 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-acpi-processor.ko 2024-04-14T23:36:48.381 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-evtchn.ko 2024-04-14T23:36:48.382 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-gntalloc.ko 2024-04-14T23:36:48.382 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-gntdev.ko 2024-04-14T23:36:48.382 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-pciback 2024-04-14T23:36:48.382 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-04-14T23:36:48.382 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-privcmd.ko 2024-04-14T23:36:48.382 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xenfs 2024-04-14T23:36:48.382 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xenfs/xenfs.ko 2024-04-14T23:36:48.382 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs 2024-04-14T23:36:48.382 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/9p 2024-04-14T23:36:48.382 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/9p/9p.ko 2024-04-14T23:36:48.382 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/affs 2024-04-14T23:36:48.382 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/affs/affs.ko 2024-04-14T23:36:48.382 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/befs 2024-04-14T23:36:48.382 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/befs/befs.ko 2024-04-14T23:36:48.383 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/binfmt_misc.ko 2024-04-14T23:36:48.383 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/btrfs 2024-04-14T23:36:48.383 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/btrfs/btrfs.ko 2024-04-14T23:36:48.383 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/cachefiles 2024-04-14T23:36:48.383 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/cachefiles/cachefiles.ko 2024-04-14T23:36:48.383 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ceph 2024-04-14T23:36:48.383 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ceph/ceph.ko 2024-04-14T23:36:48.383 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/coda 2024-04-14T23:36:48.383 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/coda/coda.ko 2024-04-14T23:36:48.383 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/cramfs 2024-04-14T23:36:48.383 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/cramfs/cramfs.ko 2024-04-14T23:36:48.383 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/dlm 2024-04-14T23:36:48.383 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/dlm/dlm.ko 2024-04-14T23:36:48.383 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ecryptfs 2024-04-14T23:36:48.383 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ecryptfs/ecryptfs.ko 2024-04-14T23:36:48.383 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/f2fs 2024-04-14T23:36:48.384 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/f2fs/f2fs.ko 2024-04-14T23:36:48.384 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fat 2024-04-14T23:36:48.384 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fat/fat.ko 2024-04-14T23:36:48.384 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fat/msdos.ko 2024-04-14T23:36:48.384 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fat/vfat.ko 2024-04-14T23:36:48.384 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fuse 2024-04-14T23:36:48.384 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fuse/cuse.ko 2024-04-14T23:36:48.384 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fuse/fuse.ko 2024-04-14T23:36:48.384 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/gfs2 2024-04-14T23:36:48.384 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/gfs2/gfs2.ko 2024-04-14T23:36:48.384 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hfs 2024-04-14T23:36:48.384 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hfs/hfs.ko 2024-04-14T23:36:48.384 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hfsplus 2024-04-14T23:36:48.384 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hfsplus/hfsplus.ko 2024-04-14T23:36:48.384 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/isofs 2024-04-14T23:36:48.385 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/isofs/isofs.ko 2024-04-14T23:36:48.385 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/jfs 2024-04-14T23:36:48.385 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/jfs/jfs.ko 2024-04-14T23:36:48.385 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/lockd 2024-04-14T23:36:48.385 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/lockd/lockd.ko 2024-04-14T23:36:48.385 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/minix 2024-04-14T23:36:48.385 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/minix/minix.ko 2024-04-14T23:36:48.385 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/netfs 2024-04-14T23:36:48.385 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/netfs/netfs.ko 2024-04-14T23:36:48.385 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs 2024-04-14T23:36:48.385 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/blocklayout 2024-04-14T23:36:48.385 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-04-14T23:36:48.385 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/filelayout 2024-04-14T23:36:48.385 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-04-14T23:36:48.385 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/flexfilelayout 2024-04-14T23:36:48.385 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-04-14T23:36:48.386 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/nfs.ko 2024-04-14T23:36:48.386 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/nfsv3.ko 2024-04-14T23:36:48.386 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/nfsv4.ko 2024-04-14T23:36:48.386 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs_common 2024-04-14T23:36:48.386 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs_common/grace.ko 2024-04-14T23:36:48.386 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs_common/nfs_acl.ko 2024-04-14T23:36:48.386 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfsd 2024-04-14T23:36:48.386 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfsd/nfsd.ko 2024-04-14T23:36:48.386 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nilfs2 2024-04-14T23:36:48.386 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nilfs2/nilfs2.ko 2024-04-14T23:36:48.386 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls 2024-04-14T23:36:48.386 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-celtic.ko 2024-04-14T23:36:48.386 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-centeuro.ko 2024-04-14T23:36:48.386 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-croatian.ko 2024-04-14T23:36:48.386 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-cyrillic.ko 2024-04-14T23:36:48.387 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-gaelic.ko 2024-04-14T23:36:48.387 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-greek.ko 2024-04-14T23:36:48.387 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-iceland.ko 2024-04-14T23:36:48.387 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-inuit.ko 2024-04-14T23:36:48.387 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-roman.ko 2024-04-14T23:36:48.387 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-romanian.ko 2024-04-14T23:36:48.387 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-turkish.ko 2024-04-14T23:36:48.387 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp1250.ko 2024-04-14T23:36:48.387 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp1251.ko 2024-04-14T23:36:48.387 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp1255.ko 2024-04-14T23:36:48.387 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp737.ko 2024-04-14T23:36:48.387 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp775.ko 2024-04-14T23:36:48.387 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp850.ko 2024-04-14T23:36:48.387 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp852.ko 2024-04-14T23:36:48.387 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp855.ko 2024-04-14T23:36:48.387 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp857.ko 2024-04-14T23:36:48.388 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp860.ko 2024-04-14T23:36:48.388 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp861.ko 2024-04-14T23:36:48.388 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp862.ko 2024-04-14T23:36:48.388 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp863.ko 2024-04-14T23:36:48.388 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp864.ko 2024-04-14T23:36:48.388 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp865.ko 2024-04-14T23:36:48.388 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp866.ko 2024-04-14T23:36:48.388 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp869.ko 2024-04-14T23:36:48.388 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp874.ko 2024-04-14T23:36:48.388 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp932.ko 2024-04-14T23:36:48.388 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp936.ko 2024-04-14T23:36:48.388 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp949.ko 2024-04-14T23:36:48.388 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp950.ko 2024-04-14T23:36:48.388 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_euc-jp.ko 2024-04-14T23:36:48.388 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-1.ko 2024-04-14T23:36:48.388 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-13.ko 2024-04-14T23:36:48.389 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-14.ko 2024-04-14T23:36:48.389 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-15.ko 2024-04-14T23:36:48.389 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-2.ko 2024-04-14T23:36:48.389 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-3.ko 2024-04-14T23:36:48.389 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-4.ko 2024-04-14T23:36:48.389 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-5.ko 2024-04-14T23:36:48.389 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-6.ko 2024-04-14T23:36:48.389 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-7.ko 2024-04-14T23:36:48.389 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-9.ko 2024-04-14T23:36:48.389 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_koi8-r.ko 2024-04-14T23:36:48.389 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_koi8-ru.ko 2024-04-14T23:36:48.389 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_koi8-u.ko 2024-04-14T23:36:48.389 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_ucs2_utils.ko 2024-04-14T23:36:48.389 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_utf8.ko 2024-04-14T23:36:48.389 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2 2024-04-14T23:36:48.389 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/cluster 2024-04-14T23:36:48.390 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-04-14T23:36:48.390 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/dlm 2024-04-14T23:36:48.390 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-04-14T23:36:48.390 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/dlmfs 2024-04-14T23:36:48.390 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-04-14T23:36:48.390 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ocfs2.ko 2024-04-14T23:36:48.390 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-04-14T23:36:48.390 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-04-14T23:36:48.390 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-04-14T23:36:48.390 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/overlayfs 2024-04-14T23:36:48.390 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/overlayfs/overlay.ko 2024-04-14T23:36:48.390 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/pstore 2024-04-14T23:36:48.390 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/pstore/ramoops.ko 2024-04-14T23:36:48.390 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/reiserfs 2024-04-14T23:36:48.390 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/reiserfs/reiserfs.ko 2024-04-14T23:36:48.391 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/romfs 2024-04-14T23:36:48.391 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/romfs/romfs.ko 2024-04-14T23:36:48.391 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb 2024-04-14T23:36:48.391 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/client 2024-04-14T23:36:48.391 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/client/cifs.ko 2024-04-14T23:36:48.391 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/common 2024-04-14T23:36:48.391 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/common/cifs_arc4.ko 2024-04-14T23:36:48.391 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/common/cifs_md4.ko 2024-04-14T23:36:48.391 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/squashfs 2024-04-14T23:36:48.391 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/squashfs/squashfs.ko 2024-04-14T23:36:48.391 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/sysv 2024-04-14T23:36:48.391 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/sysv/sysv.ko 2024-04-14T23:36:48.391 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/udf 2024-04-14T23:36:48.391 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/udf/udf.ko 2024-04-14T23:36:48.392 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ufs 2024-04-14T23:36:48.392 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ufs/ufs.ko 2024-04-14T23:36:48.392 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/xfs 2024-04-14T23:36:48.392 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/xfs/xfs.ko 2024-04-14T23:36:48.392 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel 2024-04-14T23:36:48.392 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/rcu 2024-04-14T23:36:48.392 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/rcu/rcutorture.ko 2024-04-14T23:36:48.392 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/torture.ko 2024-04-14T23:36:48.392 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/trace 2024-04-14T23:36:48.392 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-04-14T23:36:48.392 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib 2024-04-14T23:36:48.392 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/asn1_encoder.ko 2024-04-14T23:36:48.392 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crc-itu-t.ko 2024-04-14T23:36:48.392 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crc8.ko 2024-04-14T23:36:48.393 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crypto 2024-04-14T23:36:48.393 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crypto/libarc4.ko 2024-04-14T23:36:48.393 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crypto/libdes.ko 2024-04-14T23:36:48.393 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/libcrc32c.ko 2024-04-14T23:36:48.393 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/lru_cache.ko 2024-04-14T23:36:48.393 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/lz4 2024-04-14T23:36:48.393 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/lz4/lz4_compress.ko 2024-04-14T23:36:48.393 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/lz4/lz4hc_compress.ko 2024-04-14T23:36:48.393 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/math 2024-04-14T23:36:48.393 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/math/cordic.ko 2024-04-14T23:36:48.393 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/raid6 2024-04-14T23:36:48.393 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/raid6/raid6_pq.ko 2024-04-14T23:36:48.393 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/reed_solomon 2024-04-14T23:36:48.393 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/reed_solomon/reed_solomon.ko 2024-04-14T23:36:48.393 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/ts_bm.ko 2024-04-14T23:36:48.393 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/ts_fsm.ko 2024-04-14T23:36:48.394 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/ts_kmp.ko 2024-04-14T23:36:48.394 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/zstd 2024-04-14T23:36:48.394 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/zstd/zstd_compress.ko 2024-04-14T23:36:48.394 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/mm 2024-04-14T23:36:48.394 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/mm/hwpoison-inject.ko 2024-04-14T23:36:48.394 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net 2024-04-14T23:36:48.394 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan 2024-04-14T23:36:48.394 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/6lowpan.ko 2024-04-14T23:36:48.394 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_dest.ko 2024-04-14T23:36:48.394 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_fragment.ko 2024-04-14T23:36:48.394 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_hop.ko 2024-04-14T23:36:48.394 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_ipv6.ko 2024-04-14T23:36:48.394 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_mobility.ko 2024-04-14T23:36:48.394 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_routing.ko 2024-04-14T23:36:48.395 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_udp.ko 2024-04-14T23:36:48.395 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802 2024-04-14T23:36:48.395 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/garp.ko 2024-04-14T23:36:48.395 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/mrp.ko 2024-04-14T23:36:48.395 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/p8022.ko 2024-04-14T23:36:48.395 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/psnap.ko 2024-04-14T23:36:48.395 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/stp.ko 2024-04-14T23:36:48.395 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/8021q 2024-04-14T23:36:48.395 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/8021q/8021q.ko 2024-04-14T23:36:48.395 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p 2024-04-14T23:36:48.395 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/9pnet.ko 2024-04-14T23:36:48.395 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/9pnet_fd.ko 2024-04-14T23:36:48.395 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/9pnet_rdma.ko 2024-04-14T23:36:48.395 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/9pnet_virtio.ko 2024-04-14T23:36:48.395 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/batman-adv 2024-04-14T23:36:48.395 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/batman-adv/batman-adv.ko 2024-04-14T23:36:48.396 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge 2024-04-14T23:36:48.396 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/br_netfilter.ko 2024-04-14T23:36:48.396 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/bridge.ko 2024-04-14T23:36:48.396 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter 2024-04-14T23:36:48.396 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-04-14T23:36:48.396 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_among.ko 2024-04-14T23:36:48.396 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_arp.ko 2024-04-14T23:36:48.396 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-04-14T23:36:48.396 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-04-14T23:36:48.396 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_ip.ko 2024-04-14T23:36:48.396 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-04-14T23:36:48.396 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_limit.ko 2024-04-14T23:36:48.396 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_log.ko 2024-04-14T23:36:48.396 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_mark.ko 2024-04-14T23:36:48.396 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-04-14T23:36:48.397 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-04-14T23:36:48.397 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-04-14T23:36:48.397 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-04-14T23:36:48.397 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_snat.ko 2024-04-14T23:36:48.397 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_stp.ko 2024-04-14T23:36:48.397 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-04-14T23:36:48.397 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-04-14T23:36:48.397 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-04-14T23:36:48.397 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-04-14T23:36:48.397 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebtables.ko 2024-04-14T23:36:48.397 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-04-14T23:36:48.397 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-04-14T23:36:48.397 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-04-14T23:36:48.397 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ceph 2024-04-14T23:36:48.397 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ceph/libceph.ko 2024-04-14T23:36:48.398 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/core 2024-04-14T23:36:48.398 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/core/failover.ko 2024-04-14T23:36:48.398 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/core/pktgen.ko 2024-04-14T23:36:48.398 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp 2024-04-14T23:36:48.398 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/dccp.ko 2024-04-14T23:36:48.398 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/dccp_diag.ko 2024-04-14T23:36:48.398 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/dccp_ipv4.ko 2024-04-14T23:36:48.398 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/dccp_ipv6.ko 2024-04-14T23:36:48.398 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dns_resolver 2024-04-14T23:36:48.398 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dns_resolver/dns_resolver.ko 2024-04-14T23:36:48.398 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa 2024-04-14T23:36:48.398 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/dsa_core.ko 2024-04-14T23:36:48.398 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/tag_brcm.ko 2024-04-14T23:36:48.398 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/tag_dsa.ko 2024-04-14T23:36:48.399 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/tag_none.ko 2024-04-14T23:36:48.399 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/tag_trailer.ko 2024-04-14T23:36:48.399 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154 2024-04-14T23:36:48.399 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/6lowpan 2024-04-14T23:36:48.399 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-04-14T23:36:48.399 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/ieee802154.ko 2024-04-14T23:36:48.399 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/ieee802154_socket.ko 2024-04-14T23:36:48.399 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4 2024-04-14T23:36:48.399 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ah4.ko 2024-04-14T23:36:48.399 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/esp4.ko 2024-04-14T23:36:48.399 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/fou.ko 2024-04-14T23:36:48.399 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/gre.ko 2024-04-14T23:36:48.399 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/inet_diag.ko 2024-04-14T23:36:48.399 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ip_gre.ko 2024-04-14T23:36:48.399 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ip_tunnel.ko 2024-04-14T23:36:48.399 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ip_vti.ko 2024-04-14T23:36:48.400 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ipcomp.ko 2024-04-14T23:36:48.400 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ipip.ko 2024-04-14T23:36:48.400 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter 2024-04-14T23:36:48.400 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/arp_tables.ko 2024-04-14T23:36:48.400 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-04-14T23:36:48.400 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-04-14T23:36:48.400 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-04-14T23:36:48.400 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-04-14T23:36:48.400 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-04-14T23:36:48.400 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-04-14T23:36:48.400 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-04-14T23:36:48.400 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-04-14T23:36:48.400 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-04-14T23:36:48.400 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_security.ko 2024-04-14T23:36:48.400 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-04-14T23:36:48.401 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-04-14T23:36:48.401 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-04-14T23:36:48.401 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-04-14T23:36:48.401 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-04-14T23:36:48.401 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-04-14T23:36:48.401 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-04-14T23:36:48.401 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-04-14T23:36:48.401 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-04-14T23:36:48.401 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-04-14T23:36:48.401 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_bic.ko 2024-04-14T23:36:48.401 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_dctcp.ko 2024-04-14T23:36:48.401 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_diag.ko 2024-04-14T23:36:48.401 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_highspeed.ko 2024-04-14T23:36:48.401 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_htcp.ko 2024-04-14T23:36:48.402 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_hybla.ko 2024-04-14T23:36:48.402 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_illinois.ko 2024-04-14T23:36:48.402 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_lp.ko 2024-04-14T23:36:48.402 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_scalable.ko 2024-04-14T23:36:48.402 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_vegas.ko 2024-04-14T23:36:48.402 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_veno.ko 2024-04-14T23:36:48.402 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_westwood.ko 2024-04-14T23:36:48.402 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_yeah.ko 2024-04-14T23:36:48.402 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tunnel4.ko 2024-04-14T23:36:48.402 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/udp_diag.ko 2024-04-14T23:36:48.402 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/udp_tunnel.ko 2024-04-14T23:36:48.402 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/xfrm4_tunnel.ko 2024-04-14T23:36:48.402 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6 2024-04-14T23:36:48.402 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ah6.ko 2024-04-14T23:36:48.402 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/esp6.ko 2024-04-14T23:36:48.403 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/fou6.ko 2024-04-14T23:36:48.403 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ip6_tunnel.ko 2024-04-14T23:36:48.403 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-04-14T23:36:48.403 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ip6_vti.ko 2024-04-14T23:36:48.403 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ipcomp6.ko 2024-04-14T23:36:48.403 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter 2024-04-14T23:36:48.403 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-04-14T23:36:48.403 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-04-14T23:36:48.403 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-04-14T23:36:48.403 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-04-14T23:36:48.403 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-04-14T23:36:48.403 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-04-14T23:36:48.403 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-04-14T23:36:48.403 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-04-14T23:36:48.404 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-04-14T23:36:48.404 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-04-14T23:36:48.404 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-04-14T23:36:48.404 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-04-14T23:36:48.404 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-04-14T23:36:48.404 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-04-14T23:36:48.404 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-04-14T23:36:48.404 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-04-14T23:36:48.404 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-04-14T23:36:48.404 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-04-14T23:36:48.404 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-04-14T23:36:48.404 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-04-14T23:36:48.404 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-04-14T23:36:48.404 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-04-14T23:36:48.404 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-04-14T23:36:48.405 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-04-14T23:36:48.405 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/sit.ko 2024-04-14T23:36:48.405 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/tunnel6.ko 2024-04-14T23:36:48.405 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/xfrm6_tunnel.ko 2024-04-14T23:36:48.405 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/key 2024-04-14T23:36:48.405 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/key/af_key.ko 2024-04-14T23:36:48.405 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp 2024-04-14T23:36:48.405 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_core.ko 2024-04-14T23:36:48.405 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_debugfs.ko 2024-04-14T23:36:48.405 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_eth.ko 2024-04-14T23:36:48.405 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_ip.ko 2024-04-14T23:36:48.405 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_ip6.ko 2024-04-14T23:36:48.405 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_netlink.ko 2024-04-14T23:36:48.405 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_ppp.ko 2024-04-14T23:36:48.405 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/llc 2024-04-14T23:36:48.406 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/llc/llc.ko 2024-04-14T23:36:48.406 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/mac802154 2024-04-14T23:36:48.406 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/mac802154/mac802154.ko 2024-04-14T23:36:48.406 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/mpls 2024-04-14T23:36:48.406 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/mpls/mpls_gso.ko 2024-04-14T23:36:48.406 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter 2024-04-14T23:36:48.406 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset 2024-04-14T23:36:48.406 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set.ko 2024-04-14T23:36:48.406 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-04-14T23:36:48.406 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-04-14T23:36:48.406 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-04-14T23:36:48.406 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-04-14T23:36:48.406 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-04-14T23:36:48.406 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-04-14T23:36:48.406 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-04-14T23:36:48.407 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-04-14T23:36:48.407 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-04-14T23:36:48.407 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-04-14T23:36:48.407 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-04-14T23:36:48.407 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-04-14T23:36:48.407 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-04-14T23:36:48.407 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-04-14T23:36:48.407 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-04-14T23:36:48.407 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs 2024-04-14T23:36:48.407 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs.ko 2024-04-14T23:36:48.407 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-04-14T23:36:48.407 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-04-14T23:36:48.407 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-04-14T23:36:48.407 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-04-14T23:36:48.407 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-04-14T23:36:48.408 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-04-14T23:36:48.408 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-04-14T23:36:48.408 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-04-14T23:36:48.408 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-04-14T23:36:48.408 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-04-14T23:36:48.408 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-04-14T23:36:48.408 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-04-14T23:36:48.408 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-04-14T23:36:48.408 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conncount.ko 2024-04-14T23:36:48.408 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack.ko 2024-04-14T23:36:48.408 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-04-14T23:36:48.408 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-04-14T23:36:48.408 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-04-14T23:36:48.408 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_h323.ko 2024-04-14T23:36:48.409 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_irc.ko 2024-04-14T23:36:48.409 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-04-14T23:36:48.409 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-04-14T23:36:48.409 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-04-14T23:36:48.409 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_sane.ko 2024-04-14T23:36:48.409 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_sip.ko 2024-04-14T23:36:48.409 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-04-14T23:36:48.409 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-04-14T23:36:48.409 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_dup_netdev.ko 2024-04-14T23:36:48.409 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_flow_table.ko 2024-04-14T23:36:48.409 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_flow_table_inet.ko 2024-04-14T23:36:48.409 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_log_syslog.ko 2024-04-14T23:36:48.409 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat.ko 2024-04-14T23:36:48.409 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_amanda.ko 2024-04-14T23:36:48.409 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_ftp.ko 2024-04-14T23:36:48.410 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_irc.ko 2024-04-14T23:36:48.410 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_sip.ko 2024-04-14T23:36:48.410 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_tftp.ko 2024-04-14T23:36:48.410 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_synproxy_core.ko 2024-04-14T23:36:48.410 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_tables.ko 2024-04-14T23:36:48.410 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink.ko 2024-04-14T23:36:48.410 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_acct.ko 2024-04-14T23:36:48.410 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_log.ko 2024-04-14T23:36:48.410 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_osf.ko 2024-04-14T23:36:48.410 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_queue.ko 2024-04-14T23:36:48.410 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_chain_nat.ko 2024-04-14T23:36:48.410 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_compat.ko 2024-04-14T23:36:48.410 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_ct.ko 2024-04-14T23:36:48.410 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_dup_netdev.ko 2024-04-14T23:36:48.411 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_fib.ko 2024-04-14T23:36:48.411 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_fwd_netdev.ko 2024-04-14T23:36:48.411 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_hash.ko 2024-04-14T23:36:48.411 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_limit.ko 2024-04-14T23:36:48.411 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_log.ko 2024-04-14T23:36:48.411 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_masq.ko 2024-04-14T23:36:48.411 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_nat.ko 2024-04-14T23:36:48.411 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_numgen.ko 2024-04-14T23:36:48.411 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_queue.ko 2024-04-14T23:36:48.411 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_quota.ko 2024-04-14T23:36:48.411 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_redir.ko 2024-04-14T23:36:48.411 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_reject.ko 2024-04-14T23:36:48.411 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_reject_inet.ko 2024-04-14T23:36:48.411 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_socket.ko 2024-04-14T23:36:48.411 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_tunnel.ko 2024-04-14T23:36:48.412 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_AUDIT.ko 2024-04-14T23:36:48.412 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_CHECKSUM.ko 2024-04-14T23:36:48.412 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_CLASSIFY.ko 2024-04-14T23:36:48.412 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-04-14T23:36:48.412 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_CT.ko 2024-04-14T23:36:48.412 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_DSCP.ko 2024-04-14T23:36:48.412 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_HL.ko 2024-04-14T23:36:48.412 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_HMARK.ko 2024-04-14T23:36:48.412 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_IDLETIMER.ko 2024-04-14T23:36:48.412 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_LED.ko 2024-04-14T23:36:48.412 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_LOG.ko 2024-04-14T23:36:48.412 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_MASQUERADE.ko 2024-04-14T23:36:48.412 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_NETMAP.ko 2024-04-14T23:36:48.412 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_NFLOG.ko 2024-04-14T23:36:48.412 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_NFQUEUE.ko 2024-04-14T23:36:48.413 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_RATEEST.ko 2024-04-14T23:36:48.413 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_REDIRECT.ko 2024-04-14T23:36:48.413 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_SECMARK.ko 2024-04-14T23:36:48.413 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TCPMSS.ko 2024-04-14T23:36:48.413 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-04-14T23:36:48.413 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TEE.ko 2024-04-14T23:36:48.413 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TPROXY.ko 2024-04-14T23:36:48.413 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TRACE.ko 2024-04-14T23:36:48.413 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_addrtype.ko 2024-04-14T23:36:48.413 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_bpf.ko 2024-04-14T23:36:48.413 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_cgroup.ko 2024-04-14T23:36:48.413 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_cluster.ko 2024-04-14T23:36:48.413 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_comment.ko 2024-04-14T23:36:48.413 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_connbytes.ko 2024-04-14T23:36:48.413 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_connlabel.ko 2024-04-14T23:36:48.414 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_connlimit.ko 2024-04-14T23:36:48.414 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_connmark.ko 2024-04-14T23:36:48.414 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_conntrack.ko 2024-04-14T23:36:48.414 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_cpu.ko 2024-04-14T23:36:48.414 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_dccp.ko 2024-04-14T23:36:48.414 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_devgroup.ko 2024-04-14T23:36:48.414 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_dscp.ko 2024-04-14T23:36:48.414 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_ecn.ko 2024-04-14T23:36:48.414 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_esp.ko 2024-04-14T23:36:48.414 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_hashlimit.ko 2024-04-14T23:36:48.414 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_helper.ko 2024-04-14T23:36:48.414 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_hl.ko 2024-04-14T23:36:48.414 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_ipcomp.ko 2024-04-14T23:36:48.414 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_iprange.ko 2024-04-14T23:36:48.414 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_ipvs.ko 2024-04-14T23:36:48.415 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_l2tp.ko 2024-04-14T23:36:48.415 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_length.ko 2024-04-14T23:36:48.415 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_limit.ko 2024-04-14T23:36:48.415 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_mac.ko 2024-04-14T23:36:48.415 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_mark.ko 2024-04-14T23:36:48.415 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_multiport.ko 2024-04-14T23:36:48.415 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_nat.ko 2024-04-14T23:36:48.415 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_nfacct.ko 2024-04-14T23:36:48.415 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_osf.ko 2024-04-14T23:36:48.415 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_owner.ko 2024-04-14T23:36:48.415 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_physdev.ko 2024-04-14T23:36:48.415 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_pkttype.ko 2024-04-14T23:36:48.415 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_policy.ko 2024-04-14T23:36:48.415 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_quota.ko 2024-04-14T23:36:48.416 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_rateest.ko 2024-04-14T23:36:48.416 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_realm.ko 2024-04-14T23:36:48.416 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_recent.ko 2024-04-14T23:36:48.416 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_sctp.ko 2024-04-14T23:36:48.416 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_set.ko 2024-04-14T23:36:48.416 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_socket.ko 2024-04-14T23:36:48.416 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_state.ko 2024-04-14T23:36:48.416 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_statistic.ko 2024-04-14T23:36:48.416 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_string.ko 2024-04-14T23:36:48.416 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_tcpmss.ko 2024-04-14T23:36:48.416 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_time.ko 2024-04-14T23:36:48.416 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_u32.ko 2024-04-14T23:36:48.416 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netlink 2024-04-14T23:36:48.416 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netlink/netlink_diag.ko 2024-04-14T23:36:48.416 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/nsh 2024-04-14T23:36:48.417 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/nsh/nsh.ko 2024-04-14T23:36:48.417 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch 2024-04-14T23:36:48.417 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/openvswitch.ko 2024-04-14T23:36:48.417 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/vport-geneve.ko 2024-04-14T23:36:48.417 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/vport-gre.ko 2024-04-14T23:36:48.417 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/vport-vxlan.ko 2024-04-14T23:36:48.417 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/packet 2024-04-14T23:36:48.417 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/packet/af_packet_diag.ko 2024-04-14T23:36:48.417 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rds 2024-04-14T23:36:48.417 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rds/rds.ko 2024-04-14T23:36:48.417 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rds/rds_rdma.ko 2024-04-14T23:36:48.417 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rds/rds_tcp.ko 2024-04-14T23:36:48.417 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched 2024-04-14T23:36:48.417 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_csum.ko 2024-04-14T23:36:48.417 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_gact.ko 2024-04-14T23:36:48.418 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_mirred.ko 2024-04-14T23:36:48.418 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_nat.ko 2024-04-14T23:36:48.418 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_pedit.ko 2024-04-14T23:36:48.418 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_police.ko 2024-04-14T23:36:48.418 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_simple.ko 2024-04-14T23:36:48.418 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_skbedit.ko 2024-04-14T23:36:48.418 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_vlan.ko 2024-04-14T23:36:48.418 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_basic.ko 2024-04-14T23:36:48.418 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_bpf.ko 2024-04-14T23:36:48.418 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_flow.ko 2024-04-14T23:36:48.418 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_fw.ko 2024-04-14T23:36:48.418 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_route.ko 2024-04-14T23:36:48.418 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_u32.ko 2024-04-14T23:36:48.418 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_cmp.ko 2024-04-14T23:36:48.418 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_ipset.ko 2024-04-14T23:36:48.419 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_meta.ko 2024-04-14T23:36:48.419 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_nbyte.ko 2024-04-14T23:36:48.419 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_text.ko 2024-04-14T23:36:48.419 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_u32.ko 2024-04-14T23:36:48.419 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_choke.ko 2024-04-14T23:36:48.419 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_codel.ko 2024-04-14T23:36:48.419 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_drr.ko 2024-04-14T23:36:48.419 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_fq.ko 2024-04-14T23:36:48.419 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_gred.ko 2024-04-14T23:36:48.419 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_hfsc.ko 2024-04-14T23:36:48.419 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_hhf.ko 2024-04-14T23:36:48.419 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_htb.ko 2024-04-14T23:36:48.419 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_ingress.ko 2024-04-14T23:36:48.419 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_mqprio.ko 2024-04-14T23:36:48.420 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_mqprio_lib.ko 2024-04-14T23:36:48.420 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_multiq.ko 2024-04-14T23:36:48.420 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_netem.ko 2024-04-14T23:36:48.420 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_pie.ko 2024-04-14T23:36:48.420 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_plug.ko 2024-04-14T23:36:48.420 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_prio.ko 2024-04-14T23:36:48.420 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_qfq.ko 2024-04-14T23:36:48.420 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_red.ko 2024-04-14T23:36:48.420 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_sfb.ko 2024-04-14T23:36:48.420 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_sfq.ko 2024-04-14T23:36:48.420 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_tbf.ko 2024-04-14T23:36:48.420 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_teql.ko 2024-04-14T23:36:48.420 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sctp 2024-04-14T23:36:48.420 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sctp/sctp.ko 2024-04-14T23:36:48.420 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sctp/sctp_diag.ko 2024-04-14T23:36:48.421 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc 2024-04-14T23:36:48.421 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/auth_gss 2024-04-14T23:36:48.421 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-04-14T23:36:48.421 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-04-14T23:36:48.421 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/sunrpc.ko 2024-04-14T23:36:48.421 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/xprtrdma 2024-04-14T23:36:48.421 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-04-14T23:36:48.421 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/tipc 2024-04-14T23:36:48.421 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/tipc/tipc.ko 2024-04-14T23:36:48.421 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/tipc/tipc_diag.ko 2024-04-14T23:36:48.421 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/unix 2024-04-14T23:36:48.421 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/unix/unix_diag.ko 2024-04-14T23:36:48.421 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock 2024-04-14T23:36:48.421 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-04-14T23:36:48.421 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-04-14T23:36:48.422 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vsock.ko 2024-04-14T23:36:48.422 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vsock_diag.ko 2024-04-14T23:36:48.422 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vsock_loopback.ko 2024-04-14T23:36:48.422 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/xfrm 2024-04-14T23:36:48.422 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/xfrm/xfrm_ipcomp.ko 2024-04-14T23:36:48.422 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security 2024-04-14T23:36:48.422 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys 2024-04-14T23:36:48.422 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys/encrypted-keys 2024-04-14T23:36:48.422 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-04-14T23:36:48.422 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys/trusted-keys 2024-04-14T23:36:48.422 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys/trusted-keys/trusted.ko 2024-04-14T23:36:48.422 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.alias 2024-04-14T23:36:48.422 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.alias.bin 2024-04-14T23:36:48.422 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.builtin 2024-04-14T23:36:48.423 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.builtin.alias.bin 2024-04-14T23:36:48.423 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.builtin.bin 2024-04-14T23:36:48.423 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.builtin.modinfo 2024-04-14T23:36:48.423 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.dep 2024-04-14T23:36:48.423 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.dep.bin 2024-04-14T23:36:48.423 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.devname 2024-04-14T23:36:48.423 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.order 2024-04-14T23:36:48.423 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.softdep 2024-04-14T23:36:48.423 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.symbols 2024-04-14T23:36:48.423 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.symbols.bin 2024-04-14T23:36:48.423 INFO:teuthology.orchestra.run.smithi060.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/vmlinuz 2024-04-14T23:36:48.427 DEBUG:teuthology.task.kernel:get_image_version: 6.9.0-rc3-g3f60ed14f7c1 2024-04-14T23:36:48.427 DEBUG:teuthology.orchestra.run.smithi060:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-04-14T23:36:48.452 INFO:teuthology.orchestra.run.smithi060.stdout:postinstall scriptlet (using /bin/sh): 2024-04-14T23:36:48.452 INFO:teuthology.orchestra.run.smithi060.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-14T23:36:48.452 INFO:teuthology.orchestra.run.smithi060.stdout: /usr/bin/kernel-install add 6.9.0-rc3-g3f60ed14f7c1 /lib/modules/6.9.0-rc3-g3f60ed14f7c1/vmlinuz 2024-04-14T23:36:48.452 INFO:teuthology.orchestra.run.smithi060.stdout:fi 2024-04-14T23:36:48.452 INFO:teuthology.orchestra.run.smithi060.stdout:for file in vmlinuz System.map config; do 2024-04-14T23:36:48.452 INFO:teuthology.orchestra.run.smithi060.stdout: if ! cmp --silent "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/${file}" "/boot/${file}-6.9.0-rc3-g3f60ed14f7c1"; then 2024-04-14T23:36:48.452 INFO:teuthology.orchestra.run.smithi060.stdout: cp "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/${file}" "/boot/${file}-6.9.0-rc3-g3f60ed14f7c1" 2024-04-14T23:36:48.452 INFO:teuthology.orchestra.run.smithi060.stdout: fi 2024-04-14T23:36:48.452 INFO:teuthology.orchestra.run.smithi060.stdout:done 2024-04-14T23:36:48.452 INFO:teuthology.orchestra.run.smithi060.stdout:if [ -d "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/dtb" ] && \ 2024-04-14T23:36:48.453 INFO:teuthology.orchestra.run.smithi060.stdout: ! diff -rq "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/dtb" "/boot/dtb-6.9.0-rc3-g3f60ed14f7c1" >/dev/null 2>&1; then 2024-04-14T23:36:48.453 INFO:teuthology.orchestra.run.smithi060.stdout: rm -rf "/boot/dtb-6.9.0-rc3-g3f60ed14f7c1" 2024-04-14T23:36:48.453 INFO:teuthology.orchestra.run.smithi060.stdout: cp -r "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/dtb" "/boot/dtb-6.9.0-rc3-g3f60ed14f7c1" 2024-04-14T23:36:48.453 INFO:teuthology.orchestra.run.smithi060.stdout:fi 2024-04-14T23:36:48.453 INFO:teuthology.orchestra.run.smithi060.stdout:if [ ! -e "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.dep" ]; then 2024-04-14T23:36:48.453 INFO:teuthology.orchestra.run.smithi060.stdout: /usr/sbin/depmod 6.9.0-rc3-g3f60ed14f7c1 2024-04-14T23:36:48.453 INFO:teuthology.orchestra.run.smithi060.stdout:fi 2024-04-14T23:36:48.453 INFO:teuthology.orchestra.run.smithi060.stdout:preuninstall scriptlet (using /bin/sh): 2024-04-14T23:36:48.453 INFO:teuthology.orchestra.run.smithi060.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-14T23:36:48.453 INFO:teuthology.orchestra.run.smithi060.stdout:kernel-install remove 6.9.0-rc3-g3f60ed14f7c1 2024-04-14T23:36:48.453 INFO:teuthology.orchestra.run.smithi060.stdout:fi 2024-04-14T23:36:48.454 DEBUG:teuthology.orchestra.run.smithi060:> sudo rpm -qi grub2-tools 2024-04-14T23:36:48.546 INFO:teuthology.orchestra.run.smithi060.stdout:Name : grub2-tools 2024-04-14T23:36:48.546 INFO:teuthology.orchestra.run.smithi060.stdout:Epoch : 1 2024-04-14T23:36:48.546 INFO:teuthology.orchestra.run.smithi060.stdout:Version : 2.06 2024-04-14T23:36:48.546 INFO:teuthology.orchestra.run.smithi060.stdout:Release : 68.el9 2024-04-14T23:36:48.546 INFO:teuthology.orchestra.run.smithi060.stdout:Architecture: x86_64 2024-04-14T23:36:48.546 INFO:teuthology.orchestra.run.smithi060.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-14T23:36:48.546 INFO:teuthology.orchestra.run.smithi060.stdout:Group : Unspecified 2024-04-14T23:36:48.546 INFO:teuthology.orchestra.run.smithi060.stdout:Size : 8264254 2024-04-14T23:36:48.546 INFO:teuthology.orchestra.run.smithi060.stdout:License : GPLv3+ 2024-04-14T23:36:48.547 INFO:teuthology.orchestra.run.smithi060.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-14T23:36:48.547 INFO:teuthology.orchestra.run.smithi060.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-14T23:36:48.547 INFO:teuthology.orchestra.run.smithi060.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-14T23:36:48.547 INFO:teuthology.orchestra.run.smithi060.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-14T23:36:48.547 INFO:teuthology.orchestra.run.smithi060.stdout:Packager : builder@centos.org 2024-04-14T23:36:48.547 INFO:teuthology.orchestra.run.smithi060.stdout:Vendor : CentOS 2024-04-14T23:36:48.547 INFO:teuthology.orchestra.run.smithi060.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-14T23:36:48.547 INFO:teuthology.orchestra.run.smithi060.stdout:Summary : Support tools for GRUB. 2024-04-14T23:36:48.547 INFO:teuthology.orchestra.run.smithi060.stdout:Description : 2024-04-14T23:36:48.547 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:36:48.547 INFO:teuthology.orchestra.run.smithi060.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-14T23:36:48.547 INFO:teuthology.orchestra.run.smithi060.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-14T23:36:48.547 INFO:teuthology.orchestra.run.smithi060.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-14T23:36:48.547 INFO:teuthology.orchestra.run.smithi060.stdout:hardware devices. 2024-04-14T23:36:48.547 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:36:48.547 INFO:teuthology.orchestra.run.smithi060.stdout:This subpackage provides tools for support of all platforms. 2024-04-14T23:36:48.548 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-14T23:36:48.548 INFO:teuthology.task.kernel:Updating grub on smithi060 to boot 6.9.0-rc3-g3f60ed14f7c1 2024-04-14T23:36:48.548 DEBUG:teuthology.orchestra.run.smithi060:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-14T23:36:49.120 INFO:teuthology.orchestra.run.smithi060.stderr:Generating grub configuration file ... 2024-04-14T23:36:50.500 INFO:teuthology.orchestra.run.smithi060.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-14T23:36:50.531 INFO:teuthology.orchestra.run.smithi060.stderr:done 2024-04-14T23:36:50.533 DEBUG:teuthology.orchestra.run.smithi060:> mktemp 2024-04-14T23:36:50.547 INFO:teuthology.orchestra.run.smithi060.stdout:/tmp/tmp.sovnFpO0DW 2024-04-14T23:36:50.548 DEBUG:teuthology.orchestra.run.smithi060:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.sovnFpO0DW 2024-04-14T23:36:50.619 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0666 /tmp/tmp.sovnFpO0DW 2024-04-14T23:36:50.762 DEBUG:teuthology.orchestra.remote:smithi060:/tmp/tmp.sovnFpO0DW is 6KB 2024-04-14T23:36:50.810 DEBUG:teuthology.orchestra.run.smithi060:> rm -fr /tmp/tmp.sovnFpO0DW 2024-04-14T23:36:50.824 DEBUG:teuthology.orchestra.run.smithi060:> sudo /bin/ls /boot/loader/entries || true 2024-04-14T23:36:50.894 INFO:teuthology.orchestra.run.smithi060.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-14T23:36:50.894 INFO:teuthology.orchestra.run.smithi060.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-14T23:36:50.894 INFO:teuthology.orchestra.run.smithi060.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-14T23:36:50.894 INFO:teuthology.orchestra.run.smithi060.stdout:15949e3c88704300882482eddfd7baae-6.9.0-rc3-g3f60ed14f7c1.conf 2024-04-14T23:36:50.895 DEBUG:teuthology.orchestra.run.smithi060:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.9.0-rc3-g3f60ed14f7c1 2024-04-14T23:36:51.071 DEBUG:teuthology.orchestra.run.smithi060:> sudo shutdown -r now 2024-04-14T23:37:21.074 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-04-14T23:37:21.074 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-04-14T23:37:21.075 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-14T23:37:39.614 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.60 2024-04-14T23:37:48.615 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-04-14T23:37:48.616 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-14T23:38:48.618 DEBUG:teuthology.orchestra.remote:timed out 2024-04-14T23:39:00.619 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-04-14T23:39:00.621 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-14T23:39:01.078 DEBUG:teuthology.orchestra.run.smithi060:> true 2024-04-14T23:39:01.858 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-04-14T23:39:01.858 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.9.0-rc3-g3f60ed14f7c1"... 2024-04-14T23:39:01.859 DEBUG:teuthology.orchestra.run.smithi060:> uname -r 2024-04-14T23:39:01.916 INFO:teuthology.orchestra.run.smithi060.stdout:6.9.0-rc3-g3f60ed14f7c1 2024-04-14T23:39:01.916 DEBUG:teuthology.task.kernel:current kernel version is 6.9.0-rc3-g3f60ed14f7c1 vs 6.9.0-rc3-g3f60ed14f7c1 2024-04-14T23:39:01.916 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-14T23:39:01.916 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-14T23:39:02.919 DEBUG:teuthology.parallel:result is None 2024-04-14T23:39:02.919 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-14T23:39:02.926 INFO:teuthology.task.internal:Creating test directory... 2024-04-14T23:39:02.927 DEBUG:teuthology.orchestra.run.smithi060:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-14T23:39:02.934 DEBUG:teuthology.orchestra.run.smithi186:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-14T23:39:02.986 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-14T23:39:02.993 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-14T23:39:03.020 INFO:teuthology.task.internal:Creating archive directory... 2024-04-14T23:39:03.020 DEBUG:teuthology.orchestra.run.smithi060:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-14T23:39:03.028 DEBUG:teuthology.orchestra.run.smithi186:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-14T23:39:03.090 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-14T23:39:03.096 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-14T23:39:03.096 DEBUG:teuthology.orchestra.run.smithi186:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-14T23:39:03.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T23:39:03.112 DEBUG:teuthology.orchestra.run.smithi060:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-14T23:39:03.120 DEBUG:teuthology.orchestra.run.smithi186:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-14T23:39:03.202 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-14T23:39:03.228 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-14T23:39:03.251 INFO:teuthology.orchestra.run.smithi060.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-14T23:39:03.328 INFO:teuthology.orchestra.run.smithi060.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-14T23:39:03.333 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-14T23:39:03.341 INFO:teuthology.task.internal:Configuring sudo... 2024-04-14T23:39:03.341 DEBUG:teuthology.orchestra.run.smithi060:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-14T23:39:03.346 DEBUG:teuthology.orchestra.run.smithi186:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-14T23:39:03.437 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-14T23:39:03.446 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-14T23:39:03.446 DEBUG:teuthology.orchestra.run.smithi060:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-14T23:39:03.453 DEBUG:teuthology.orchestra.run.smithi186:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-14T23:39:03.502 DEBUG:teuthology.orchestra.run.smithi060:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-14T23:39:03.588 DEBUG:teuthology.orchestra.run.smithi060:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-14T23:39:03.695 DEBUG:teuthology.orchestra.run.smithi060:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-14T23:39:03.785 DEBUG:teuthology.orchestra.run.smithi060:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-14T23:39:03.891 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-04-14T23:39:03.892 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-14T23:39:03.991 DEBUG:teuthology.orchestra.run.smithi186:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-14T23:39:04.034 DEBUG:teuthology.orchestra.run.smithi186:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-14T23:39:04.103 DEBUG:teuthology.orchestra.run.smithi186:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-14T23:39:04.145 DEBUG:teuthology.orchestra.run.smithi186:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-14T23:39:04.190 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-14T23:39:04.190 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-14T23:39:04.258 DEBUG:teuthology.orchestra.run.smithi060:> sudo service rsyslog restart 2024-04-14T23:39:04.266 DEBUG:teuthology.orchestra.run.smithi186:> sudo service rsyslog restart 2024-04-14T23:39:04.336 INFO:teuthology.orchestra.run.smithi186.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-14T23:39:04.403 INFO:teuthology.orchestra.run.smithi060.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-14T23:39:04.690 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-14T23:39:04.698 INFO:teuthology.task.internal:Starting timer... 2024-04-14T23:39:04.698 INFO:teuthology.run_tasks:Running task pcp... 2024-04-14T23:39:04.740 INFO:teuthology.run_tasks:Running task selinux... 2024-04-14T23:39:04.774 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-04-14T23:39:04.775 DEBUG:teuthology.orchestra.run.smithi060:> sudo service auditd rotate 2024-04-14T23:39:04.909 INFO:teuthology.orchestra.run.smithi060.stdout:Rotating logs: 2024-04-14T23:39:04.913 DEBUG:teuthology.orchestra.run.smithi186:> sudo service auditd rotate 2024-04-14T23:39:04.978 INFO:teuthology.orchestra.run.smithi186.stdout:Rotating logs: 2024-04-14T23:39:04.979 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-14T23:39:04.979 DEBUG:teuthology.orchestra.run.smithi060:> /usr/sbin/getenforce 2024-04-14T23:39:05.033 INFO:teuthology.orchestra.run.smithi060.stdout:Permissive 2024-04-14T23:39:05.033 DEBUG:teuthology.orchestra.run.smithi186:> /usr/sbin/getenforce 2024-04-14T23:39:05.063 INFO:teuthology.orchestra.run.smithi186.stdout:Permissive 2024-04-14T23:39:05.063 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi060.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi186.front.sepia.ceph.com': 'permissive'} 2024-04-14T23:39:05.063 DEBUG:teuthology.orchestra.run.smithi060:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-14T23:39:05.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T23:39:05.153 DEBUG:teuthology.orchestra.run.smithi186:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-14T23:39:05.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T23:39:05.181 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-14T23:39:05.181 DEBUG:teuthology.orchestra.run.smithi060:> sudo /usr/sbin/setenforce permissive 2024-04-14T23:39:05.263 DEBUG:teuthology.orchestra.run.smithi186:> sudo /usr/sbin/setenforce permissive 2024-04-14T23:39:05.298 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-14T23:39:05.308 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-14T23:39:05.314 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-14T23:39:05.416 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-14T23:39:05.431 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-14T23:39:05.433 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi060.front.sepia.ceph.com,smithi186.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-14T23:45:18.303 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi060.front.sepia.ceph.com'), Remote(name='ubuntu@smithi186.front.sepia.ceph.com')] 2024-04-14T23:45:18.304 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-04-14T23:45:18.305 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-14T23:45:18.465 DEBUG:teuthology.orchestra.run.smithi060:> true 2024-04-14T23:45:18.629 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-04-14T23:45:18.629 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-04-14T23:45:18.630 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-14T23:45:18.721 DEBUG:teuthology.orchestra.run.smithi186:> true 2024-04-14T23:45:18.790 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-04-14T23:45:18.790 INFO:teuthology.run_tasks:Running task clock... 2024-04-14T23:45:18.801 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-14T23:45:18.801 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-14T23:45:18.802 DEBUG:teuthology.orchestra.run.smithi060:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-14T23:45:18.810 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-14T23:45:18.810 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-14T23:45:18.864 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-14T23:45:18.882 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-14T23:45:18.911 INFO:teuthology.orchestra.run.smithi060.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-14T23:45:18.912 INFO:teuthology.orchestra.run.smithi186.stderr:sudo: ntpd: command not found 2024-04-14T23:45:18.926 INFO:teuthology.orchestra.run.smithi186.stdout:506 Cannot talk to daemon 2024-04-14T23:45:18.943 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-14T23:45:18.960 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-14T23:45:18.967 INFO:teuthology.orchestra.run.smithi060.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-14T23:45:19.004 INFO:teuthology.orchestra.run.smithi186.stderr:bash: line 1: ntpq: command not found 2024-04-14T23:45:19.008 INFO:teuthology.orchestra.run.smithi186.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-14T23:45:19.008 INFO:teuthology.orchestra.run.smithi186.stdout:=============================================================================== 2024-04-14T23:45:19.008 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-14T23:45:19.008 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-14T23:45:19.008 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-14T23:45:19.008 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-14T23:45:19.066 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: ntpd: command not found 2024-04-14T23:45:19.111 INFO:teuthology.orchestra.run.smithi060.stdout:506 Cannot talk to daemon 2024-04-14T23:45:19.166 INFO:teuthology.orchestra.run.smithi060.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-14T23:45:19.218 INFO:teuthology.orchestra.run.smithi060.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-14T23:45:19.369 INFO:teuthology.orchestra.run.smithi060.stderr:bash: line 1: ntpq: command not found 2024-04-14T23:45:19.378 INFO:teuthology.orchestra.run.smithi060.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-14T23:45:19.378 INFO:teuthology.orchestra.run.smithi060.stdout:=============================================================================== 2024-04-14T23:45:19.378 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-14T23:45:19.378 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-14T23:45:19.378 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-14T23:45:19.378 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-14T23:45:19.380 INFO:teuthology.run_tasks:Running task install... 2024-04-14T23:45:19.388 DEBUG:teuthology.task.install:project ceph 2024-04-14T23:45:19.389 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'a9a752df26c63acad72e1b3569fd79a515ca0765'}} 2024-04-14T23:45:19.389 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'a9a752df26c63acad72e1b3569fd79a515ca0765'} 2024-04-14T23:45:19.389 INFO:teuthology.task.install:Using flavor: default 2024-04-14T23:45:19.394 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-14T23:45:19.395 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-14T23:45:19.395 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'a9a752df26c63acad72e1b3569fd79a515ca0765', 'tag': None, 'wait_for_package': False} 2024-04-14T23:45:19.396 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 2024-04-14T23:45:19.398 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'a9a752df26c63acad72e1b3569fd79a515ca0765', 'tag': None, 'wait_for_package': False} 2024-04-14T23:45:19.398 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 2024-04-14T23:45:19.539 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/squid/a9a752df26c63acad72e1b3569fd79a515ca0765/centos/9/flavors/default/ 2024-04-14T23:45:19.539 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2114.ga9a752df 2024-04-14T23:45:19.611 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/squid/a9a752df26c63acad72e1b3569fd79a515ca0765/centos/9/flavors/default/ 2024-04-14T23:45:19.611 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2114.ga9a752df 2024-04-14T23:45:19.654 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/squid/a9a752df26c63acad72e1b3569fd79a515ca0765/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/a9a752df26c63acad72e1b3569fd79a515ca0765/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/a9a752df26c63acad72e1b3569fd79a515ca0765/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-14T23:45:19.654 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-04-14T23:45:19.654 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-14T23:45:19.732 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/squid/a9a752df26c63acad72e1b3569fd79a515ca0765/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/a9a752df26c63acad72e1b3569fd79a515ca0765/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/a9a752df26c63acad72e1b3569fd79a515ca0765/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-14T23:45:19.732 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-14T23:45:19.733 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-14T23:45:19.764 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-14T23:45:19.764 DEBUG:teuthology.orchestra.run.smithi060:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/a9a752df26c63acad72e1b3569fd79a515ca0765/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-14T23:45:19.765 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-14T23:45:19.765 DEBUG:teuthology.orchestra.run.smithi186:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/a9a752df26c63acad72e1b3569fd79a515ca0765/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-14T23:45:19.836 DEBUG:teuthology.orchestra.run.smithi186:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-14T23:45:19.911 DEBUG:teuthology.orchestra.run.smithi060:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-14T23:45:19.919 DEBUG:teuthology.orchestra.run.smithi186:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-14T23:45:19.987 INFO:teuthology.orchestra.run.smithi186.stdout:check_obsoletes = 1 2024-04-14T23:45:19.988 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum clean all 2024-04-14T23:45:20.053 DEBUG:teuthology.orchestra.run.smithi060:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-14T23:45:20.148 INFO:teuthology.orchestra.run.smithi060.stdout:check_obsoletes = 1 2024-04-14T23:45:20.152 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum clean all 2024-04-14T23:45:20.226 INFO:teuthology.orchestra.run.smithi186.stdout:74 files removed 2024-04-14T23:45:20.254 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-14T23:45:20.526 INFO:teuthology.orchestra.run.smithi060.stdout:74 files removed 2024-04-14T23:45:20.567 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-14T23:45:20.955 INFO:teuthology.orchestra.run.smithi186.stdout:ceph packages for x86_64 203 kB/s | 83 kB 00:00 2024-04-14T23:45:21.414 INFO:teuthology.orchestra.run.smithi186.stdout:ceph noarch packages 27 kB/s | 11 kB 00:00 2024-04-14T23:45:21.602 INFO:teuthology.orchestra.run.smithi060.stdout:ceph packages for x86_64 164 kB/s | 83 kB 00:00 2024-04-14T23:45:21.956 INFO:teuthology.orchestra.run.smithi186.stdout:ceph source packages 3.7 kB/s | 1.9 kB 00:00 2024-04-14T23:45:22.376 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS-9 - AppStream 21 MB/s | 8.1 MB 00:00 2024-04-14T23:45:22.461 INFO:teuthology.orchestra.run.smithi060.stdout:ceph noarch packages 15 kB/s | 11 kB 00:00 2024-04-14T23:45:23.411 INFO:teuthology.orchestra.run.smithi060.stdout:ceph source packages 2.2 kB/s | 1.9 kB 00:00 2024-04-14T23:45:24.067 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS-9 - BaseOS 6.9 MB/s | 2.5 MB 00:00 2024-04-14T23:45:24.087 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS-9 - AppStream 14 MB/s | 8.1 MB 00:00 2024-04-14T23:45:25.932 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS-9 - BaseOS 6.7 MB/s | 2.5 MB 00:00 2024-04-14T23:45:25.958 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - BaseOS 5.5 MB/s | 8.0 MB 00:01 2024-04-14T23:45:26.986 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-04-14T23:45:29.262 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - AppStream 19 MB/s | 19 MB 00:00 2024-04-14T23:45:34.038 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - CRB 7.8 MB/s | 6.2 MB 00:00 2024-04-14T23:45:35.731 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - Extras packages 56 kB/s | 16 kB 00:00 2024-04-14T23:45:38.114 INFO:teuthology.orchestra.run.smithi060.stdout:Extra Packages for Enterprise Linux 9.7 MB/s | 21 MB 00:02 2024-04-14T23:45:44.613 INFO:teuthology.orchestra.run.smithi060.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-04-14T23:45:47.380 INFO:teuthology.orchestra.run.smithi060.stderr:Modular dependency problems: 2024-04-14T23:45:47.381 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-04-14T23:45:47.381 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 1: conflicting requests 2024-04-14T23:45:47.381 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.381 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 2: conflicting requests 2024-04-14T23:45:47.381 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.381 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 3: conflicting requests 2024-04-14T23:45:47.381 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.381 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 4: conflicting requests 2024-04-14T23:45:47.381 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.381 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 5: conflicting requests 2024-04-14T23:45:47.381 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.381 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 6: conflicting requests 2024-04-14T23:45:47.382 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.382 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 7: conflicting requests 2024-04-14T23:45:47.382 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.382 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 8: conflicting requests 2024-04-14T23:45:47.382 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.382 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 9: conflicting requests 2024-04-14T23:45:47.382 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.382 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 10: conflicting requests 2024-04-14T23:45:47.382 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.382 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 11: conflicting requests 2024-04-14T23:45:47.382 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.382 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 12: conflicting requests 2024-04-14T23:45:47.383 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.383 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 13: conflicting requests 2024-04-14T23:45:47.383 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.383 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 14: conflicting requests 2024-04-14T23:45:47.383 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.383 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 15: conflicting requests 2024-04-14T23:45:47.383 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.383 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 16: conflicting requests 2024-04-14T23:45:47.383 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.383 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 17: conflicting requests 2024-04-14T23:45:47.383 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.383 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 18: conflicting requests 2024-04-14T23:45:47.383 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.384 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 19: conflicting requests 2024-04-14T23:45:47.384 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.384 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 20: conflicting requests 2024-04-14T23:45:47.384 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.384 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 21: conflicting requests 2024-04-14T23:45:47.384 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.384 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 22: conflicting requests 2024-04-14T23:45:47.384 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.384 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 23: conflicting requests 2024-04-14T23:45:47.384 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.384 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 24: conflicting requests 2024-04-14T23:45:47.384 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.384 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 25: conflicting requests 2024-04-14T23:45:47.385 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.385 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 26: conflicting requests 2024-04-14T23:45:47.385 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.385 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 27: conflicting requests 2024-04-14T23:45:47.385 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.385 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 28: conflicting requests 2024-04-14T23:45:47.385 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.385 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 29: conflicting requests 2024-04-14T23:45:47.385 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.385 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 30: conflicting requests 2024-04-14T23:45:47.385 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.385 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 31: conflicting requests 2024-04-14T23:45:47.386 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.386 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 32: conflicting requests 2024-04-14T23:45:47.386 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.386 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 33: conflicting requests 2024-04-14T23:45:47.386 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.386 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 34: conflicting requests 2024-04-14T23:45:47.386 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.386 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 35: conflicting requests 2024-04-14T23:45:47.386 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.386 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 36: conflicting requests 2024-04-14T23:45:47.386 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.387 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 37: conflicting requests 2024-04-14T23:45:47.387 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.387 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.387 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.387 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 38: cannot install the best candidate for the job 2024-04-14T23:45:47.387 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.387 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.387 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.387 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.387 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 39: conflicting requests 2024-04-14T23:45:47.387 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.387 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.387 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.388 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.388 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.388 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.388 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 40: cannot install the best candidate for the job 2024-04-14T23:45:47.388 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.388 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.389 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.389 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.389 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 41: cannot install the best candidate for the job 2024-04-14T23:45:47.389 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.389 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.389 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.389 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.389 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 42: cannot install the best candidate for the job 2024-04-14T23:45:47.389 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.389 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.389 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.390 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.390 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 43: cannot install the best candidate for the job 2024-04-14T23:45:47.390 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.390 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.390 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.390 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.391 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 44: cannot install the best candidate for the job 2024-04-14T23:45:47.391 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.391 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.391 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.391 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.391 INFO:teuthology.orchestra.run.smithi060.stderr: Problem 45: cannot install the best candidate for the job 2024-04-14T23:45:47.391 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.391 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.391 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.391 INFO:teuthology.orchestra.run.smithi060.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-14T23:45:47.392 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - AppStream 953 kB/s | 19 MB 00:20 2024-04-14T23:45:48.003 INFO:teuthology.orchestra.run.smithi060.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-14T23:45:48.004 INFO:teuthology.orchestra.run.smithi060.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-14T23:45:48.025 INFO:teuthology.orchestra.run.smithi060.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-04-14T23:45:48.026 INFO:teuthology.orchestra.run.smithi060.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-14T23:45:48.028 INFO:teuthology.orchestra.run.smithi060.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-04-14T23:45:48.029 INFO:teuthology.orchestra.run.smithi060.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-04-14T23:45:48.032 INFO:teuthology.orchestra.run.smithi060.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-14T23:45:48.035 INFO:teuthology.orchestra.run.smithi060.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-04-14T23:45:48.037 INFO:teuthology.orchestra.run.smithi060.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-04-14T23:45:48.143 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-04-14T23:45:48.168 INFO:teuthology.orchestra.run.smithi060.stdout:========================================================================================= 2024-04-14T23:45:48.168 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-04-14T23:45:48.168 INFO:teuthology.orchestra.run.smithi060.stdout:========================================================================================= 2024-04-14T23:45:48.169 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-04-14T23:45:48.169 INFO:teuthology.orchestra.run.smithi060.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-14T23:45:48.169 INFO:teuthology.orchestra.run.smithi060.stdout: ceph x86_64 2:19.0.0-2114.ga9a752df.el7 ceph 6.6 k 2024-04-14T23:45:48.169 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-base x86_64 2:19.0.0-2114.ga9a752df.el7 ceph 5.3 M 2024-04-14T23:45:48.169 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-fuse x86_64 2:19.0.0-2114.ga9a752df.el7 ceph 891 k 2024-04-14T23:45:48.169 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2114.ga9a752df.el7 ceph 145 k 2024-04-14T23:45:48.169 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr x86_64 2:19.0.0-2114.ga9a752df.el7 ceph 1.5 M 2024-04-14T23:45:48.169 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2114.ga9a752df.el7 ceph-noarch 139 k 2024-04-14T23:45:48.169 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2114.ga9a752df.el7 ceph-noarch 3.5 M 2024-04-14T23:45:48.169 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2114.ga9a752df.el7 ceph-noarch 7.4 M 2024-04-14T23:45:48.169 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-rook noarch 2:19.0.0-2114.ga9a752df.el7 ceph-noarch 49 k 2024-04-14T23:45:48.169 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-radosgw x86_64 2:19.0.0-2114.ga9a752df.el7 ceph 9.9 M 2024-04-14T23:45:48.169 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-test x86_64 2:19.0.0-2114.ga9a752df.el7 ceph 48 M 2024-04-14T23:45:48.170 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-volume noarch 2:19.0.0-2114.ga9a752df.el7 ceph-noarch 264 k 2024-04-14T23:45:48.170 INFO:teuthology.orchestra.run.smithi060.stdout: cephadm noarch 2:19.0.0-2114.ga9a752df.el7 ceph-noarch 749 k 2024-04-14T23:45:48.170 INFO:teuthology.orchestra.run.smithi060.stdout: cephfs-mirror x86_64 2:19.0.0-2114.ga9a752df.el7 ceph 222 k 2024-04-14T23:45:48.170 INFO:teuthology.orchestra.run.smithi060.stdout: cephfs-shell x86_64 2:19.0.0-2114.ga9a752df.el7 ceph 27 k 2024-04-14T23:45:48.170 INFO:teuthology.orchestra.run.smithi060.stdout: cephfs-top noarch 2:19.0.0-2114.ga9a752df.el7 ceph-noarch 18 k 2024-04-14T23:45:48.170 INFO:teuthology.orchestra.run.smithi060.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-14T23:45:48.170 INFO:teuthology.orchestra.run.smithi060.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-14T23:45:48.170 INFO:teuthology.orchestra.run.smithi060.stdout: libaio-devel x86_64 0.3.112-1.el8 CentOS-BaseOS 18 k 2024-04-14T23:45:48.170 INFO:teuthology.orchestra.run.smithi060.stdout: libcephfs-devel x86_64 2:19.0.0-2114.ga9a752df.el7 ceph 32 k 2024-04-14T23:45:48.170 INFO:teuthology.orchestra.run.smithi060.stdout: libcephfs2 x86_64 2:19.0.0-2114.ga9a752df.el7 ceph 746 k 2024-04-14T23:45:48.170 INFO:teuthology.orchestra.run.smithi060.stdout: librados-devel x86_64 2:19.0.0-2114.ga9a752df.el7 ceph 127 k 2024-04-14T23:45:48.170 INFO:teuthology.orchestra.run.smithi060.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-14T23:45:48.170 INFO:teuthology.orchestra.run.smithi060.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-14T23:45:48.171 INFO:teuthology.orchestra.run.smithi060.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-04-14T23:45:48.171 INFO:teuthology.orchestra.run.smithi060.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-14T23:45:48.171 INFO:teuthology.orchestra.run.smithi060.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-14T23:45:48.171 INFO:teuthology.orchestra.run.smithi060.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-14T23:45:48.171 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cephfs x86_64 2:19.0.0-2114.ga9a752df.el7 ceph 162 k 2024-04-14T23:45:48.171 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rados x86_64 2:19.0.0-2114.ga9a752df.el7 ceph 320 k 2024-04-14T23:45:48.171 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rbd x86_64 2:19.0.0-2114.ga9a752df.el7 ceph 298 k 2024-04-14T23:45:48.171 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rgw x86_64 2:19.0.0-2114.ga9a752df.el7 ceph 99 k 2024-04-14T23:45:48.171 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-fuse x86_64 2:19.0.0-2114.ga9a752df.el7 ceph 85 k 2024-04-14T23:45:48.171 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-mirror x86_64 2:19.0.0-2114.ga9a752df.el7 ceph 3.1 M 2024-04-14T23:45:48.171 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-nbd x86_64 2:19.0.0-2114.ga9a752df.el7 ceph 171 k 2024-04-14T23:45:48.171 INFO:teuthology.orchestra.run.smithi060.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-14T23:45:48.171 INFO:teuthology.orchestra.run.smithi060.stdout:Upgrading: 2024-04-14T23:45:48.171 INFO:teuthology.orchestra.run.smithi060.stdout: NetworkManager x86_64 1:1.47.4-1.el9 baseos 2.3 M 2024-04-14T23:45:48.172 INFO:teuthology.orchestra.run.smithi060.stdout: NetworkManager-libnm x86_64 1:1.47.4-1.el9 baseos 1.8 M 2024-04-14T23:45:48.172 INFO:teuthology.orchestra.run.smithi060.stdout: NetworkManager-team x86_64 1:1.47.4-1.el9 baseos 42 k 2024-04-14T23:45:48.172 INFO:teuthology.orchestra.run.smithi060.stdout: NetworkManager-tui x86_64 1:1.47.4-1.el9 baseos 249 k 2024-04-14T23:45:48.172 INFO:teuthology.orchestra.run.smithi060.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-04-14T23:45:48.172 INFO:teuthology.orchestra.run.smithi060.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-04-14T23:45:48.172 INFO:teuthology.orchestra.run.smithi060.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-04-14T23:45:48.172 INFO:teuthology.orchestra.run.smithi060.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-04-14T23:45:48.172 INFO:teuthology.orchestra.run.smithi060.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-04-14T23:45:48.172 INFO:teuthology.orchestra.run.smithi060.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-04-14T23:45:48.172 INFO:teuthology.orchestra.run.smithi060.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-04-14T23:45:48.172 INFO:teuthology.orchestra.run.smithi060.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-04-14T23:45:48.172 INFO:teuthology.orchestra.run.smithi060.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-04-14T23:45:48.172 INFO:teuthology.orchestra.run.smithi060.stdout: librados2 x86_64 2:19.0.0-2114.ga9a752df.el7 ceph 3.4 M 2024-04-14T23:45:48.173 INFO:teuthology.orchestra.run.smithi060.stdout: librbd1 x86_64 2:19.0.0-2114.ga9a752df.el7 ceph 3.1 M 2024-04-14T23:45:48.173 INFO:teuthology.orchestra.run.smithi060.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-04-14T23:45:48.173 INFO:teuthology.orchestra.run.smithi060.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-04-14T23:45:48.173 INFO:teuthology.orchestra.run.smithi060.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-04-14T23:45:48.173 INFO:teuthology.orchestra.run.smithi060.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-04-14T23:45:48.173 INFO:teuthology.orchestra.run.smithi060.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-04-14T23:45:48.173 INFO:teuthology.orchestra.run.smithi060.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-04-14T23:45:48.173 INFO:teuthology.orchestra.run.smithi060.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-04-14T23:45:48.173 INFO:teuthology.orchestra.run.smithi060.stdout: xfsdump x86_64 3.1.12-4.el9 baseos 323 k 2024-04-14T23:45:48.173 INFO:teuthology.orchestra.run.smithi060.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-04-14T23:45:48.173 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-04-14T23:45:48.173 INFO:teuthology.orchestra.run.smithi060.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-14T23:45:48.173 INFO:teuthology.orchestra.run.smithi060.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-14T23:45:48.174 INFO:teuthology.orchestra.run.smithi060.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-14T23:45:48.174 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-common x86_64 2:19.0.0-2114.ga9a752df.el7 ceph 21 M 2024-04-14T23:45:48.174 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2114.ga9a752df.el7 ceph-noarch 32 k 2024-04-14T23:45:48.174 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mds x86_64 2:19.0.0-2114.ga9a752df.el7 ceph 2.4 M 2024-04-14T23:45:48.174 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2114.ga9a752df.el7 ceph-noarch 248 k 2024-04-14T23:45:48.174 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mon x86_64 2:19.0.0-2114.ga9a752df.el7 ceph 4.7 M 2024-04-14T23:45:48.174 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-osd x86_64 2:19.0.0-2114.ga9a752df.el7 ceph 17 M 2024-04-14T23:45:48.174 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2114.ga9a752df.el7 ceph-noarch 15 k 2024-04-14T23:45:48.174 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-selinux x86_64 2:19.0.0-2114.ga9a752df.el7 ceph 25 k 2024-04-14T23:45:48.174 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-14T23:45:48.174 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-14T23:45:48.174 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-14T23:45:48.174 INFO:teuthology.orchestra.run.smithi060.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-14T23:45:48.174 INFO:teuthology.orchestra.run.smithi060.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-14T23:45:48.175 INFO:teuthology.orchestra.run.smithi060.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-14T23:45:48.175 INFO:teuthology.orchestra.run.smithi060.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-14T23:45:48.175 INFO:teuthology.orchestra.run.smithi060.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-14T23:45:48.175 INFO:teuthology.orchestra.run.smithi060.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-14T23:45:48.175 INFO:teuthology.orchestra.run.smithi060.stdout: libcephsqlite x86_64 2:19.0.0-2114.ga9a752df.el7 ceph 164 k 2024-04-14T23:45:48.175 INFO:teuthology.orchestra.run.smithi060.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-14T23:45:48.175 INFO:teuthology.orchestra.run.smithi060.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-14T23:45:48.175 INFO:teuthology.orchestra.run.smithi060.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-14T23:45:48.175 INFO:teuthology.orchestra.run.smithi060.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-14T23:45:48.175 INFO:teuthology.orchestra.run.smithi060.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-14T23:45:48.175 INFO:teuthology.orchestra.run.smithi060.stdout: libradosstriper1 x86_64 2:19.0.0-2114.ga9a752df.el7 ceph 501 k 2024-04-14T23:45:48.175 INFO:teuthology.orchestra.run.smithi060.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-14T23:45:48.176 INFO:teuthology.orchestra.run.smithi060.stdout: librgw2 x86_64 2:19.0.0-2114.ga9a752df.el7 ceph 5.2 M 2024-04-14T23:45:48.176 INFO:teuthology.orchestra.run.smithi060.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-14T23:45:48.176 INFO:teuthology.orchestra.run.smithi060.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-14T23:45:48.176 INFO:teuthology.orchestra.run.smithi060.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-14T23:45:48.176 INFO:teuthology.orchestra.run.smithi060.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-14T23:45:48.176 INFO:teuthology.orchestra.run.smithi060.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-14T23:45:48.176 INFO:teuthology.orchestra.run.smithi060.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-14T23:45:48.176 INFO:teuthology.orchestra.run.smithi060.stdout: oniguruma x86_64 6.9.6-1.el9.5 baseos 218 k 2024-04-14T23:45:48.176 INFO:teuthology.orchestra.run.smithi060.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-14T23:45:48.176 INFO:teuthology.orchestra.run.smithi060.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-14T23:45:48.176 INFO:teuthology.orchestra.run.smithi060.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-14T23:45:48.176 INFO:teuthology.orchestra.run.smithi060.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-14T23:45:48.176 INFO:teuthology.orchestra.run.smithi060.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-14T23:45:48.176 INFO:teuthology.orchestra.run.smithi060.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-14T23:45:48.177 INFO:teuthology.orchestra.run.smithi060.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-14T23:45:48.177 INFO:teuthology.orchestra.run.smithi060.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-14T23:45:48.177 INFO:teuthology.orchestra.run.smithi060.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-14T23:45:48.177 INFO:teuthology.orchestra.run.smithi060.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-14T23:45:48.177 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-14T23:45:48.177 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ceph-argparse x86_64 2:19.0.0-2114.ga9a752df.el7 ceph 45 k 2024-04-14T23:45:48.177 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ceph-common x86_64 2:19.0.0-2114.ga9a752df.el7 ceph 131 k 2024-04-14T23:45:48.177 INFO:teuthology.orchestra.run.smithi060.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-14T23:45:48.177 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-14T23:45:48.177 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-14T23:45:48.177 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-14T23:45:48.177 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-14T23:45:48.177 INFO:teuthology.orchestra.run.smithi060.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-14T23:45:48.177 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-14T23:45:48.178 INFO:teuthology.orchestra.run.smithi060.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-14T23:45:48.178 INFO:teuthology.orchestra.run.smithi060.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-14T23:45:48.178 INFO:teuthology.orchestra.run.smithi060.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-14T23:45:48.178 INFO:teuthology.orchestra.run.smithi060.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-14T23:45:48.178 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-14T23:45:48.178 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-14T23:45:48.178 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-14T23:45:48.178 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-14T23:45:48.178 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-14T23:45:48.178 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-14T23:45:48.178 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-14T23:45:48.178 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-14T23:45:48.178 INFO:teuthology.orchestra.run.smithi060.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-14T23:45:48.179 INFO:teuthology.orchestra.run.smithi060.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-14T23:45:48.179 INFO:teuthology.orchestra.run.smithi060.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-14T23:45:48.179 INFO:teuthology.orchestra.run.smithi060.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-14T23:45:48.179 INFO:teuthology.orchestra.run.smithi060.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-14T23:45:48.179 INFO:teuthology.orchestra.run.smithi060.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-14T23:45:48.179 INFO:teuthology.orchestra.run.smithi060.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-14T23:45:48.179 INFO:teuthology.orchestra.run.smithi060.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-14T23:45:48.179 INFO:teuthology.orchestra.run.smithi060.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-14T23:45:48.179 INFO:teuthology.orchestra.run.smithi060.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-14T23:45:48.179 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-14T23:45:48.179 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-14T23:45:48.179 INFO:teuthology.orchestra.run.smithi060.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-14T23:45:48.179 INFO:teuthology.orchestra.run.smithi060.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-14T23:45:48.180 INFO:teuthology.orchestra.run.smithi060.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-14T23:45:48.180 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-14T23:45:48.180 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-14T23:45:48.180 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-14T23:45:48.180 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-14T23:45:48.180 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-14T23:45:48.180 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-14T23:45:48.180 INFO:teuthology.orchestra.run.smithi060.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-14T23:45:48.180 INFO:teuthology.orchestra.run.smithi060.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-14T23:45:48.180 INFO:teuthology.orchestra.run.smithi060.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-14T23:45:48.180 INFO:teuthology.orchestra.run.smithi060.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-14T23:45:48.180 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-14T23:45:48.180 INFO:teuthology.orchestra.run.smithi060.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-14T23:45:48.181 INFO:teuthology.orchestra.run.smithi060.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-14T23:45:48.181 INFO:teuthology.orchestra.run.smithi060.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-14T23:45:48.181 INFO:teuthology.orchestra.run.smithi060.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-14T23:45:48.181 INFO:teuthology.orchestra.run.smithi060.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-14T23:45:48.181 INFO:teuthology.orchestra.run.smithi060.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-14T23:45:48.181 INFO:teuthology.orchestra.run.smithi060.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-14T23:45:48.181 INFO:teuthology.orchestra.run.smithi060.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-14T23:45:48.181 INFO:teuthology.orchestra.run.smithi060.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-14T23:45:48.181 INFO:teuthology.orchestra.run.smithi060.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-14T23:45:48.181 INFO:teuthology.orchestra.run.smithi060.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-14T23:45:48.181 INFO:teuthology.orchestra.run.smithi060.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-14T23:45:48.181 INFO:teuthology.orchestra.run.smithi060.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-14T23:45:48.181 INFO:teuthology.orchestra.run.smithi060.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-14T23:45:48.181 INFO:teuthology.orchestra.run.smithi060.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-14T23:45:48.182 INFO:teuthology.orchestra.run.smithi060.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-14T23:45:48.182 INFO:teuthology.orchestra.run.smithi060.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-14T23:45:48.182 INFO:teuthology.orchestra.run.smithi060.stdout: zlib-devel x86_64 1.2.11-40.el9 appstream 45 k 2024-04-14T23:45:48.182 INFO:teuthology.orchestra.run.smithi060.stdout:Installing weak dependencies: 2024-04-14T23:45:48.182 INFO:teuthology.orchestra.run.smithi060.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-14T23:45:48.182 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:45:48.182 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-04-14T23:45:48.182 INFO:teuthology.orchestra.run.smithi060.stdout:========================================================================================= 2024-04-14T23:45:48.182 INFO:teuthology.orchestra.run.smithi060.stdout:Install 150 Packages 2024-04-14T23:45:48.182 INFO:teuthology.orchestra.run.smithi060.stdout:Upgrade 24 Packages 2024-04-14T23:45:48.182 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:45:48.183 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 228 M 2024-04-14T23:45:48.183 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-04-14T23:45:49.059 INFO:teuthology.orchestra.run.smithi060.stdout:(1/174): ceph-19.0.0-2114.ga9a752df.el7.x86_64. 25 kB/s | 6.6 kB 00:00 2024-04-14T23:45:49.296 INFO:teuthology.orchestra.run.smithi060.stdout:(2/174): ceph-fuse-19.0.0-2114.ga9a752df.el7.x8 3.7 MB/s | 891 kB 00:00 2024-04-14T23:45:49.377 INFO:teuthology.orchestra.run.smithi060.stdout:(3/174): ceph-immutable-object-cache-19.0.0-211 1.8 MB/s | 145 kB 00:00 2024-04-14T23:45:49.702 INFO:teuthology.orchestra.run.smithi060.stdout:(4/174): ceph-mds-19.0.0-2114.ga9a752df.el7.x86 7.3 MB/s | 2.4 MB 00:00 2024-04-14T23:45:49.919 INFO:teuthology.orchestra.run.smithi060.stdout:(5/174): ceph-mgr-19.0.0-2114.ga9a752df.el7.x86 6.9 MB/s | 1.5 MB 00:00 2024-04-14T23:45:50.128 INFO:teuthology.orchestra.run.smithi060.stdout:(6/174): ceph-base-19.0.0-2114.ga9a752df.el7.x8 4.0 MB/s | 5.3 MB 00:01 2024-04-14T23:45:50.495 INFO:teuthology.orchestra.run.smithi060.stdout:(7/174): ceph-mon-19.0.0-2114.ga9a752df.el7.x86 8.2 MB/s | 4.7 MB 00:00 2024-04-14T23:45:50.770 INFO:teuthology.orchestra.run.smithi060.stdout:(8/174): ceph-common-19.0.0-2114.ga9a752df.el7. 10 MB/s | 21 MB 00:01 2024-04-14T23:45:50.853 INFO:teuthology.orchestra.run.smithi060.stdout:(9/174): ceph-selinux-19.0.0-2114.ga9a752df.el7 307 kB/s | 25 kB 00:00 2024-04-14T23:45:51.508 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.2 MB 00:00 2024-04-14T23:45:51.624 INFO:teuthology.orchestra.run.smithi060.stdout:(10/174): ceph-radosgw-19.0.0-2114.ga9a752df.el 8.7 MB/s | 9.9 MB 00:01 2024-04-14T23:45:51.690 INFO:teuthology.orchestra.run.smithi060.stdout:(11/174): cephfs-mirror-19.0.0-2114.ga9a752df.e 3.5 MB/s | 222 kB 00:00 2024-04-14T23:45:51.754 INFO:teuthology.orchestra.run.smithi060.stdout:(12/174): cephfs-shell-19.0.0-2114.ga9a752df.el 411 kB/s | 27 kB 00:00 2024-04-14T23:45:51.821 INFO:teuthology.orchestra.run.smithi060.stdout:(13/174): libcephfs-devel-19.0.0-2114.ga9a752df 483 kB/s | 32 kB 00:00 2024-04-14T23:45:51.938 INFO:teuthology.orchestra.run.smithi060.stdout:(14/174): libcephfs2-19.0.0-2114.ga9a752df.el7. 6.3 MB/s | 746 kB 00:00 2024-04-14T23:45:52.022 INFO:teuthology.orchestra.run.smithi060.stdout:(15/174): libcephsqlite-19.0.0-2114.ga9a752df.e 1.9 MB/s | 164 kB 00:00 2024-04-14T23:45:52.130 INFO:teuthology.orchestra.run.smithi060.stdout:(16/174): librados-devel-19.0.0-2114.ga9a752df. 1.2 MB/s | 127 kB 00:00 2024-04-14T23:45:52.256 INFO:teuthology.orchestra.run.smithi060.stdout:(17/174): libradosstriper1-19.0.0-2114.ga9a752d 3.9 MB/s | 501 kB 00:00 2024-04-14T23:45:52.939 INFO:teuthology.orchestra.run.smithi060.stdout:(18/174): ceph-osd-19.0.0-2114.ga9a752df.el7.x8 6.0 MB/s | 17 MB 00:02 2024-04-14T23:45:53.017 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - Extras packages 49 kB/s | 16 kB 00:00 2024-04-14T23:45:53.073 INFO:teuthology.orchestra.run.smithi060.stdout:(19/174): librgw2-19.0.0-2114.ga9a752df.el7.x86 6.3 MB/s | 5.2 MB 00:00 2024-04-14T23:45:53.115 INFO:teuthology.orchestra.run.smithi060.stdout:(20/174): python3-ceph-argparse-19.0.0-2114.ga9 259 kB/s | 45 kB 00:00 2024-04-14T23:45:53.140 INFO:teuthology.orchestra.run.smithi060.stdout:(21/174): python3-ceph-common-19.0.0-2114.ga9a7 1.9 MB/s | 131 kB 00:00 2024-04-14T23:45:53.182 INFO:teuthology.orchestra.run.smithi060.stdout:(22/174): python3-cephfs-19.0.0-2114.ga9a752df. 2.4 MB/s | 162 kB 00:00 2024-04-14T23:45:53.224 INFO:teuthology.orchestra.run.smithi060.stdout:(23/174): python3-rados-19.0.0-2114.ga9a752df.e 3.7 MB/s | 320 kB 00:00 2024-04-14T23:45:53.257 INFO:teuthology.orchestra.run.smithi060.stdout:(24/174): python3-rbd-19.0.0-2114.ga9a752df.el7 3.9 MB/s | 298 kB 00:00 2024-04-14T23:45:53.274 INFO:teuthology.orchestra.run.smithi060.stdout:(25/174): python3-rgw-19.0.0-2114.ga9a752df.el7 2.0 MB/s | 99 kB 00:00 2024-04-14T23:45:53.324 INFO:teuthology.orchestra.run.smithi060.stdout:(26/174): rbd-fuse-19.0.0-2114.ga9a752df.el7.x8 1.3 MB/s | 85 kB 00:00 2024-04-14T23:45:53.417 INFO:teuthology.orchestra.run.smithi060.stdout:(27/174): rbd-nbd-19.0.0-2114.ga9a752df.el7.x86 1.8 MB/s | 171 kB 00:00 2024-04-14T23:45:53.542 INFO:teuthology.orchestra.run.smithi060.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2114.g 255 kB/s | 32 kB 00:00 2024-04-14T23:45:53.634 INFO:teuthology.orchestra.run.smithi060.stdout:(29/174): rbd-mirror-19.0.0-2114.ga9a752df.el7. 8.6 MB/s | 3.1 MB 00:00 2024-04-14T23:45:53.668 INFO:teuthology.orchestra.run.smithi060.stdout:(30/174): ceph-mgr-cephadm-19.0.0-2114.ga9a752d 1.1 MB/s | 139 kB 00:00 2024-04-14T23:45:54.068 INFO:teuthology.orchestra.run.smithi060.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2114.ga9a75 8.1 MB/s | 3.5 MB 00:00 2024-04-14T23:45:54.593 INFO:teuthology.orchestra.run.smithi060.stdout:(32/174): ceph-test-19.0.0-2114.ga9a752df.el7.x 13 MB/s | 48 MB 00:03 2024-04-14T23:45:54.627 INFO:teuthology.orchestra.run.smithi060.stdout:(33/174): ceph-mgr-modules-core-19.0.0-2114.ga9 445 kB/s | 248 kB 00:00 2024-04-14T23:45:54.669 INFO:teuthology.orchestra.run.smithi060.stdout:(34/174): ceph-mgr-rook-19.0.0-2114.ga9a752df.e 660 kB/s | 49 kB 00:00 2024-04-14T23:45:54.710 INFO:teuthology.orchestra.run.smithi060.stdout:(35/174): ceph-prometheus-alerts-19.0.0-2114.ga 186 kB/s | 15 kB 00:00 2024-04-14T23:45:54.760 INFO:teuthology.orchestra.run.smithi060.stdout:(36/174): ceph-volume-19.0.0-2114.ga9a752df.el7 2.8 MB/s | 264 kB 00:00 2024-04-14T23:45:54.828 INFO:teuthology.orchestra.run.smithi060.stdout:(37/174): cephadm-19.0.0-2114.ga9a752df.el7.noa 6.3 MB/s | 749 kB 00:00 2024-04-14T23:45:54.869 INFO:teuthology.orchestra.run.smithi060.stdout:(38/174): cephfs-top-19.0.0-2114.ga9a752df.el7. 169 kB/s | 18 kB 00:00 2024-04-14T23:45:55.029 INFO:teuthology.orchestra.run.smithi060.stdout:(39/174): ceph-mgr-diskprediction-local-19.0.0- 5.4 MB/s | 7.4 MB 00:01 2024-04-14T23:45:55.087 INFO:teuthology.orchestra.run.smithi060.stdout:(40/174): libaio-devel-0.3.112-1.el8.x86_64.rpm 81 kB/s | 18 kB 00:00 2024-04-14T23:45:55.129 INFO:teuthology.orchestra.run.smithi060.stdout:(41/174): fuse-2.9.9-15.el9.x86_64.rpm 312 kB/s | 80 kB 00:00 2024-04-14T23:45:55.379 INFO:teuthology.orchestra.run.smithi060.stdout:(42/174): jq-1.6-16.el9.x86_64.rpm 538 kB/s | 187 kB 00:00 2024-04-14T23:45:55.463 INFO:teuthology.orchestra.run.smithi060.stdout:(43/174): libquadmath-11.4.1-3.el9.x86_64.rpm 587 kB/s | 195 kB 00:00 2024-04-14T23:45:55.638 INFO:teuthology.orchestra.run.smithi060.stdout:(44/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 845 kB/s | 218 kB 00:00 2024-04-14T23:45:55.722 INFO:teuthology.orchestra.run.smithi060.stdout:(45/174): libgfortran-11.4.1-3.el9.x86_64.rpm 1.2 MB/s | 803 kB 00:00 2024-04-14T23:45:55.747 INFO:teuthology.orchestra.run.smithi060.stdout:(46/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 894 kB/s | 253 kB 00:00 2024-04-14T23:45:55.856 INFO:teuthology.orchestra.run.smithi060.stdout:(47/174): python3-ply-3.11-14.el9.noarch.rpm 993 kB/s | 106 kB 00:00 2024-04-14T23:45:55.998 INFO:teuthology.orchestra.run.smithi060.stdout:(48/174): python3-pycparser-2.20-6.el9.noarch.r 543 kB/s | 135 kB 00:00 2024-04-14T23:45:56.031 INFO:teuthology.orchestra.run.smithi060.stdout:(49/174): python3-requests-2.25.1-8.el9.noarch. 716 kB/s | 125 kB 00:00 2024-04-14T23:45:56.132 INFO:teuthology.orchestra.run.smithi060.stdout:(50/174): python3-urllib3-1.26.5-5.el9.noarch.r 1.6 MB/s | 215 kB 00:00 2024-04-14T23:45:56.290 INFO:teuthology.orchestra.run.smithi060.stdout:(51/174): python3-cryptography-36.0.1-4.el9.x86 1.9 MB/s | 1.2 MB 00:00 2024-04-14T23:45:56.765 INFO:teuthology.orchestra.run.smithi060.stdout:(52/174): boost-program-options-1.75.0-8.el9.x8 225 kB/s | 107 kB 00:00 2024-04-14T23:45:56.807 INFO:teuthology.orchestra.run.smithi060.stdout:(53/174): automake-1.16.2-8.el9.noarch.rpm 889 kB/s | 689 kB 00:00 2024-04-14T23:45:56.874 INFO:teuthology.orchestra.run.smithi060.stdout:(54/174): elfutils-libelf-devel-0.190-2.el9.x86 199 kB/s | 21 kB 00:00 2024-04-14T23:45:56.925 INFO:teuthology.orchestra.run.smithi060.stdout:(55/174): flex-2.6.4-9.el9.x86_64.rpm 2.6 MB/s | 313 kB 00:00 2024-04-14T23:45:56.983 INFO:teuthology.orchestra.run.smithi060.stdout:(56/174): flexiblas-3.0.4-8.el9.x86_64.rpm 302 kB/s | 33 kB 00:00 2024-04-14T23:45:57.025 INFO:teuthology.orchestra.run.smithi060.stdout:(57/174): bison-3.7.4-5.el9.x86_64.rpm 1.0 MB/s | 943 kB 00:00 2024-04-14T23:45:57.092 INFO:teuthology.orchestra.run.smithi060.stdout:(58/174): flexiblas-openblas-openmp-3.0.4-8.el9 169 kB/s | 18 kB 00:00 2024-04-14T23:45:57.143 INFO:teuthology.orchestra.run.smithi060.stdout:(59/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 160 kB 00:00 2024-04-14T23:45:57.201 INFO:teuthology.orchestra.run.smithi060.stdout:(60/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 418 kB/s | 45 kB 00:00 2024-04-14T23:45:57.343 INFO:teuthology.orchestra.run.smithi060.stdout:(61/174): librdkafka-1.6.1-102.el9.x86_64.rpm 3.2 MB/s | 662 kB 00:00 2024-04-14T23:45:57.452 INFO:teuthology.orchestra.run.smithi060.stdout:(62/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 5.7 MB/s | 3.0 MB 00:00 2024-04-14T23:45:57.493 INFO:teuthology.orchestra.run.smithi060.stdout:(63/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 170 kB/s | 25 kB 00:00 2024-04-14T23:45:57.560 INFO:teuthology.orchestra.run.smithi060.stdout:(64/174): libtool-2.4.6-45.el9.x86_64.rpm 1.6 MB/s | 582 kB 00:00 2024-04-14T23:45:57.586 INFO:teuthology.orchestra.run.smithi060.stdout:(65/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 356 kB/s | 47 kB 00:00 2024-04-14T23:45:57.619 INFO:teuthology.orchestra.run.smithi060.stdout:(66/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.3 MB/s | 292 kB 00:00 2024-04-14T23:45:57.670 INFO:teuthology.orchestra.run.smithi060.stdout:(67/174): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 188 kB 00:00 2024-04-14T23:45:57.695 INFO:teuthology.orchestra.run.smithi060.stdout:(68/174): openblas-0.3.21-2.el9.x86_64.rpm 325 kB/s | 35 kB 00:00 2024-04-14T23:45:57.862 INFO:teuthology.orchestra.run.smithi060.stdout:(69/174): postgresql-13.14-1.el9.x86_64.rpm 9.5 MB/s | 1.6 MB 00:00 2024-04-14T23:45:58.604 INFO:teuthology.orchestra.run.smithi060.stdout:(70/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 4.4 MB/s | 4.1 MB 00:00 2024-04-14T23:45:58.605 INFO:teuthology.orchestra.run.smithi186.stdout:Extra Packages for Enterprise Linux 3.8 MB/s | 21 MB 00:05 2024-04-14T23:45:58.871 INFO:teuthology.orchestra.run.smithi060.stdout:(71/174): postgresql-private-libs-13.14-1.el9.x 502 kB/s | 133 kB 00:00 2024-04-14T23:45:59.163 INFO:teuthology.orchestra.run.smithi060.stdout:(72/174): postgresql-contrib-13.14-1.el9.x86_64 701 kB/s | 911 kB 00:01 2024-04-14T23:46:00.563 INFO:teuthology.orchestra.run.smithi060.stdout:(73/174): protobuf-3.14.0-13.el9.x86_64.rpm 738 kB/s | 1.0 MB 00:01 2024-04-14T23:46:00.813 INFO:teuthology.orchestra.run.smithi060.stdout:(74/174): openblas-openmp-0.3.21-2.el9.x86_64.r 1.5 MB/s | 4.7 MB 00:03 2024-04-14T23:46:00.838 INFO:teuthology.orchestra.run.smithi060.stdout:(75/174): python3-attrs-20.3.0-7.el9.noarch.rpm 319 kB/s | 87 kB 00:00 2024-04-14T23:46:01.197 INFO:teuthology.orchestra.run.smithi060.stdout:(76/174): python3-devel-3.9.17-2.el9.x86_64.rpm 686 kB/s | 245 kB 00:00 2024-04-14T23:46:01.547 INFO:teuthology.orchestra.run.smithi060.stdout:(77/174): python3-jinja2-2.11.3-5.el9.noarch.rp 710 kB/s | 248 kB 00:00 2024-04-14T23:46:01.656 INFO:teuthology.orchestra.run.smithi060.stdout:(78/174): python3-jsonpatch-1.21-16.el9.noarch. 238 kB/s | 26 kB 00:00 2024-04-14T23:46:01.764 INFO:teuthology.orchestra.run.smithi060.stdout:(79/174): python3-jsonpointer-2.0-4.el9.noarch. 178 kB/s | 19 kB 00:00 2024-04-14T23:46:02.040 INFO:teuthology.orchestra.run.smithi060.stdout:(80/174): python3-mako-1.1.4-6.el9.noarch.rpm 627 kB/s | 172 kB 00:00 2024-04-14T23:46:02.148 INFO:teuthology.orchestra.run.smithi060.stdout:(81/174): python3-markupsafe-1.1.1-12.el9.x86_6 323 kB/s | 35 kB 00:00 2024-04-14T23:46:03.818 INFO:teuthology.orchestra.run.smithi186.stdout:lab-extras 17 kB/s | 1.7 kB 00:00 2024-04-14T23:46:06.405 INFO:teuthology.orchestra.run.smithi186.stderr:Modular dependency problems: 2024-04-14T23:46:06.405 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-14T23:46:06.405 INFO:teuthology.orchestra.run.smithi186.stderr: Problem 1: conflicting requests 2024-04-14T23:46:06.405 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.405 INFO:teuthology.orchestra.run.smithi186.stderr: Problem 2: conflicting requests 2024-04-14T23:46:06.405 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.405 INFO:teuthology.orchestra.run.smithi186.stderr: Problem 3: conflicting requests 2024-04-14T23:46:06.405 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.406 INFO:teuthology.orchestra.run.smithi186.stderr: Problem 4: conflicting requests 2024-04-14T23:46:06.406 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.406 INFO:teuthology.orchestra.run.smithi186.stderr: Problem 5: conflicting requests 2024-04-14T23:46:06.406 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.406 INFO:teuthology.orchestra.run.smithi186.stderr: Problem 6: conflicting requests 2024-04-14T23:46:06.406 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.406 INFO:teuthology.orchestra.run.smithi186.stderr: Problem 7: conflicting requests 2024-04-14T23:46:06.406 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.406 INFO:teuthology.orchestra.run.smithi186.stderr: Problem 8: conflicting requests 2024-04-14T23:46:06.406 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.406 INFO:teuthology.orchestra.run.smithi186.stderr: Problem 9: conflicting requests 2024-04-14T23:46:06.406 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.407 INFO:teuthology.orchestra.run.smithi186.stderr: Problem 10: conflicting requests 2024-04-14T23:46:06.407 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.407 INFO:teuthology.orchestra.run.smithi186.stderr: Problem 11: conflicting requests 2024-04-14T23:46:06.407 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.407 INFO:teuthology.orchestra.run.smithi186.stderr: Problem 12: conflicting requests 2024-04-14T23:46:06.407 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.407 INFO:teuthology.orchestra.run.smithi186.stderr: Problem 13: conflicting requests 2024-04-14T23:46:06.407 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.407 INFO:teuthology.orchestra.run.smithi186.stderr: Problem 14: conflicting requests 2024-04-14T23:46:06.407 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.407 INFO:teuthology.orchestra.run.smithi186.stderr: Problem 15: conflicting requests 2024-04-14T23:46:06.407 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.407 INFO:teuthology.orchestra.run.smithi186.stderr: Problem 16: conflicting requests 2024-04-14T23:46:06.407 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.408 INFO:teuthology.orchestra.run.smithi186.stderr: Problem 17: conflicting requests 2024-04-14T23:46:06.408 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.408 INFO:teuthology.orchestra.run.smithi186.stderr: Problem 18: conflicting requests 2024-04-14T23:46:06.408 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.408 INFO:teuthology.orchestra.run.smithi186.stderr: Problem 19: conflicting requests 2024-04-14T23:46:06.408 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.408 INFO:teuthology.orchestra.run.smithi186.stderr: Problem 20: conflicting requests 2024-04-14T23:46:06.408 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.408 INFO:teuthology.orchestra.run.smithi186.stderr: Problem 21: conflicting requests 2024-04-14T23:46:06.408 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.408 INFO:teuthology.orchestra.run.smithi186.stderr: Problem 22: conflicting requests 2024-04-14T23:46:06.408 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.408 INFO:teuthology.orchestra.run.smithi186.stderr: Problem 23: conflicting requests 2024-04-14T23:46:06.408 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.409 INFO:teuthology.orchestra.run.smithi186.stderr: Problem 24: conflicting requests 2024-04-14T23:46:06.409 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.409 INFO:teuthology.orchestra.run.smithi186.stderr: Problem 25: conflicting requests 2024-04-14T23:46:06.409 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.409 INFO:teuthology.orchestra.run.smithi186.stderr: Problem 26: conflicting requests 2024-04-14T23:46:06.409 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.409 INFO:teuthology.orchestra.run.smithi186.stderr: Problem 27: conflicting requests 2024-04-14T23:46:06.409 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.409 INFO:teuthology.orchestra.run.smithi186.stderr: Problem 28: conflicting requests 2024-04-14T23:46:06.409 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.409 INFO:teuthology.orchestra.run.smithi186.stderr: Problem 29: conflicting requests 2024-04-14T23:46:06.409 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.409 INFO:teuthology.orchestra.run.smithi186.stderr: Problem 30: conflicting requests 2024-04-14T23:46:06.410 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.410 INFO:teuthology.orchestra.run.smithi186.stderr: Problem 31: conflicting requests 2024-04-14T23:46:06.410 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.410 INFO:teuthology.orchestra.run.smithi186.stderr: Problem 32: conflicting requests 2024-04-14T23:46:06.410 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.410 INFO:teuthology.orchestra.run.smithi186.stderr: Problem 33: conflicting requests 2024-04-14T23:46:06.410 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.410 INFO:teuthology.orchestra.run.smithi186.stderr: Problem 34: conflicting requests 2024-04-14T23:46:06.410 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.410 INFO:teuthology.orchestra.run.smithi186.stderr: Problem 35: conflicting requests 2024-04-14T23:46:06.410 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.410 INFO:teuthology.orchestra.run.smithi186.stderr: Problem 36: conflicting requests 2024-04-14T23:46:06.410 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.411 INFO:teuthology.orchestra.run.smithi186.stderr: Problem 37: conflicting requests 2024-04-14T23:46:06.411 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.411 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.411 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.411 INFO:teuthology.orchestra.run.smithi186.stderr: Problem 38: cannot install the best candidate for the job 2024-04-14T23:46:06.411 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.411 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.411 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.411 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.411 INFO:teuthology.orchestra.run.smithi186.stderr: Problem 39: conflicting requests 2024-04-14T23:46:06.411 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.411 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.411 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.412 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.412 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.412 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.412 INFO:teuthology.orchestra.run.smithi186.stderr: Problem 40: cannot install the best candidate for the job 2024-04-14T23:46:06.412 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.412 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.412 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.412 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.412 INFO:teuthology.orchestra.run.smithi186.stderr: Problem 41: cannot install the best candidate for the job 2024-04-14T23:46:06.412 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.412 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.412 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.412 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.412 INFO:teuthology.orchestra.run.smithi186.stderr: Problem 42: cannot install the best candidate for the job 2024-04-14T23:46:06.413 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.413 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.413 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.413 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.413 INFO:teuthology.orchestra.run.smithi186.stderr: Problem 43: cannot install the best candidate for the job 2024-04-14T23:46:06.413 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.413 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.413 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.413 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.413 INFO:teuthology.orchestra.run.smithi186.stderr: Problem 44: cannot install the best candidate for the job 2024-04-14T23:46:06.413 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.413 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.413 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.413 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.413 INFO:teuthology.orchestra.run.smithi186.stderr: Problem 45: cannot install the best candidate for the job 2024-04-14T23:46:06.414 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.414 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.414 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.414 INFO:teuthology.orchestra.run.smithi186.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-14T23:46:06.814 INFO:teuthology.orchestra.run.smithi186.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-14T23:46:06.815 INFO:teuthology.orchestra.run.smithi186.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-14T23:46:06.828 INFO:teuthology.orchestra.run.smithi186.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-04-14T23:46:06.829 INFO:teuthology.orchestra.run.smithi186.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-14T23:46:06.830 INFO:teuthology.orchestra.run.smithi186.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-04-14T23:46:06.831 INFO:teuthology.orchestra.run.smithi186.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-04-14T23:46:06.833 INFO:teuthology.orchestra.run.smithi186.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-14T23:46:06.835 INFO:teuthology.orchestra.run.smithi186.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-04-14T23:46:06.837 INFO:teuthology.orchestra.run.smithi186.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-04-14T23:46:06.848 INFO:teuthology.orchestra.run.smithi060.stdout:(82/174): postgresql-server-13.14-1.el9.x86_64. 742 kB/s | 5.8 MB 00:07 2024-04-14T23:46:06.932 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-04-14T23:46:06.965 INFO:teuthology.orchestra.run.smithi186.stdout:========================================================================================= 2024-04-14T23:46:06.966 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-04-14T23:46:06.966 INFO:teuthology.orchestra.run.smithi186.stdout:========================================================================================= 2024-04-14T23:46:06.966 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-04-14T23:46:06.966 INFO:teuthology.orchestra.run.smithi186.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-14T23:46:06.966 INFO:teuthology.orchestra.run.smithi186.stdout: ceph x86_64 2:19.0.0-2114.ga9a752df.el7 ceph 6.6 k 2024-04-14T23:46:06.966 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-base x86_64 2:19.0.0-2114.ga9a752df.el7 ceph 5.3 M 2024-04-14T23:46:06.966 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-fuse x86_64 2:19.0.0-2114.ga9a752df.el7 ceph 891 k 2024-04-14T23:46:06.966 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2114.ga9a752df.el7 ceph 145 k 2024-04-14T23:46:06.966 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr x86_64 2:19.0.0-2114.ga9a752df.el7 ceph 1.5 M 2024-04-14T23:46:06.966 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2114.ga9a752df.el7 ceph-noarch 139 k 2024-04-14T23:46:06.966 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2114.ga9a752df.el7 ceph-noarch 3.5 M 2024-04-14T23:46:06.966 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2114.ga9a752df.el7 ceph-noarch 7.4 M 2024-04-14T23:46:06.966 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-rook noarch 2:19.0.0-2114.ga9a752df.el7 ceph-noarch 49 k 2024-04-14T23:46:06.967 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-radosgw x86_64 2:19.0.0-2114.ga9a752df.el7 ceph 9.9 M 2024-04-14T23:46:06.967 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-test x86_64 2:19.0.0-2114.ga9a752df.el7 ceph 48 M 2024-04-14T23:46:06.967 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-volume noarch 2:19.0.0-2114.ga9a752df.el7 ceph-noarch 264 k 2024-04-14T23:46:06.967 INFO:teuthology.orchestra.run.smithi186.stdout: cephadm noarch 2:19.0.0-2114.ga9a752df.el7 ceph-noarch 749 k 2024-04-14T23:46:06.967 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-mirror x86_64 2:19.0.0-2114.ga9a752df.el7 ceph 222 k 2024-04-14T23:46:06.967 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-shell x86_64 2:19.0.0-2114.ga9a752df.el7 ceph 27 k 2024-04-14T23:46:06.967 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-top noarch 2:19.0.0-2114.ga9a752df.el7 ceph-noarch 18 k 2024-04-14T23:46:06.967 INFO:teuthology.orchestra.run.smithi186.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-14T23:46:06.967 INFO:teuthology.orchestra.run.smithi186.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-14T23:46:06.967 INFO:teuthology.orchestra.run.smithi186.stdout: libaio-devel x86_64 0.3.112-1.el8 CentOS-BaseOS 18 k 2024-04-14T23:46:06.967 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs-devel x86_64 2:19.0.0-2114.ga9a752df.el7 ceph 32 k 2024-04-14T23:46:06.967 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs2 x86_64 2:19.0.0-2114.ga9a752df.el7 ceph 746 k 2024-04-14T23:46:06.967 INFO:teuthology.orchestra.run.smithi186.stdout: librados-devel x86_64 2:19.0.0-2114.ga9a752df.el7 ceph 127 k 2024-04-14T23:46:06.968 INFO:teuthology.orchestra.run.smithi186.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-14T23:46:06.968 INFO:teuthology.orchestra.run.smithi186.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-14T23:46:06.968 INFO:teuthology.orchestra.run.smithi186.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-04-14T23:46:06.968 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-14T23:46:06.968 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-14T23:46:06.968 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-14T23:46:06.968 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cephfs x86_64 2:19.0.0-2114.ga9a752df.el7 ceph 162 k 2024-04-14T23:46:06.968 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rados x86_64 2:19.0.0-2114.ga9a752df.el7 ceph 320 k 2024-04-14T23:46:06.968 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rbd x86_64 2:19.0.0-2114.ga9a752df.el7 ceph 298 k 2024-04-14T23:46:06.968 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rgw x86_64 2:19.0.0-2114.ga9a752df.el7 ceph 99 k 2024-04-14T23:46:06.968 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-fuse x86_64 2:19.0.0-2114.ga9a752df.el7 ceph 85 k 2024-04-14T23:46:06.968 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-mirror x86_64 2:19.0.0-2114.ga9a752df.el7 ceph 3.1 M 2024-04-14T23:46:06.968 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-nbd x86_64 2:19.0.0-2114.ga9a752df.el7 ceph 171 k 2024-04-14T23:46:06.968 INFO:teuthology.orchestra.run.smithi186.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-14T23:46:06.969 INFO:teuthology.orchestra.run.smithi186.stdout:Upgrading: 2024-04-14T23:46:06.969 INFO:teuthology.orchestra.run.smithi186.stdout: NetworkManager x86_64 1:1.47.4-1.el9 baseos 2.3 M 2024-04-14T23:46:06.969 INFO:teuthology.orchestra.run.smithi186.stdout: NetworkManager-libnm x86_64 1:1.47.4-1.el9 baseos 1.8 M 2024-04-14T23:46:06.969 INFO:teuthology.orchestra.run.smithi186.stdout: NetworkManager-team x86_64 1:1.47.4-1.el9 baseos 42 k 2024-04-14T23:46:06.969 INFO:teuthology.orchestra.run.smithi186.stdout: NetworkManager-tui x86_64 1:1.47.4-1.el9 baseos 249 k 2024-04-14T23:46:06.969 INFO:teuthology.orchestra.run.smithi186.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-04-14T23:46:06.969 INFO:teuthology.orchestra.run.smithi186.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-04-14T23:46:06.969 INFO:teuthology.orchestra.run.smithi186.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-04-14T23:46:06.969 INFO:teuthology.orchestra.run.smithi186.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-04-14T23:46:06.969 INFO:teuthology.orchestra.run.smithi186.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-04-14T23:46:06.969 INFO:teuthology.orchestra.run.smithi186.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-04-14T23:46:06.969 INFO:teuthology.orchestra.run.smithi186.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-04-14T23:46:06.969 INFO:teuthology.orchestra.run.smithi186.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-04-14T23:46:06.969 INFO:teuthology.orchestra.run.smithi186.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-04-14T23:46:06.970 INFO:teuthology.orchestra.run.smithi186.stdout: librados2 x86_64 2:19.0.0-2114.ga9a752df.el7 ceph 3.4 M 2024-04-14T23:46:06.970 INFO:teuthology.orchestra.run.smithi186.stdout: librbd1 x86_64 2:19.0.0-2114.ga9a752df.el7 ceph 3.1 M 2024-04-14T23:46:06.970 INFO:teuthology.orchestra.run.smithi186.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-04-14T23:46:06.970 INFO:teuthology.orchestra.run.smithi186.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-04-14T23:46:06.970 INFO:teuthology.orchestra.run.smithi186.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-04-14T23:46:06.970 INFO:teuthology.orchestra.run.smithi186.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-04-14T23:46:06.970 INFO:teuthology.orchestra.run.smithi186.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-04-14T23:46:06.970 INFO:teuthology.orchestra.run.smithi186.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-04-14T23:46:06.970 INFO:teuthology.orchestra.run.smithi186.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-04-14T23:46:06.970 INFO:teuthology.orchestra.run.smithi186.stdout: xfsdump x86_64 3.1.12-4.el9 baseos 323 k 2024-04-14T23:46:06.970 INFO:teuthology.orchestra.run.smithi186.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-04-14T23:46:06.970 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-04-14T23:46:06.970 INFO:teuthology.orchestra.run.smithi186.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-14T23:46:06.970 INFO:teuthology.orchestra.run.smithi186.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-14T23:46:06.971 INFO:teuthology.orchestra.run.smithi186.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-14T23:46:06.971 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-common x86_64 2:19.0.0-2114.ga9a752df.el7 ceph 21 M 2024-04-14T23:46:06.971 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2114.ga9a752df.el7 ceph-noarch 32 k 2024-04-14T23:46:06.971 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mds x86_64 2:19.0.0-2114.ga9a752df.el7 ceph 2.4 M 2024-04-14T23:46:06.971 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2114.ga9a752df.el7 ceph-noarch 248 k 2024-04-14T23:46:06.971 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mon x86_64 2:19.0.0-2114.ga9a752df.el7 ceph 4.7 M 2024-04-14T23:46:06.971 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-osd x86_64 2:19.0.0-2114.ga9a752df.el7 ceph 17 M 2024-04-14T23:46:06.971 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2114.ga9a752df.el7 ceph-noarch 15 k 2024-04-14T23:46:06.971 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-selinux x86_64 2:19.0.0-2114.ga9a752df.el7 ceph 25 k 2024-04-14T23:46:06.971 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-14T23:46:06.971 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-14T23:46:06.971 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-14T23:46:06.972 INFO:teuthology.orchestra.run.smithi186.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-14T23:46:06.972 INFO:teuthology.orchestra.run.smithi186.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-14T23:46:06.972 INFO:teuthology.orchestra.run.smithi186.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-14T23:46:06.972 INFO:teuthology.orchestra.run.smithi186.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-14T23:46:06.972 INFO:teuthology.orchestra.run.smithi186.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-14T23:46:06.972 INFO:teuthology.orchestra.run.smithi186.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-14T23:46:06.972 INFO:teuthology.orchestra.run.smithi186.stdout: libcephsqlite x86_64 2:19.0.0-2114.ga9a752df.el7 ceph 164 k 2024-04-14T23:46:06.972 INFO:teuthology.orchestra.run.smithi186.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-14T23:46:06.972 INFO:teuthology.orchestra.run.smithi186.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-14T23:46:06.973 INFO:teuthology.orchestra.run.smithi186.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-14T23:46:06.973 INFO:teuthology.orchestra.run.smithi186.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-14T23:46:06.973 INFO:teuthology.orchestra.run.smithi186.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-14T23:46:06.973 INFO:teuthology.orchestra.run.smithi186.stdout: libradosstriper1 x86_64 2:19.0.0-2114.ga9a752df.el7 ceph 501 k 2024-04-14T23:46:06.973 INFO:teuthology.orchestra.run.smithi186.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-14T23:46:06.973 INFO:teuthology.orchestra.run.smithi186.stdout: librgw2 x86_64 2:19.0.0-2114.ga9a752df.el7 ceph 5.2 M 2024-04-14T23:46:06.973 INFO:teuthology.orchestra.run.smithi186.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-14T23:46:06.973 INFO:teuthology.orchestra.run.smithi186.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-14T23:46:06.973 INFO:teuthology.orchestra.run.smithi186.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-14T23:46:06.973 INFO:teuthology.orchestra.run.smithi186.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-14T23:46:06.973 INFO:teuthology.orchestra.run.smithi186.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-14T23:46:06.973 INFO:teuthology.orchestra.run.smithi186.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-14T23:46:06.973 INFO:teuthology.orchestra.run.smithi186.stdout: oniguruma x86_64 6.9.6-1.el9.5 baseos 218 k 2024-04-14T23:46:06.973 INFO:teuthology.orchestra.run.smithi186.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-14T23:46:06.974 INFO:teuthology.orchestra.run.smithi186.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-14T23:46:06.974 INFO:teuthology.orchestra.run.smithi186.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-14T23:46:06.974 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-14T23:46:06.974 INFO:teuthology.orchestra.run.smithi186.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-14T23:46:06.974 INFO:teuthology.orchestra.run.smithi186.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-14T23:46:06.974 INFO:teuthology.orchestra.run.smithi186.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-14T23:46:06.974 INFO:teuthology.orchestra.run.smithi186.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-14T23:46:06.974 INFO:teuthology.orchestra.run.smithi186.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-14T23:46:06.974 INFO:teuthology.orchestra.run.smithi186.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-14T23:46:06.974 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-14T23:46:06.974 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-argparse x86_64 2:19.0.0-2114.ga9a752df.el7 ceph 45 k 2024-04-14T23:46:06.974 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-common x86_64 2:19.0.0-2114.ga9a752df.el7 ceph 131 k 2024-04-14T23:46:06.974 INFO:teuthology.orchestra.run.smithi186.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-14T23:46:06.975 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-14T23:46:06.975 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-14T23:46:06.975 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-14T23:46:06.975 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-14T23:46:06.975 INFO:teuthology.orchestra.run.smithi186.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-14T23:46:06.975 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-14T23:46:06.975 INFO:teuthology.orchestra.run.smithi186.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-14T23:46:06.975 INFO:teuthology.orchestra.run.smithi186.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-14T23:46:06.975 INFO:teuthology.orchestra.run.smithi186.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-14T23:46:06.975 INFO:teuthology.orchestra.run.smithi186.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-14T23:46:06.975 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-14T23:46:06.975 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-14T23:46:06.975 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-14T23:46:06.975 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-14T23:46:06.975 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-14T23:46:06.976 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-14T23:46:06.976 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-14T23:46:06.976 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-14T23:46:06.976 INFO:teuthology.orchestra.run.smithi186.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-14T23:46:06.976 INFO:teuthology.orchestra.run.smithi186.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-14T23:46:06.976 INFO:teuthology.orchestra.run.smithi186.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-14T23:46:06.976 INFO:teuthology.orchestra.run.smithi186.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-14T23:46:06.976 INFO:teuthology.orchestra.run.smithi186.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-14T23:46:06.976 INFO:teuthology.orchestra.run.smithi186.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-14T23:46:06.976 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-14T23:46:06.976 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-14T23:46:06.976 INFO:teuthology.orchestra.run.smithi186.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-14T23:46:06.976 INFO:teuthology.orchestra.run.smithi186.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-14T23:46:06.976 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-14T23:46:06.977 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-14T23:46:06.977 INFO:teuthology.orchestra.run.smithi186.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-14T23:46:06.977 INFO:teuthology.orchestra.run.smithi186.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-14T23:46:06.977 INFO:teuthology.orchestra.run.smithi186.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-14T23:46:06.977 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-14T23:46:06.977 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-14T23:46:06.977 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-14T23:46:06.977 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-14T23:46:06.977 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-14T23:46:06.977 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-14T23:46:06.977 INFO:teuthology.orchestra.run.smithi186.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-14T23:46:06.977 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-14T23:46:06.977 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-14T23:46:06.977 INFO:teuthology.orchestra.run.smithi186.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-14T23:46:06.977 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-14T23:46:06.978 INFO:teuthology.orchestra.run.smithi186.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-14T23:46:06.978 INFO:teuthology.orchestra.run.smithi186.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-14T23:46:06.978 INFO:teuthology.orchestra.run.smithi186.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-14T23:46:06.978 INFO:teuthology.orchestra.run.smithi186.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-14T23:46:06.978 INFO:teuthology.orchestra.run.smithi186.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-14T23:46:06.978 INFO:teuthology.orchestra.run.smithi186.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-14T23:46:06.978 INFO:teuthology.orchestra.run.smithi186.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-14T23:46:06.978 INFO:teuthology.orchestra.run.smithi186.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-14T23:46:06.978 INFO:teuthology.orchestra.run.smithi186.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-14T23:46:06.978 INFO:teuthology.orchestra.run.smithi186.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-14T23:46:06.978 INFO:teuthology.orchestra.run.smithi186.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-14T23:46:06.978 INFO:teuthology.orchestra.run.smithi186.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-14T23:46:06.978 INFO:teuthology.orchestra.run.smithi186.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-14T23:46:06.978 INFO:teuthology.orchestra.run.smithi186.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-14T23:46:06.979 INFO:teuthology.orchestra.run.smithi186.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-14T23:46:06.979 INFO:teuthology.orchestra.run.smithi186.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-14T23:46:06.979 INFO:teuthology.orchestra.run.smithi186.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-14T23:46:06.979 INFO:teuthology.orchestra.run.smithi186.stdout: zlib-devel x86_64 1.2.11-40.el9 appstream 45 k 2024-04-14T23:46:06.979 INFO:teuthology.orchestra.run.smithi186.stdout:Installing weak dependencies: 2024-04-14T23:46:06.979 INFO:teuthology.orchestra.run.smithi186.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-14T23:46:06.979 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-14T23:46:06.979 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-04-14T23:46:06.979 INFO:teuthology.orchestra.run.smithi186.stdout:========================================================================================= 2024-04-14T23:46:06.979 INFO:teuthology.orchestra.run.smithi186.stdout:Install 150 Packages 2024-04-14T23:46:06.979 INFO:teuthology.orchestra.run.smithi186.stdout:Upgrade 24 Packages 2024-04-14T23:46:06.979 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-14T23:46:06.980 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 228 M 2024-04-14T23:46:06.980 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-04-14T23:46:07.224 INFO:teuthology.orchestra.run.smithi060.stdout:(83/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 670 kB/s | 250 kB 00:00 2024-04-14T23:46:07.574 INFO:teuthology.orchestra.run.smithi060.stdout:(84/174): python3-oauthlib-3.1.1-5.el9.noarch.r 637 kB/s | 222 kB 00:00 2024-04-14T23:46:07.775 INFO:teuthology.orchestra.run.smithi060.stdout:(85/174): python3-packaging-20.9-5.el9.noarch.r 385 kB/s | 77 kB 00:00 2024-04-14T23:46:07.883 INFO:teuthology.orchestra.run.smithi060.stdout:(86/174): python3-prettytable-0.7.2-27.el9.noar 397 kB/s | 42 kB 00:00 2024-04-14T23:46:08.245 INFO:teuthology.orchestra.run.smithi186.stdout:(1/174): ceph-19.0.0-2114.ga9a752df.el7.x86_64. 29 kB/s | 6.6 kB 00:00 2024-04-14T23:46:08.317 INFO:teuthology.orchestra.run.smithi060.stdout:(87/174): python3-protobuf-3.14.0-13.el9.noarch 622 kB/s | 269 kB 00:00 2024-04-14T23:46:08.592 INFO:teuthology.orchestra.run.smithi060.stdout:(88/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 578 kB/s | 159 kB 00:00 2024-04-14T23:46:08.786 INFO:teuthology.orchestra.run.smithi186.stdout:(2/174): ceph-fuse-19.0.0-2114.ga9a752df.el7.x8 1.6 MB/s | 891 kB 00:00 2024-04-14T23:46:08.912 INFO:teuthology.orchestra.run.smithi186.stdout:(3/174): ceph-immutable-object-cache-19.0.0-211 1.1 MB/s | 145 kB 00:00 2024-04-14T23:46:09.025 INFO:teuthology.orchestra.run.smithi060.stdout:(89/174): python3-pyasn1-modules-0.4.8-6.el9.no 645 kB/s | 279 kB 00:00 2024-04-14T23:46:09.226 INFO:teuthology.orchestra.run.smithi060.stdout:(90/174): python3-babel-2.9.1-2.el9.noarch.rpm 726 kB/s | 6.0 MB 00:08 2024-04-14T23:46:09.343 INFO:teuthology.orchestra.run.smithi060.stdout:(91/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 726 kB/s | 5.1 MB 00:07 2024-04-14T23:46:09.368 INFO:teuthology.orchestra.run.smithi060.stdout:(92/174): python3-pytz-2021.1-5.el9.noarch.rpm 149 kB/s | 51 kB 00:00 2024-04-14T23:46:09.393 INFO:teuthology.orchestra.run.smithi060.stdout:(93/174): python3-requests-oauthlib-1.3.0-12.el 323 kB/s | 54 kB 00:00 2024-04-14T23:46:09.477 INFO:teuthology.orchestra.run.smithi060.stdout:(94/174): python3-toml-0.10.2-6.el9.noarch.rpm 389 kB/s | 42 kB 00:00 2024-04-14T23:46:09.527 INFO:teuthology.orchestra.run.smithi060.stdout:(95/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 332 kB/s | 44 kB 00:00 2024-04-14T23:46:09.554 INFO:teuthology.orchestra.run.smithi186.stdout:(4/174): ceph-mds-19.0.0-2114.ga9a752df.el7.x86 3.7 MB/s | 2.4 MB 00:00 2024-04-14T23:46:09.594 INFO:teuthology.orchestra.run.smithi060.stdout:(96/174): qatlib-24.02.0-1.el9.x86_64.rpm 1.9 MB/s | 221 kB 00:00 2024-04-14T23:46:09.636 INFO:teuthology.orchestra.run.smithi060.stdout:(97/174): qatlib-service-24.02.0-1.el9.x86_64.r 332 kB/s | 36 kB 00:00 2024-04-14T23:46:09.703 INFO:teuthology.orchestra.run.smithi060.stdout:(98/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 438 kB/s | 47 kB 00:00 2024-04-14T23:46:09.787 INFO:teuthology.orchestra.run.smithi186.stdout:(5/174): ceph-base-19.0.0-2114.ga9a752df.el7.x8 3.0 MB/s | 5.3 MB 00:01 2024-04-14T23:46:09.812 INFO:teuthology.orchestra.run.smithi060.stdout:(99/174): uuid-1.6.2-55.el9.x86_64.rpm 538 kB/s | 58 kB 00:00 2024-04-14T23:46:09.904 INFO:teuthology.orchestra.run.smithi186.stdout:(6/174): ceph-mgr-19.0.0-2114.ga9a752df.el7.x86 4.3 MB/s | 1.5 MB 00:00 2024-04-14T23:46:09.920 INFO:teuthology.orchestra.run.smithi060.stdout:(100/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 872 kB/s | 94 kB 00:00 2024-04-14T23:46:10.029 INFO:teuthology.orchestra.run.smithi060.stdout:(101/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 593 kB/s | 64 kB 00:00 2024-04-14T23:46:10.263 INFO:teuthology.orchestra.run.smithi060.stdout:(102/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 18 MB/s | 16 MB 00:00 2024-04-14T23:46:10.288 INFO:teuthology.orchestra.run.smithi060.stdout:(103/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 175 kB/s | 45 kB 00:00 2024-04-14T23:46:10.321 INFO:teuthology.orchestra.run.smithi060.stdout:(104/174): socat-1.7.4.1-5.el9.x86_64.rpm 445 kB/s | 305 kB 00:00 2024-04-14T23:46:10.371 INFO:teuthology.orchestra.run.smithi186.stdout:(7/174): ceph-mon-19.0.0-2114.ga9a752df.el7.x86 8.0 MB/s | 4.7 MB 00:00 2024-04-14T23:46:10.388 INFO:teuthology.orchestra.run.smithi060.stdout:(105/174): lua-devel-5.4.4-4.el9.x86_64.rpm 180 kB/s | 22 kB 00:00 2024-04-14T23:46:10.689 INFO:teuthology.orchestra.run.smithi060.stdout:(106/174): protobuf-compiler-3.14.0-13.el9.x86_ 2.1 MB/s | 863 kB 00:00 2024-04-14T23:46:11.197 INFO:teuthology.orchestra.run.smithi060.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 39 kB/s | 19 kB 00:00 2024-04-14T23:46:11.389 INFO:teuthology.orchestra.run.smithi060.stdout:(108/174): gperftools-libs-2.9.1-2.el9.x86_64.r 309 kB/s | 309 kB 00:00 2024-04-14T23:46:11.413 INFO:teuthology.orchestra.run.smithi186.stdout:(8/174): ceph-common-19.0.0-2114.ga9a752df.el7. 6.1 MB/s | 21 MB 00:03 2024-04-14T23:46:11.498 INFO:teuthology.orchestra.run.smithi060.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 241 kB/s | 26 kB 00:00 2024-04-14T23:46:11.530 INFO:teuthology.orchestra.run.smithi186.stdout:(9/174): ceph-radosgw-19.0.0-2114.ga9a752df.el7 8.5 MB/s | 9.9 MB 00:01 2024-04-14T23:46:11.547 INFO:teuthology.orchestra.run.smithi186.stdout:(10/174): ceph-selinux-19.0.0-2114.ga9a752df.el 189 kB/s | 25 kB 00:00 2024-04-14T23:46:11.615 INFO:teuthology.orchestra.run.smithi060.stdout:(110/174): abseil-cpp-20211102.0-3.el9.x86_64.r 424 kB/s | 548 kB 00:01 2024-04-14T23:46:11.640 INFO:teuthology.orchestra.run.smithi060.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 344 kB/s | 49 kB 00:00 2024-04-14T23:46:11.664 INFO:teuthology.orchestra.run.smithi186.stdout:(11/174): cephfs-mirror-19.0.0-2114.ga9a752df.e 1.9 MB/s | 222 kB 00:00 2024-04-14T23:46:11.824 INFO:teuthology.orchestra.run.smithi060.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 324 kB/s | 67 kB 00:00 2024-04-14T23:46:11.831 INFO:teuthology.orchestra.run.smithi186.stdout:(12/174): ceph-osd-19.0.0-2114.ga9a752df.el7.x8 8.7 MB/s | 17 MB 00:01 2024-04-14T23:46:11.856 INFO:teuthology.orchestra.run.smithi186.stdout:(13/174): cephfs-shell-19.0.0-2114.ga9a752df.el 141 kB/s | 27 kB 00:00 2024-04-14T23:46:11.898 INFO:teuthology.orchestra.run.smithi186.stdout:(14/174): libcephfs-devel-19.0.0-2114.ga9a752df 476 kB/s | 32 kB 00:00 2024-04-14T23:46:11.932 INFO:teuthology.orchestra.run.smithi060.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 519 kB/s | 151 kB 00:00 2024-04-14T23:46:11.948 INFO:teuthology.orchestra.run.smithi186.stdout:(15/174): libcephfs2-19.0.0-2114.ga9a752df.el7. 7.9 MB/s | 746 kB 00:00 2024-04-14T23:46:11.999 INFO:teuthology.orchestra.run.smithi186.stdout:(16/174): libcephsqlite-19.0.0-2114.ga9a752df.e 1.6 MB/s | 164 kB 00:00 2024-04-14T23:46:12.032 INFO:teuthology.orchestra.run.smithi186.stdout:(17/174): librados-devel-19.0.0-2114.ga9a752df. 1.5 MB/s | 127 kB 00:00 2024-04-14T23:46:12.099 INFO:teuthology.orchestra.run.smithi186.stdout:(18/174): libradosstriper1-19.0.0-2114.ga9a752d 4.9 MB/s | 501 kB 00:00 2024-04-14T23:46:12.216 INFO:teuthology.orchestra.run.smithi186.stdout:(19/174): python3-ceph-argparse-19.0.0-2114.ga9 387 kB/s | 45 kB 00:00 2024-04-14T23:46:12.316 INFO:teuthology.orchestra.run.smithi186.stdout:(20/174): python3-ceph-common-19.0.0-2114.ga9a7 1.3 MB/s | 131 kB 00:00 2024-04-14T23:46:12.417 INFO:teuthology.orchestra.run.smithi186.stdout:(21/174): python3-cephfs-19.0.0-2114.ga9a752df. 1.6 MB/s | 162 kB 00:00 2024-04-14T23:46:12.523 INFO:teuthology.orchestra.run.smithi186.stdout:(22/174): python3-rados-19.0.0-2114.ga9a752df.e 2.9 MB/s | 320 kB 00:00 2024-04-14T23:46:12.601 INFO:teuthology.orchestra.run.smithi186.stdout:(23/174): librgw2-19.0.0-2114.ga9a752df.el7.x86 9.1 MB/s | 5.2 MB 00:00 2024-04-14T23:46:12.624 INFO:teuthology.orchestra.run.smithi186.stdout:(24/174): python3-rbd-19.0.0-2114.ga9a752df.el7 2.9 MB/s | 298 kB 00:00 2024-04-14T23:46:12.682 INFO:teuthology.orchestra.run.smithi060.stdout:(114/174): python3-asyncssh-2.13.2-5.el9.noarch 732 kB/s | 548 kB 00:00 2024-04-14T23:46:12.691 INFO:teuthology.orchestra.run.smithi186.stdout:(25/174): python3-rgw-19.0.0-2114.ga9a752df.el7 1.0 MB/s | 99 kB 00:00 2024-04-14T23:46:12.717 INFO:teuthology.orchestra.run.smithi186.stdout:(26/174): rbd-fuse-19.0.0-2114.ga9a752df.el7.x8 923 kB/s | 85 kB 00:00 2024-04-14T23:46:12.791 INFO:teuthology.orchestra.run.smithi060.stdout:(115/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 403 kB/s | 43 kB 00:00 2024-04-14T23:46:12.826 INFO:teuthology.orchestra.run.smithi186.stdout:(27/174): rbd-nbd-19.0.0-2114.ga9a752df.el7.x86 1.5 MB/s | 171 kB 00:00 2024-04-14T23:46:12.908 INFO:teuthology.orchestra.run.smithi060.stdout:(116/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 774 kB/s | 838 kB 00:01 2024-04-14T23:46:12.925 INFO:teuthology.orchestra.run.smithi060.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 242 kB/s | 32 kB 00:00 2024-04-14T23:46:12.934 INFO:teuthology.orchestra.run.smithi186.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2114.g 293 kB/s | 32 kB 00:00 2024-04-14T23:46:13.026 INFO:teuthology.orchestra.run.smithi060.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 129 kB/s | 15 kB 00:00 2024-04-14T23:46:13.068 INFO:teuthology.orchestra.run.smithi186.stdout:(29/174): rbd-mirror-19.0.0-2114.ga9a752df.el7. 8.2 MB/s | 3.1 MB 00:00 2024-04-14T23:46:13.093 INFO:teuthology.orchestra.run.smithi186.stdout:(30/174): ceph-mgr-cephadm-19.0.0-2114.ga9a752d 880 kB/s | 139 kB 00:00 2024-04-14T23:46:13.217 INFO:teuthology.orchestra.run.smithi060.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 590 kB/s | 172 kB 00:00 2024-04-14T23:46:13.544 INFO:teuthology.orchestra.run.smithi186.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2114.ga9a75 7.4 MB/s | 3.5 MB 00:00 2024-04-14T23:46:13.559 INFO:teuthology.orchestra.run.smithi060.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 673 kB/s | 358 kB 00:00 2024-04-14T23:46:13.627 INFO:teuthology.orchestra.run.smithi186.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2114.ga9 2.9 MB/s | 248 kB 00:00 2024-04-14T23:46:13.676 INFO:teuthology.orchestra.run.smithi060.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 662 kB/s | 303 kB 00:00 2024-04-14T23:46:13.705 INFO:teuthology.orchestra.run.smithi060.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 250 kB/s | 35 kB 00:00 2024-04-14T23:46:13.719 INFO:teuthology.orchestra.run.smithi186.stdout:(33/174): ceph-mgr-rook-19.0.0-2114.ga9a752df.e 539 kB/s | 49 kB 00:00 2024-04-14T23:46:13.861 INFO:teuthology.orchestra.run.smithi186.stdout:(34/174): ceph-mgr-diskprediction-local-19.0.0- 9.6 MB/s | 7.4 MB 00:00 2024-04-14T23:46:13.906 INFO:teuthology.orchestra.run.smithi186.stdout:(35/174): ceph-prometheus-alerts-19.0.0-2114.ga 82 kB/s | 15 kB 00:00 2024-04-14T23:46:13.995 INFO:teuthology.orchestra.run.smithi186.stdout:(36/174): ceph-volume-19.0.0-2114.ga9a752df.el7 1.9 MB/s | 264 kB 00:00 2024-04-14T23:46:14.062 INFO:teuthology.orchestra.run.smithi186.stdout:(37/174): cephadm-19.0.0-2114.ga9a752df.el7.noa 4.7 MB/s | 749 kB 00:00 2024-04-14T23:46:14.087 INFO:teuthology.orchestra.run.smithi186.stdout:(38/174): cephfs-top-19.0.0-2114.ga9a752df.el7. 199 kB/s | 18 kB 00:00 2024-04-14T23:46:14.162 INFO:teuthology.orchestra.run.smithi186.stdout:(39/174): libaio-devel-0.3.112-1.el8.x86_64.rpm 175 kB/s | 18 kB 00:00 2024-04-14T23:46:14.213 INFO:teuthology.orchestra.run.smithi186.stdout:(40/174): fuse-2.9.9-15.el9.x86_64.rpm 641 kB/s | 80 kB 00:00 2024-04-14T23:46:14.226 INFO:teuthology.orchestra.run.smithi060.stdout:(123/174): python3-google-auth-2.28.2-1.el9.noa 387 kB/s | 203 kB 00:00 2024-04-14T23:46:14.321 INFO:teuthology.orchestra.run.smithi186.stdout:(41/174): jq-1.6-16.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-04-14T23:46:14.405 INFO:teuthology.orchestra.run.smithi186.stdout:(42/174): libquadmath-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 195 kB 00:00 2024-04-14T23:46:14.493 INFO:teuthology.orchestra.run.smithi060.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 544 kB/s | 144 kB 00:00 2024-04-14T23:46:14.602 INFO:teuthology.orchestra.run.smithi060.stdout:(125/174): python3-jaraco-8.2.1-3.el9.noarch.rp 99 kB/s | 11 kB 00:00 2024-04-14T23:46:14.711 INFO:teuthology.orchestra.run.smithi060.stdout:(126/174): python3-jaraco-classes-3.2.1-5.el9.n 164 kB/s | 18 kB 00:00 2024-04-14T23:46:14.836 INFO:teuthology.orchestra.run.smithi060.stdout:(127/174): python3-jaraco-collections-3.0.0-8.e 186 kB/s | 23 kB 00:00 2024-04-14T23:46:14.881 INFO:teuthology.orchestra.run.smithi186.stdout:(43/174): ceph-test-19.0.0-2114.ga9a752df.el7.x 14 MB/s | 48 MB 00:03 2024-04-14T23:46:14.906 INFO:teuthology.orchestra.run.smithi186.stdout:(44/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 436 kB/s | 218 kB 00:00 2024-04-14T23:46:14.943 INFO:teuthology.orchestra.run.smithi060.stdout:(128/174): python3-jaraco-functools-3.5.0-2.el9 184 kB/s | 19 kB 00:00 2024-04-14T23:46:14.947 INFO:teuthology.orchestra.run.smithi186.stdout:(45/174): libgfortran-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 803 kB 00:00 2024-04-14T23:46:15.048 INFO:teuthology.orchestra.run.smithi186.stdout:(46/174): python3-ply-3.11-14.el9.noarch.rpm 1.0 MB/s | 106 kB 00:00 2024-04-14T23:46:15.051 INFO:teuthology.orchestra.run.smithi060.stdout:(129/174): python3-jaraco-text-3.2.0-6.el9.noar 182 kB/s | 20 kB 00:00 2024-04-14T23:46:15.090 INFO:teuthology.orchestra.run.smithi186.stdout:(47/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.2 MB/s | 253 kB 00:00 2024-04-14T23:46:15.115 INFO:teuthology.orchestra.run.smithi186.stdout:(48/174): python3-pycparser-2.20-6.el9.noarch.r 2.0 MB/s | 135 kB 00:00 2024-04-14T23:46:15.150 INFO:teuthology.orchestra.run.smithi186.stdout:(49/174): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-04-14T23:46:15.224 INFO:teuthology.orchestra.run.smithi186.stdout:(50/174): python3-cryptography-36.0.1-4.el9.x86 3.9 MB/s | 1.2 MB 00:00 2024-04-14T23:46:15.250 INFO:teuthology.orchestra.run.smithi186.stdout:(51/174): python3-urllib3-1.26.5-5.el9.noarch.r 1.6 MB/s | 215 kB 00:00 2024-04-14T23:46:15.333 INFO:teuthology.orchestra.run.smithi186.stdout:(52/174): automake-1.16.2-8.el9.noarch.rpm 3.7 MB/s | 689 kB 00:00 2024-04-14T23:46:15.375 INFO:teuthology.orchestra.run.smithi186.stdout:(53/174): elfutils-libelf-devel-0.190-2.el9.x86 519 kB/s | 21 kB 00:00 2024-04-14T23:46:15.400 INFO:teuthology.orchestra.run.smithi186.stdout:(54/174): boost-program-options-1.75.0-8.el9.x8 710 kB/s | 107 kB 00:00 2024-04-14T23:46:15.426 INFO:teuthology.orchestra.run.smithi186.stdout:(55/174): flex-2.6.4-9.el9.x86_64.rpm 6.1 MB/s | 313 kB 00:00 2024-04-14T23:46:15.451 INFO:teuthology.orchestra.run.smithi186.stdout:(56/174): flexiblas-3.0.4-8.el9.x86_64.rpm 651 kB/s | 33 kB 00:00 2024-04-14T23:46:15.485 INFO:teuthology.orchestra.run.smithi186.stdout:(57/174): bison-3.7.4-5.el9.x86_64.rpm 3.5 MB/s | 943 kB 00:00 2024-04-14T23:46:15.510 INFO:teuthology.orchestra.run.smithi186.stdout:(58/174): flexiblas-openblas-openmp-3.0.4-8.el9 311 kB/s | 18 kB 00:00 2024-04-14T23:46:15.535 INFO:teuthology.orchestra.run.smithi186.stdout:(59/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 160 kB 00:00 2024-04-14T23:46:15.560 INFO:teuthology.orchestra.run.smithi186.stdout:(60/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 902 kB/s | 45 kB 00:00 2024-04-14T23:46:15.602 INFO:teuthology.orchestra.run.smithi186.stdout:(61/174): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 662 kB 00:00 2024-04-14T23:46:15.653 INFO:teuthology.orchestra.run.smithi186.stdout:(62/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 13 MB/s | 3.0 MB 00:00 2024-04-14T23:46:15.678 INFO:teuthology.orchestra.run.smithi186.stdout:(63/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 338 kB/s | 25 kB 00:00 2024-04-14T23:46:15.703 INFO:teuthology.orchestra.run.smithi186.stdout:(64/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 944 kB/s | 47 kB 00:00 2024-04-14T23:46:15.729 INFO:teuthology.orchestra.run.smithi186.stdout:(65/174): libtool-2.4.6-45.el9.x86_64.rpm 3.4 MB/s | 582 kB 00:00 2024-04-14T23:46:15.754 INFO:teuthology.orchestra.run.smithi186.stdout:(66/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.8 MB/s | 292 kB 00:00 2024-04-14T23:46:15.779 INFO:teuthology.orchestra.run.smithi186.stdout:(67/174): openblas-0.3.21-2.el9.x86_64.rpm 703 kB/s | 35 kB 00:00 2024-04-14T23:46:15.804 INFO:teuthology.orchestra.run.smithi186.stdout:(68/174): lua-5.4.4-4.el9.x86_64.rpm 1.8 MB/s | 188 kB 00:00 2024-04-14T23:46:15.863 INFO:teuthology.orchestra.run.smithi186.stdout:(69/174): postgresql-13.14-1.el9.x86_64.rpm 27 MB/s | 1.6 MB 00:00 2024-04-14T23:46:15.930 INFO:teuthology.orchestra.run.smithi186.stdout:(70/174): openblas-openmp-0.3.21-2.el9.x86_64.r 27 MB/s | 4.7 MB 00:00 2024-04-14T23:46:15.963 INFO:teuthology.orchestra.run.smithi186.stdout:(71/174): postgresql-contrib-13.14-1.el9.x86_64 8.9 MB/s | 911 kB 00:00 2024-04-14T23:46:15.991 INFO:teuthology.orchestra.run.smithi186.stdout:(72/174): postgresql-private-libs-13.14-1.el9.x 2.2 MB/s | 133 kB 00:00 2024-04-14T23:46:16.058 INFO:teuthology.orchestra.run.smithi186.stdout:(73/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 15 MB/s | 4.1 MB 00:00 2024-04-14T23:46:16.100 INFO:teuthology.orchestra.run.smithi186.stdout:(74/174): protobuf-3.14.0-13.el9.x86_64.rpm 9.3 MB/s | 1.0 MB 00:00 2024-04-14T23:46:16.125 INFO:teuthology.orchestra.run.smithi186.stdout:(75/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 87 kB 00:00 2024-04-14T23:46:16.217 INFO:teuthology.orchestra.run.smithi186.stdout:(76/174): postgresql-server-13.14-1.el9.x86_64. 23 MB/s | 5.8 MB 00:00 2024-04-14T23:46:16.242 INFO:teuthology.orchestra.run.smithi186.stdout:(77/174): python3-devel-3.9.17-2.el9.x86_64.rpm 2.0 MB/s | 245 kB 00:00 2024-04-14T23:46:16.268 INFO:teuthology.orchestra.run.smithi186.stdout:(78/174): python3-jinja2-2.11.3-5.el9.noarch.rp 4.8 MB/s | 248 kB 00:00 2024-04-14T23:46:16.293 INFO:teuthology.orchestra.run.smithi186.stdout:(79/174): python3-jsonpatch-1.21-16.el9.noarch. 510 kB/s | 26 kB 00:00 2024-04-14T23:46:16.302 INFO:teuthology.orchestra.run.smithi060.stdout:(130/174): python3-grpcio-1.46.7-10.el9.x86_64. 803 kB/s | 2.0 MB 00:02 2024-04-14T23:46:16.318 INFO:teuthology.orchestra.run.smithi186.stdout:(80/174): python3-jsonpointer-2.0-4.el9.noarch. 381 kB/s | 19 kB 00:00 2024-04-14T23:46:16.344 INFO:teuthology.orchestra.run.smithi186.stdout:(81/174): python3-mako-1.1.4-6.el9.noarch.rpm 3.3 MB/s | 172 kB 00:00 2024-04-14T23:46:16.369 INFO:teuthology.orchestra.run.smithi186.stdout:(82/174): python3-markupsafe-1.1.1-12.el9.x86_6 695 kB/s | 35 kB 00:00 2024-04-14T23:46:16.411 INFO:teuthology.orchestra.run.smithi060.stdout:(131/174): python3-kubernetes-26.1.0-2.el9.noar 770 kB/s | 1.0 MB 00:01 2024-04-14T23:46:16.436 INFO:teuthology.orchestra.run.smithi060.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 349 kB/s | 46 kB 00:00 2024-04-14T23:46:16.452 INFO:teuthology.orchestra.run.smithi186.stdout:(83/174): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 6.0 MB 00:00 2024-04-14T23:46:16.478 INFO:teuthology.orchestra.run.smithi186.stdout:(84/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.3 MB/s | 250 kB 00:00 2024-04-14T23:46:16.503 INFO:teuthology.orchestra.run.smithi186.stdout:(85/174): python3-oauthlib-3.1.1-5.el9.noarch.r 4.3 MB/s | 222 kB 00:00 2024-04-14T23:46:16.528 INFO:teuthology.orchestra.run.smithi186.stdout:(86/174): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-04-14T23:46:16.553 INFO:teuthology.orchestra.run.smithi186.stdout:(87/174): python3-prettytable-0.7.2-27.el9.noar 841 kB/s | 42 kB 00:00 2024-04-14T23:46:16.579 INFO:teuthology.orchestra.run.smithi186.stdout:(88/174): python3-protobuf-3.14.0-13.el9.noarch 5.3 MB/s | 269 kB 00:00 2024-04-14T23:46:16.604 INFO:teuthology.orchestra.run.smithi186.stdout:(89/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 159 kB 00:00 2024-04-14T23:46:16.629 INFO:teuthology.orchestra.run.smithi186.stdout:(90/174): python3-pyasn1-modules-0.4.8-6.el9.no 5.4 MB/s | 279 kB 00:00 2024-04-14T23:46:16.637 INFO:teuthology.orchestra.run.smithi060.stdout:(133/174): libarrow-9.0.0-10.el9.x86_64.rpm 832 kB/s | 4.4 MB 00:05 2024-04-14T23:46:16.653 INFO:teuthology.orchestra.run.smithi060.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 267 kB/s | 58 kB 00:00 2024-04-14T23:46:16.654 INFO:teuthology.orchestra.run.smithi186.stdout:(91/174): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-04-14T23:46:16.679 INFO:teuthology.orchestra.run.smithi060.stdout:(135/174): python3-more-itertools-8.12.0-2.el9. 295 kB/s | 79 kB 00:00 2024-04-14T23:46:16.680 INFO:teuthology.orchestra.run.smithi186.stdout:(92/174): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-04-14T23:46:16.754 INFO:teuthology.orchestra.run.smithi060.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 165 kB/s | 16 kB 00:00 2024-04-14T23:46:16.755 INFO:teuthology.orchestra.run.smithi186.stdout:(93/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 12 MB/s | 5.1 MB 00:00 2024-04-14T23:46:16.781 INFO:teuthology.orchestra.run.smithi186.stdout:(94/174): python3-toml-0.10.2-6.el9.noarch.rpm 416 kB/s | 42 kB 00:00 2024-04-14T23:46:16.806 INFO:teuthology.orchestra.run.smithi186.stdout:(95/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 884 kB/s | 44 kB 00:00 2024-04-14T23:46:16.831 INFO:teuthology.orchestra.run.smithi186.stdout:(96/174): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-04-14T23:46:16.854 INFO:teuthology.orchestra.run.smithi060.stdout:(137/174): python3-pyperclip-1.8.2-2.el9.noarch 284 kB/s | 28 kB 00:00 2024-04-14T23:46:16.857 INFO:teuthology.orchestra.run.smithi186.stdout:(97/174): qatlib-service-24.02.0-1.el9.x86_64.r 715 kB/s | 36 kB 00:00 2024-04-14T23:46:16.879 INFO:teuthology.orchestra.run.smithi060.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 450 kB/s | 90 kB 00:00 2024-04-14T23:46:16.881 INFO:teuthology.orchestra.run.smithi186.stdout:(98/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 949 kB/s | 47 kB 00:00 2024-04-14T23:46:16.910 INFO:teuthology.orchestra.run.smithi186.stdout:(99/174): socat-1.7.4.1-5.el9.x86_64.rpm 6.0 MB/s | 305 kB 00:00 2024-04-14T23:46:16.932 INFO:teuthology.orchestra.run.smithi186.stdout:(100/174): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-04-14T23:46:16.963 INFO:teuthology.orchestra.run.smithi060.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 285 kB/s | 31 kB 00:00 2024-04-14T23:46:16.974 INFO:teuthology.orchestra.run.smithi186.stdout:(101/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.4 MB/s | 94 kB 00:00 2024-04-14T23:46:17.000 INFO:teuthology.orchestra.run.smithi186.stdout:(102/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 954 kB/s | 64 kB 00:00 2024-04-14T23:46:17.025 INFO:teuthology.orchestra.run.smithi186.stdout:(103/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 901 kB/s | 45 kB 00:00 2024-04-14T23:46:17.050 INFO:teuthology.orchestra.run.smithi186.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 445 kB/s | 22 kB 00:00 2024-04-14T23:46:17.072 INFO:teuthology.orchestra.run.smithi060.stdout:(140/174): python3-pecan-1.4.2-3.el9.noarch.rpm 627 kB/s | 272 kB 00:00 2024-04-14T23:46:17.130 INFO:teuthology.orchestra.run.smithi060.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 357 kB/s | 59 kB 00:00 2024-04-14T23:46:17.164 INFO:teuthology.orchestra.run.smithi060.stdout:(142/174): python3-routes-2.5.1-5.el9.noarch.rp 664 kB/s | 188 kB 00:00 2024-04-14T23:46:17.189 INFO:teuthology.orchestra.run.smithi060.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 307 kB/s | 36 kB 00:00 2024-04-14T23:46:17.240 INFO:teuthology.orchestra.run.smithi060.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 468 kB/s | 51 kB 00:00 2024-04-14T23:46:17.300 INFO:teuthology.orchestra.run.smithi186.stdout:(105/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 25 MB/s | 16 MB 00:00 2024-04-14T23:46:17.342 INFO:teuthology.orchestra.run.smithi186.stdout:(106/174): protobuf-compiler-3.14.0-13.el9.x86_ 2.7 MB/s | 863 kB 00:00 2024-04-14T23:46:17.382 INFO:teuthology.orchestra.run.smithi060.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 469 kB/s | 90 kB 00:00 2024-04-14T23:46:17.432 INFO:teuthology.orchestra.run.smithi060.stdout:(146/174): python3-werkzeug-2.0.3-3.el9.1.noarc 2.2 MB/s | 427 kB 00:00 2024-04-14T23:46:17.434 INFO:teuthology.orchestra.run.smithi186.stdout:(107/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.4 MB/s | 548 kB 00:00 2024-04-14T23:46:17.460 INFO:teuthology.orchestra.run.smithi186.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 166 kB/s | 19 kB 00:00 2024-04-14T23:46:17.491 INFO:teuthology.orchestra.run.smithi060.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 185 kB/s | 20 kB 00:00 2024-04-14T23:46:17.541 INFO:teuthology.orchestra.run.smithi060.stdout:(148/174): python3-webob-1.8.7-6.el9.noarch.rpm 611 kB/s | 230 kB 00:00 2024-04-14T23:46:17.552 INFO:teuthology.orchestra.run.smithi186.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 283 kB/s | 26 kB 00:00 2024-04-14T23:46:17.574 INFO:teuthology.orchestra.run.smithi060.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 1.3 MB/s | 192 kB 00:00 2024-04-14T23:46:17.710 INFO:teuthology.orchestra.run.smithi186.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 307 kB/s | 49 kB 00:00 2024-04-14T23:46:17.777 INFO:teuthology.orchestra.run.smithi186.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-04-14T23:46:17.919 INFO:teuthology.orchestra.run.smithi186.stdout:(112/174): luarocks-3.9.2-1.el9.noarch.rpm 1.0 MB/s | 151 kB 00:00 2024-04-14T23:46:17.969 INFO:teuthology.orchestra.run.smithi186.stdout:(113/174): gperftools-libs-2.9.1-2.el9.x86_64.r 462 kB/s | 309 kB 00:00 2024-04-14T23:46:18.036 INFO:teuthology.orchestra.run.smithi186.stdout:(114/174): libarrow-9.0.0-10.el9.x86_64.rpm 7.3 MB/s | 4.4 MB 00:00 2024-04-14T23:46:18.195 INFO:teuthology.orchestra.run.smithi186.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.0 MB/s | 838 kB 00:00 2024-04-14T23:46:18.220 INFO:teuthology.orchestra.run.smithi186.stdout:(116/174): python3-asyncssh-2.13.2-5.el9.noarch 2.1 MB/s | 548 kB 00:00 2024-04-14T23:46:18.246 INFO:teuthology.orchestra.run.smithi186.stdout:(117/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 208 kB/s | 43 kB 00:00 2024-04-14T23:46:18.271 INFO:teuthology.orchestra.run.smithi186.stdout:(118/174): python3-cachetools-4.2.4-1.el9.noarc 427 kB/s | 32 kB 00:00 2024-04-14T23:46:18.305 INFO:teuthology.orchestra.run.smithi186.stdout:(119/174): python3-certifi-2021.10.8-2.el9.noar 179 kB/s | 15 kB 00:00 2024-04-14T23:46:18.372 INFO:teuthology.orchestra.run.smithi186.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.3 MB/s | 172 kB 00:00 2024-04-14T23:46:18.480 INFO:teuthology.orchestra.run.smithi186.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.7 MB/s | 303 kB 00:00 2024-04-14T23:46:18.530 INFO:teuthology.orchestra.run.smithi186.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 222 kB/s | 35 kB 00:00 2024-04-14T23:46:18.556 INFO:teuthology.orchestra.run.smithi186.stdout:(123/174): python3-cherrypy-18.6.1-2.el9.noarch 1.2 MB/s | 358 kB 00:00 2024-04-14T23:46:18.656 INFO:teuthology.orchestra.run.smithi186.stdout:(124/174): python3-google-auth-2.28.2-1.el9.noa 1.1 MB/s | 203 kB 00:00 2024-04-14T23:46:18.681 INFO:teuthology.orchestra.run.smithi186.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 1.1 MB/s | 144 kB 00:00 2024-04-14T23:46:18.748 INFO:teuthology.orchestra.run.smithi186.stdout:(126/174): python3-jaraco-classes-3.2.1-5.el9.n 266 kB/s | 18 kB 00:00 2024-04-14T23:46:18.849 INFO:teuthology.orchestra.run.smithi186.stdout:(127/174): python3-grpcio-1.46.7-10.el9.x86_64. 6.4 MB/s | 2.0 MB 00:00 2024-04-14T23:46:18.874 INFO:teuthology.orchestra.run.smithi186.stdout:(128/174): python3-jaraco-8.2.1-3.el9.noarch.rp 49 kB/s | 11 kB 00:00 2024-04-14T23:46:18.899 INFO:teuthology.orchestra.run.smithi186.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 154 kB/s | 23 kB 00:00 2024-04-14T23:46:18.949 INFO:teuthology.orchestra.run.smithi186.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 193 kB/s | 19 kB 00:00 2024-04-14T23:46:19.000 INFO:teuthology.orchestra.run.smithi186.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 156 kB/s | 20 kB 00:00 2024-04-14T23:46:19.042 INFO:teuthology.orchestra.run.smithi060.stdout:(150/174): librbd1-19.0.0-2114.ga9a752df.el7.x8 2.1 MB/s | 3.1 MB 00:01 2024-04-14T23:46:19.150 INFO:teuthology.orchestra.run.smithi186.stdout:(132/174): python3-kubernetes-26.1.0-2.el9.noar 4.1 MB/s | 1.0 MB 00:00 2024-04-14T23:46:19.175 INFO:teuthology.orchestra.run.smithi186.stdout:(133/174): python3-logutils-0.3.5-21.el9.noarch 205 kB/s | 46 kB 00:00 2024-04-14T23:46:19.201 INFO:teuthology.orchestra.run.smithi186.stdout:(134/174): python3-more-itertools-8.12.0-2.el9. 392 kB/s | 79 kB 00:00 2024-04-14T23:46:19.226 INFO:teuthology.orchestra.run.smithi186.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 761 kB/s | 58 kB 00:00 2024-04-14T23:46:19.350 INFO:teuthology.orchestra.run.smithi060.stdout:(151/174): librados2-19.0.0-2114.ga9a752df.el7. 1.9 MB/s | 3.4 MB 00:01 2024-04-14T23:46:19.385 INFO:teuthology.orchestra.run.smithi186.stdout:(136/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 567 kB/s | 90 kB 00:00 2024-04-14T23:46:19.477 INFO:teuthology.orchestra.run.smithi186.stdout:(137/174): python3-pyperclip-1.8.2-2.el9.noarch 308 kB/s | 28 kB 00:00 2024-04-14T23:46:19.510 INFO:teuthology.orchestra.run.smithi186.stdout:(138/174): python3-pecan-1.4.2-3.el9.noarch.rpm 813 kB/s | 272 kB 00:00 2024-04-14T23:46:19.536 INFO:teuthology.orchestra.run.smithi186.stdout:(139/174): python3-portend-3.1.0-2.el9.noarch.r 49 kB/s | 16 kB 00:00 2024-04-14T23:46:19.567 INFO:teuthology.orchestra.run.smithi060.stdout:(152/174): thrift-0.15.0-2.el9.x86_64.rpm 780 kB/s | 1.6 MB 00:02 2024-04-14T23:46:19.653 INFO:teuthology.orchestra.run.smithi186.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 175 kB/s | 31 kB 00:00 2024-04-14T23:46:19.678 INFO:teuthology.orchestra.run.smithi186.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 1.1 MB/s | 188 kB 00:00 2024-04-14T23:46:19.718 INFO:teuthology.orchestra.run.smithi060.stdout:(153/174): NetworkManager-team-1.47.4-1.el9.x86 280 kB/s | 42 kB 00:00 2024-04-14T23:46:19.761 INFO:teuthology.orchestra.run.smithi186.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 262 kB/s | 59 kB 00:00 2024-04-14T23:46:19.820 INFO:teuthology.orchestra.run.smithi186.stdout:(143/174): python3-typing-extensions-4.4.0-2.el 359 kB/s | 51 kB 00:00 2024-04-14T23:46:19.845 INFO:teuthology.orchestra.run.smithi186.stdout:(144/174): python3-tempora-5.0.0-2.el9.noarch.r 186 kB/s | 36 kB 00:00 2024-04-14T23:46:19.901 INFO:teuthology.orchestra.run.smithi060.stdout:(154/174): NetworkManager-tui-1.47.4-1.el9.x86_ 1.3 MB/s | 249 kB 00:00 2024-04-14T23:46:19.945 INFO:teuthology.orchestra.run.smithi186.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 1.2 MB/s | 230 kB 00:00 2024-04-14T23:46:19.968 INFO:teuthology.orchestra.run.smithi060.stdout:(155/174): acl-2.3.1-4.el9.x86_64.rpm 1.1 MB/s | 71 kB 00:00 2024-04-14T23:46:19.970 INFO:teuthology.orchestra.run.smithi186.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 596 kB/s | 90 kB 00:00 2024-04-14T23:46:20.035 INFO:teuthology.orchestra.run.smithi060.stdout:(156/174): elfutils-debuginfod-client-0.190-2.e 556 kB/s | 36 kB 00:00 2024-04-14T23:46:20.038 INFO:teuthology.orchestra.run.smithi186.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 217 kB/s | 20 kB 00:00 2024-04-14T23:46:20.094 INFO:teuthology.orchestra.run.smithi060.stdout:(157/174): NetworkManager-1.47.4-1.el9.x86_64.r 2.2 MB/s | 2.3 MB 00:01 2024-04-14T23:46:20.138 INFO:teuthology.orchestra.run.smithi186.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 1.1 MB/s | 192 kB 00:00 2024-04-14T23:46:20.152 INFO:teuthology.orchestra.run.smithi060.stdout:(158/174): NetworkManager-libnm-1.47.4-1.el9.x8 2.3 MB/s | 1.8 MB 00:00 2024-04-14T23:46:20.163 INFO:teuthology.orchestra.run.smithi186.stdout:(149/174): python3-werkzeug-2.0.3-3.el9.1.noarc 1.3 MB/s | 427 kB 00:00 2024-04-14T23:46:20.219 INFO:teuthology.orchestra.run.smithi060.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 1.0 MB/s | 192 kB 00:00 2024-04-14T23:46:20.244 INFO:teuthology.orchestra.run.smithi060.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 254 kB/s | 23 kB 00:00 2024-04-14T23:46:20.270 INFO:teuthology.orchestra.run.smithi060.stdout:(161/174): elfutils-libs-0.190-2.el9.x86_64.rpm 1.4 MB/s | 256 kB 00:00 2024-04-14T23:46:20.288 INFO:teuthology.orchestra.run.smithi186.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 6.3 MB/s | 1.6 MB 00:00 2024-04-14T23:46:20.295 INFO:teuthology.orchestra.run.smithi060.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.4 MB/s | 108 kB 00:00 2024-04-14T23:46:20.345 INFO:teuthology.orchestra.run.smithi060.stdout:(163/174): libmount-2.37.4-18.el9.x86_64.rpm 1.8 MB/s | 136 kB 00:00 2024-04-14T23:46:20.371 INFO:teuthology.orchestra.run.smithi060.stdout:(164/174): libsmartcols-2.37.4-18.el9.x86_64.rp 860 kB/s | 64 kB 00:00 2024-04-14T23:46:20.396 INFO:teuthology.orchestra.run.smithi060.stdout:(165/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.0 MB/s | 155 kB 00:00 2024-04-14T23:46:20.421 INFO:teuthology.orchestra.run.smithi060.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 382 kB/s | 29 kB 00:00 2024-04-14T23:46:20.639 INFO:teuthology.orchestra.run.smithi186.stdout:(151/174): librados2-19.0.0-2114.ga9a752df.el7. 6.8 MB/s | 3.4 MB 00:00 2024-04-14T23:46:20.713 INFO:teuthology.orchestra.run.smithi060.stdout:(167/174): openssl-3.0.7-27.el9.x86_64.rpm 3.5 MB/s | 1.2 MB 00:00 2024-04-14T23:46:20.747 INFO:teuthology.orchestra.run.smithi186.stdout:(152/174): NetworkManager-1.47.4-1.el9.x86_64.r 5.0 MB/s | 2.3 MB 00:00 2024-04-14T23:46:20.814 INFO:teuthology.orchestra.run.smithi186.stdout:(153/174): librbd1-19.0.0-2114.ga9a752df.el7.x8 4.8 MB/s | 3.1 MB 00:00 2024-04-14T23:46:20.855 INFO:teuthology.orchestra.run.smithi060.stdout:(168/174): util-linux-core-2.37.4-18.el9.x86_64 3.2 MB/s | 465 kB 00:00 2024-04-14T23:46:20.923 INFO:teuthology.orchestra.run.smithi186.stdout:(154/174): NetworkManager-team-1.47.4-1.el9.x86 238 kB/s | 42 kB 00:00 2024-04-14T23:46:20.940 INFO:teuthology.orchestra.run.smithi060.stdout:(169/174): util-linux-user-2.37.4-18.el9.x86_64 376 kB/s | 31 kB 00:00 2024-04-14T23:46:20.948 INFO:teuthology.orchestra.run.smithi186.stdout:(155/174): NetworkManager-tui-1.47.4-1.el9.x86_ 1.8 MB/s | 249 kB 00:00 2024-04-14T23:46:20.990 INFO:teuthology.orchestra.run.smithi186.stdout:(156/174): NetworkManager-libnm-1.47.4-1.el9.x8 5.2 MB/s | 1.8 MB 00:00 2024-04-14T23:46:21.006 INFO:teuthology.orchestra.run.smithi060.stdout:(170/174): util-linux-2.37.4-18.el9.x86_64.rpm 3.9 MB/s | 2.3 MB 00:00 2024-04-14T23:46:21.007 INFO:teuthology.orchestra.run.smithi186.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 623 kB/s | 36 kB 00:00 2024-04-14T23:46:21.032 INFO:teuthology.orchestra.run.smithi186.stdout:(158/174): acl-2.3.1-4.el9.x86_64.rpm 654 kB/s | 71 kB 00:00 2024-04-14T23:46:21.058 INFO:teuthology.orchestra.run.smithi186.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.8 MB/s | 192 kB 00:00 2024-04-14T23:46:21.074 INFO:teuthology.orchestra.run.smithi186.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 549 kB/s | 23 kB 00:00 2024-04-14T23:46:21.081 INFO:teuthology.orchestra.run.smithi060.stdout:(171/174): xfsdump-3.1.12-4.el9.x86_64.rpm 2.2 MB/s | 323 kB 00:00 2024-04-14T23:46:21.108 INFO:teuthology.orchestra.run.smithi186.stdout:(161/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.5 MB/s | 256 kB 00:00 2024-04-14T23:46:21.259 INFO:teuthology.orchestra.run.smithi186.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 540 kB/s | 108 kB 00:00 2024-04-14T23:46:21.282 INFO:teuthology.orchestra.run.smithi060.stdout:(172/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 3.9 MB/s | 1.1 MB 00:00 2024-04-14T23:46:21.286 INFO:teuthology.orchestra.run.smithi186.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 733 kB/s | 155 kB 00:00 2024-04-14T23:46:21.311 INFO:teuthology.orchestra.run.smithi186.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 669 kB/s | 136 kB 00:00 2024-04-14T23:46:21.336 INFO:teuthology.orchestra.run.smithi186.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 822 kB/s | 64 kB 00:00 2024-04-14T23:46:21.362 INFO:teuthology.orchestra.run.smithi186.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 380 kB/s | 29 kB 00:00 2024-04-14T23:46:21.434 INFO:teuthology.orchestra.run.smithi060.stdout:(173/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 2.1 MB/s | 2.2 MB 00:01 2024-04-14T23:46:21.507 INFO:teuthology.orchestra.run.smithi060.stdout:(174/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 175 kB/s | 74 kB 00:00 2024-04-14T23:46:21.518 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-04-14T23:46:21.518 INFO:teuthology.orchestra.run.smithi060.stdout:Total 6.8 MB/s | 228 MB 00:33 2024-04-14T23:46:21.546 INFO:teuthology.orchestra.run.smithi186.stdout:(167/174): openssl-3.0.7-27.el9.x86_64.rpm 5.2 MB/s | 1.2 MB 00:00 2024-04-14T23:46:21.646 INFO:teuthology.orchestra.run.smithi186.stdout:(168/174): util-linux-core-2.37.4-18.el9.x86_64 4.5 MB/s | 465 kB 00:00 2024-04-14T23:46:21.688 INFO:teuthology.orchestra.run.smithi186.stdout:(169/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 6.1 MB/s | 2.2 MB 00:00 2024-04-14T23:46:21.713 INFO:teuthology.orchestra.run.smithi186.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 466 kB/s | 31 kB 00:00 2024-04-14T23:46:21.772 INFO:teuthology.orchestra.run.smithi186.stdout:(171/174): xfsdump-3.1.12-4.el9.x86_64.rpm 3.8 MB/s | 323 kB 00:00 2024-04-14T23:46:21.847 INFO:teuthology.orchestra.run.smithi186.stdout:(172/174): util-linux-2.37.4-18.el9.x86_64.rpm 4.7 MB/s | 2.3 MB 00:00 2024-04-14T23:46:21.881 INFO:teuthology.orchestra.run.smithi186.stdout:(173/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 6.4 MB/s | 1.1 MB 00:00 2024-04-14T23:46:21.906 INFO:teuthology.orchestra.run.smithi186.stdout:(174/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 556 kB/s | 74 kB 00:00 2024-04-14T23:46:21.908 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-04-14T23:46:21.910 INFO:teuthology.orchestra.run.smithi186.stdout:Total 15 MB/s | 228 MB 00:14 2024-04-14T23:46:23.302 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-04-14T23:46:23.581 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-04-14T23:46:23.581 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-04-14T23:46:25.450 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-04-14T23:46:25.451 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-04-14T23:46:25.473 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-04-14T23:46:25.885 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-04-14T23:46:25.885 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-04-14T23:46:28.618 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-04-14T23:46:28.618 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-04-14T23:46:32.519 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-04-14T23:46:32.643 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-04-14T23:46:32.677 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-04-14T23:46:32.717 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-04-14T23:46:32.727 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-04-14T23:46:32.774 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-04-14T23:46:32.828 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-04-14T23:46:32.862 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-04-14T23:46:32.988 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-04-14T23:46:33.090 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-04-14T23:46:33.424 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-04-14T23:46:33.480 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : librados2-2:19.0.0-2114.ga9a752df.el7.x86_64 10/198 2024-04-14T23:46:33.585 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librados2-2:19.0.0-2114.ga9a752df.el7.x86_64 10/198 2024-04-14T23:46:33.623 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libcephfs2-2:19.0.0-2114.ga9a752df.el7.x86_64 11/198 2024-04-14T23:46:33.698 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libcephfs2-2:19.0.0-2114.ga9a752df.el7.x86_64 11/198 2024-04-14T23:46:33.727 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rados-2:19.0.0-2114.ga9a752df.el7.x86_64 12/198 2024-04-14T23:46:33.769 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-04-14T23:46:33.944 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-04-14T23:46:34.001 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-04-14T23:46:34.037 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libcephsqlite-2:19.0.0-2114.ga9a752df.el7.x86_64 16/198 2024-04-14T23:46:34.141 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2114.ga9a752df.el7.x86_64 16/198 2024-04-14T23:46:34.171 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libradosstriper1-2:19.0.0-2114.ga9a752df.el7.x86 17/198 2024-04-14T23:46:34.218 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2114.ga9a752df.el7.x86 17/198 2024-04-14T23:46:34.283 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-04-14T23:46:34.306 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-04-14T23:46:34.322 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-04-14T23:46:34.481 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-04-14T23:46:34.521 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-04-14T23:46:34.560 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-04-14T23:46:34.665 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-04-14T23:46:34.780 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 23/198 2024-04-14T23:46:34.832 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 23/198 2024-04-14T23:46:34.903 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 24/198 2024-04-14T23:46:34.967 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : NetworkManager-1:1.47.4-1.el9.x86_64 24/198 2024-04-14T23:46:35.183 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 24/198 2024-04-14T23:46:35.288 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-04-14T23:46:35.382 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-04-14T23:46:35.413 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-04-14T23:46:35.448 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-04-14T23:46:35.512 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 29/198 2024-04-14T23:46:35.547 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 30/198 2024-04-14T23:46:35.599 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 31/198 2024-04-14T23:46:35.640 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 32/198 2024-04-14T23:46:35.668 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : lua-5.4.4-4.el9.x86_64 33/198 2024-04-14T23:46:35.736 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 34/198 2024-04-14T23:46:35.790 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 35/198 2024-04-14T23:46:35.820 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 36/198 2024-04-14T23:46:35.856 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 37/198 2024-04-14T23:46:35.931 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 38/198 2024-04-14T23:46:35.967 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-ceph-common-2:19.0.0-2114.ga9a752df.el7. 39/198 2024-04-14T23:46:36.010 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-ceph-argparse-2:19.0.0-2114.ga9a752df.el 40/198 2024-04-14T23:46:36.047 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cephfs-2:19.0.0-2114.ga9a752df.el7.x86_6 41/198 2024-04-14T23:46:36.115 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-04-14T23:46:36.170 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-04-14T23:46:36.243 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-04-14T23:46:36.279 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-04-14T23:46:36.379 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-04-14T23:46:36.393 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-04-14T23:46:36.431 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-04-14T23:46:36.441 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-04-14T23:46:36.480 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-04-14T23:46:36.526 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-04-14T23:46:36.624 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-04-14T23:46:36.726 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-04-14T23:46:36.789 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-04-14T23:46:36.820 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-04-14T23:46:36.916 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-04-14T23:46:36.965 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-04-14T23:46:37.069 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-04-14T23:46:37.141 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-04-14T23:46:37.188 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-04-14T23:46:37.201 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-04-14T23:46:37.237 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-04-14T23:46:37.286 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-04-14T23:46:37.324 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librados-devel-2:19.0.0-2114.ga9a752df.el7.x86_6 53/198 2024-04-14T23:46:37.342 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-04-14T23:46:37.353 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-04-14T23:46:37.393 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : librados2-2:19.0.0-2114.ga9a752df.el7.x86_64 10/198 2024-04-14T23:46:37.557 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librados2-2:19.0.0-2114.ga9a752df.el7.x86_64 10/198 2024-04-14T23:46:37.586 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libcephfs2-2:19.0.0-2114.ga9a752df.el7.x86_64 11/198 2024-04-14T23:46:37.604 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-04-14T23:46:37.710 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: libcephfs2-2:19.0.0-2114.ga9a752df.el7.x86_64 11/198 2024-04-14T23:46:37.743 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-rados-2:19.0.0-2114.ga9a752df.el7.x86_64 12/198 2024-04-14T23:46:37.801 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-04-14T23:46:37.815 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-04-14T23:46:37.876 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-04-14T23:46:37.921 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-04-14T23:46:38.064 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-04-14T23:46:38.107 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-04-14T23:46:38.152 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-04-14T23:46:38.162 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-04-14T23:46:38.216 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-04-14T23:46:38.234 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-04-14T23:46:38.241 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libcephsqlite-2:19.0.0-2114.ga9a752df.el7.x86_64 16/198 2024-04-14T23:46:38.266 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-04-14T23:46:38.302 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-04-14T23:46:38.349 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-04-14T23:46:38.394 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-04-14T23:46:38.403 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2114.ga9a752df.el7.x86_64 16/198 2024-04-14T23:46:38.433 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libradosstriper1-2:19.0.0-2114.ga9a752df.el7.x86 17/198 2024-04-14T23:46:38.436 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-04-14T23:46:38.479 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-04-14T23:46:38.559 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2114.ga9a752df.el7.x86 17/198 2024-04-14T23:46:38.612 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-04-14T23:46:38.614 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-04-14T23:46:38.644 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-04-14T23:46:38.660 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-04-14T23:46:38.690 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-04-14T23:46:38.709 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-04-14T23:46:38.731 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-04-14T23:46:38.826 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-04-14T23:46:38.865 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-04-14T23:46:38.913 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-04-14T23:46:38.947 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-04-14T23:46:38.972 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-04-14T23:46:39.011 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-04-14T23:46:39.045 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-04-14T23:46:39.091 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-04-14T23:46:39.136 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-04-14T23:46:39.172 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-04-14T23:46:39.205 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-04-14T23:46:39.232 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 81/198 2024-04-14T23:46:39.267 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 82/198 2024-04-14T23:46:39.306 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : uuid-1.6.2-55.el9.x86_64 83/198 2024-04-14T23:46:39.338 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 84/198 2024-04-14T23:46:39.368 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 85/198 2024-04-14T23:46:39.402 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 86/198 2024-04-14T23:46:39.429 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-04-14T23:46:39.435 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 87/198 2024-04-14T23:46:39.477 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 88/198 2024-04-14T23:46:39.520 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 89/198 2024-04-14T23:46:39.609 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 23/198 2024-04-14T23:46:39.644 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 90/198 2024-04-14T23:46:39.706 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 23/198 2024-04-14T23:46:39.897 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 91/198 2024-04-14T23:46:39.910 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 24/198 2024-04-14T23:46:39.970 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : NetworkManager-1:1.47.4-1.el9.x86_64 24/198 2024-04-14T23:46:40.134 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 92/198 2024-04-14T23:46:40.189 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 93/198 2024-04-14T23:46:40.304 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-packaging-20.9-5.el9.noarch 94/198 2024-04-14T23:46:40.332 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 24/198 2024-04-14T23:46:40.358 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 95/198 2024-04-14T23:46:40.529 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-04-14T23:46:40.654 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-04-14T23:46:40.701 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-04-14T23:46:40.749 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-04-14T23:46:40.849 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 29/198 2024-04-14T23:46:40.897 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 30/198 2024-04-14T23:46:40.961 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 31/198 2024-04-14T23:46:41.015 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 32/198 2024-04-14T23:46:41.050 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : lua-5.4.4-4.el9.x86_64 33/198 2024-04-14T23:46:41.084 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 96/198 2024-04-14T23:46:41.130 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 97/198 2024-04-14T23:46:41.164 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 98/198 2024-04-14T23:46:41.173 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 34/198 2024-04-14T23:46:41.263 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 99/198 2024-04-14T23:46:41.266 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 35/198 2024-04-14T23:46:41.302 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 36/198 2024-04-14T23:46:41.322 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 100/198 2024-04-14T23:46:41.363 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 37/198 2024-04-14T23:46:41.410 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 101/198 2024-04-14T23:46:41.442 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 102/198 2024-04-14T23:46:41.494 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 38/198 2024-04-14T23:46:41.540 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-ceph-common-2:19.0.0-2114.ga9a752df.el7. 39/198 2024-04-14T23:46:41.583 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-ceph-argparse-2:19.0.0-2114.ga9a752df.el 40/198 2024-04-14T23:46:41.619 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : openblas-0.3.21-2.el9.x86_64 103/198 2024-04-14T23:46:41.620 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cephfs-2:19.0.0-2114.ga9a752df.el7.x86_6 41/198 2024-04-14T23:46:41.648 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 104/198 2024-04-14T23:46:41.734 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 105/198 2024-04-14T23:46:41.734 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-04-14T23:46:41.807 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-04-14T23:46:41.903 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-04-14T23:46:42.107 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-04-14T23:46:42.178 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-04-14T23:46:42.371 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 106/198 2024-04-14T23:46:42.486 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 107/198 2024-04-14T23:46:42.895 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-04-14T23:46:43.004 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-04-14T23:46:43.124 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-04-14T23:46:43.166 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-04-14T23:46:43.194 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-04-14T23:46:43.238 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-04-14T23:46:43.306 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-04-14T23:46:43.363 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : librados-devel-2:19.0.0-2114.ga9a752df.el7.x86_6 53/198 2024-04-14T23:46:43.407 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-04-14T23:46:43.714 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 108/198 2024-04-14T23:46:43.794 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 109/198 2024-04-14T23:46:43.821 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 110/198 2024-04-14T23:46:43.834 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-04-14T23:46:44.073 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-04-14T23:46:44.131 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 111/198 2024-04-14T23:46:44.159 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : librbd1-2:19.0.0-2114.ga9a752df.el7.x86_64 112/198 2024-04-14T23:46:44.179 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-04-14T23:46:44.233 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librbd1-2:19.0.0-2114.ga9a752df.el7.x86_64 112/198 2024-04-14T23:46:44.247 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-04-14T23:46:44.264 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rbd-2:19.0.0-2114.ga9a752df.el7.x86_64 113/198 2024-04-14T23:46:44.300 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 114/198 2024-04-14T23:46:44.465 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-04-14T23:46:44.525 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-04-14T23:46:44.615 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-04-14T23:46:44.728 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-04-14T23:46:44.760 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-04-14T23:46:44.790 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librgw2-2:19.0.0-2114.ga9a752df.el7.x86_64 116/198 2024-04-14T23:46:44.858 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librgw2-2:19.0.0-2114.ga9a752df.el7.x86_64 116/198 2024-04-14T23:46:44.889 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rgw-2:19.0.0-2114.ga9a752df.el7.x86_64 117/198 2024-04-14T23:46:45.282 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-04-14T23:46:45.329 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-04-14T23:46:45.420 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-04-14T23:46:45.500 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-04-14T23:46:45.582 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-04-14T23:46:45.654 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-04-14T23:46:45.867 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-04-14T23:46:45.929 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-04-14T23:46:45.966 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-04-14T23:46:46.028 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-04-14T23:46:46.192 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-04-14T23:46:46.231 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-04-14T23:46:46.280 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-04-14T23:46:46.314 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-04-14T23:46:46.347 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-04-14T23:46:46.528 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-04-14T23:46:46.710 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-04-14T23:46:46.753 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-04-14T23:46:46.790 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 81/198 2024-04-14T23:46:46.825 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 82/198 2024-04-14T23:46:46.874 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : uuid-1.6.2-55.el9.x86_64 83/198 2024-04-14T23:46:46.922 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 84/198 2024-04-14T23:46:46.973 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 85/198 2024-04-14T23:46:47.019 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 86/198 2024-04-14T23:46:47.079 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 87/198 2024-04-14T23:46:47.093 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-common-2:19.0.0-2114.ga9a752df.el7.x86_64 118/198 2024-04-14T23:46:47.123 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 88/198 2024-04-14T23:46:47.127 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-common-2:19.0.0-2114.ga9a752df.el7.x86_64 118/198 2024-04-14T23:46:47.165 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 89/198 2024-04-14T23:46:47.332 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 90/198 2024-04-14T23:46:47.391 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 91/198 2024-04-14T23:46:47.431 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 92/198 2024-04-14T23:46:47.485 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 93/198 2024-04-14T23:46:47.643 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-packaging-20.9-5.el9.noarch 94/198 2024-04-14T23:46:47.665 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-common-2:19.0.0-2114.ga9a752df.el7.x86_64 118/198 2024-04-14T23:46:47.698 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-base-2:19.0.0-2114.ga9a752df.el7.x86_64 119/198 2024-04-14T23:46:47.714 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 95/198 2024-04-14T23:46:47.762 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-base-2:19.0.0-2114.ga9a752df.el7.x86_64 119/198 2024-04-14T23:46:47.763 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-14T23:46:47.763 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-14T23:46:47.763 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-14T23:46:47.790 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-selinux-2:19.0.0-2114.ga9a752df.el7.x86_64 120/198 2024-04-14T23:46:48.722 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 96/198 2024-04-14T23:46:48.779 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 97/198 2024-04-14T23:46:48.813 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 98/198 2024-04-14T23:46:48.947 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 99/198 2024-04-14T23:46:49.017 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 100/198 2024-04-14T23:46:49.127 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 101/198 2024-04-14T23:46:49.164 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 102/198 2024-04-14T23:46:49.405 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : openblas-0.3.21-2.el9.x86_64 103/198 2024-04-14T23:46:49.435 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 104/198 2024-04-14T23:46:49.533 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 105/198 2024-04-14T23:46:50.405 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 106/198 2024-04-14T23:46:50.547 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 107/198 2024-04-14T23:46:52.188 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 108/198 2024-04-14T23:46:52.275 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 109/198 2024-04-14T23:46:52.309 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 110/198 2024-04-14T23:46:52.683 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 111/198 2024-04-14T23:46:52.712 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : librbd1-2:19.0.0-2114.ga9a752df.el7.x86_64 112/198 2024-04-14T23:46:52.833 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librbd1-2:19.0.0-2114.ga9a752df.el7.x86_64 112/198 2024-04-14T23:46:52.877 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-rbd-2:19.0.0-2114.ga9a752df.el7.x86_64 113/198 2024-04-14T23:46:52.932 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 114/198 2024-04-14T23:46:53.517 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-04-14T23:46:53.556 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : librgw2-2:19.0.0-2114.ga9a752df.el7.x86_64 116/198 2024-04-14T23:46:53.657 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librgw2-2:19.0.0-2114.ga9a752df.el7.x86_64 116/198 2024-04-14T23:46:53.683 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-rgw-2:19.0.0-2114.ga9a752df.el7.x86_64 117/198 2024-04-14T23:46:56.417 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-common-2:19.0.0-2114.ga9a752df.el7.x86_64 118/198 2024-04-14T23:46:56.464 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-common-2:19.0.0-2114.ga9a752df.el7.x86_64 118/198 2024-04-14T23:46:57.148 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-common-2:19.0.0-2114.ga9a752df.el7.x86_64 118/198 2024-04-14T23:46:57.193 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-base-2:19.0.0-2114.ga9a752df.el7.x86_64 119/198 2024-04-14T23:46:57.348 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-base-2:19.0.0-2114.ga9a752df.el7.x86_64 119/198 2024-04-14T23:46:57.348 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-14T23:46:57.348 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-14T23:46:57.348 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:46:57.385 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-selinux-2:19.0.0-2114.ga9a752df.el7.x86_64 120/198 2024-04-14T23:46:59.116 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2114.ga9a752df.el7.x86_64 120/198 2024-04-14T23:46:59.117 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /sys 2024-04-14T23:46:59.117 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /proc 2024-04-14T23:46:59.117 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /mnt 2024-04-14T23:46:59.117 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /var/tmp 2024-04-14T23:46:59.117 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /home 2024-04-14T23:46:59.117 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /root 2024-04-14T23:46:59.117 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /tmp 2024-04-14T23:46:59.117 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-14T23:46:59.282 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-volume-2:19.0.0-2114.ga9a752df.el7.noarch 121/198 2024-04-14T23:46:59.298 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-volume-2:19.0.0-2114.ga9a752df.el7.noarch 121/198 2024-04-14T23:46:59.298 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-14T23:46:59.298 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-14T23:46:59.299 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-14T23:47:00.768 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-osd-2:19.0.0-2114.ga9a752df.el7.x86_64 122/198 2024-04-14T23:47:00.794 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-osd-2:19.0.0-2114.ga9a752df.el7.x86_64 122/198 2024-04-14T23:47:00.795 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-14T23:47:00.795 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-14T23:47:00.795 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-14T23:47:00.795 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-14T23:47:00.795 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-14T23:47:01.027 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mds-2:19.0.0-2114.ga9a752df.el7.x86_64 123/198 2024-04-14T23:47:01.050 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mds-2:19.0.0-2114.ga9a752df.el7.x86_64 123/198 2024-04-14T23:47:01.050 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-14T23:47:01.050 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-14T23:47:01.050 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-14T23:47:01.050 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-14T23:47:01.050 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-14T23:47:01.503 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mon-2:19.0.0-2114.ga9a752df.el7.x86_64 124/198 2024-04-14T23:47:01.526 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mon-2:19.0.0-2114.ga9a752df.el7.x86_64 124/198 2024-04-14T23:47:01.526 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-14T23:47:01.526 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-14T23:47:01.526 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-14T23:47:01.526 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-14T23:47:01.526 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-14T23:47:01.650 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : automake-1.16.2-8.el9.noarch 125/198 2024-04-14T23:47:01.707 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-ply-3.11-14.el9.noarch 126/198 2024-04-14T23:47:01.762 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 127/198 2024-04-14T23:47:01.955 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 128/198 2024-04-14T23:47:02.004 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 129/198 2024-04-14T23:47:02.082 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 130/198 2024-04-14T23:47:02.171 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 131/198 2024-04-14T23:47:02.318 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 132/198 2024-04-14T23:47:02.362 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 133/198 2024-04-14T23:47:02.395 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 134/198 2024-04-14T23:47:02.429 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 135/198 2024-04-14T23:47:02.464 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : jq-1.6-16.el9.x86_64 136/198 2024-04-14T23:47:02.492 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : fuse-2.9.9-15.el9.x86_64 137/198 2024-04-14T23:47:02.918 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: cephadm-2:19.0.0-2114.ga9a752df.el7.noarch 138/198 2024-04-14T23:47:02.957 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : cephadm-2:19.0.0-2114.ga9a752df.el7.noarch 138/198 2024-04-14T23:47:02.989 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2114.ga9a752df.e 139/198 2024-04-14T23:47:03.049 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2114.ga9a752df. 140/198 2024-04-14T23:47:03.072 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2114.ga9a752df.el7.noa 141/198 2024-04-14T23:47:04.106 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2114.ga9a752df.el7.noa 141/198 2024-04-14T23:47:04.141 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2114.ga9a752df.el7.n 142/198 2024-04-14T23:47:05.118 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2114.ga9a752df.el7.n 142/198 2024-04-14T23:47:05.157 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2114.ga9a 143/198 2024-04-14T23:47:05.259 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2114.ga9a 143/198 2024-04-14T23:47:05.421 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2114.ga9a752df.el 144/198 2024-04-14T23:47:05.460 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-2:19.0.0-2114.ga9a752df.el7.x86_64 145/198 2024-04-14T23:47:05.483 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2114.ga9a752df.el7.x86_64 145/198 2024-04-14T23:47:05.483 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-14T23:47:05.484 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-14T23:47:05.484 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-14T23:47:05.484 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-14T23:47:05.484 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-14T23:47:05.527 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-rook-2:19.0.0-2114.ga9a752df.el7.noarch 146/198 2024-04-14T23:47:05.539 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2114.ga9a752df.el7.noarch 146/198 2024-04-14T23:47:05.638 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-2:19.0.0-2114.ga9a752df.el7.x86_64 147/198 2024-04-14T23:47:10.066 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-fuse-2:19.0.0-2114.ga9a752df.el7.x86_64 148/198 2024-04-14T23:47:10.139 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-test-2:19.0.0-2114.ga9a752df.el7.x86_64 149/198 2024-04-14T23:47:10.770 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2114.ga9a752df.el7.x86_64 120/198 2024-04-14T23:47:10.770 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /sys 2024-04-14T23:47:10.770 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /proc 2024-04-14T23:47:10.771 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /mnt 2024-04-14T23:47:10.771 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /var/tmp 2024-04-14T23:47:10.771 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /home 2024-04-14T23:47:10.771 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /root 2024-04-14T23:47:10.771 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /tmp 2024-04-14T23:47:10.771 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:47:10.989 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-volume-2:19.0.0-2114.ga9a752df.el7.noarch 121/198 2024-04-14T23:47:11.000 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libtool-2.4.6-45.el9.x86_64 150/198 2024-04-14T23:47:11.035 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-radosgw-2:19.0.0-2114.ga9a752df.el7.x86_64 151/198 2024-04-14T23:47:11.037 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-volume-2:19.0.0-2114.ga9a752df.el7.noarch 121/198 2024-04-14T23:47:11.037 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-14T23:47:11.037 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-14T23:47:11.037 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:47:11.058 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2114.ga9a752df.el7.x86_64 151/198 2024-04-14T23:47:11.058 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-14T23:47:11.059 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-14T23:47:11.059 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-14T23:47:11.059 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-14T23:47:11.059 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-14T23:47:11.103 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2114.ga9a75 152/198 2024-04-14T23:47:11.125 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2114.ga9a75 152/198 2024-04-14T23:47:11.125 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-14T23:47:11.125 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-14T23:47:11.125 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-14T23:47:11.170 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : cephfs-mirror-2:19.0.0-2114.ga9a752df.el7.x86_64 153/198 2024-04-14T23:47:11.192 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2114.ga9a752df.el7.x86_64 153/198 2024-04-14T23:47:11.192 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-14T23:47:11.193 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-14T23:47:11.193 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-14T23:47:11.193 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-14T23:47:11.193 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-14T23:47:11.495 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : rbd-mirror-2:19.0.0-2114.ga9a752df.el7.x86_64 154/198 2024-04-14T23:47:11.518 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2114.ga9a752df.el7.x86_64 154/198 2024-04-14T23:47:11.518 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-14T23:47:11.519 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-14T23:47:11.519 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-14T23:47:11.519 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-14T23:47:11.519 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-14T23:47:11.570 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : rbd-fuse-2:19.0.0-2114.ga9a752df.el7.x86_64 155/198 2024-04-14T23:47:11.600 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : rbd-nbd-2:19.0.0-2114.ga9a752df.el7.x86_64 156/198 2024-04-14T23:47:11.636 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 157/198 2024-04-14T23:47:11.832 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : cephfs-shell-2:19.0.0-2114.ga9a752df.el7.x86_64 158/198 2024-04-14T23:47:11.884 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 159/198 2024-04-14T23:47:11.923 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-04-14T23:47:12.000 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-04-14T23:47:12.044 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-04-14T23:47:12.068 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libcephfs-devel-2:19.0.0-2114.ga9a752df.el7.x86_ 163/198 2024-04-14T23:47:12.109 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-04-14T23:47:12.138 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : NetworkManager-team-1:1.47.4-1.el9.x86_64 165/198 2024-04-14T23:47:12.197 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : NetworkManager-tui-1:1.47.4-1.el9.x86_64 166/198 2024-04-14T23:47:12.266 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-04-14T23:47:12.304 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-04-14T23:47:12.717 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-osd-2:19.0.0-2114.ga9a752df.el7.x86_64 122/198 2024-04-14T23:47:12.824 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-14T23:47:12.825 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-osd-2:19.0.0-2114.ga9a752df.el7.x86_64 122/198 2024-04-14T23:47:12.825 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-14T23:47:12.826 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-14T23:47:12.826 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-14T23:47:12.826 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-14T23:47:12.826 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:47:12.869 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-14T23:47:12.889 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-14T23:47:13.091 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mds-2:19.0.0-2114.ga9a752df.el7.x86_64 123/198 2024-04-14T23:47:13.187 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mds-2:19.0.0-2114.ga9a752df.el7.x86_64 123/198 2024-04-14T23:47:13.187 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-14T23:47:13.187 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-14T23:47:13.187 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-14T23:47:13.188 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-14T23:47:13.188 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:47:13.701 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mon-2:19.0.0-2114.ga9a752df.el7.x86_64 124/198 2024-04-14T23:47:13.714 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : cephfs-top-2:19.0.0-2114.ga9a752df.el7.noarch 170/198 2024-04-14T23:47:13.796 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mon-2:19.0.0-2114.ga9a752df.el7.x86_64 124/198 2024-04-14T23:47:13.797 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-14T23:47:13.797 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-14T23:47:13.797 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-14T23:47:13.797 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-14T23:47:13.797 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:47:13.805 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-04-14T23:47:13.885 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : flex-2.6.4-9.el9.x86_64 172/198 2024-04-14T23:47:13.934 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : bison-3.7.4-5.el9.x86_64 173/198 2024-04-14T23:47:13.943 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : automake-1.16.2-8.el9.noarch 125/198 2024-04-14T23:47:13.960 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 174/198 2024-04-14T23:47:13.985 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-04-14T23:47:14.007 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-ply-3.11-14.el9.noarch 126/198 2024-04-14T23:47:14.010 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-04-14T23:47:14.045 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-04-14T23:47:14.046 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-04-14T23:47:14.089 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 127/198 2024-04-14T23:47:14.141 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-04-14T23:47:14.171 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-04-14T23:47:14.171 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-04-14T23:47:14.247 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-04-14T23:47:14.273 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-04-14T23:47:14.307 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-04-14T23:47:14.332 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-04-14T23:47:14.338 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 128/198 2024-04-14T23:47:14.357 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-04-14T23:47:14.397 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 129/198 2024-04-14T23:47:14.407 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-04-14T23:47:14.447 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-04-14T23:47:14.479 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-04-14T23:47:14.486 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 130/198 2024-04-14T23:47:14.504 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-04-14T23:47:14.529 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-04-14T23:47:14.555 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-04-14T23:47:14.588 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-04-14T23:47:14.596 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-14T23:47:14.596 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-14T23:47:14.597 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 131/198 2024-04-14T23:47:14.768 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 132/198 2024-04-14T23:47:14.784 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-14T23:47:14.784 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-04-14T23:47:14.832 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 133/198 2024-04-14T23:47:14.883 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 134/198 2024-04-14T23:47:14.884 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-04-14T23:47:14.916 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-04-14T23:47:14.920 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 135/198 2024-04-14T23:47:14.949 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-04-14T23:47:14.961 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : jq-1.6-16.el9.x86_64 136/198 2024-04-14T23:47:14.977 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-04-14T23:47:14.996 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : fuse-2.9.9-15.el9.x86_64 137/198 2024-04-14T23:47:15.001 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-04-14T23:47:15.074 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-04-14T23:47:15.703 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: cephadm-2:19.0.0-2114.ga9a752df.el7.noarch 138/198 2024-04-14T23:47:15.733 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : cephadm-2:19.0.0-2114.ga9a752df.el7.noarch 138/198 2024-04-14T23:47:15.790 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2114.ga9a752df.e 139/198 2024-04-14T23:47:15.904 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2114.ga9a752df. 140/198 2024-04-14T23:47:15.959 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2114.ga9a752df.el7.noa 141/198 2024-04-14T23:47:17.320 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2114.ga9a752df.el7.noa 141/198 2024-04-14T23:47:17.361 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2114.ga9a752df.el7.n 142/198 2024-04-14T23:47:18.520 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2114.ga9a752df.el7.n 142/198 2024-04-14T23:47:18.553 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2114.ga9a 143/198 2024-04-14T23:47:18.707 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2114.ga9a 143/198 2024-04-14T23:47:18.892 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2114.ga9a752df.el 144/198 2024-04-14T23:47:18.913 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-2:19.0.0-2114.ga9a752df.el7.x86_64 145/198 2024-04-14T23:47:18.986 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2114.ga9a752df.el7.x86_64 145/198 2024-04-14T23:47:18.987 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-14T23:47:18.987 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-14T23:47:18.987 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-14T23:47:18.987 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-14T23:47:18.987 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:47:19.039 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-rook-2:19.0.0-2114.ga9a752df.el7.noarch 146/198 2024-04-14T23:47:19.076 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2114.ga9a752df.el7.noarch 146/198 2024-04-14T23:47:19.195 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-2:19.0.0-2114.ga9a752df.el7.x86_64 147/198 2024-04-14T23:47:20.820 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-04-14T23:47:20.820 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-2:19.0.0-2114.ga9a752df.el7.x86_64 1/198 2024-04-14T23:47:20.820 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-base-2:19.0.0-2114.ga9a752df.el7.x86_64 2/198 2024-04-14T23:47:20.820 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-common-2:19.0.0-2114.ga9a752df.el7.x86_64 3/198 2024-04-14T23:47:20.820 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-fuse-2:19.0.0-2114.ga9a752df.el7.x86_64 4/198 2024-04-14T23:47:20.820 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2114.ga9a75 5/198 2024-04-14T23:47:20.820 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mds-2:19.0.0-2114.ga9a752df.el7.x86_64 6/198 2024-04-14T23:47:20.820 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-2:19.0.0-2114.ga9a752df.el7.x86_64 7/198 2024-04-14T23:47:20.820 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mon-2:19.0.0-2114.ga9a752df.el7.x86_64 8/198 2024-04-14T23:47:20.821 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-osd-2:19.0.0-2114.ga9a752df.el7.x86_64 9/198 2024-04-14T23:47:20.821 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-radosgw-2:19.0.0-2114.ga9a752df.el7.x86_64 10/198 2024-04-14T23:47:20.821 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-selinux-2:19.0.0-2114.ga9a752df.el7.x86_64 11/198 2024-04-14T23:47:20.821 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-test-2:19.0.0-2114.ga9a752df.el7.x86_64 12/198 2024-04-14T23:47:20.821 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : cephfs-mirror-2:19.0.0-2114.ga9a752df.el7.x86_64 13/198 2024-04-14T23:47:20.821 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : cephfs-shell-2:19.0.0-2114.ga9a752df.el7.x86_64 14/198 2024-04-14T23:47:20.821 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libcephfs-devel-2:19.0.0-2114.ga9a752df.el7.x86_ 15/198 2024-04-14T23:47:20.821 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libcephfs2-2:19.0.0-2114.ga9a752df.el7.x86_64 16/198 2024-04-14T23:47:20.821 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libcephsqlite-2:19.0.0-2114.ga9a752df.el7.x86_64 17/198 2024-04-14T23:47:20.821 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librados-devel-2:19.0.0-2114.ga9a752df.el7.x86_6 18/198 2024-04-14T23:47:20.821 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libradosstriper1-2:19.0.0-2114.ga9a752df.el7.x86 19/198 2024-04-14T23:47:20.821 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librgw2-2:19.0.0-2114.ga9a752df.el7.x86_64 20/198 2024-04-14T23:47:20.821 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2114.ga9a752df.el 21/198 2024-04-14T23:47:20.821 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-ceph-common-2:19.0.0-2114.ga9a752df.el7. 22/198 2024-04-14T23:47:20.822 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cephfs-2:19.0.0-2114.ga9a752df.el7.x86_6 23/198 2024-04-14T23:47:20.822 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rados-2:19.0.0-2114.ga9a752df.el7.x86_64 24/198 2024-04-14T23:47:20.822 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rbd-2:19.0.0-2114.ga9a752df.el7.x86_64 25/198 2024-04-14T23:47:20.822 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rgw-2:19.0.0-2114.ga9a752df.el7.x86_64 26/198 2024-04-14T23:47:20.822 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : rbd-fuse-2:19.0.0-2114.ga9a752df.el7.x86_64 27/198 2024-04-14T23:47:20.822 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : rbd-mirror-2:19.0.0-2114.ga9a752df.el7.x86_64 28/198 2024-04-14T23:47:20.822 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : rbd-nbd-2:19.0.0-2114.ga9a752df.el7.x86_64 29/198 2024-04-14T23:47:20.822 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2114.ga9a752df. 30/198 2024-04-14T23:47:20.822 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2114.ga9a752df.el7.noa 31/198 2024-04-14T23:47:20.822 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2114.ga9a752df.el7.n 32/198 2024-04-14T23:47:20.822 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2114.ga9a 33/198 2024-04-14T23:47:20.822 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2114.ga9a752df.el 34/198 2024-04-14T23:47:20.822 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2114.ga9a752df.el7.noarch 35/198 2024-04-14T23:47:20.822 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2114.ga9a752df.e 36/198 2024-04-14T23:47:20.823 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-volume-2:19.0.0-2114.ga9a752df.el7.noarch 37/198 2024-04-14T23:47:20.823 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : cephadm-2:19.0.0-2114.ga9a752df.el7.noarch 38/198 2024-04-14T23:47:20.823 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : cephfs-top-2:19.0.0-2114.ga9a752df.el7.noarch 39/198 2024-04-14T23:47:20.823 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 40/198 2024-04-14T23:47:20.823 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 41/198 2024-04-14T23:47:20.823 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : jq-1.6-16.el9.x86_64 42/198 2024-04-14T23:47:20.823 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 43/198 2024-04-14T23:47:20.823 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 44/198 2024-04-14T23:47:20.823 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 45/198 2024-04-14T23:47:20.823 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 46/198 2024-04-14T23:47:20.823 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 47/198 2024-04-14T23:47:20.823 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-ply-3.11-14.el9.noarch 48/198 2024-04-14T23:47:20.823 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 49/198 2024-04-14T23:47:20.823 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 50/198 2024-04-14T23:47:20.824 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 51/198 2024-04-14T23:47:20.824 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : automake-1.16.2-8.el9.noarch 52/198 2024-04-14T23:47:20.824 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : bison-3.7.4-5.el9.x86_64 53/198 2024-04-14T23:47:20.824 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 54/198 2024-04-14T23:47:20.824 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 55/198 2024-04-14T23:47:20.824 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : flex-2.6.4-9.el9.x86_64 56/198 2024-04-14T23:47:20.824 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 57/198 2024-04-14T23:47:20.824 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 58/198 2024-04-14T23:47:20.824 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 59/198 2024-04-14T23:47:20.824 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 60/198 2024-04-14T23:47:20.824 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 61/198 2024-04-14T23:47:20.824 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 62/198 2024-04-14T23:47:20.824 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 63/198 2024-04-14T23:47:20.824 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 64/198 2024-04-14T23:47:20.825 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 65/198 2024-04-14T23:47:20.825 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 66/198 2024-04-14T23:47:20.825 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : lua-5.4.4-4.el9.x86_64 67/198 2024-04-14T23:47:20.825 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 68/198 2024-04-14T23:47:20.825 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 69/198 2024-04-14T23:47:20.825 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 70/198 2024-04-14T23:47:20.825 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : postgresql-13.14-1.el9.x86_64 71/198 2024-04-14T23:47:20.825 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 72/198 2024-04-14T23:47:20.825 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 73/198 2024-04-14T23:47:20.825 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 74/198 2024-04-14T23:47:20.825 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 75/198 2024-04-14T23:47:20.825 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 76/198 2024-04-14T23:47:20.825 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 77/198 2024-04-14T23:47:20.825 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 78/198 2024-04-14T23:47:20.826 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 79/198 2024-04-14T23:47:20.826 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 80/198 2024-04-14T23:47:20.826 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 81/198 2024-04-14T23:47:20.826 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 82/198 2024-04-14T23:47:20.826 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 83/198 2024-04-14T23:47:20.826 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 84/198 2024-04-14T23:47:20.826 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 85/198 2024-04-14T23:47:20.826 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 86/198 2024-04-14T23:47:20.826 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 87/198 2024-04-14T23:47:20.826 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 88/198 2024-04-14T23:47:20.826 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 89/198 2024-04-14T23:47:20.826 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 90/198 2024-04-14T23:47:20.827 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 91/198 2024-04-14T23:47:20.827 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 92/198 2024-04-14T23:47:20.827 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 93/198 2024-04-14T23:47:20.827 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 94/198 2024-04-14T23:47:20.827 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 95/198 2024-04-14T23:47:20.827 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 96/198 2024-04-14T23:47:20.827 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 97/198 2024-04-14T23:47:20.827 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 98/198 2024-04-14T23:47:20.827 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 99/198 2024-04-14T23:47:20.827 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 100/198 2024-04-14T23:47:20.827 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 101/198 2024-04-14T23:47:20.827 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 102/198 2024-04-14T23:47:20.827 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 103/198 2024-04-14T23:47:20.827 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 104/198 2024-04-14T23:47:20.828 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-04-14T23:47:20.828 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-04-14T23:47:20.828 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-04-14T23:47:20.828 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-04-14T23:47:20.828 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-04-14T23:47:20.828 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-04-14T23:47:20.828 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-04-14T23:47:20.828 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-04-14T23:47:20.828 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-04-14T23:47:20.828 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-04-14T23:47:20.828 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-04-14T23:47:20.828 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-04-14T23:47:20.828 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-04-14T23:47:20.828 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-04-14T23:47:20.829 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-04-14T23:47:20.829 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-04-14T23:47:20.829 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-04-14T23:47:20.829 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-04-14T23:47:20.829 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-04-14T23:47:20.829 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-04-14T23:47:20.829 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-04-14T23:47:20.829 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-04-14T23:47:20.829 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-04-14T23:47:20.829 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-04-14T23:47:20.829 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-04-14T23:47:20.829 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-04-14T23:47:20.829 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-04-14T23:47:20.829 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-04-14T23:47:20.829 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-04-14T23:47:20.830 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-04-14T23:47:20.830 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-04-14T23:47:20.830 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-04-14T23:47:20.830 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-04-14T23:47:20.830 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-04-14T23:47:20.830 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-04-14T23:47:20.830 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-04-14T23:47:20.830 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-04-14T23:47:20.830 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-04-14T23:47:20.830 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-04-14T23:47:20.830 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-04-14T23:47:20.830 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-04-14T23:47:20.831 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-04-14T23:47:20.831 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-04-14T23:47:20.831 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-04-14T23:47:20.831 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-04-14T23:47:20.831 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-04-14T23:47:20.831 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librados2-2:19.0.0-2114.ga9a752df.el7.x86_64 151/198 2024-04-14T23:47:20.831 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-04-14T23:47:20.832 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librbd1-2:19.0.0-2114.ga9a752df.el7.x86_64 153/198 2024-04-14T23:47:20.832 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-04-14T23:47:20.832 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 155/198 2024-04-14T23:47:20.832 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 156/198 2024-04-14T23:47:20.832 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 157/198 2024-04-14T23:47:20.832 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 158/198 2024-04-14T23:47:20.832 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 159/198 2024-04-14T23:47:20.833 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 160/198 2024-04-14T23:47:20.833 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 161/198 2024-04-14T23:47:20.833 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 162/198 2024-04-14T23:47:20.833 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : acl-2.3.1-4.el9.x86_64 163/198 2024-04-14T23:47:20.833 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : acl-2.3.1-3.el9.x86_64 164/198 2024-04-14T23:47:20.833 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 165/198 2024-04-14T23:47:20.833 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 166/198 2024-04-14T23:47:20.834 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 167/198 2024-04-14T23:47:20.834 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 168/198 2024-04-14T23:47:20.834 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 169/198 2024-04-14T23:47:20.834 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 170/198 2024-04-14T23:47:20.834 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 171/198 2024-04-14T23:47:20.834 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 172/198 2024-04-14T23:47:20.834 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 173/198 2024-04-14T23:47:20.834 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 174/198 2024-04-14T23:47:20.834 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 175/198 2024-04-14T23:47:20.835 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 176/198 2024-04-14T23:47:20.835 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 177/198 2024-04-14T23:47:20.835 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 178/198 2024-04-14T23:47:20.835 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 179/198 2024-04-14T23:47:20.835 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 180/198 2024-04-14T23:47:20.835 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 181/198 2024-04-14T23:47:20.835 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 182/198 2024-04-14T23:47:20.835 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 183/198 2024-04-14T23:47:20.835 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 184/198 2024-04-14T23:47:20.835 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 185/198 2024-04-14T23:47:20.835 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 186/198 2024-04-14T23:47:20.835 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 187/198 2024-04-14T23:47:20.835 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 188/198 2024-04-14T23:47:20.835 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 189/198 2024-04-14T23:47:20.835 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 190/198 2024-04-14T23:47:20.836 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 191/198 2024-04-14T23:47:20.836 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 192/198 2024-04-14T23:47:20.836 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 193/198 2024-04-14T23:47:20.836 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 194/198 2024-04-14T23:47:20.836 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-04-14T23:47:20.836 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-04-14T23:47:20.836 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-04-14T23:47:23.987 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-04-14T23:47:23.988 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-14T23:47:23.988 INFO:teuthology.orchestra.run.smithi186.stdout:Upgraded: 2024-04-14T23:47:23.988 INFO:teuthology.orchestra.run.smithi186.stdout: NetworkManager-1:1.47.4-1.el9.x86_64 2024-04-14T23:47:23.988 INFO:teuthology.orchestra.run.smithi186.stdout: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 2024-04-14T23:47:23.988 INFO:teuthology.orchestra.run.smithi186.stdout: NetworkManager-team-1:1.47.4-1.el9.x86_64 2024-04-14T23:47:23.988 INFO:teuthology.orchestra.run.smithi186.stdout: NetworkManager-tui-1:1.47.4-1.el9.x86_64 2024-04-14T23:47:23.988 INFO:teuthology.orchestra.run.smithi186.stdout: acl-2.3.1-4.el9.x86_64 2024-04-14T23:47:23.988 INFO:teuthology.orchestra.run.smithi186.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-04-14T23:47:23.988 INFO:teuthology.orchestra.run.smithi186.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-04-14T23:47:23.988 INFO:teuthology.orchestra.run.smithi186.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-04-14T23:47:23.988 INFO:teuthology.orchestra.run.smithi186.stdout: libacl-2.3.1-4.el9.x86_64 2024-04-14T23:47:23.988 INFO:teuthology.orchestra.run.smithi186.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-04-14T23:47:23.989 INFO:teuthology.orchestra.run.smithi186.stdout: libblkid-2.37.4-18.el9.x86_64 2024-04-14T23:47:23.989 INFO:teuthology.orchestra.run.smithi186.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-04-14T23:47:23.989 INFO:teuthology.orchestra.run.smithi186.stdout: libmount-2.37.4-18.el9.x86_64 2024-04-14T23:47:23.989 INFO:teuthology.orchestra.run.smithi186.stdout: librados2-2:19.0.0-2114.ga9a752df.el7.x86_64 2024-04-14T23:47:23.989 INFO:teuthology.orchestra.run.smithi186.stdout: librbd1-2:19.0.0-2114.ga9a752df.el7.x86_64 2024-04-14T23:47:23.989 INFO:teuthology.orchestra.run.smithi186.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-04-14T23:47:23.989 INFO:teuthology.orchestra.run.smithi186.stdout: libuuid-2.37.4-18.el9.x86_64 2024-04-14T23:47:23.989 INFO:teuthology.orchestra.run.smithi186.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-04-14T23:47:23.989 INFO:teuthology.orchestra.run.smithi186.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-04-14T23:47:23.989 INFO:teuthology.orchestra.run.smithi186.stdout: util-linux-2.37.4-18.el9.x86_64 2024-04-14T23:47:23.989 INFO:teuthology.orchestra.run.smithi186.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-04-14T23:47:23.989 INFO:teuthology.orchestra.run.smithi186.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-04-14T23:47:23.989 INFO:teuthology.orchestra.run.smithi186.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-04-14T23:47:23.989 INFO:teuthology.orchestra.run.smithi186.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-04-14T23:47:23.990 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-04-14T23:47:23.990 INFO:teuthology.orchestra.run.smithi186.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-14T23:47:23.990 INFO:teuthology.orchestra.run.smithi186.stdout: automake-1.16.2-8.el9.noarch 2024-04-14T23:47:23.990 INFO:teuthology.orchestra.run.smithi186.stdout: bison-3.7.4-5.el9.x86_64 2024-04-14T23:47:23.990 INFO:teuthology.orchestra.run.smithi186.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-14T23:47:23.990 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-2:19.0.0-2114.ga9a752df.el7.x86_64 2024-04-14T23:47:23.990 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-base-2:19.0.0-2114.ga9a752df.el7.x86_64 2024-04-14T23:47:23.990 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-common-2:19.0.0-2114.ga9a752df.el7.x86_64 2024-04-14T23:47:23.990 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-fuse-2:19.0.0-2114.ga9a752df.el7.x86_64 2024-04-14T23:47:23.990 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-grafana-dashboards-2:19.0.0-2114.ga9a752df.el7.noarch 2024-04-14T23:47:23.990 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-immutable-object-cache-2:19.0.0-2114.ga9a752df.el7.x86_64 2024-04-14T23:47:23.990 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mds-2:19.0.0-2114.ga9a752df.el7.x86_64 2024-04-14T23:47:23.990 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-2:19.0.0-2114.ga9a752df.el7.x86_64 2024-04-14T23:47:23.991 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-cephadm-2:19.0.0-2114.ga9a752df.el7.noarch 2024-04-14T23:47:23.991 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-dashboard-2:19.0.0-2114.ga9a752df.el7.noarch 2024-04-14T23:47:23.991 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2114.ga9a752df.el7.noarch 2024-04-14T23:47:23.991 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-modules-core-2:19.0.0-2114.ga9a752df.el7.noarch 2024-04-14T23:47:23.991 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-rook-2:19.0.0-2114.ga9a752df.el7.noarch 2024-04-14T23:47:23.991 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mon-2:19.0.0-2114.ga9a752df.el7.x86_64 2024-04-14T23:47:23.991 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-osd-2:19.0.0-2114.ga9a752df.el7.x86_64 2024-04-14T23:47:23.991 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-prometheus-alerts-2:19.0.0-2114.ga9a752df.el7.noarch 2024-04-14T23:47:23.991 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-radosgw-2:19.0.0-2114.ga9a752df.el7.x86_64 2024-04-14T23:47:23.991 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-selinux-2:19.0.0-2114.ga9a752df.el7.x86_64 2024-04-14T23:47:23.991 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-test-2:19.0.0-2114.ga9a752df.el7.x86_64 2024-04-14T23:47:23.991 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-volume-2:19.0.0-2114.ga9a752df.el7.noarch 2024-04-14T23:47:23.991 INFO:teuthology.orchestra.run.smithi186.stdout: cephadm-2:19.0.0-2114.ga9a752df.el7.noarch 2024-04-14T23:47:23.991 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-mirror-2:19.0.0-2114.ga9a752df.el7.x86_64 2024-04-14T23:47:23.992 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-shell-2:19.0.0-2114.ga9a752df.el7.x86_64 2024-04-14T23:47:23.992 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-top-2:19.0.0-2114.ga9a752df.el7.noarch 2024-04-14T23:47:23.992 INFO:teuthology.orchestra.run.smithi186.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-14T23:47:23.992 INFO:teuthology.orchestra.run.smithi186.stdout: flex-2.6.4-9.el9.x86_64 2024-04-14T23:47:23.992 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-14T23:47:23.992 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-14T23:47:23.992 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-14T23:47:23.992 INFO:teuthology.orchestra.run.smithi186.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-14T23:47:23.992 INFO:teuthology.orchestra.run.smithi186.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-14T23:47:23.992 INFO:teuthology.orchestra.run.smithi186.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-14T23:47:23.992 INFO:teuthology.orchestra.run.smithi186.stdout: jq-1.6-16.el9.x86_64 2024-04-14T23:47:23.992 INFO:teuthology.orchestra.run.smithi186.stdout: libaio-devel-0.3.112-1.el8.x86_64 2024-04-14T23:47:23.992 INFO:teuthology.orchestra.run.smithi186.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-14T23:47:23.992 INFO:teuthology.orchestra.run.smithi186.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-14T23:47:23.993 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs-devel-2:19.0.0-2114.ga9a752df.el7.x86_64 2024-04-14T23:47:23.993 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs2-2:19.0.0-2114.ga9a752df.el7.x86_64 2024-04-14T23:47:23.993 INFO:teuthology.orchestra.run.smithi186.stdout: libcephsqlite-2:19.0.0-2114.ga9a752df.el7.x86_64 2024-04-14T23:47:23.993 INFO:teuthology.orchestra.run.smithi186.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-14T23:47:23.993 INFO:teuthology.orchestra.run.smithi186.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-14T23:47:23.993 INFO:teuthology.orchestra.run.smithi186.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-14T23:47:23.993 INFO:teuthology.orchestra.run.smithi186.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-14T23:47:23.993 INFO:teuthology.orchestra.run.smithi186.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-14T23:47:23.993 INFO:teuthology.orchestra.run.smithi186.stdout: librados-devel-2:19.0.0-2114.ga9a752df.el7.x86_64 2024-04-14T23:47:23.993 INFO:teuthology.orchestra.run.smithi186.stdout: libradosstriper1-2:19.0.0-2114.ga9a752df.el7.x86_64 2024-04-14T23:47:23.993 INFO:teuthology.orchestra.run.smithi186.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-14T23:47:23.993 INFO:teuthology.orchestra.run.smithi186.stdout: librgw2-2:19.0.0-2114.ga9a752df.el7.x86_64 2024-04-14T23:47:23.993 INFO:teuthology.orchestra.run.smithi186.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-14T23:47:23.993 INFO:teuthology.orchestra.run.smithi186.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-14T23:47:23.994 INFO:teuthology.orchestra.run.smithi186.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-14T23:47:23.994 INFO:teuthology.orchestra.run.smithi186.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-14T23:47:23.994 INFO:teuthology.orchestra.run.smithi186.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-14T23:47:23.994 INFO:teuthology.orchestra.run.smithi186.stdout: lua-5.4.4-4.el9.x86_64 2024-04-14T23:47:23.994 INFO:teuthology.orchestra.run.smithi186.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-14T23:47:23.994 INFO:teuthology.orchestra.run.smithi186.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-14T23:47:23.994 INFO:teuthology.orchestra.run.smithi186.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-14T23:47:23.995 INFO:teuthology.orchestra.run.smithi186.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-14T23:47:23.995 INFO:teuthology.orchestra.run.smithi186.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-14T23:47:23.995 INFO:teuthology.orchestra.run.smithi186.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-04-14T23:47:23.995 INFO:teuthology.orchestra.run.smithi186.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-14T23:47:23.995 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-13.14-1.el9.x86_64 2024-04-14T23:47:23.995 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-14T23:47:23.995 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-14T23:47:23.995 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-14T23:47:23.995 INFO:teuthology.orchestra.run.smithi186.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-14T23:47:23.995 INFO:teuthology.orchestra.run.smithi186.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-14T23:47:23.995 INFO:teuthology.orchestra.run.smithi186.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-14T23:47:23.995 INFO:teuthology.orchestra.run.smithi186.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-14T23:47:23.995 INFO:teuthology.orchestra.run.smithi186.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-14T23:47:23.995 INFO:teuthology.orchestra.run.smithi186.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-14T23:47:23.996 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-14T23:47:23.996 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-argparse-2:19.0.0-2114.ga9a752df.el7.x86_64 2024-04-14T23:47:23.996 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-common-2:19.0.0-2114.ga9a752df.el7.x86_64 2024-04-14T23:47:23.996 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cephfs-2:19.0.0-2114.ga9a752df.el7.x86_64 2024-04-14T23:47:23.996 INFO:teuthology.orchestra.run.smithi186.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-14T23:47:23.996 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-14T23:47:23.996 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-14T23:47:23.996 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-14T23:47:23.996 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-14T23:47:23.996 INFO:teuthology.orchestra.run.smithi186.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-14T23:47:23.996 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-14T23:47:23.996 INFO:teuthology.orchestra.run.smithi186.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-14T23:47:23.996 INFO:teuthology.orchestra.run.smithi186.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-14T23:47:23.997 INFO:teuthology.orchestra.run.smithi186.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-14T23:47:23.997 INFO:teuthology.orchestra.run.smithi186.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-14T23:47:23.997 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-14T23:47:23.997 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-14T23:47:23.997 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-14T23:47:23.997 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-14T23:47:23.997 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-14T23:47:23.997 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-14T23:47:23.997 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-14T23:47:23.997 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-14T23:47:23.997 INFO:teuthology.orchestra.run.smithi186.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-14T23:47:23.997 INFO:teuthology.orchestra.run.smithi186.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-14T23:47:23.997 INFO:teuthology.orchestra.run.smithi186.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-14T23:47:23.997 INFO:teuthology.orchestra.run.smithi186.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-14T23:47:23.998 INFO:teuthology.orchestra.run.smithi186.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-14T23:47:23.998 INFO:teuthology.orchestra.run.smithi186.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-14T23:47:23.998 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-14T23:47:23.998 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-14T23:47:23.998 INFO:teuthology.orchestra.run.smithi186.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-14T23:47:23.998 INFO:teuthology.orchestra.run.smithi186.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-14T23:47:23.998 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-14T23:47:23.998 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ply-3.11-14.el9.noarch 2024-04-14T23:47:23.998 INFO:teuthology.orchestra.run.smithi186.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-14T23:47:23.998 INFO:teuthology.orchestra.run.smithi186.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-14T23:47:23.998 INFO:teuthology.orchestra.run.smithi186.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-14T23:47:23.998 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-14T23:47:23.998 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-14T23:47:23.998 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-14T23:47:23.999 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-14T23:47:23.999 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-14T23:47:23.999 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-14T23:47:23.999 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rados-2:19.0.0-2114.ga9a752df.el7.x86_64 2024-04-14T23:47:23.999 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rbd-2:19.0.0-2114.ga9a752df.el7.x86_64 2024-04-14T23:47:23.999 INFO:teuthology.orchestra.run.smithi186.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-14T23:47:23.999 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-14T23:47:23.999 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-14T23:47:23.999 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rgw-2:19.0.0-2114.ga9a752df.el7.x86_64 2024-04-14T23:47:23.999 INFO:teuthology.orchestra.run.smithi186.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-14T23:47:23.999 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-14T23:47:23.999 INFO:teuthology.orchestra.run.smithi186.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-14T23:47:23.999 INFO:teuthology.orchestra.run.smithi186.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-14T23:47:23.999 INFO:teuthology.orchestra.run.smithi186.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-14T23:47:24.000 INFO:teuthology.orchestra.run.smithi186.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-14T23:47:24.000 INFO:teuthology.orchestra.run.smithi186.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-14T23:47:24.000 INFO:teuthology.orchestra.run.smithi186.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-14T23:47:24.000 INFO:teuthology.orchestra.run.smithi186.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-14T23:47:24.000 INFO:teuthology.orchestra.run.smithi186.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-14T23:47:24.000 INFO:teuthology.orchestra.run.smithi186.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-14T23:47:24.000 INFO:teuthology.orchestra.run.smithi186.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-14T23:47:24.000 INFO:teuthology.orchestra.run.smithi186.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-14T23:47:24.000 INFO:teuthology.orchestra.run.smithi186.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-14T23:47:24.000 INFO:teuthology.orchestra.run.smithi186.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-14T23:47:24.000 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-fuse-2:19.0.0-2114.ga9a752df.el7.x86_64 2024-04-14T23:47:24.000 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-mirror-2:19.0.0-2114.ga9a752df.el7.x86_64 2024-04-14T23:47:24.000 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-nbd-2:19.0.0-2114.ga9a752df.el7.x86_64 2024-04-14T23:47:24.000 INFO:teuthology.orchestra.run.smithi186.stdout: re2-1:20211101-3.el9.x86_64 2024-04-14T23:47:24.001 INFO:teuthology.orchestra.run.smithi186.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-14T23:47:24.001 INFO:teuthology.orchestra.run.smithi186.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-14T23:47:24.001 INFO:teuthology.orchestra.run.smithi186.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-14T23:47:24.001 INFO:teuthology.orchestra.run.smithi186.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-14T23:47:24.001 INFO:teuthology.orchestra.run.smithi186.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-14T23:47:24.001 INFO:teuthology.orchestra.run.smithi186.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-04-14T23:47:24.001 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-14T23:47:24.001 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-04-14T23:47:24.279 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-fuse-2:19.0.0-2114.ga9a752df.el7.x86_64 148/198 2024-04-14T23:47:24.376 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-test-2:19.0.0-2114.ga9a752df.el7.x86_64 149/198 2024-04-14T23:47:24.486 DEBUG:teuthology.parallel:result is None 2024-04-14T23:47:25.366 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libtool-2.4.6-45.el9.x86_64 150/198 2024-04-14T23:47:25.396 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-radosgw-2:19.0.0-2114.ga9a752df.el7.x86_64 151/198 2024-04-14T23:47:25.489 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2114.ga9a752df.el7.x86_64 151/198 2024-04-14T23:47:25.489 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-14T23:47:25.489 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-14T23:47:25.489 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-14T23:47:25.489 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-14T23:47:25.489 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:47:25.539 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2114.ga9a75 152/198 2024-04-14T23:47:25.638 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2114.ga9a75 152/198 2024-04-14T23:47:25.638 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-14T23:47:25.638 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-14T23:47:25.638 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:47:25.742 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : cephfs-mirror-2:19.0.0-2114.ga9a752df.el7.x86_64 153/198 2024-04-14T23:47:25.851 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2114.ga9a752df.el7.x86_64 153/198 2024-04-14T23:47:25.852 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-14T23:47:25.852 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-14T23:47:25.852 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-14T23:47:25.852 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-14T23:47:25.852 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:47:26.232 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : rbd-mirror-2:19.0.0-2114.ga9a752df.el7.x86_64 154/198 2024-04-14T23:47:26.306 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2114.ga9a752df.el7.x86_64 154/198 2024-04-14T23:47:26.306 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-14T23:47:26.307 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-14T23:47:26.307 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-14T23:47:26.307 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-14T23:47:26.307 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:47:26.372 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : rbd-fuse-2:19.0.0-2114.ga9a752df.el7.x86_64 155/198 2024-04-14T23:47:26.407 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : rbd-nbd-2:19.0.0-2114.ga9a752df.el7.x86_64 156/198 2024-04-14T23:47:26.447 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 157/198 2024-04-14T23:47:26.751 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : cephfs-shell-2:19.0.0-2114.ga9a752df.el7.x86_64 158/198 2024-04-14T23:47:26.821 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 159/198 2024-04-14T23:47:26.890 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-04-14T23:47:26.968 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-04-14T23:47:27.013 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-04-14T23:47:27.057 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libcephfs-devel-2:19.0.0-2114.ga9a752df.el7.x86_ 163/198 2024-04-14T23:47:27.101 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-04-14T23:47:27.154 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : NetworkManager-team-1:1.47.4-1.el9.x86_64 165/198 2024-04-14T23:47:27.242 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : NetworkManager-tui-1:1.47.4-1.el9.x86_64 166/198 2024-04-14T23:47:27.360 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-04-14T23:47:27.402 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-04-14T23:47:28.348 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-14T23:47:28.398 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-14T23:47:28.461 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-14T23:47:29.919 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : cephfs-top-2:19.0.0-2114.ga9a752df.el7.noarch 170/198 2024-04-14T23:47:30.043 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-04-14T23:47:30.169 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : flex-2.6.4-9.el9.x86_64 172/198 2024-04-14T23:47:30.205 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : bison-3.7.4-5.el9.x86_64 173/198 2024-04-14T23:47:30.230 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 174/198 2024-04-14T23:47:30.263 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-04-14T23:47:30.290 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-04-14T23:47:30.349 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-04-14T23:47:30.350 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-04-14T23:47:30.511 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-04-14T23:47:30.561 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-04-14T23:47:30.561 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-04-14T23:47:30.708 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-04-14T23:47:30.750 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-04-14T23:47:30.806 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-04-14T23:47:30.847 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-04-14T23:47:30.879 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-04-14T23:47:30.978 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-04-14T23:47:31.054 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-04-14T23:47:31.109 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-04-14T23:47:31.142 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-04-14T23:47:31.176 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-04-14T23:47:31.209 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-04-14T23:47:31.243 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-04-14T23:47:31.282 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-14T23:47:31.282 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-14T23:47:31.627 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-14T23:47:31.627 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-04-14T23:47:31.766 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-04-14T23:47:31.796 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-04-14T23:47:31.851 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-04-14T23:47:31.891 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-04-14T23:47:31.931 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-04-14T23:47:32.054 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-04-14T23:47:32.361 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-04-14T23:47:32.361 INFO:teuthology.orchestra.run.smithi060.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-04-14T23:47:32.361 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:47:37.994 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-2:19.0.0-2114.ga9a752df.el7.x86_64 1/198 2024-04-14T23:47:37.994 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-base-2:19.0.0-2114.ga9a752df.el7.x86_64 2/198 2024-04-14T23:47:37.994 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-common-2:19.0.0-2114.ga9a752df.el7.x86_64 3/198 2024-04-14T23:47:37.994 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-fuse-2:19.0.0-2114.ga9a752df.el7.x86_64 4/198 2024-04-14T23:47:37.994 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2114.ga9a75 5/198 2024-04-14T23:47:37.994 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mds-2:19.0.0-2114.ga9a752df.el7.x86_64 6/198 2024-04-14T23:47:37.994 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-2:19.0.0-2114.ga9a752df.el7.x86_64 7/198 2024-04-14T23:47:37.995 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mon-2:19.0.0-2114.ga9a752df.el7.x86_64 8/198 2024-04-14T23:47:37.995 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-osd-2:19.0.0-2114.ga9a752df.el7.x86_64 9/198 2024-04-14T23:47:37.995 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-radosgw-2:19.0.0-2114.ga9a752df.el7.x86_64 10/198 2024-04-14T23:47:37.995 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-selinux-2:19.0.0-2114.ga9a752df.el7.x86_64 11/198 2024-04-14T23:47:37.995 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-test-2:19.0.0-2114.ga9a752df.el7.x86_64 12/198 2024-04-14T23:47:37.995 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : cephfs-mirror-2:19.0.0-2114.ga9a752df.el7.x86_64 13/198 2024-04-14T23:47:37.996 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : cephfs-shell-2:19.0.0-2114.ga9a752df.el7.x86_64 14/198 2024-04-14T23:47:37.996 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libcephfs-devel-2:19.0.0-2114.ga9a752df.el7.x86_ 15/198 2024-04-14T23:47:37.996 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libcephfs2-2:19.0.0-2114.ga9a752df.el7.x86_64 16/198 2024-04-14T23:47:37.996 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libcephsqlite-2:19.0.0-2114.ga9a752df.el7.x86_64 17/198 2024-04-14T23:47:37.996 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librados-devel-2:19.0.0-2114.ga9a752df.el7.x86_6 18/198 2024-04-14T23:47:37.996 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libradosstriper1-2:19.0.0-2114.ga9a752df.el7.x86 19/198 2024-04-14T23:47:37.996 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librgw2-2:19.0.0-2114.ga9a752df.el7.x86_64 20/198 2024-04-14T23:47:37.996 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2114.ga9a752df.el 21/198 2024-04-14T23:47:37.996 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-ceph-common-2:19.0.0-2114.ga9a752df.el7. 22/198 2024-04-14T23:47:37.996 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cephfs-2:19.0.0-2114.ga9a752df.el7.x86_6 23/198 2024-04-14T23:47:37.996 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-rados-2:19.0.0-2114.ga9a752df.el7.x86_64 24/198 2024-04-14T23:47:37.996 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-rbd-2:19.0.0-2114.ga9a752df.el7.x86_64 25/198 2024-04-14T23:47:37.996 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-rgw-2:19.0.0-2114.ga9a752df.el7.x86_64 26/198 2024-04-14T23:47:37.996 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : rbd-fuse-2:19.0.0-2114.ga9a752df.el7.x86_64 27/198 2024-04-14T23:47:37.997 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : rbd-mirror-2:19.0.0-2114.ga9a752df.el7.x86_64 28/198 2024-04-14T23:47:37.997 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : rbd-nbd-2:19.0.0-2114.ga9a752df.el7.x86_64 29/198 2024-04-14T23:47:37.997 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2114.ga9a752df. 30/198 2024-04-14T23:47:37.997 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2114.ga9a752df.el7.noa 31/198 2024-04-14T23:47:37.997 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2114.ga9a752df.el7.n 32/198 2024-04-14T23:47:37.997 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2114.ga9a 33/198 2024-04-14T23:47:37.997 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2114.ga9a752df.el 34/198 2024-04-14T23:47:37.997 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2114.ga9a752df.el7.noarch 35/198 2024-04-14T23:47:37.997 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2114.ga9a752df.e 36/198 2024-04-14T23:47:37.997 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-volume-2:19.0.0-2114.ga9a752df.el7.noarch 37/198 2024-04-14T23:47:37.997 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : cephadm-2:19.0.0-2114.ga9a752df.el7.noarch 38/198 2024-04-14T23:47:37.997 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : cephfs-top-2:19.0.0-2114.ga9a752df.el7.noarch 39/198 2024-04-14T23:47:37.997 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 40/198 2024-04-14T23:47:37.998 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 41/198 2024-04-14T23:47:37.998 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : jq-1.6-16.el9.x86_64 42/198 2024-04-14T23:47:37.998 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 43/198 2024-04-14T23:47:37.998 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 44/198 2024-04-14T23:47:37.998 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 45/198 2024-04-14T23:47:37.998 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 46/198 2024-04-14T23:47:37.998 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 47/198 2024-04-14T23:47:37.998 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-ply-3.11-14.el9.noarch 48/198 2024-04-14T23:47:37.998 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 49/198 2024-04-14T23:47:37.998 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 50/198 2024-04-14T23:47:37.998 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 51/198 2024-04-14T23:47:37.998 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : automake-1.16.2-8.el9.noarch 52/198 2024-04-14T23:47:37.998 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : bison-3.7.4-5.el9.x86_64 53/198 2024-04-14T23:47:37.998 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 54/198 2024-04-14T23:47:37.999 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 55/198 2024-04-14T23:47:37.999 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : flex-2.6.4-9.el9.x86_64 56/198 2024-04-14T23:47:37.999 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 57/198 2024-04-14T23:47:37.999 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 58/198 2024-04-14T23:47:37.999 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 59/198 2024-04-14T23:47:37.999 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 60/198 2024-04-14T23:47:37.999 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 61/198 2024-04-14T23:47:37.999 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 62/198 2024-04-14T23:47:37.999 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 63/198 2024-04-14T23:47:37.999 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 64/198 2024-04-14T23:47:37.999 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 65/198 2024-04-14T23:47:37.999 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 66/198 2024-04-14T23:47:37.999 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : lua-5.4.4-4.el9.x86_64 67/198 2024-04-14T23:47:37.999 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 68/198 2024-04-14T23:47:38.000 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 69/198 2024-04-14T23:47:38.000 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 70/198 2024-04-14T23:47:38.000 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : postgresql-13.14-1.el9.x86_64 71/198 2024-04-14T23:47:38.000 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 72/198 2024-04-14T23:47:38.000 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 73/198 2024-04-14T23:47:38.000 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 74/198 2024-04-14T23:47:38.000 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 75/198 2024-04-14T23:47:38.000 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 76/198 2024-04-14T23:47:38.000 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 77/198 2024-04-14T23:47:38.000 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 78/198 2024-04-14T23:47:38.000 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 79/198 2024-04-14T23:47:38.000 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 80/198 2024-04-14T23:47:38.001 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 81/198 2024-04-14T23:47:38.001 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 82/198 2024-04-14T23:47:38.001 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 83/198 2024-04-14T23:47:38.001 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 84/198 2024-04-14T23:47:38.001 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 85/198 2024-04-14T23:47:38.001 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 86/198 2024-04-14T23:47:38.001 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 87/198 2024-04-14T23:47:38.001 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 88/198 2024-04-14T23:47:38.001 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 89/198 2024-04-14T23:47:38.001 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 90/198 2024-04-14T23:47:38.001 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 91/198 2024-04-14T23:47:38.001 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 92/198 2024-04-14T23:47:38.001 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 93/198 2024-04-14T23:47:38.002 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 94/198 2024-04-14T23:47:38.002 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 95/198 2024-04-14T23:47:38.002 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 96/198 2024-04-14T23:47:38.002 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 97/198 2024-04-14T23:47:38.002 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 98/198 2024-04-14T23:47:38.002 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 99/198 2024-04-14T23:47:38.002 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 100/198 2024-04-14T23:47:38.002 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 101/198 2024-04-14T23:47:38.002 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 102/198 2024-04-14T23:47:38.002 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 103/198 2024-04-14T23:47:38.003 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 104/198 2024-04-14T23:47:38.003 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-04-14T23:47:38.003 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-04-14T23:47:38.003 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-04-14T23:47:38.003 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-04-14T23:47:38.003 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-04-14T23:47:38.003 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-04-14T23:47:38.003 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-04-14T23:47:38.003 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-04-14T23:47:38.003 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-04-14T23:47:38.003 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-04-14T23:47:38.003 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-04-14T23:47:38.003 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-04-14T23:47:38.004 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-04-14T23:47:38.004 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-04-14T23:47:38.004 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-04-14T23:47:38.004 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-04-14T23:47:38.004 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-04-14T23:47:38.004 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-04-14T23:47:38.004 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-04-14T23:47:38.004 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-04-14T23:47:38.004 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-04-14T23:47:38.004 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-04-14T23:47:38.004 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-04-14T23:47:38.004 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-04-14T23:47:38.004 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-04-14T23:47:38.004 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-04-14T23:47:38.005 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-04-14T23:47:38.005 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-04-14T23:47:38.005 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-04-14T23:47:38.005 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-04-14T23:47:38.005 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-04-14T23:47:38.005 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-04-14T23:47:38.005 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-04-14T23:47:38.005 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-04-14T23:47:38.005 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-04-14T23:47:38.005 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-04-14T23:47:38.005 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-04-14T23:47:38.005 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-04-14T23:47:38.005 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-04-14T23:47:38.005 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-04-14T23:47:38.006 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-04-14T23:47:38.006 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-04-14T23:47:38.006 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-04-14T23:47:38.006 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-04-14T23:47:38.006 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-04-14T23:47:38.006 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-04-14T23:47:38.006 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librados2-2:19.0.0-2114.ga9a752df.el7.x86_64 151/198 2024-04-14T23:47:38.006 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-04-14T23:47:38.006 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librbd1-2:19.0.0-2114.ga9a752df.el7.x86_64 153/198 2024-04-14T23:47:38.006 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-04-14T23:47:38.006 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 155/198 2024-04-14T23:47:38.006 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 156/198 2024-04-14T23:47:38.006 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 157/198 2024-04-14T23:47:38.007 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 158/198 2024-04-14T23:47:38.007 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 159/198 2024-04-14T23:47:38.007 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 160/198 2024-04-14T23:47:38.007 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 161/198 2024-04-14T23:47:38.007 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 162/198 2024-04-14T23:47:38.007 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : acl-2.3.1-4.el9.x86_64 163/198 2024-04-14T23:47:38.007 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : acl-2.3.1-3.el9.x86_64 164/198 2024-04-14T23:47:38.007 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 165/198 2024-04-14T23:47:38.007 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 166/198 2024-04-14T23:47:38.007 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 167/198 2024-04-14T23:47:38.007 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 168/198 2024-04-14T23:47:38.007 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 169/198 2024-04-14T23:47:38.007 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 170/198 2024-04-14T23:47:38.007 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 171/198 2024-04-14T23:47:38.008 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 172/198 2024-04-14T23:47:38.008 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 173/198 2024-04-14T23:47:38.008 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 174/198 2024-04-14T23:47:38.008 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 175/198 2024-04-14T23:47:38.008 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 176/198 2024-04-14T23:47:38.008 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 177/198 2024-04-14T23:47:38.008 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 178/198 2024-04-14T23:47:38.008 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 179/198 2024-04-14T23:47:38.008 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 180/198 2024-04-14T23:47:38.008 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 181/198 2024-04-14T23:47:38.008 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 182/198 2024-04-14T23:47:38.008 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 183/198 2024-04-14T23:47:38.009 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 184/198 2024-04-14T23:47:38.009 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 185/198 2024-04-14T23:47:38.009 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 186/198 2024-04-14T23:47:38.009 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 187/198 2024-04-14T23:47:38.009 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 188/198 2024-04-14T23:47:38.009 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 189/198 2024-04-14T23:47:38.009 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 190/198 2024-04-14T23:47:38.009 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 191/198 2024-04-14T23:47:38.009 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 192/198 2024-04-14T23:47:38.009 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 193/198 2024-04-14T23:47:38.009 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 194/198 2024-04-14T23:47:38.009 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-04-14T23:47:38.009 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-04-14T23:47:38.009 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-04-14T23:47:40.610 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-04-14T23:47:40.610 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:47:40.610 INFO:teuthology.orchestra.run.smithi060.stdout:Upgraded: 2024-04-14T23:47:40.610 INFO:teuthology.orchestra.run.smithi060.stdout: NetworkManager-1:1.47.4-1.el9.x86_64 2024-04-14T23:47:40.610 INFO:teuthology.orchestra.run.smithi060.stdout: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 2024-04-14T23:47:40.610 INFO:teuthology.orchestra.run.smithi060.stdout: NetworkManager-team-1:1.47.4-1.el9.x86_64 2024-04-14T23:47:40.610 INFO:teuthology.orchestra.run.smithi060.stdout: NetworkManager-tui-1:1.47.4-1.el9.x86_64 2024-04-14T23:47:40.610 INFO:teuthology.orchestra.run.smithi060.stdout: acl-2.3.1-4.el9.x86_64 2024-04-14T23:47:40.610 INFO:teuthology.orchestra.run.smithi060.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-04-14T23:47:40.611 INFO:teuthology.orchestra.run.smithi060.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-04-14T23:47:40.611 INFO:teuthology.orchestra.run.smithi060.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-04-14T23:47:40.611 INFO:teuthology.orchestra.run.smithi060.stdout: libacl-2.3.1-4.el9.x86_64 2024-04-14T23:47:40.611 INFO:teuthology.orchestra.run.smithi060.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-04-14T23:47:40.611 INFO:teuthology.orchestra.run.smithi060.stdout: libblkid-2.37.4-18.el9.x86_64 2024-04-14T23:47:40.611 INFO:teuthology.orchestra.run.smithi060.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-04-14T23:47:40.611 INFO:teuthology.orchestra.run.smithi060.stdout: libmount-2.37.4-18.el9.x86_64 2024-04-14T23:47:40.611 INFO:teuthology.orchestra.run.smithi060.stdout: librados2-2:19.0.0-2114.ga9a752df.el7.x86_64 2024-04-14T23:47:40.611 INFO:teuthology.orchestra.run.smithi060.stdout: librbd1-2:19.0.0-2114.ga9a752df.el7.x86_64 2024-04-14T23:47:40.611 INFO:teuthology.orchestra.run.smithi060.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-04-14T23:47:40.611 INFO:teuthology.orchestra.run.smithi060.stdout: libuuid-2.37.4-18.el9.x86_64 2024-04-14T23:47:40.611 INFO:teuthology.orchestra.run.smithi060.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-04-14T23:47:40.611 INFO:teuthology.orchestra.run.smithi060.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-04-14T23:47:40.612 INFO:teuthology.orchestra.run.smithi060.stdout: util-linux-2.37.4-18.el9.x86_64 2024-04-14T23:47:40.612 INFO:teuthology.orchestra.run.smithi060.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-04-14T23:47:40.612 INFO:teuthology.orchestra.run.smithi060.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-04-14T23:47:40.612 INFO:teuthology.orchestra.run.smithi060.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-04-14T23:47:40.612 INFO:teuthology.orchestra.run.smithi060.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-04-14T23:47:40.612 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-04-14T23:47:40.612 INFO:teuthology.orchestra.run.smithi060.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-14T23:47:40.612 INFO:teuthology.orchestra.run.smithi060.stdout: automake-1.16.2-8.el9.noarch 2024-04-14T23:47:40.612 INFO:teuthology.orchestra.run.smithi060.stdout: bison-3.7.4-5.el9.x86_64 2024-04-14T23:47:40.612 INFO:teuthology.orchestra.run.smithi060.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-14T23:47:40.612 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-2:19.0.0-2114.ga9a752df.el7.x86_64 2024-04-14T23:47:40.612 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-base-2:19.0.0-2114.ga9a752df.el7.x86_64 2024-04-14T23:47:40.612 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-common-2:19.0.0-2114.ga9a752df.el7.x86_64 2024-04-14T23:47:40.613 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-fuse-2:19.0.0-2114.ga9a752df.el7.x86_64 2024-04-14T23:47:40.613 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-grafana-dashboards-2:19.0.0-2114.ga9a752df.el7.noarch 2024-04-14T23:47:40.613 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-immutable-object-cache-2:19.0.0-2114.ga9a752df.el7.x86_64 2024-04-14T23:47:40.613 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mds-2:19.0.0-2114.ga9a752df.el7.x86_64 2024-04-14T23:47:40.613 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-2:19.0.0-2114.ga9a752df.el7.x86_64 2024-04-14T23:47:40.613 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-cephadm-2:19.0.0-2114.ga9a752df.el7.noarch 2024-04-14T23:47:40.613 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-dashboard-2:19.0.0-2114.ga9a752df.el7.noarch 2024-04-14T23:47:40.613 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2114.ga9a752df.el7.noarch 2024-04-14T23:47:40.613 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-modules-core-2:19.0.0-2114.ga9a752df.el7.noarch 2024-04-14T23:47:40.613 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-rook-2:19.0.0-2114.ga9a752df.el7.noarch 2024-04-14T23:47:40.613 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mon-2:19.0.0-2114.ga9a752df.el7.x86_64 2024-04-14T23:47:40.613 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-osd-2:19.0.0-2114.ga9a752df.el7.x86_64 2024-04-14T23:47:40.613 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-prometheus-alerts-2:19.0.0-2114.ga9a752df.el7.noarch 2024-04-14T23:47:40.613 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-radosgw-2:19.0.0-2114.ga9a752df.el7.x86_64 2024-04-14T23:47:40.614 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-selinux-2:19.0.0-2114.ga9a752df.el7.x86_64 2024-04-14T23:47:40.614 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-test-2:19.0.0-2114.ga9a752df.el7.x86_64 2024-04-14T23:47:40.614 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-volume-2:19.0.0-2114.ga9a752df.el7.noarch 2024-04-14T23:47:40.614 INFO:teuthology.orchestra.run.smithi060.stdout: cephadm-2:19.0.0-2114.ga9a752df.el7.noarch 2024-04-14T23:47:40.614 INFO:teuthology.orchestra.run.smithi060.stdout: cephfs-mirror-2:19.0.0-2114.ga9a752df.el7.x86_64 2024-04-14T23:47:40.614 INFO:teuthology.orchestra.run.smithi060.stdout: cephfs-shell-2:19.0.0-2114.ga9a752df.el7.x86_64 2024-04-14T23:47:40.614 INFO:teuthology.orchestra.run.smithi060.stdout: cephfs-top-2:19.0.0-2114.ga9a752df.el7.noarch 2024-04-14T23:47:40.614 INFO:teuthology.orchestra.run.smithi060.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-14T23:47:40.614 INFO:teuthology.orchestra.run.smithi060.stdout: flex-2.6.4-9.el9.x86_64 2024-04-14T23:47:40.614 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-14T23:47:40.615 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-14T23:47:40.615 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-14T23:47:40.615 INFO:teuthology.orchestra.run.smithi060.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-14T23:47:40.615 INFO:teuthology.orchestra.run.smithi060.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-14T23:47:40.615 INFO:teuthology.orchestra.run.smithi060.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-14T23:47:40.615 INFO:teuthology.orchestra.run.smithi060.stdout: jq-1.6-16.el9.x86_64 2024-04-14T23:47:40.615 INFO:teuthology.orchestra.run.smithi060.stdout: libaio-devel-0.3.112-1.el8.x86_64 2024-04-14T23:47:40.615 INFO:teuthology.orchestra.run.smithi060.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-14T23:47:40.615 INFO:teuthology.orchestra.run.smithi060.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-14T23:47:40.615 INFO:teuthology.orchestra.run.smithi060.stdout: libcephfs-devel-2:19.0.0-2114.ga9a752df.el7.x86_64 2024-04-14T23:47:40.615 INFO:teuthology.orchestra.run.smithi060.stdout: libcephfs2-2:19.0.0-2114.ga9a752df.el7.x86_64 2024-04-14T23:47:40.615 INFO:teuthology.orchestra.run.smithi060.stdout: libcephsqlite-2:19.0.0-2114.ga9a752df.el7.x86_64 2024-04-14T23:47:40.615 INFO:teuthology.orchestra.run.smithi060.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-14T23:47:40.616 INFO:teuthology.orchestra.run.smithi060.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-14T23:47:40.616 INFO:teuthology.orchestra.run.smithi060.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-14T23:47:40.616 INFO:teuthology.orchestra.run.smithi060.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-14T23:47:40.616 INFO:teuthology.orchestra.run.smithi060.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-14T23:47:40.616 INFO:teuthology.orchestra.run.smithi060.stdout: librados-devel-2:19.0.0-2114.ga9a752df.el7.x86_64 2024-04-14T23:47:40.616 INFO:teuthology.orchestra.run.smithi060.stdout: libradosstriper1-2:19.0.0-2114.ga9a752df.el7.x86_64 2024-04-14T23:47:40.616 INFO:teuthology.orchestra.run.smithi060.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-14T23:47:40.616 INFO:teuthology.orchestra.run.smithi060.stdout: librgw2-2:19.0.0-2114.ga9a752df.el7.x86_64 2024-04-14T23:47:40.616 INFO:teuthology.orchestra.run.smithi060.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-14T23:47:40.616 INFO:teuthology.orchestra.run.smithi060.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-14T23:47:40.616 INFO:teuthology.orchestra.run.smithi060.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-14T23:47:40.616 INFO:teuthology.orchestra.run.smithi060.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-14T23:47:40.616 INFO:teuthology.orchestra.run.smithi060.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-14T23:47:40.617 INFO:teuthology.orchestra.run.smithi060.stdout: lua-5.4.4-4.el9.x86_64 2024-04-14T23:47:40.617 INFO:teuthology.orchestra.run.smithi060.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-14T23:47:40.617 INFO:teuthology.orchestra.run.smithi060.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-14T23:47:40.617 INFO:teuthology.orchestra.run.smithi060.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-14T23:47:40.617 INFO:teuthology.orchestra.run.smithi060.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-14T23:47:40.617 INFO:teuthology.orchestra.run.smithi060.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-14T23:47:40.617 INFO:teuthology.orchestra.run.smithi060.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-04-14T23:47:40.617 INFO:teuthology.orchestra.run.smithi060.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-14T23:47:40.617 INFO:teuthology.orchestra.run.smithi060.stdout: postgresql-13.14-1.el9.x86_64 2024-04-14T23:47:40.617 INFO:teuthology.orchestra.run.smithi060.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-14T23:47:40.617 INFO:teuthology.orchestra.run.smithi060.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-14T23:47:40.617 INFO:teuthology.orchestra.run.smithi060.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-14T23:47:40.618 INFO:teuthology.orchestra.run.smithi060.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-14T23:47:40.618 INFO:teuthology.orchestra.run.smithi060.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-14T23:47:40.618 INFO:teuthology.orchestra.run.smithi060.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-14T23:47:40.618 INFO:teuthology.orchestra.run.smithi060.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-14T23:47:40.618 INFO:teuthology.orchestra.run.smithi060.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-14T23:47:40.618 INFO:teuthology.orchestra.run.smithi060.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-14T23:47:40.618 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-14T23:47:40.618 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ceph-argparse-2:19.0.0-2114.ga9a752df.el7.x86_64 2024-04-14T23:47:40.618 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ceph-common-2:19.0.0-2114.ga9a752df.el7.x86_64 2024-04-14T23:47:40.618 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cephfs-2:19.0.0-2114.ga9a752df.el7.x86_64 2024-04-14T23:47:40.618 INFO:teuthology.orchestra.run.smithi060.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-14T23:47:40.618 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-14T23:47:40.619 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-14T23:47:40.619 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-14T23:47:40.619 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-14T23:47:40.619 INFO:teuthology.orchestra.run.smithi060.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-14T23:47:40.619 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-14T23:47:40.619 INFO:teuthology.orchestra.run.smithi060.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-14T23:47:40.619 INFO:teuthology.orchestra.run.smithi060.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-14T23:47:40.619 INFO:teuthology.orchestra.run.smithi060.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-14T23:47:40.619 INFO:teuthology.orchestra.run.smithi060.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-14T23:47:40.620 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-14T23:47:40.620 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-14T23:47:40.620 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-14T23:47:40.620 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-14T23:47:40.620 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-14T23:47:40.620 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-14T23:47:40.620 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-14T23:47:40.620 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-14T23:47:40.620 INFO:teuthology.orchestra.run.smithi060.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-14T23:47:40.620 INFO:teuthology.orchestra.run.smithi060.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-14T23:47:40.620 INFO:teuthology.orchestra.run.smithi060.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-14T23:47:40.620 INFO:teuthology.orchestra.run.smithi060.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-14T23:47:40.620 INFO:teuthology.orchestra.run.smithi060.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-14T23:47:40.620 INFO:teuthology.orchestra.run.smithi060.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-14T23:47:40.621 INFO:teuthology.orchestra.run.smithi060.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-14T23:47:40.621 INFO:teuthology.orchestra.run.smithi060.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-14T23:47:40.621 INFO:teuthology.orchestra.run.smithi060.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-14T23:47:40.621 INFO:teuthology.orchestra.run.smithi060.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-14T23:47:40.621 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-14T23:47:40.621 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ply-3.11-14.el9.noarch 2024-04-14T23:47:40.621 INFO:teuthology.orchestra.run.smithi060.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-14T23:47:40.621 INFO:teuthology.orchestra.run.smithi060.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-14T23:47:40.621 INFO:teuthology.orchestra.run.smithi060.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-14T23:47:40.621 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-14T23:47:40.621 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-14T23:47:40.621 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-14T23:47:40.621 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-14T23:47:40.621 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-14T23:47:40.622 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-14T23:47:40.622 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rados-2:19.0.0-2114.ga9a752df.el7.x86_64 2024-04-14T23:47:40.622 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rbd-2:19.0.0-2114.ga9a752df.el7.x86_64 2024-04-14T23:47:40.622 INFO:teuthology.orchestra.run.smithi060.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-14T23:47:40.622 INFO:teuthology.orchestra.run.smithi060.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-14T23:47:40.622 INFO:teuthology.orchestra.run.smithi060.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-14T23:47:40.622 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rgw-2:19.0.0-2114.ga9a752df.el7.x86_64 2024-04-14T23:47:40.622 INFO:teuthology.orchestra.run.smithi060.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-14T23:47:40.622 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-14T23:47:40.622 INFO:teuthology.orchestra.run.smithi060.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-14T23:47:40.622 INFO:teuthology.orchestra.run.smithi060.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-14T23:47:40.622 INFO:teuthology.orchestra.run.smithi060.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-14T23:47:40.622 INFO:teuthology.orchestra.run.smithi060.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-14T23:47:40.622 INFO:teuthology.orchestra.run.smithi060.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-14T23:47:40.623 INFO:teuthology.orchestra.run.smithi060.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-14T23:47:40.623 INFO:teuthology.orchestra.run.smithi060.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-14T23:47:40.623 INFO:teuthology.orchestra.run.smithi060.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-14T23:47:40.623 INFO:teuthology.orchestra.run.smithi060.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-14T23:47:40.623 INFO:teuthology.orchestra.run.smithi060.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-14T23:47:40.623 INFO:teuthology.orchestra.run.smithi060.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-14T23:47:40.623 INFO:teuthology.orchestra.run.smithi060.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-14T23:47:40.623 INFO:teuthology.orchestra.run.smithi060.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-14T23:47:40.623 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-fuse-2:19.0.0-2114.ga9a752df.el7.x86_64 2024-04-14T23:47:40.623 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-mirror-2:19.0.0-2114.ga9a752df.el7.x86_64 2024-04-14T23:47:40.623 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-nbd-2:19.0.0-2114.ga9a752df.el7.x86_64 2024-04-14T23:47:40.623 INFO:teuthology.orchestra.run.smithi060.stdout: re2-1:20211101-3.el9.x86_64 2024-04-14T23:47:40.623 INFO:teuthology.orchestra.run.smithi060.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-14T23:47:40.623 INFO:teuthology.orchestra.run.smithi060.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-14T23:47:40.624 INFO:teuthology.orchestra.run.smithi060.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-14T23:47:40.624 INFO:teuthology.orchestra.run.smithi060.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-14T23:47:40.624 INFO:teuthology.orchestra.run.smithi060.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-14T23:47:40.624 INFO:teuthology.orchestra.run.smithi060.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-04-14T23:47:40.624 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:47:40.624 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-04-14T23:47:41.285 DEBUG:teuthology.parallel:result is None 2024-04-14T23:47:41.286 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 2024-04-14T23:47:41.505 DEBUG:teuthology.orchestra.run.smithi060:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-14T23:47:41.564 INFO:teuthology.orchestra.run.smithi060.stdout:19.0.0-2114.ga9a752df.el7 2024-04-14T23:47:41.564 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2114.ga9a752df.el7 2024-04-14T23:47:41.564 INFO:teuthology.task.install:The correct ceph version 19.0.0-2114.ga9a752df is installed. 2024-04-14T23:47:41.567 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 2024-04-14T23:47:41.707 DEBUG:teuthology.orchestra.run.smithi186:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-14T23:47:41.733 INFO:teuthology.orchestra.run.smithi186.stdout:19.0.0-2114.ga9a752df.el7 2024-04-14T23:47:41.733 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2114.ga9a752df.el7 2024-04-14T23:47:41.733 INFO:teuthology.task.install:The correct ceph version 19.0.0-2114.ga9a752df is installed. 2024-04-14T23:47:41.735 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-14T23:47:41.736 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-04-14T23:47:41.736 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-14T23:47:41.835 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-14T23:47:41.836 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-14T23:47:41.870 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-14T23:47:41.870 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-04-14T23:47:41.870 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/usr/bin/daemon-helper 2024-04-14T23:47:41.949 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-14T23:47:42.063 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-14T23:47:42.063 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/daemon-helper 2024-04-14T23:47:42.094 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-14T23:47:42.165 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-14T23:47:42.166 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-04-14T23:47:42.166 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-14T23:47:42.241 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-14T23:47:42.354 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-14T23:47:42.354 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-14T23:47:42.386 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-14T23:47:42.458 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-14T23:47:42.458 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-04-14T23:47:42.458 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/usr/bin/stdin-killer 2024-04-14T23:47:42.539 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-14T23:47:42.661 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-14T23:47:42.661 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/stdin-killer 2024-04-14T23:47:42.693 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-14T23:47:42.765 INFO:teuthology.run_tasks:Running task ceph... 2024-04-14T23:47:42.858 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-14T23:47:42.858 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 777 /var/log/ceph 2024-04-14T23:47:42.863 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 777 /var/log/ceph 2024-04-14T23:47:42.938 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-14T23:47:42.938 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-14T23:47:42.942 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-14T23:47:43.010 INFO:tasks.ceph:Creating extra log directories... 2024-04-14T23:47:43.011 DEBUG:teuthology.orchestra.run.smithi060:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-14T23:47:43.015 DEBUG:teuthology.orchestra.run.smithi186:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-14T23:47:43.091 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-04-14T23:47:43.091 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-04-14T23:47:43.091 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-04-14T23:47:43.091 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-04-14T23:47:43.091 DEBUG:teuthology.orchestra.run.smithi060:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-04-14T23:47:43.133 DEBUG:teuthology.orchestra.run.smithi060:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-04-14T23:47:43.345 DEBUG:teuthology.orchestra.run.smithi060:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-04-14T23:47:43.417 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-14T23:47:43.417 DEBUG:teuthology.orchestra.run.smithi186:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-04-14T23:47:43.435 DEBUG:teuthology.orchestra.run.smithi186:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-04-14T23:47:43.558 DEBUG:teuthology.orchestra.run.smithi186:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-04-14T23:47:43.593 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-14T23:47:43.593 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': False, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'Replacing daemon mds'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-14T23:47:43.593 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-04-14_21:24:03-fs-squid-distro-default-smithi/7656585', 'branch': 'squid', 'description': 'fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7656585', 'kernel': {'client': {'branch': 'testing'}, 'kdb': 1, 'sha1': 'distro'}, 'ktype': 'upstream', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'mgrmodules': {'sequential': [{'print': 'Enabling mgr modules'}]}, 'name': 'teuthology-2024-04-14_21:24:03-fs-squid-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'cephfs': {'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': False, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'Replacing daemon mds'], 'sha1': 'a9a752df26c63acad72e1b3569fd79a515ca0765'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'a9a752df26c63acad72e1b3569fd79a515ca0765'}}, 'kclient': {'syntax': 'v1'}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'a9a752df26c63acad72e1b3569fd79a515ca0765'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']], 'seed': 840, 'sha1': 'a9a752df26c63acad72e1b3569fd79a515ca0765', 'sleep_before_teardown': 0, 'subset': '427/512', 'suite': 'fs', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_a9a752df26c63acad72e1b3569fd79a515ca0765/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'a9a752df26c63acad72e1b3569fd79a515ca0765', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi060.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFjV/pqyaEiSS62epvH8tgHmy+F/PJpE6WM74CJKJ2mX8YiPrJ7j+BVMPex/LpLCEZB9HKVb/OF6pAfvG9ajCuE=', 'smithi186.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLQ6oA9Y2BUpihFO/opwukGeh9/GE1Y47E4ZKJRZd++cP5yuL+qzcFkgQ6J4vfQSLvhNz3d0ALuw0/VE76drTXE='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'branch': 'testing'}, 'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'a9a752df26c63acad72e1b3569fd79a515ca0765'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': False, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'Replacing daemon mds'], 'sha1': 'a9a752df26c63acad72e1b3569fd79a515ca0765', 'cluster': 'ceph'}}, {'sequential': ['mgrmodules']}, {'kclient': None}, {'mds_thrash': None}, {'workunit': {'clients': {'all': ['fs/snaps']}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': '6c637841c215537a4502385240412f1966e0faab', 'timestamp': '2024-04-14_21:24:03', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-14T23:47:43.594 DEBUG:teuthology.orchestra.run.smithi060:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-14T23:47:43.599 DEBUG:teuthology.orchestra.run.smithi186:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-14T23:47:43.644 DEBUG:teuthology.orchestra.run.smithi060:> sudo install -d -m0777 -- /var/run/ceph 2024-04-14T23:47:43.685 DEBUG:teuthology.orchestra.run.smithi186:> sudo install -d -m0777 -- /var/run/ceph 2024-04-14T23:47:43.762 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-04-14T23:47:43.762 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/scratch_devs of=/dev/stdout 2024-04-14T23:47:43.804 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-14T23:47:43.805 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_1 2024-04-14T23:47:43.887 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-14T23:47:43.887 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-14T23:47:43.887 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 6h/6d Inode: 633 Links: 1 2024-04-14T23:47:43.887 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-14T23:47:43.887 INFO:teuthology.orchestra.run.smithi060.stdout:Context: system_u:object_r:device_t:s0 2024-04-14T23:47:43.887 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2024-04-14 23:47:32.826460499 +0000 2024-04-14T23:47:43.888 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2024-04-14 23:44:43.978552032 +0000 2024-04-14T23:47:43.888 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2024-04-14 23:44:43.978552032 +0000 2024-04-14T23:47:43.888 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: 2024-04-14 23:44:43.978552032 +0000 2024-04-14T23:47:43.888 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-14T23:47:44.021 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2024-04-14T23:47:44.022 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2024-04-14T23:47:44.022 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 0.000334052 s, 1.5 MB/s 2024-04-14T23:47:44.026 DEBUG:teuthology.orchestra.run.smithi060:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-14T23:47:44.071 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_2 2024-04-14T23:47:44.156 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-14T23:47:44.156 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-14T23:47:44.156 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 6h/6d Inode: 652 Links: 1 2024-04-14T23:47:44.156 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-14T23:47:44.156 INFO:teuthology.orchestra.run.smithi060.stdout:Context: system_u:object_r:device_t:s0 2024-04-14T23:47:44.156 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2024-04-14 23:47:32.828460479 +0000 2024-04-14T23:47:44.157 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2024-04-14 23:44:44.575546155 +0000 2024-04-14T23:47:44.157 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2024-04-14 23:44:44.575546155 +0000 2024-04-14T23:47:44.157 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: 2024-04-14 23:44:44.575546155 +0000 2024-04-14T23:47:44.157 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-14T23:47:44.266 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2024-04-14T23:47:44.266 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2024-04-14T23:47:44.266 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 0.00021953 s, 2.3 MB/s 2024-04-14T23:47:44.269 DEBUG:teuthology.orchestra.run.smithi060:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-14T23:47:44.313 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_3 2024-04-14T23:47:44.392 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-14T23:47:44.392 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-14T23:47:44.392 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 6h/6d Inode: 671 Links: 1 2024-04-14T23:47:44.392 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-14T23:47:44.392 INFO:teuthology.orchestra.run.smithi060.stdout:Context: system_u:object_r:device_t:s0 2024-04-14T23:47:44.392 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2024-04-14 23:47:32.829460469 +0000 2024-04-14T23:47:44.392 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2024-04-14 23:44:45.168540317 +0000 2024-04-14T23:47:44.393 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2024-04-14 23:44:45.168540317 +0000 2024-04-14T23:47:44.393 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: 2024-04-14 23:44:45.168540317 +0000 2024-04-14T23:47:44.393 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-14T23:47:44.504 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2024-04-14T23:47:44.504 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2024-04-14T23:47:44.504 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 0.000197341 s, 2.6 MB/s 2024-04-14T23:47:44.508 DEBUG:teuthology.orchestra.run.smithi060:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-14T23:47:44.550 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_4 2024-04-14T23:47:44.633 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-14T23:47:44.633 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-14T23:47:44.633 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 6h/6d Inode: 690 Links: 1 2024-04-14T23:47:44.633 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-14T23:47:44.633 INFO:teuthology.orchestra.run.smithi060.stdout:Context: system_u:object_r:device_t:s0 2024-04-14T23:47:44.633 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2024-04-14 23:47:32.829460469 +0000 2024-04-14T23:47:44.633 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2024-04-14 23:44:45.752534567 +0000 2024-04-14T23:47:44.634 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2024-04-14 23:44:45.752534567 +0000 2024-04-14T23:47:44.634 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: 2024-04-14 23:44:45.752534567 +0000 2024-04-14T23:47:44.634 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-14T23:47:44.769 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2024-04-14T23:47:44.770 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2024-04-14T23:47:44.770 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 0.000416194 s, 1.2 MB/s 2024-04-14T23:47:44.775 DEBUG:teuthology.orchestra.run.smithi060:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-14T23:47:44.830 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-14T23:47:44.830 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-14T23:47:44.830 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/scratch_devs of=/dev/stdout 2024-04-14T23:47:44.849 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-14T23:47:44.849 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_1 2024-04-14T23:47:44.907 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-14T23:47:44.907 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-14T23:47:44.908 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 662 Links: 1 2024-04-14T23:47:44.908 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-14T23:47:44.908 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-04-14T23:47:44.908 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-04-14 23:47:15.409019642 +0000 2024-04-14T23:47:44.908 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-04-14 23:43:16.218926313 +0000 2024-04-14T23:47:44.908 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-04-14 23:43:16.218926313 +0000 2024-04-14T23:47:44.908 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-04-14 23:43:16.218926313 +0000 2024-04-14T23:47:44.908 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-14T23:47:44.977 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-04-14T23:47:44.977 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-04-14T23:47:44.977 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000275773 s, 1.9 MB/s 2024-04-14T23:47:44.978 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-14T23:47:45.036 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_2 2024-04-14T23:47:45.093 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-14T23:47:45.093 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-14T23:47:45.093 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 682 Links: 1 2024-04-14T23:47:45.093 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-14T23:47:45.093 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-04-14T23:47:45.094 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-04-14 23:47:15.410019625 +0000 2024-04-14T23:47:45.094 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-04-14 23:43:16.557920314 +0000 2024-04-14T23:47:45.094 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-04-14 23:43:16.557920314 +0000 2024-04-14T23:47:45.094 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-04-14 23:43:16.557920314 +0000 2024-04-14T23:47:45.094 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-14T23:47:45.161 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-04-14T23:47:45.161 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-04-14T23:47:45.161 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000195344 s, 2.6 MB/s 2024-04-14T23:47:45.162 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-14T23:47:45.219 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_3 2024-04-14T23:47:45.275 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-14T23:47:45.275 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-14T23:47:45.276 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-14T23:47:45.276 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-14T23:47:45.276 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-04-14T23:47:45.276 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-04-14 23:47:15.410019625 +0000 2024-04-14T23:47:45.276 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-04-14 23:43:16.906914138 +0000 2024-04-14T23:47:45.276 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-04-14 23:43:16.906914138 +0000 2024-04-14T23:47:45.276 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-04-14 23:43:16.906914138 +0000 2024-04-14T23:47:45.276 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-14T23:47:45.342 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-04-14T23:47:45.342 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-04-14T23:47:45.342 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.00019248 s, 2.7 MB/s 2024-04-14T23:47:45.343 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-14T23:47:45.400 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_4 2024-04-14T23:47:45.455 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-14T23:47:45.455 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-14T23:47:45.455 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 720 Links: 1 2024-04-14T23:47:45.455 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-14T23:47:45.455 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-04-14T23:47:45.456 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-04-14 23:47:15.411019607 +0000 2024-04-14T23:47:45.456 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-04-14 23:43:17.254907979 +0000 2024-04-14T23:47:45.456 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-04-14 23:43:17.254907979 +0000 2024-04-14T23:47:45.456 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-04-14 23:43:17.254907979 +0000 2024-04-14T23:47:45.456 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-14T23:47:45.520 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-04-14T23:47:45.521 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-04-14T23:47:45.521 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000172871 s, 3.0 MB/s 2024-04-14T23:47:45.522 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-14T23:47:45.578 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-14T23:47:45.578 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi060.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi186.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-14T23:47:45.578 INFO:tasks.ceph:Generating config... 2024-04-14T23:47:45.579 INFO:tasks.ceph:[client] client mount timeout = 600 2024-04-14T23:47:45.579 INFO:tasks.ceph:[client] client_shutdown_timeout = 120 2024-04-14T23:47:45.579 INFO:tasks.ceph:[client] debug client = 20 2024-04-14T23:47:45.579 INFO:tasks.ceph:[client] debug ms = 1 2024-04-14T23:47:45.579 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-04-14T23:47:45.580 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-04-14T23:47:45.580 INFO:tasks.ceph:[global] ms die on skipped message = False 2024-04-14T23:47:45.580 INFO:tasks.ceph:[mds] debug mds = 20 2024-04-14T23:47:45.580 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-04-14T23:47:45.580 INFO:tasks.ceph:[mds] debug ms = 1 2024-04-14T23:47:45.580 INFO:tasks.ceph:[mds] mds bal fragment size max = 10000 2024-04-14T23:47:45.580 INFO:tasks.ceph:[mds] mds bal merge size = 5 2024-04-14T23:47:45.580 INFO:tasks.ceph:[mds] mds bal split bits = 3 2024-04-14T23:47:45.580 INFO:tasks.ceph:[mds] mds bal split size = 100 2024-04-14T23:47:45.580 INFO:tasks.ceph:[mds] mds debug frag = True 2024-04-14T23:47:45.580 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-04-14T23:47:45.580 INFO:tasks.ceph:[mds] mds oft prefetch dirfrags = True 2024-04-14T23:47:45.580 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-04-14T23:47:45.580 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-04-14T23:47:45.580 INFO:tasks.ceph:[mds] mds_dir_prefetch = False 2024-04-14T23:47:45.581 INFO:tasks.ceph:[mds] mds_sleep_rank_change = 5000000.0 2024-04-14T23:47:45.581 INFO:tasks.ceph:[mds] mon client directed command retry = 5 2024-04-14T23:47:45.581 INFO:tasks.ceph:[mds] ms inject delay max = 1 2024-04-14T23:47:45.581 INFO:tasks.ceph:[mds] ms inject delay probability = 0.005 2024-04-14T23:47:45.581 INFO:tasks.ceph:[mds] ms inject delay type = client mds osd 2024-04-14T23:47:45.581 INFO:tasks.ceph:[mds] ms inject socket failures = 2500 2024-04-14T23:47:45.581 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-04-14T23:47:45.581 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-04-14T23:47:45.581 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-04-14T23:47:45.581 INFO:tasks.ceph:[mgr] debug client = 20 2024-04-14T23:47:45.581 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-14T23:47:45.581 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-14T23:47:45.582 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-14T23:47:45.582 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-14T23:47:45.582 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-14T23:47:45.582 INFO:tasks.ceph:[mon] mon down mkfs grace = 300 2024-04-14T23:47:45.582 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-04-14T23:47:45.582 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-14T23:47:45.582 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-04-14T23:47:45.582 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-14T23:47:45.582 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-04-14T23:47:45.582 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-04-14T23:47:45.582 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-14T23:47:45.582 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-14T23:47:45.582 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-04-14T23:47:45.582 INFO:tasks.ceph:[osd] mon client directed command retry = 5 2024-04-14T23:47:45.582 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-14T23:47:45.583 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-14T23:47:45.583 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-14T23:47:45.583 INFO:tasks.ceph:[osd] ms inject delay max = 1 2024-04-14T23:47:45.583 INFO:tasks.ceph:[osd] ms inject delay probability = 0.005 2024-04-14T23:47:45.583 INFO:tasks.ceph:[osd] ms inject delay type = client mds 2024-04-14T23:47:45.583 INFO:tasks.ceph:[osd] ms inject socket failures = 2500 2024-04-14T23:47:45.583 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-04-14T23:47:45.583 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-14T23:47:45.583 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-04-14T23:47:45.583 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-14T23:47:45.583 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-04-14T23:47:45.583 INFO:tasks.ceph:Setting up mon.a... 2024-04-14T23:47:45.584 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-14T23:47:45.745 INFO:teuthology.orchestra.run.smithi060.stdout:creating /etc/ceph/ceph.keyring 2024-04-14T23:47:45.753 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-14T23:47:45.888 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-14T23:47:45.990 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.60'), ('mon.b', '172.21.15.186'), ('mon.c', '[v2:172.21.15.186:3301,v1:172.21.15.186:6790]')] 2024-04-14T23:47:45.991 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.60,172.21.15.186,[v2:172.21.15.186:3301,v1:172.21.15.186:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.e': {}, 'mds.b': {}, 'mds.d': {}, 'mds.f': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': False, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-04-14T23:47:45.992 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-04-14T23:47:45.992 DEBUG:teuthology.orchestra.run.smithi060:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-14T23:47:46.048 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.60 --add b 172.21.15.186 --addv c '[v2:172.21.15.186:3301,v1:172.21.15.186:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-04-14T23:47:46.164 INFO:teuthology.orchestra.run.smithi060.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-14T23:47:46.165 INFO:teuthology.orchestra.run.smithi060.stdout:monmaptool: generated fsid 5847db56-a76d-4486-b563-d83f1186af92 2024-04-14T23:47:46.165 INFO:teuthology.orchestra.run.smithi060.stdout:setting min_mon_release = quincy 2024-04-14T23:47:46.165 INFO:teuthology.orchestra.run.smithi060.stdout:epoch 0 2024-04-14T23:47:46.165 INFO:teuthology.orchestra.run.smithi060.stdout:fsid 5847db56-a76d-4486-b563-d83f1186af92 2024-04-14T23:47:46.165 INFO:teuthology.orchestra.run.smithi060.stdout:last_changed 2024-04-14T23:47:46.161032+0000 2024-04-14T23:47:46.165 INFO:teuthology.orchestra.run.smithi060.stdout:created 2024-04-14T23:47:46.161032+0000 2024-04-14T23:47:46.165 INFO:teuthology.orchestra.run.smithi060.stdout:min_mon_release 17 (quincy) 2024-04-14T23:47:46.165 INFO:teuthology.orchestra.run.smithi060.stdout:election_strategy: 1 2024-04-14T23:47:46.165 INFO:teuthology.orchestra.run.smithi060.stdout:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-04-14T23:47:46.165 INFO:teuthology.orchestra.run.smithi060.stdout:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-14T23:47:46.165 INFO:teuthology.orchestra.run.smithi060.stdout:2: [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] mon.c 2024-04-14T23:47:46.165 INFO:teuthology.orchestra.run.smithi060.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-04-14T23:47:46.168 DEBUG:teuthology.orchestra.run.smithi060:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-14T23:47:46.221 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 5847db56-a76d-4486-b563-d83f1186af92... 2024-04-14T23:47:46.222 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-14T23:47:46.227 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-14T23:47:46.280 INFO:teuthology.orchestra.run.smithi186.stdout:[global] 2024-04-14T23:47:46.280 INFO:teuthology.orchestra.run.smithi186.stdout: chdir = "" 2024-04-14T23:47:46.280 INFO:teuthology.orchestra.run.smithi186.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-14T23:47:46.280 INFO:teuthology.orchestra.run.smithi186.stdout: auth supported = cephx 2024-04-14T23:47:46.280 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-14T23:47:46.280 INFO:teuthology.orchestra.run.smithi186.stdout: filestore xattr use omap = true 2024-04-14T23:47:46.280 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-14T23:47:46.280 INFO:teuthology.orchestra.run.smithi186.stdout: mon clock drift allowed = 1.000 2024-04-14T23:47:46.280 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-14T23:47:46.280 INFO:teuthology.orchestra.run.smithi186.stdout: osd crush chooseleaf type = 0 2024-04-14T23:47:46.281 INFO:teuthology.orchestra.run.smithi186.stdout: auth debug = true 2024-04-14T23:47:46.281 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-14T23:47:46.281 INFO:teuthology.orchestra.run.smithi186.stdout: ms die on old message = true 2024-04-14T23:47:46.281 INFO:teuthology.orchestra.run.smithi186.stdout: ms die on bug = true 2024-04-14T23:47:46.281 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-14T23:47:46.281 INFO:teuthology.orchestra.run.smithi186.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-14T23:47:46.281 INFO:teuthology.orchestra.run.smithi186.stdout: mon pg warn max object skew = 0 2024-04-14T23:47:46.281 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-14T23:47:46.281 INFO:teuthology.orchestra.run.smithi186.stdout: # disable pg_autoscaler by default for new pools 2024-04-14T23:47:46.281 INFO:teuthology.orchestra.run.smithi186.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-14T23:47:46.281 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-14T23:47:46.281 INFO:teuthology.orchestra.run.smithi186.stdout: osd pool default size = 2 2024-04-14T23:47:46.281 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-14T23:47:46.281 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd allow primary affinity = true 2024-04-14T23:47:46.282 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd allow pg remap = true 2024-04-14T23:47:46.282 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on legacy crush tunables = false 2024-04-14T23:47:46.282 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on crush straw calc version zero = false 2024-04-14T23:47:46.282 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on no sortbitwise = false 2024-04-14T23:47:46.282 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on osd down out interval zero = false 2024-04-14T23:47:46.282 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on too few osds = false 2024-04-14T23:47:46.282 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-14T23:47:46.282 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-14T23:47:46.282 INFO:teuthology.orchestra.run.smithi186.stdout: mon_allow_pool_size_one = true 2024-04-14T23:47:46.282 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-14T23:47:46.282 INFO:teuthology.orchestra.run.smithi186.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-14T23:47:46.282 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-14T23:47:46.282 INFO:teuthology.orchestra.run.smithi186.stdout: osd default data pool replay window = 5 2024-04-14T23:47:46.282 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-14T23:47:46.282 INFO:teuthology.orchestra.run.smithi186.stdout: mon allow pool delete = true 2024-04-14T23:47:46.283 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-14T23:47:46.283 INFO:teuthology.orchestra.run.smithi186.stdout: mon cluster log file level = debug 2024-04-14T23:47:46.283 INFO:teuthology.orchestra.run.smithi186.stdout: debug asserts on shutdown = true 2024-04-14T23:47:46.283 INFO:teuthology.orchestra.run.smithi186.stdout: mon health detail to clog = false 2024-04-14T23:47:46.283 INFO:teuthology.orchestra.run.smithi186.stdout: mon host = "172.21.15.60,172.21.15.186,[v2:172.21.15.186:3301,v1:172.21.15.186:6790]" 2024-04-14T23:47:46.283 INFO:teuthology.orchestra.run.smithi186.stdout: ms die on skipped message = False 2024-04-14T23:47:46.283 INFO:teuthology.orchestra.run.smithi186.stdout: fsid = 5847db56-a76d-4486-b563-d83f1186af92 2024-04-14T23:47:46.283 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-14T23:47:46.283 INFO:teuthology.orchestra.run.smithi186.stdout:[osd] 2024-04-14T23:47:46.283 INFO:teuthology.orchestra.run.smithi186.stdout: osd journal size = 100 2024-04-14T23:47:46.283 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-14T23:47:46.283 INFO:teuthology.orchestra.run.smithi186.stdout: osd scrub load threshold = 5.0 2024-04-14T23:47:46.284 INFO:teuthology.orchestra.run.smithi186.stdout: osd scrub max interval = 600 2024-04-14T23:47:46.284 INFO:teuthology.orchestra.run.smithi186.stdout: osd mclock profile = high_recovery_ops 2024-04-14T23:47:46.284 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-14T23:47:46.284 INFO:teuthology.orchestra.run.smithi186.stdout: osd recover clone overlap = true 2024-04-14T23:47:46.284 INFO:teuthology.orchestra.run.smithi186.stdout: osd recovery max chunk = 1048576 2024-04-14T23:47:46.284 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-14T23:47:46.284 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug shutdown = true 2024-04-14T23:47:46.284 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug op order = true 2024-04-14T23:47:46.284 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug verify stray on activate = true 2024-04-14T23:47:46.284 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug trim objects = true 2024-04-14T23:47:46.284 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-14T23:47:46.284 INFO:teuthology.orchestra.run.smithi186.stdout: osd open classes on start = true 2024-04-14T23:47:46.284 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug pg log writeout = true 2024-04-14T23:47:46.284 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-14T23:47:46.285 INFO:teuthology.orchestra.run.smithi186.stdout: osd deep scrub update digest min age = 30 2024-04-14T23:47:46.285 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-14T23:47:46.285 INFO:teuthology.orchestra.run.smithi186.stdout: osd map max advance = 10 2024-04-14T23:47:46.285 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-14T23:47:46.285 INFO:teuthology.orchestra.run.smithi186.stdout: journal zero on create = true 2024-04-14T23:47:46.285 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-14T23:47:46.285 INFO:teuthology.orchestra.run.smithi186.stdout: filestore ondisk finisher threads = 3 2024-04-14T23:47:46.285 INFO:teuthology.orchestra.run.smithi186.stdout: filestore apply finisher threads = 3 2024-04-14T23:47:46.285 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-14T23:47:46.286 INFO:teuthology.orchestra.run.smithi186.stdout: bdev debug aio = true 2024-04-14T23:47:46.286 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug misdirected ops = true 2024-04-14T23:47:46.286 INFO:teuthology.orchestra.run.smithi186.stdout: bluestore block size = 96636764160 2024-04-14T23:47:46.286 INFO:teuthology.orchestra.run.smithi186.stdout: bluestore compression mode = aggressive 2024-04-14T23:47:46.286 INFO:teuthology.orchestra.run.smithi186.stdout: bluestore fsck on mount = True 2024-04-14T23:47:46.286 INFO:teuthology.orchestra.run.smithi186.stdout: debug bluefs = 20 2024-04-14T23:47:46.286 INFO:teuthology.orchestra.run.smithi186.stdout: debug bluestore = 20 2024-04-14T23:47:46.286 INFO:teuthology.orchestra.run.smithi186.stdout: debug ms = 1 2024-04-14T23:47:46.286 INFO:teuthology.orchestra.run.smithi186.stdout: debug osd = 20 2024-04-14T23:47:46.286 INFO:teuthology.orchestra.run.smithi186.stdout: debug rocksdb = 10 2024-04-14T23:47:46.286 INFO:teuthology.orchestra.run.smithi186.stdout: mon client directed command retry = 5 2024-04-14T23:47:46.286 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd backfillfull_ratio = 0.85 2024-04-14T23:47:46.287 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd full ratio = 0.9 2024-04-14T23:47:46.287 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd nearfull ratio = 0.8 2024-04-14T23:47:46.287 INFO:teuthology.orchestra.run.smithi186.stdout: ms inject delay max = 1 2024-04-14T23:47:46.287 INFO:teuthology.orchestra.run.smithi186.stdout: ms inject delay probability = 0.005 2024-04-14T23:47:46.287 INFO:teuthology.orchestra.run.smithi186.stdout: ms inject delay type = client mds 2024-04-14T23:47:46.287 INFO:teuthology.orchestra.run.smithi186.stdout: ms inject socket failures = 2500 2024-04-14T23:47:46.287 INFO:teuthology.orchestra.run.smithi186.stdout: osd blocked scrub grace period = 3600 2024-04-14T23:47:46.287 INFO:teuthology.orchestra.run.smithi186.stdout: osd failsafe full ratio = 0.95 2024-04-14T23:47:46.287 INFO:teuthology.orchestra.run.smithi186.stdout: osd max markdown count = 1000 2024-04-14T23:47:46.287 INFO:teuthology.orchestra.run.smithi186.stdout: osd objectstore = bluestore 2024-04-14T23:47:46.287 INFO:teuthology.orchestra.run.smithi186.stdout: osd op complaint time = 180 2024-04-14T23:47:46.287 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-14T23:47:46.287 INFO:teuthology.orchestra.run.smithi186.stdout:[mgr] 2024-04-14T23:47:46.287 INFO:teuthology.orchestra.run.smithi186.stdout: debug ms = 1 2024-04-14T23:47:46.288 INFO:teuthology.orchestra.run.smithi186.stdout: debug mgr = 20 2024-04-14T23:47:46.288 INFO:teuthology.orchestra.run.smithi186.stdout: debug mon = 20 2024-04-14T23:47:46.288 INFO:teuthology.orchestra.run.smithi186.stdout: debug auth = 20 2024-04-14T23:47:46.288 INFO:teuthology.orchestra.run.smithi186.stdout: mon reweight min pgs per osd = 4 2024-04-14T23:47:46.288 INFO:teuthology.orchestra.run.smithi186.stdout: mon reweight min bytes per osd = 10 2024-04-14T23:47:46.288 INFO:teuthology.orchestra.run.smithi186.stdout: mgr/telemetry/nag = false 2024-04-14T23:47:46.288 INFO:teuthology.orchestra.run.smithi186.stdout: debug client = 20 2024-04-14T23:47:46.288 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-14T23:47:46.288 INFO:teuthology.orchestra.run.smithi186.stdout:[mon] 2024-04-14T23:47:46.288 INFO:teuthology.orchestra.run.smithi186.stdout: debug ms = 1 2024-04-14T23:47:46.288 INFO:teuthology.orchestra.run.smithi186.stdout: debug mon = 20 2024-04-14T23:47:46.288 INFO:teuthology.orchestra.run.smithi186.stdout: debug paxos = 20 2024-04-14T23:47:46.288 INFO:teuthology.orchestra.run.smithi186.stdout: debug auth = 20 2024-04-14T23:47:46.288 INFO:teuthology.orchestra.run.smithi186.stdout: mon data avail warn = 5 2024-04-14T23:47:46.288 INFO:teuthology.orchestra.run.smithi186.stdout: mon mgr mkfs grace = 240 2024-04-14T23:47:46.289 INFO:teuthology.orchestra.run.smithi186.stdout: mon reweight min pgs per osd = 4 2024-04-14T23:47:46.289 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd reporter subtree level = osd 2024-04-14T23:47:46.289 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd prime pg temp = true 2024-04-14T23:47:46.289 INFO:teuthology.orchestra.run.smithi186.stdout: mon reweight min bytes per osd = 10 2024-04-14T23:47:46.289 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-14T23:47:46.289 INFO:teuthology.orchestra.run.smithi186.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-14T23:47:46.289 INFO:teuthology.orchestra.run.smithi186.stdout: auth mon ticket ttl = 660 # 11m 2024-04-14T23:47:46.289 INFO:teuthology.orchestra.run.smithi186.stdout: auth service ticket ttl = 240 # 4m 2024-04-14T23:47:46.289 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-14T23:47:46.289 INFO:teuthology.orchestra.run.smithi186.stdout: # don't complain about insecure global_id in the test suite 2024-04-14T23:47:46.289 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-14T23:47:46.289 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-14T23:47:46.289 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-14T23:47:46.289 INFO:teuthology.orchestra.run.smithi186.stdout: # 1m isn't quite enough 2024-04-14T23:47:46.289 INFO:teuthology.orchestra.run.smithi186.stdout: mon_down_mkfs_grace = 2m 2024-04-14T23:47:46.290 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-14T23:47:46.290 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_filestore_osds = false 2024-04-14T23:47:46.290 INFO:teuthology.orchestra.run.smithi186.stdout: mon down mkfs grace = 300 2024-04-14T23:47:46.290 INFO:teuthology.orchestra.run.smithi186.stdout: mon op complaint time = 120 2024-04-14T23:47:46.290 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-14T23:47:46.290 INFO:teuthology.orchestra.run.smithi186.stdout:[client] 2024-04-14T23:47:46.290 INFO:teuthology.orchestra.run.smithi186.stdout: rgw cache enabled = true 2024-04-14T23:47:46.290 INFO:teuthology.orchestra.run.smithi186.stdout: rgw enable ops log = true 2024-04-14T23:47:46.290 INFO:teuthology.orchestra.run.smithi186.stdout: rgw enable usage log = true 2024-04-14T23:47:46.290 INFO:teuthology.orchestra.run.smithi186.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-14T23:47:46.290 INFO:teuthology.orchestra.run.smithi186.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-14T23:47:46.290 INFO:teuthology.orchestra.run.smithi186.stdout: client mount timeout = 600 2024-04-14T23:47:46.292 INFO:teuthology.orchestra.run.smithi186.stdout: client_shutdown_timeout = 120 2024-04-14T23:47:46.292 INFO:teuthology.orchestra.run.smithi186.stdout: debug client = 20 2024-04-14T23:47:46.292 INFO:teuthology.orchestra.run.smithi186.stdout: debug ms = 1 2024-04-14T23:47:46.292 INFO:teuthology.orchestra.run.smithi186.stdout: rados mon op timeout = 900 2024-04-14T23:47:46.292 INFO:teuthology.orchestra.run.smithi186.stdout: rados osd op timeout = 900 2024-04-14T23:47:46.292 INFO:teuthology.orchestra.run.smithi186.stdout:[mon.a] 2024-04-14T23:47:46.292 INFO:teuthology.orchestra.run.smithi186.stdout:[mon.b] 2024-04-14T23:47:46.292 INFO:teuthology.orchestra.run.smithi186.stdout:[mon.c] 2024-04-14T23:47:46.292 INFO:teuthology.orchestra.run.smithi186.stdout:[mds.a] 2024-04-14T23:47:46.292 INFO:teuthology.orchestra.run.smithi186.stdout:[mds.c] 2024-04-14T23:47:46.292 INFO:teuthology.orchestra.run.smithi186.stdout:[mds.e] 2024-04-14T23:47:46.292 INFO:teuthology.orchestra.run.smithi186.stdout:[mds.b] 2024-04-14T23:47:46.292 INFO:teuthology.orchestra.run.smithi186.stdout:[mds.d] 2024-04-14T23:47:46.293 INFO:teuthology.orchestra.run.smithi186.stdout:[mds.f] 2024-04-14T23:47:46.293 INFO:teuthology.orchestra.run.smithi186.stdout:[mds] 2024-04-14T23:47:46.293 INFO:teuthology.orchestra.run.smithi186.stdout: debug mds = 20 2024-04-14T23:47:46.293 INFO:teuthology.orchestra.run.smithi186.stdout: debug mds balancer = 20 2024-04-14T23:47:46.293 INFO:teuthology.orchestra.run.smithi186.stdout: debug ms = 1 2024-04-14T23:47:46.293 INFO:teuthology.orchestra.run.smithi186.stdout: mds bal fragment size max = 10000 2024-04-14T23:47:46.293 INFO:teuthology.orchestra.run.smithi186.stdout: mds bal merge size = 5 2024-04-14T23:47:46.293 INFO:teuthology.orchestra.run.smithi186.stdout: mds bal split bits = 3 2024-04-14T23:47:46.293 INFO:teuthology.orchestra.run.smithi186.stdout: mds bal split size = 100 2024-04-14T23:47:46.293 INFO:teuthology.orchestra.run.smithi186.stdout: mds debug frag = True 2024-04-14T23:47:46.293 INFO:teuthology.orchestra.run.smithi186.stdout: mds debug scatterstat = True 2024-04-14T23:47:46.293 INFO:teuthology.orchestra.run.smithi186.stdout: mds oft prefetch dirfrags = True 2024-04-14T23:47:46.293 INFO:teuthology.orchestra.run.smithi186.stdout: mds op complaint time = 180 2024-04-14T23:47:46.293 INFO:teuthology.orchestra.run.smithi186.stdout: mds verify scatter = True 2024-04-14T23:47:46.293 INFO:teuthology.orchestra.run.smithi186.stdout: mds_dir_prefetch = False 2024-04-14T23:47:46.294 INFO:teuthology.orchestra.run.smithi186.stdout: mds_sleep_rank_change = 5000000.0 2024-04-14T23:47:46.294 INFO:teuthology.orchestra.run.smithi186.stdout: mon client directed command retry = 5 2024-04-14T23:47:46.294 INFO:teuthology.orchestra.run.smithi186.stdout: ms inject delay max = 1 2024-04-14T23:47:46.294 INFO:teuthology.orchestra.run.smithi186.stdout: ms inject delay probability = 0.005 2024-04-14T23:47:46.294 INFO:teuthology.orchestra.run.smithi186.stdout: ms inject delay type = client mds osd 2024-04-14T23:47:46.294 INFO:teuthology.orchestra.run.smithi186.stdout: ms inject socket failures = 2500 2024-04-14T23:47:46.294 INFO:teuthology.orchestra.run.smithi186.stdout: osd op complaint time = 180 2024-04-14T23:47:46.294 INFO:teuthology.orchestra.run.smithi186.stdout: rados mon op timeout = 900 2024-04-14T23:47:46.294 INFO:teuthology.orchestra.run.smithi186.stdout: rados osd op timeout = 900 2024-04-14T23:47:46.403 INFO:teuthology.orchestra.run.smithi060.stdout:[global] 2024-04-14T23:47:46.403 INFO:teuthology.orchestra.run.smithi060.stdout: chdir = "" 2024-04-14T23:47:46.403 INFO:teuthology.orchestra.run.smithi060.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-14T23:47:46.403 INFO:teuthology.orchestra.run.smithi060.stdout: auth supported = cephx 2024-04-14T23:47:46.403 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:47:46.403 INFO:teuthology.orchestra.run.smithi060.stdout: filestore xattr use omap = true 2024-04-14T23:47:46.403 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:47:46.404 INFO:teuthology.orchestra.run.smithi060.stdout: mon clock drift allowed = 1.000 2024-04-14T23:47:46.404 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:47:46.404 INFO:teuthology.orchestra.run.smithi060.stdout: osd crush chooseleaf type = 0 2024-04-14T23:47:46.404 INFO:teuthology.orchestra.run.smithi060.stdout: auth debug = true 2024-04-14T23:47:46.404 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:47:46.404 INFO:teuthology.orchestra.run.smithi060.stdout: ms die on old message = true 2024-04-14T23:47:46.404 INFO:teuthology.orchestra.run.smithi060.stdout: ms die on bug = true 2024-04-14T23:47:46.404 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:47:46.404 INFO:teuthology.orchestra.run.smithi060.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-14T23:47:46.404 INFO:teuthology.orchestra.run.smithi060.stdout: mon pg warn max object skew = 0 2024-04-14T23:47:46.404 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:47:46.404 INFO:teuthology.orchestra.run.smithi060.stdout: # disable pg_autoscaler by default for new pools 2024-04-14T23:47:46.404 INFO:teuthology.orchestra.run.smithi060.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-14T23:47:46.404 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:47:46.404 INFO:teuthology.orchestra.run.smithi060.stdout: osd pool default size = 2 2024-04-14T23:47:46.405 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:47:46.405 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd allow primary affinity = true 2024-04-14T23:47:46.405 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd allow pg remap = true 2024-04-14T23:47:46.405 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on legacy crush tunables = false 2024-04-14T23:47:46.405 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on crush straw calc version zero = false 2024-04-14T23:47:46.405 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on no sortbitwise = false 2024-04-14T23:47:46.405 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on osd down out interval zero = false 2024-04-14T23:47:46.405 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on too few osds = false 2024-04-14T23:47:46.405 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-14T23:47:46.405 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-14T23:47:46.405 INFO:teuthology.orchestra.run.smithi060.stdout: mon_allow_pool_size_one = true 2024-04-14T23:47:46.405 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:47:46.405 INFO:teuthology.orchestra.run.smithi060.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-14T23:47:46.405 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:47:46.405 INFO:teuthology.orchestra.run.smithi060.stdout: osd default data pool replay window = 5 2024-04-14T23:47:46.406 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:47:46.406 INFO:teuthology.orchestra.run.smithi060.stdout: mon allow pool delete = true 2024-04-14T23:47:46.406 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:47:46.406 INFO:teuthology.orchestra.run.smithi060.stdout: mon cluster log file level = debug 2024-04-14T23:47:46.406 INFO:teuthology.orchestra.run.smithi060.stdout: debug asserts on shutdown = true 2024-04-14T23:47:46.406 INFO:teuthology.orchestra.run.smithi060.stdout: mon health detail to clog = false 2024-04-14T23:47:46.406 INFO:teuthology.orchestra.run.smithi060.stdout: mon host = "172.21.15.60,172.21.15.186,[v2:172.21.15.186:3301,v1:172.21.15.186:6790]" 2024-04-14T23:47:46.406 INFO:teuthology.orchestra.run.smithi060.stdout: ms die on skipped message = False 2024-04-14T23:47:46.406 INFO:teuthology.orchestra.run.smithi060.stdout: fsid = 5847db56-a76d-4486-b563-d83f1186af92 2024-04-14T23:47:46.406 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:47:46.406 INFO:teuthology.orchestra.run.smithi060.stdout:[osd] 2024-04-14T23:47:46.406 INFO:teuthology.orchestra.run.smithi060.stdout: osd journal size = 100 2024-04-14T23:47:46.406 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:47:46.406 INFO:teuthology.orchestra.run.smithi060.stdout: osd scrub load threshold = 5.0 2024-04-14T23:47:46.407 INFO:teuthology.orchestra.run.smithi060.stdout: osd scrub max interval = 600 2024-04-14T23:47:46.407 INFO:teuthology.orchestra.run.smithi060.stdout: osd mclock profile = high_recovery_ops 2024-04-14T23:47:46.407 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:47:46.407 INFO:teuthology.orchestra.run.smithi060.stdout: osd recover clone overlap = true 2024-04-14T23:47:46.407 INFO:teuthology.orchestra.run.smithi060.stdout: osd recovery max chunk = 1048576 2024-04-14T23:47:46.407 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:47:46.407 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug shutdown = true 2024-04-14T23:47:46.407 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug op order = true 2024-04-14T23:47:46.407 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug verify stray on activate = true 2024-04-14T23:47:46.407 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug trim objects = true 2024-04-14T23:47:46.407 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:47:46.407 INFO:teuthology.orchestra.run.smithi060.stdout: osd open classes on start = true 2024-04-14T23:47:46.407 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug pg log writeout = true 2024-04-14T23:47:46.407 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:47:46.407 INFO:teuthology.orchestra.run.smithi060.stdout: osd deep scrub update digest min age = 30 2024-04-14T23:47:46.408 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:47:46.408 INFO:teuthology.orchestra.run.smithi060.stdout: osd map max advance = 10 2024-04-14T23:47:46.408 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:47:46.408 INFO:teuthology.orchestra.run.smithi060.stdout: journal zero on create = true 2024-04-14T23:47:46.408 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:47:46.408 INFO:teuthology.orchestra.run.smithi060.stdout: filestore ondisk finisher threads = 3 2024-04-14T23:47:46.408 INFO:teuthology.orchestra.run.smithi060.stdout: filestore apply finisher threads = 3 2024-04-14T23:47:46.408 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:47:46.408 INFO:teuthology.orchestra.run.smithi060.stdout: bdev debug aio = true 2024-04-14T23:47:46.408 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug misdirected ops = true 2024-04-14T23:47:46.408 INFO:teuthology.orchestra.run.smithi060.stdout: bluestore block size = 96636764160 2024-04-14T23:47:46.408 INFO:teuthology.orchestra.run.smithi060.stdout: bluestore compression mode = aggressive 2024-04-14T23:47:46.408 INFO:teuthology.orchestra.run.smithi060.stdout: bluestore fsck on mount = True 2024-04-14T23:47:46.408 INFO:teuthology.orchestra.run.smithi060.stdout: debug bluefs = 20 2024-04-14T23:47:46.408 INFO:teuthology.orchestra.run.smithi060.stdout: debug bluestore = 20 2024-04-14T23:47:46.408 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2024-04-14T23:47:46.409 INFO:teuthology.orchestra.run.smithi060.stdout: debug osd = 20 2024-04-14T23:47:46.409 INFO:teuthology.orchestra.run.smithi060.stdout: debug rocksdb = 10 2024-04-14T23:47:46.409 INFO:teuthology.orchestra.run.smithi060.stdout: mon client directed command retry = 5 2024-04-14T23:47:46.409 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd backfillfull_ratio = 0.85 2024-04-14T23:47:46.409 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd full ratio = 0.9 2024-04-14T23:47:46.409 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd nearfull ratio = 0.8 2024-04-14T23:47:46.409 INFO:teuthology.orchestra.run.smithi060.stdout: ms inject delay max = 1 2024-04-14T23:47:46.409 INFO:teuthology.orchestra.run.smithi060.stdout: ms inject delay probability = 0.005 2024-04-14T23:47:46.409 INFO:teuthology.orchestra.run.smithi060.stdout: ms inject delay type = client mds 2024-04-14T23:47:46.409 INFO:teuthology.orchestra.run.smithi060.stdout: ms inject socket failures = 2500 2024-04-14T23:47:46.409 INFO:teuthology.orchestra.run.smithi060.stdout: osd blocked scrub grace period = 3600 2024-04-14T23:47:46.409 INFO:teuthology.orchestra.run.smithi060.stdout: osd failsafe full ratio = 0.95 2024-04-14T23:47:46.409 INFO:teuthology.orchestra.run.smithi060.stdout: osd max markdown count = 1000 2024-04-14T23:47:46.409 INFO:teuthology.orchestra.run.smithi060.stdout: osd objectstore = bluestore 2024-04-14T23:47:46.409 INFO:teuthology.orchestra.run.smithi060.stdout: osd op complaint time = 180 2024-04-14T23:47:46.410 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:47:46.410 INFO:teuthology.orchestra.run.smithi060.stdout:[mgr] 2024-04-14T23:47:46.410 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2024-04-14T23:47:46.410 INFO:teuthology.orchestra.run.smithi060.stdout: debug mgr = 20 2024-04-14T23:47:46.410 INFO:teuthology.orchestra.run.smithi060.stdout: debug mon = 20 2024-04-14T23:47:46.410 INFO:teuthology.orchestra.run.smithi060.stdout: debug auth = 20 2024-04-14T23:47:46.410 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min pgs per osd = 4 2024-04-14T23:47:46.410 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min bytes per osd = 10 2024-04-14T23:47:46.410 INFO:teuthology.orchestra.run.smithi060.stdout: mgr/telemetry/nag = false 2024-04-14T23:47:46.410 INFO:teuthology.orchestra.run.smithi060.stdout: debug client = 20 2024-04-14T23:47:46.410 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:47:46.410 INFO:teuthology.orchestra.run.smithi060.stdout:[mon] 2024-04-14T23:47:46.410 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2024-04-14T23:47:46.410 INFO:teuthology.orchestra.run.smithi060.stdout: debug mon = 20 2024-04-14T23:47:46.411 INFO:teuthology.orchestra.run.smithi060.stdout: debug paxos = 20 2024-04-14T23:47:46.411 INFO:teuthology.orchestra.run.smithi060.stdout: debug auth = 20 2024-04-14T23:47:46.411 INFO:teuthology.orchestra.run.smithi060.stdout: mon data avail warn = 5 2024-04-14T23:47:46.411 INFO:teuthology.orchestra.run.smithi060.stdout: mon mgr mkfs grace = 240 2024-04-14T23:47:46.411 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min pgs per osd = 4 2024-04-14T23:47:46.411 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd reporter subtree level = osd 2024-04-14T23:47:46.411 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd prime pg temp = true 2024-04-14T23:47:46.411 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min bytes per osd = 10 2024-04-14T23:47:46.411 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:47:46.411 INFO:teuthology.orchestra.run.smithi060.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-14T23:47:46.411 INFO:teuthology.orchestra.run.smithi060.stdout: auth mon ticket ttl = 660 # 11m 2024-04-14T23:47:46.411 INFO:teuthology.orchestra.run.smithi060.stdout: auth service ticket ttl = 240 # 4m 2024-04-14T23:47:46.411 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:47:46.411 INFO:teuthology.orchestra.run.smithi060.stdout: # don't complain about insecure global_id in the test suite 2024-04-14T23:47:46.411 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-14T23:47:46.412 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-14T23:47:46.412 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:47:46.412 INFO:teuthology.orchestra.run.smithi060.stdout: # 1m isn't quite enough 2024-04-14T23:47:46.412 INFO:teuthology.orchestra.run.smithi060.stdout: mon_down_mkfs_grace = 2m 2024-04-14T23:47:46.412 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:47:46.412 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_filestore_osds = false 2024-04-14T23:47:46.412 INFO:teuthology.orchestra.run.smithi060.stdout: mon down mkfs grace = 300 2024-04-14T23:47:46.412 INFO:teuthology.orchestra.run.smithi060.stdout: mon op complaint time = 120 2024-04-14T23:47:46.412 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:47:46.412 INFO:teuthology.orchestra.run.smithi060.stdout:[client] 2024-04-14T23:47:46.412 INFO:teuthology.orchestra.run.smithi060.stdout: rgw cache enabled = true 2024-04-14T23:47:46.412 INFO:teuthology.orchestra.run.smithi060.stdout: rgw enable ops log = true 2024-04-14T23:47:46.412 INFO:teuthology.orchestra.run.smithi060.stdout: rgw enable usage log = true 2024-04-14T23:47:46.412 INFO:teuthology.orchestra.run.smithi060.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-14T23:47:46.412 INFO:teuthology.orchestra.run.smithi060.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-14T23:47:46.413 INFO:teuthology.orchestra.run.smithi060.stdout: client mount timeout = 600 2024-04-14T23:47:46.413 INFO:teuthology.orchestra.run.smithi060.stdout: client_shutdown_timeout = 120 2024-04-14T23:47:46.413 INFO:teuthology.orchestra.run.smithi060.stdout: debug client = 20 2024-04-14T23:47:46.413 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2024-04-14T23:47:46.413 INFO:teuthology.orchestra.run.smithi060.stdout: rados mon op timeout = 900 2024-04-14T23:47:46.413 INFO:teuthology.orchestra.run.smithi060.stdout: rados osd op timeout = 900 2024-04-14T23:47:46.413 INFO:teuthology.orchestra.run.smithi060.stdout:[mon.a] 2024-04-14T23:47:46.413 INFO:teuthology.orchestra.run.smithi060.stdout:[mon.b] 2024-04-14T23:47:46.413 INFO:teuthology.orchestra.run.smithi060.stdout:[mon.c] 2024-04-14T23:47:46.413 INFO:teuthology.orchestra.run.smithi060.stdout:[mds.a] 2024-04-14T23:47:46.413 INFO:teuthology.orchestra.run.smithi060.stdout:[mds.c] 2024-04-14T23:47:46.413 INFO:teuthology.orchestra.run.smithi060.stdout:[mds.e] 2024-04-14T23:47:46.413 INFO:teuthology.orchestra.run.smithi060.stdout:[mds.b] 2024-04-14T23:47:46.414 INFO:teuthology.orchestra.run.smithi060.stdout:[mds.d] 2024-04-14T23:47:46.414 INFO:teuthology.orchestra.run.smithi060.stdout:[mds.f] 2024-04-14T23:47:46.414 INFO:teuthology.orchestra.run.smithi060.stdout:[mds] 2024-04-14T23:47:46.414 INFO:teuthology.orchestra.run.smithi060.stdout: debug mds = 20 2024-04-14T23:47:46.414 INFO:teuthology.orchestra.run.smithi060.stdout: debug mds balancer = 20 2024-04-14T23:47:46.414 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2024-04-14T23:47:46.414 INFO:teuthology.orchestra.run.smithi060.stdout: mds bal fragment size max = 10000 2024-04-14T23:47:46.414 INFO:teuthology.orchestra.run.smithi060.stdout: mds bal merge size = 5 2024-04-14T23:47:46.414 INFO:teuthology.orchestra.run.smithi060.stdout: mds bal split bits = 3 2024-04-14T23:47:46.414 INFO:teuthology.orchestra.run.smithi060.stdout: mds bal split size = 100 2024-04-14T23:47:46.414 INFO:teuthology.orchestra.run.smithi060.stdout: mds debug frag = True 2024-04-14T23:47:46.414 INFO:teuthology.orchestra.run.smithi060.stdout: mds debug scatterstat = True 2024-04-14T23:47:46.414 INFO:teuthology.orchestra.run.smithi060.stdout: mds oft prefetch dirfrags = True 2024-04-14T23:47:46.414 INFO:teuthology.orchestra.run.smithi060.stdout: mds op complaint time = 180 2024-04-14T23:47:46.414 INFO:teuthology.orchestra.run.smithi060.stdout: mds verify scatter = True 2024-04-14T23:47:46.415 INFO:teuthology.orchestra.run.smithi060.stdout: mds_dir_prefetch = False 2024-04-14T23:47:46.415 INFO:teuthology.orchestra.run.smithi060.stdout: mds_sleep_rank_change = 5000000.0 2024-04-14T23:47:46.415 INFO:teuthology.orchestra.run.smithi060.stdout: mon client directed command retry = 5 2024-04-14T23:47:46.415 INFO:teuthology.orchestra.run.smithi060.stdout: ms inject delay max = 1 2024-04-14T23:47:46.415 INFO:teuthology.orchestra.run.smithi060.stdout: ms inject delay probability = 0.005 2024-04-14T23:47:46.415 INFO:teuthology.orchestra.run.smithi060.stdout: ms inject delay type = client mds osd 2024-04-14T23:47:46.415 INFO:teuthology.orchestra.run.smithi060.stdout: ms inject socket failures = 2500 2024-04-14T23:47:46.415 INFO:teuthology.orchestra.run.smithi060.stdout: osd op complaint time = 180 2024-04-14T23:47:46.415 INFO:teuthology.orchestra.run.smithi060.stdout: rados mon op timeout = 900 2024-04-14T23:47:46.415 INFO:teuthology.orchestra.run.smithi060.stdout: rados osd op timeout = 900 2024-04-14T23:47:46.452 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-14T23:47:46.452 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-14T23:47:46.600 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-14T23:47:46.600 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-04-14T23:47:46.600 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-14T23:47:46.655 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-04-14T23:47:46.655 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-14T23:47:46.711 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi060.front.sepia.ceph.com 2024-04-14T23:47:46.711 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-04-14T23:47:46.712 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-14T23:47:46.712 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-14T23:47:46.868 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-04-14T23:47:46.868 DEBUG:teuthology.orchestra.run.smithi060:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-14T23:47:46.968 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi186.front.sepia.ceph.com 2024-04-14T23:47:46.968 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-14T23:47:46.968 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-14T23:47:46.968 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-14T23:47:47.014 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-14T23:47:47.014 DEBUG:teuthology.orchestra.run.smithi186:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-14T23:47:47.070 INFO:tasks.ceph:Setting up mon nodes... 2024-04-14T23:47:47.070 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-14T23:47:47.070 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-04-14T23:47:47.255 INFO:teuthology.orchestra.run.smithi060.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-04-14T23:47:47.267 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-14T23:47:47.337 INFO:teuthology.orchestra.run.smithi186.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-14T23:47:47.339 INFO:tasks.ceph:Setting up mds nodes... 2024-04-14T23:47:47.340 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-04-14T23:47:47.541 INFO:teuthology.orchestra.run.smithi060.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-04-14T23:47:47.551 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-04-14T23:47:47.651 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-04-14T23:47:47.845 INFO:teuthology.orchestra.run.smithi060.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-04-14T23:47:47.855 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-04-14T23:47:47.948 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/mds/ceph-e && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.e /var/lib/ceph/mds/ceph-e/keyring 2024-04-14T23:47:48.122 INFO:teuthology.orchestra.run.smithi060.stdout:creating /var/lib/ceph/mds/ceph-e/keyring 2024-04-14T23:47:48.132 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-e 2024-04-14T23:47:48.232 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-04-14T23:47:48.294 INFO:teuthology.orchestra.run.smithi186.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-04-14T23:47:48.297 DEBUG:teuthology.orchestra.run.smithi186:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-04-14T23:47:48.325 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-04-14T23:47:48.423 INFO:teuthology.orchestra.run.smithi186.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-04-14T23:47:48.425 DEBUG:teuthology.orchestra.run.smithi186:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-04-14T23:47:48.452 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/mds/ceph-f && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.f /var/lib/ceph/mds/ceph-f/keyring 2024-04-14T23:47:48.547 INFO:teuthology.orchestra.run.smithi186.stdout:creating /var/lib/ceph/mds/ceph-f/keyring 2024-04-14T23:47:48.549 DEBUG:teuthology.orchestra.run.smithi186:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-f 2024-04-14T23:47:48.576 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-14T23:47:48.576 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-14T23:47:48.712 INFO:teuthology.orchestra.run.smithi060.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-14T23:47:48.773 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-14T23:47:48.774 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi060.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi186.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-14T23:47:48.774 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-14T23:47:48.873 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-14T23:47:48.874 INFO:tasks.ceph:role: osd.0 2024-04-14T23:47:48.874 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi060.front.sepia.ceph.com 2024-04-14T23:47:48.874 DEBUG:teuthology.orchestra.run.smithi060:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-14T23:47:48.969 INFO:teuthology.orchestra.run.smithi060.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-14T23:47:48.969 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-14T23:47:48.969 INFO:teuthology.orchestra.run.smithi060.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-14T23:47:48.969 INFO:teuthology.orchestra.run.smithi060.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-14T23:47:48.969 INFO:teuthology.orchestra.run.smithi060.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-14T23:47:48.969 INFO:teuthology.orchestra.run.smithi060.stdout: = sunit=0 swidth=0 blks 2024-04-14T23:47:48.969 INFO:teuthology.orchestra.run.smithi060.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-14T23:47:48.969 INFO:teuthology.orchestra.run.smithi060.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-14T23:47:48.969 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-14T23:47:48.970 INFO:teuthology.orchestra.run.smithi060.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-14T23:47:49.003 INFO:teuthology.orchestra.run.smithi060.stdout:Discarding blocks...Done. 2024-04-14T23:47:49.008 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi060.front.sepia.ceph.com -o noatime 2024-04-14T23:47:49.008 DEBUG:teuthology.orchestra.run.smithi060:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-14T23:47:49.127 DEBUG:teuthology.orchestra.run.smithi060:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-14T23:47:49.205 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-14T23:47:49.294 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-14T23:47:49.295 INFO:tasks.ceph:role: osd.1 2024-04-14T23:47:49.295 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi060.front.sepia.ceph.com 2024-04-14T23:47:49.295 DEBUG:teuthology.orchestra.run.smithi060:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-14T23:47:49.385 INFO:teuthology.orchestra.run.smithi060.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-14T23:47:49.385 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-14T23:47:49.385 INFO:teuthology.orchestra.run.smithi060.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-14T23:47:49.385 INFO:teuthology.orchestra.run.smithi060.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-14T23:47:49.386 INFO:teuthology.orchestra.run.smithi060.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-14T23:47:49.386 INFO:teuthology.orchestra.run.smithi060.stdout: = sunit=0 swidth=0 blks 2024-04-14T23:47:49.386 INFO:teuthology.orchestra.run.smithi060.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-14T23:47:49.386 INFO:teuthology.orchestra.run.smithi060.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-14T23:47:49.386 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-14T23:47:49.386 INFO:teuthology.orchestra.run.smithi060.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-14T23:47:49.419 INFO:teuthology.orchestra.run.smithi060.stdout:Discarding blocks...Done. 2024-04-14T23:47:49.425 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi060.front.sepia.ceph.com -o noatime 2024-04-14T23:47:49.425 DEBUG:teuthology.orchestra.run.smithi060:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-14T23:47:49.525 DEBUG:teuthology.orchestra.run.smithi060:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-14T23:47:49.602 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-14T23:47:49.679 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-14T23:47:49.680 INFO:tasks.ceph:role: osd.2 2024-04-14T23:47:49.680 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi060.front.sepia.ceph.com 2024-04-14T23:47:49.680 DEBUG:teuthology.orchestra.run.smithi060:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-14T23:47:49.760 INFO:teuthology.orchestra.run.smithi060.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-14T23:47:49.761 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-14T23:47:49.761 INFO:teuthology.orchestra.run.smithi060.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-14T23:47:49.761 INFO:teuthology.orchestra.run.smithi060.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-14T23:47:49.761 INFO:teuthology.orchestra.run.smithi060.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-14T23:47:49.761 INFO:teuthology.orchestra.run.smithi060.stdout: = sunit=0 swidth=0 blks 2024-04-14T23:47:49.761 INFO:teuthology.orchestra.run.smithi060.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-14T23:47:49.761 INFO:teuthology.orchestra.run.smithi060.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-14T23:47:49.761 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-14T23:47:49.761 INFO:teuthology.orchestra.run.smithi060.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-14T23:47:49.794 INFO:teuthology.orchestra.run.smithi060.stdout:Discarding blocks...Done. 2024-04-14T23:47:49.801 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi060.front.sepia.ceph.com -o noatime 2024-04-14T23:47:49.801 DEBUG:teuthology.orchestra.run.smithi060:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-04-14T23:47:49.906 DEBUG:teuthology.orchestra.run.smithi060:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-14T23:47:49.986 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-04-14T23:47:50.067 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-14T23:47:50.067 INFO:tasks.ceph:role: osd.3 2024-04-14T23:47:50.067 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi060.front.sepia.ceph.com 2024-04-14T23:47:50.068 DEBUG:teuthology.orchestra.run.smithi060:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-14T23:47:50.149 INFO:teuthology.orchestra.run.smithi060.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-14T23:47:50.150 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-14T23:47:50.150 INFO:teuthology.orchestra.run.smithi060.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-14T23:47:50.150 INFO:teuthology.orchestra.run.smithi060.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-14T23:47:50.150 INFO:teuthology.orchestra.run.smithi060.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-14T23:47:50.150 INFO:teuthology.orchestra.run.smithi060.stdout: = sunit=0 swidth=0 blks 2024-04-14T23:47:50.150 INFO:teuthology.orchestra.run.smithi060.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-14T23:47:50.150 INFO:teuthology.orchestra.run.smithi060.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-14T23:47:50.150 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-14T23:47:50.150 INFO:teuthology.orchestra.run.smithi060.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-14T23:47:50.183 INFO:teuthology.orchestra.run.smithi060.stdout:Discarding blocks...Done. 2024-04-14T23:47:50.189 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi060.front.sepia.ceph.com -o noatime 2024-04-14T23:47:50.189 DEBUG:teuthology.orchestra.run.smithi060:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-04-14T23:47:50.292 DEBUG:teuthology.orchestra.run.smithi060:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-04-14T23:47:50.367 DEBUG:teuthology.orchestra.run.smithi060:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-14T23:47:50.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:47:50.501+0000 7f12c24c4640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-04-14T23:47:50.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:47:50.502+0000 7f12c24c4640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-04-14T23:47:50.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:47:50.502+0000 7f12c24c4640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-14T23:47:50.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:47:50.502+0000 7f12c24c4640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-14T23:47:50.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:47:50.502+0000 7f12c24c4640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-14T23:47:50.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:47:50.505+0000 7f12c24c4640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-04-14T23:47:53.504 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-14T23:47:53.607 DEBUG:teuthology.orchestra.run.smithi060:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-14T23:47:53.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:47:53.748+0000 7f9893e11640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-04-14T23:47:53.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:47:53.749+0000 7f9893e11640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-04-14T23:47:53.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:47:53.749+0000 7f9893e11640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-14T23:47:53.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:47:53.749+0000 7f9893e11640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-14T23:47:53.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:47:53.749+0000 7f9893e11640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-14T23:47:53.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:47:53.751+0000 7f9893e11640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-04-14T23:47:56.979 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-14T23:47:57.090 DEBUG:teuthology.orchestra.run.smithi060:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-14T23:47:57.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:47:57.240+0000 7fb870b93640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-04-14T23:47:57.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:47:57.241+0000 7fb870b93640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-04-14T23:47:57.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:47:57.241+0000 7fb870b93640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-14T23:47:57.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:47:57.241+0000 7fb870b93640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-14T23:47:57.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:47:57.241+0000 7fb870b93640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-14T23:47:57.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:47:57.243+0000 7fb870b93640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-04-14T23:48:00.217 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-14T23:48:00.324 DEBUG:teuthology.orchestra.run.smithi060:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-14T23:48:00.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:00.463+0000 7fc3ab413640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-04-14T23:48:00.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:00.463+0000 7fc3ab413640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-04-14T23:48:00.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:00.463+0000 7fc3ab413640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-14T23:48:00.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:00.464+0000 7fc3ab413640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-14T23:48:00.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:00.464+0000 7fc3ab413640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-14T23:48:00.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:00.466+0000 7fc3ab413640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-04-14T23:48:03.689 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-14T23:48:03.792 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-04-14T23:48:03.824 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-14T23:48:03.824 INFO:tasks.ceph:role: osd.4 2024-04-14T23:48:03.824 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi186.front.sepia.ceph.com 2024-04-14T23:48:03.824 DEBUG:teuthology.orchestra.run.smithi186:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-14T23:48:03.901 INFO:teuthology.orchestra.run.smithi186.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-14T23:48:03.901 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-14T23:48:03.902 INFO:teuthology.orchestra.run.smithi186.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-14T23:48:03.902 INFO:teuthology.orchestra.run.smithi186.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-14T23:48:03.902 INFO:teuthology.orchestra.run.smithi186.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-14T23:48:03.902 INFO:teuthology.orchestra.run.smithi186.stdout: = sunit=0 swidth=0 blks 2024-04-14T23:48:03.902 INFO:teuthology.orchestra.run.smithi186.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-14T23:48:03.902 INFO:teuthology.orchestra.run.smithi186.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-14T23:48:03.902 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-14T23:48:03.902 INFO:teuthology.orchestra.run.smithi186.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-14T23:48:04.410 INFO:teuthology.orchestra.run.smithi186.stdout:Discarding blocks...Done. 2024-04-14T23:48:04.412 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi186.front.sepia.ceph.com -o noatime 2024-04-14T23:48:04.412 DEBUG:teuthology.orchestra.run.smithi186:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-04-14T23:48:04.541 DEBUG:teuthology.orchestra.run.smithi186:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-04-14T23:48:04.578 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-04-14T23:48:04.647 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-14T23:48:04.647 INFO:tasks.ceph:role: osd.5 2024-04-14T23:48:04.648 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi186.front.sepia.ceph.com 2024-04-14T23:48:04.648 DEBUG:teuthology.orchestra.run.smithi186:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-14T23:48:04.723 INFO:teuthology.orchestra.run.smithi186.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-14T23:48:04.724 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-14T23:48:04.724 INFO:teuthology.orchestra.run.smithi186.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-14T23:48:04.724 INFO:teuthology.orchestra.run.smithi186.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-14T23:48:04.724 INFO:teuthology.orchestra.run.smithi186.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-14T23:48:04.724 INFO:teuthology.orchestra.run.smithi186.stdout: = sunit=0 swidth=0 blks 2024-04-14T23:48:04.724 INFO:teuthology.orchestra.run.smithi186.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-14T23:48:04.724 INFO:teuthology.orchestra.run.smithi186.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-14T23:48:04.724 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-14T23:48:04.724 INFO:teuthology.orchestra.run.smithi186.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-14T23:48:05.181 INFO:teuthology.orchestra.run.smithi186.stdout:Discarding blocks...Done. 2024-04-14T23:48:05.183 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi186.front.sepia.ceph.com -o noatime 2024-04-14T23:48:05.183 DEBUG:teuthology.orchestra.run.smithi186:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-04-14T23:48:05.278 DEBUG:teuthology.orchestra.run.smithi186:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-04-14T23:48:05.311 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-04-14T23:48:05.380 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-14T23:48:05.380 INFO:tasks.ceph:role: osd.6 2024-04-14T23:48:05.380 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi186.front.sepia.ceph.com 2024-04-14T23:48:05.381 DEBUG:teuthology.orchestra.run.smithi186:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-14T23:48:05.456 INFO:teuthology.orchestra.run.smithi186.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-14T23:48:05.457 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-14T23:48:05.457 INFO:teuthology.orchestra.run.smithi186.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-14T23:48:05.457 INFO:teuthology.orchestra.run.smithi186.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-14T23:48:05.457 INFO:teuthology.orchestra.run.smithi186.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-14T23:48:05.457 INFO:teuthology.orchestra.run.smithi186.stdout: = sunit=0 swidth=0 blks 2024-04-14T23:48:05.457 INFO:teuthology.orchestra.run.smithi186.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-14T23:48:05.457 INFO:teuthology.orchestra.run.smithi186.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-14T23:48:05.457 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-14T23:48:05.457 INFO:teuthology.orchestra.run.smithi186.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-14T23:48:05.956 INFO:teuthology.orchestra.run.smithi186.stdout:Discarding blocks...Done. 2024-04-14T23:48:05.959 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi186.front.sepia.ceph.com -o noatime 2024-04-14T23:48:05.959 DEBUG:teuthology.orchestra.run.smithi186:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-04-14T23:48:06.103 DEBUG:teuthology.orchestra.run.smithi186:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-04-14T23:48:06.137 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-04-14T23:48:06.205 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-14T23:48:06.205 INFO:tasks.ceph:role: osd.7 2024-04-14T23:48:06.205 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi186.front.sepia.ceph.com 2024-04-14T23:48:06.206 DEBUG:teuthology.orchestra.run.smithi186:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-14T23:48:06.282 INFO:teuthology.orchestra.run.smithi186.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-14T23:48:06.282 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-14T23:48:06.282 INFO:teuthology.orchestra.run.smithi186.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-14T23:48:06.282 INFO:teuthology.orchestra.run.smithi186.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-14T23:48:06.282 INFO:teuthology.orchestra.run.smithi186.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-14T23:48:06.282 INFO:teuthology.orchestra.run.smithi186.stdout: = sunit=0 swidth=0 blks 2024-04-14T23:48:06.282 INFO:teuthology.orchestra.run.smithi186.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-14T23:48:06.282 INFO:teuthology.orchestra.run.smithi186.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-14T23:48:06.282 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-14T23:48:06.282 INFO:teuthology.orchestra.run.smithi186.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-14T23:48:06.796 INFO:teuthology.orchestra.run.smithi186.stdout:Discarding blocks...Done. 2024-04-14T23:48:06.798 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi186.front.sepia.ceph.com -o noatime 2024-04-14T23:48:06.798 DEBUG:teuthology.orchestra.run.smithi186:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-04-14T23:48:06.924 DEBUG:teuthology.orchestra.run.smithi186:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-04-14T23:48:06.962 DEBUG:teuthology.orchestra.run.smithi186:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-14T23:48:07.064 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:07.059+0000 7f854720b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-04-14T23:48:07.065 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:07.060+0000 7f854720b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-04-14T23:48:07.065 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:07.060+0000 7f854720b640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-14T23:48:07.065 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:07.060+0000 7f854720b640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-14T23:48:07.065 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:07.060+0000 7f854720b640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-14T23:48:07.065 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:07.061+0000 7f854720b640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-04-14T23:48:09.919 DEBUG:teuthology.orchestra.run.smithi186:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-14T23:48:09.950 DEBUG:teuthology.orchestra.run.smithi186:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-14T23:48:10.040 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:10.035+0000 7fb35d015640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-04-14T23:48:10.041 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:10.035+0000 7fb35d015640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-04-14T23:48:10.041 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:10.035+0000 7fb35d015640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-14T23:48:10.041 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:10.035+0000 7fb35d015640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-14T23:48:10.041 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:10.035+0000 7fb35d015640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-14T23:48:10.041 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:10.036+0000 7fb35d015640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-04-14T23:48:12.894 DEBUG:teuthology.orchestra.run.smithi186:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-14T23:48:12.925 DEBUG:teuthology.orchestra.run.smithi186:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-14T23:48:13.016 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:13.011+0000 7fcb18e12640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-04-14T23:48:13.016 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:13.011+0000 7fcb18e12640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-04-14T23:48:13.017 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:13.011+0000 7fcb18e12640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-14T23:48:13.017 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:13.011+0000 7fcb18e12640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-14T23:48:13.017 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:13.011+0000 7fcb18e12640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-14T23:48:13.017 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:13.012+0000 7fcb18e12640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-04-14T23:48:13.595 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-14T23:48:13.603 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-14T23:48:15.869 DEBUG:teuthology.orchestra.run.smithi186:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-14T23:48:15.900 DEBUG:teuthology.orchestra.run.smithi186:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-14T23:48:15.992 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:15.986+0000 7f2b5bef9640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-04-14T23:48:15.992 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:15.987+0000 7f2b5bef9640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-04-14T23:48:15.993 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:15.987+0000 7f2b5bef9640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-14T23:48:15.993 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:15.987+0000 7f2b5bef9640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-14T23:48:15.993 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:15.987+0000 7f2b5bef9640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-14T23:48:15.993 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:15.987+0000 7f2b5bef9640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-04-14T23:48:18.845 DEBUG:teuthology.orchestra.run.smithi186:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-14T23:48:18.876 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-14T23:48:18.876 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-04-14T23:48:18.876 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-04-14T23:48:18.981 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-04-14T23:48:18.981 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-04-14T23:48:19.073 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-04-14T23:48:19.073 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-04-14T23:48:19.170 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-04-14T23:48:19.170 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/mds/ceph-e/keyring of=/dev/stdout 2024-04-14T23:48:19.276 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-04-14T23:48:19.276 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-14T23:48:19.378 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-04-14T23:48:19.378 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-14T23:48:19.473 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-04-14T23:48:19.473 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-14T23:48:19.566 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-04-14T23:48:19.566 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-04-14T23:48:19.654 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-14T23:48:19.654 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-14T23:48:19.685 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-14T23:48:19.685 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-04-14T23:48:19.756 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-14T23:48:19.756 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-04-14T23:48:19.826 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-14T23:48:19.826 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/mds/ceph-f/keyring of=/dev/stdout 2024-04-14T23:48:19.895 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-14T23:48:19.895 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-04-14T23:48:19.963 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-14T23:48:19.964 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-04-14T23:48:20.032 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-14T23:48:20.032 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-04-14T23:48:20.101 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-14T23:48:20.101 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-04-14T23:48:20.168 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-04-14T23:48:20.168 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-14T23:48:20.229 INFO:tasks.ceph:Adding keys to all mons... 2024-04-14T23:48:20.229 DEBUG:teuthology.orchestra.run.smithi060:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-14T23:48:20.234 DEBUG:teuthology.orchestra.run.smithi186:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-14T23:48:20.260 INFO:teuthology.orchestra.run.smithi186.stdout:[mgr.y] 2024-04-14T23:48:20.260 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQAjaxxmOcgEDxAAAkIwPf+NKvXlPB4YuyHikg== 2024-04-14T23:48:20.260 INFO:teuthology.orchestra.run.smithi186.stdout:[mds.a] 2024-04-14T23:48:20.260 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQAjaxxmtKYNIBAAyaZK0CWAWaeiBnagJ37CzA== 2024-04-14T23:48:20.260 INFO:teuthology.orchestra.run.smithi186.stdout:[mds.c] 2024-04-14T23:48:20.260 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQAjaxxmXdgqMhAAQB+IeX7B09DZoi2cTQf0FA== 2024-04-14T23:48:20.260 INFO:teuthology.orchestra.run.smithi186.stdout:[mds.e] 2024-04-14T23:48:20.260 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQAkaxxmLewRBxAADZ3/NpGYEW9Ud/gEPUrAAA== 2024-04-14T23:48:20.260 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.0] 2024-04-14T23:48:20.261 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQAmaxxmHv3rHRAAGLzQiY60+7fXe7Td7Doyuw== 2024-04-14T23:48:20.261 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.1] 2024-04-14T23:48:20.261 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQApaxxmPhapLBAAjUJyG7HBTweVUhcm+8itag== 2024-04-14T23:48:20.261 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.2] 2024-04-14T23:48:20.261 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQAtaxxmoM9eDhAAEroY08QA1nuIKwL9W2SU8Q== 2024-04-14T23:48:20.261 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.3] 2024-04-14T23:48:20.261 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQAwaxxmPgKnGxAAEE1SVFbIYdxQGjl5RTyCkQ== 2024-04-14T23:48:20.261 INFO:teuthology.orchestra.run.smithi186.stdout:[mgr.x] 2024-04-14T23:48:20.261 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQAjaxxm/gLbExAAGN3dfBEMkYeEDZOYyiiMSw== 2024-04-14T23:48:20.261 INFO:teuthology.orchestra.run.smithi186.stdout:[mds.b] 2024-04-14T23:48:20.261 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQAkaxxmSwNSERAArAJxQkvlzWN6iWtXPogS5g== 2024-04-14T23:48:20.261 INFO:teuthology.orchestra.run.smithi186.stdout:[mds.d] 2024-04-14T23:48:20.261 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQAkaxxmVqsEGRAAyAfZM7Y4ehr3AZ42ScRO7A== 2024-04-14T23:48:20.261 INFO:teuthology.orchestra.run.smithi186.stdout:[mds.f] 2024-04-14T23:48:20.262 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQAkaxxmxP9lIBAAq+YJsCm4RqhX0LshpwBpmA== 2024-04-14T23:48:20.262 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.4] 2024-04-14T23:48:20.262 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQA3axxmXlidAxAAfUY/B8kXr+4+4EMectBKAw== 2024-04-14T23:48:20.262 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.5] 2024-04-14T23:48:20.262 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQA6axxm6IYtAhAAMEwzHmXDmNAzA39Ty/CMWA== 2024-04-14T23:48:20.262 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.6] 2024-04-14T23:48:20.262 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQA9axxm4zG+ABAAkysxnsHVdyOadKWB7VVrHw== 2024-04-14T23:48:20.262 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.7] 2024-04-14T23:48:20.262 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQA/axxm50XhOhAAuNZlzLBKY7ToQkAoEegapw== 2024-04-14T23:48:20.262 INFO:teuthology.orchestra.run.smithi186.stdout:[client.0] 2024-04-14T23:48:20.262 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQAkaxxm9i08KhAAP+s14XVGU0UoicFlMRf7Ug== 2024-04-14T23:48:20.322 INFO:teuthology.orchestra.run.smithi060.stdout:[mgr.y] 2024-04-14T23:48:20.323 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQAjaxxmOcgEDxAAAkIwPf+NKvXlPB4YuyHikg== 2024-04-14T23:48:20.323 INFO:teuthology.orchestra.run.smithi060.stdout:[mds.a] 2024-04-14T23:48:20.323 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQAjaxxmtKYNIBAAyaZK0CWAWaeiBnagJ37CzA== 2024-04-14T23:48:20.323 INFO:teuthology.orchestra.run.smithi060.stdout:[mds.c] 2024-04-14T23:48:20.323 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQAjaxxmXdgqMhAAQB+IeX7B09DZoi2cTQf0FA== 2024-04-14T23:48:20.323 INFO:teuthology.orchestra.run.smithi060.stdout:[mds.e] 2024-04-14T23:48:20.323 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQAkaxxmLewRBxAADZ3/NpGYEW9Ud/gEPUrAAA== 2024-04-14T23:48:20.323 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.0] 2024-04-14T23:48:20.323 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQAmaxxmHv3rHRAAGLzQiY60+7fXe7Td7Doyuw== 2024-04-14T23:48:20.323 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.1] 2024-04-14T23:48:20.323 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQApaxxmPhapLBAAjUJyG7HBTweVUhcm+8itag== 2024-04-14T23:48:20.323 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.2] 2024-04-14T23:48:20.323 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQAtaxxmoM9eDhAAEroY08QA1nuIKwL9W2SU8Q== 2024-04-14T23:48:20.323 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.3] 2024-04-14T23:48:20.324 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQAwaxxmPgKnGxAAEE1SVFbIYdxQGjl5RTyCkQ== 2024-04-14T23:48:20.324 INFO:teuthology.orchestra.run.smithi060.stdout:[mgr.x] 2024-04-14T23:48:20.324 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQAjaxxm/gLbExAAGN3dfBEMkYeEDZOYyiiMSw== 2024-04-14T23:48:20.324 INFO:teuthology.orchestra.run.smithi060.stdout:[mds.b] 2024-04-14T23:48:20.324 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQAkaxxmSwNSERAArAJxQkvlzWN6iWtXPogS5g== 2024-04-14T23:48:20.324 INFO:teuthology.orchestra.run.smithi060.stdout:[mds.d] 2024-04-14T23:48:20.324 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQAkaxxmVqsEGRAAyAfZM7Y4ehr3AZ42ScRO7A== 2024-04-14T23:48:20.324 INFO:teuthology.orchestra.run.smithi060.stdout:[mds.f] 2024-04-14T23:48:20.324 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQAkaxxmxP9lIBAAq+YJsCm4RqhX0LshpwBpmA== 2024-04-14T23:48:20.324 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.4] 2024-04-14T23:48:20.324 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQA3axxmXlidAxAAfUY/B8kXr+4+4EMectBKAw== 2024-04-14T23:48:20.324 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.5] 2024-04-14T23:48:20.324 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQA6axxm6IYtAhAAMEwzHmXDmNAzA39Ty/CMWA== 2024-04-14T23:48:20.324 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.6] 2024-04-14T23:48:20.324 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQA9axxm4zG+ABAAkysxnsHVdyOadKWB7VVrHw== 2024-04-14T23:48:20.325 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.7] 2024-04-14T23:48:20.325 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQA/axxm50XhOhAAuNZlzLBKY7ToQkAoEegapw== 2024-04-14T23:48:20.325 INFO:teuthology.orchestra.run.smithi060.stdout:[client.0] 2024-04-14T23:48:20.325 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQAkaxxm9i08KhAAP+s14XVGU0UoicFlMRf7Ug== 2024-04-14T23:48:20.326 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-14T23:48:20.335 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-14T23:48:20.472 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-14T23:48:20.477 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-14T23:48:20.622 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-14T23:48:20.631 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-14T23:48:20.781 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-14T23:48:20.786 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-14T23:48:20.931 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-14T23:48:20.936 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-14T23:48:21.079 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-14T23:48:21.086 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-14T23:48:21.225 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-14T23:48:21.231 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-14T23:48:21.381 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-14T23:48:21.388 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-14T23:48:21.539 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-14T23:48:21.544 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-14T23:48:21.687 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-14T23:48:21.692 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-14T23:48:21.838 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-14T23:48:21.845 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-14T23:48:21.993 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-14T23:48:21.998 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-14T23:48:22.140 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-14T23:48:22.145 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-14T23:48:22.295 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-14T23:48:22.300 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-14T23:48:22.448 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-14T23:48:22.453 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-14T23:48:22.601 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-14T23:48:22.608 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-14T23:48:22.757 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-14T23:48:22.762 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-14T23:48:22.909 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-14T23:48:22.909 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-14T23:48:23.009 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-14T23:48:23.173 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-14T23:48:23.258 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-04-14T23:48:23.285 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-14T23:48:23.383 DEBUG:teuthology.orchestra.run.smithi186:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-04-14T23:48:23.409 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-04-14T23:48:23.476 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-14T23:48:23.574 DEBUG:teuthology.orchestra.run.smithi186:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-04-14T23:48:23.600 DEBUG:teuthology.orchestra.run.smithi060:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-14T23:48:23.608 DEBUG:teuthology.orchestra.run.smithi186:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-14T23:48:23.660 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-14T23:48:23.660 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-14T23:48:23.660 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-14T23:48:23.666 INFO:tasks.ceph.mon.a:Started 2024-04-14T23:48:23.666 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-14T23:48:23.666 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-14T23:48:23.698 INFO:tasks.ceph.mon.b:Started 2024-04-14T23:48:23.699 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-14T23:48:23.699 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-14T23:48:23.701 INFO:tasks.ceph.mon.c:Started 2024-04-14T23:48:23.701 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-14T23:48:23.701 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-04-14T23:48:23.701 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-04-14T23:48:23.706 INFO:tasks.ceph.mgr.y:Started 2024-04-14T23:48:23.706 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-14T23:48:23.706 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-14T23:48:23.708 INFO:tasks.ceph.mgr.x:Started 2024-04-14T23:48:23.708 INFO:tasks.ceph:Setting crush tunables to default 2024-04-14T23:48:23.709 DEBUG:teuthology.orchestra.run.smithi060:> sudo ceph --cluster ceph osd crush tunables default 2024-04-14T23:48:23.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.887+0000 7fa123e00640 1 Processor -- start 2024-04-14T23:48:23.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.887+0000 7fa123e00640 1 -- start start 2024-04-14T23:48:23.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.888+0000 7fa123e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa1240cb470 0x7fa1240cc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:23.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.888+0000 7fa123e00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa1240c5400 0x7fa1240c5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:23.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.888+0000 7fa123e00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa12415b9d0 con 0x7fa124154e80 2024-04-14T23:48:23.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.888+0000 7fa123e00640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fa1240cbbf0 con 0x7fa1240c5400 2024-04-14T23:48:23.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.888+0000 7fa123e00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fa12415e0c0 con 0x7fa1240cb470 2024-04-14T23:48:23.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.889+0000 7fa11be00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa1240cb470 0x7fa1240cc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:23.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.889+0000 7fa11be00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa1240cb470 0x7fa1240cc7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.60:36444/0 (socket says 172.21.15.60:36444) 2024-04-14T23:48:23.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.889+0000 7fa11be00640 1 -- 172.21.15.60:0/1639710712 learned_addr learned my addr 172.21.15.60:0/1639710712 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:23.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.889+0000 7fa11be00640 1 -- 172.21.15.60:0/1639710712 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa1240cb470 msgr2=0x7fa1240cc7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-14T23:48:23.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.889+0000 7fa11be00640 1 -- 172.21.15.60:0/1639710712 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa1240cb470 msgr2=0x7fa1240cc7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-14T23:48:23.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.889+0000 7fa11be00640 1 --2- 172.21.15.60:0/1639710712 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa1240cb470 0x7fa1240cc7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-14T23:48:23.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.889+0000 7fa11be00640 1 --2- 172.21.15.60:0/1639710712 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa1240cb470 0x7fa1240cc7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-14T23:48:23.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.889+0000 7fa123400640 1 --2- 172.21.15.60:0/1639710712 >> v2:172.21.15.60:3300/0 conn(0x7fa1240c5400 0x7fa1240c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:23.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.889+0000 7fa123400640 1 -- 172.21.15.60:0/1639710712 >> v2:172.21.15.60:3300/0 conn(0x7fa1240c5400 msgr2=0x7fa1240c5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-14T23:48:23.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.889+0000 7fa123400640 1 -- 172.21.15.60:0/1639710712 >> v2:172.21.15.60:3300/0 conn(0x7fa1240c5400 msgr2=0x7fa1240c5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-14T23:48:23.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.889+0000 7fa123400640 1 --2- 172.21.15.60:0/1639710712 >> v2:172.21.15.60:3300/0 conn(0x7fa1240c5400 0x7fa1240c5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-14T23:48:23.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.889+0000 7fa123400640 1 --2- 172.21.15.60:0/1639710712 >> v2:172.21.15.60:3300/0 conn(0x7fa1240c5400 0x7fa1240c5850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-14T23:48:23.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.911+0000 7fa122000640 1 -- 172.21.15.60:0/1639710712 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4004458910 0 0) 0x7fa12415b9d0 con 0x7fa124154e80 2024-04-14T23:48:23.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.911+0000 7fa122000640 1 -- 172.21.15.60:0/1639710712 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa108003520 con 0x7fa124154e80 2024-04-14T23:48:23.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.913+0000 7fa122000640 1 -- 172.21.15.60:0/1639710712 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3912765631 0 0) 0x7fa108003520 con 0x7fa124154e80 2024-04-14T23:48:23.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.913+0000 7fa122000640 1 -- 172.21.15.60:0/1639710712 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa1240cb470 msgr2=0x7fa1240cc7a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-14T23:48:23.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.913+0000 7fa122000640 1 --2- 172.21.15.60:0/1639710712 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa1240cb470 0x7fa1240cc7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:23.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.913+0000 7fa122000640 1 -- 172.21.15.60:0/1639710712 >> v2:172.21.15.60:3300/0 conn(0x7fa1240c5400 msgr2=0x7fa1240c5850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-14T23:48:23.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.913+0000 7fa122000640 1 --2- 172.21.15.60:0/1639710712 >> v2:172.21.15.60:3300/0 conn(0x7fa1240c5400 0x7fa1240c5850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:23.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.913+0000 7fa122000640 1 -- 172.21.15.60:0/1639710712 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa12415e3a0 con 0x7fa124154e80 2024-04-14T23:48:23.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.913+0000 7fa122000640 1 -- 172.21.15.60:0/1639710712 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2814038680 0 0) 0x7fa114001670 con 0x7fa124154e80 2024-04-14T23:48:23.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.913+0000 7fa122000640 1 -- 172.21.15.60:0/1639710712 >> v1:172.21.15.60:6789/0 conn(0x7fa124154e80 legacy=0x7fa1240d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:23.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.913+0000 7fa122000640 1 --2- 172.21.15.60:0/1639710712 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa1240cb470 0x7fa1080039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:23.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.913+0000 7fa122000640 1 --2- 172.21.15.60:0/1639710712 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa108004030 0x7fa108004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:23.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.913+0000 7fa122000640 1 --2- 172.21.15.60:0/1639710712 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa108004d40 0x7fa1080051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:23.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.913+0000 7fa122000640 1 -- 172.21.15.60:0/1639710712 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fa1240cbbf0 con 0x7fa108004030 2024-04-14T23:48:23.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.913+0000 7fa122000640 1 -- 172.21.15.60:0/1639710712 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa12415e0c0 con 0x7fa108004d40 2024-04-14T23:48:23.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.914+0000 7fa122000640 1 -- 172.21.15.60:0/1639710712 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fa10800bbd0 con 0x7fa1240cb470 2024-04-14T23:48:23.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.914+0000 7fa122a00640 1 --2- 172.21.15.60:0/1639710712 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa1240cb470 0x7fa1080039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:23.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.914+0000 7fa123400640 1 --2- 172.21.15.60:0/1639710712 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa108004d40 0x7fa1080051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:23.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.914+0000 7fa11be00640 1 --2- 172.21.15.60:0/1639710712 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa108004030 0x7fa108004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:23.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.914+0000 7fa123400640 1 -- 172.21.15.60:0/1639710712 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa1240cb470 msgr2=0x7fa1080039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:23.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.914+0000 7fa123400640 1 --2- 172.21.15.60:0/1639710712 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa1240cb470 0x7fa1080039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:23.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.914+0000 7fa123400640 1 -- 172.21.15.60:0/1639710712 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa108004030 msgr2=0x7fa108004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:23.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.914+0000 7fa123400640 1 --2- 172.21.15.60:0/1639710712 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa108004030 0x7fa108004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:23.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.914+0000 7fa123400640 1 -- 172.21.15.60:0/1639710712 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa108003520 con 0x7fa108004d40 2024-04-14T23:48:23.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.916+0000 7fa123400640 1 --2- 172.21.15.60:0/1639710712 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa108004d40 0x7fa1080051c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa11c009fd0 tx=0x7fa108005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=913530473116aa03 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:23.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.920+0000 7fa122000640 1 -- 172.21.15.60:0/1639710712 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa11c039070 con 0x7fa108004d40 2024-04-14T23:48:23.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.920+0000 7fa122000640 1 -- 172.21.15.60:0/1639710712 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa11c02ccb0 con 0x7fa108004d40 2024-04-14T23:48:23.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.920+0000 7fa122000640 1 -- 172.21.15.60:0/1639710712 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa11c009bd0 con 0x7fa108004d40 2024-04-14T23:48:23.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.920+0000 7fa123e00640 1 -- 172.21.15.60:0/1639710712 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa108004d40 msgr2=0x7fa1080051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:23.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.920+0000 7fa123e00640 1 --2- 172.21.15.60:0/1639710712 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa108004d40 0x7fa1080051c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa11c009fd0 tx=0x7fa108005830 comp rx=0 tx=0).stop 2024-04-14T23:48:23.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.920+0000 7fa123400640 1 -- 172.21.15.60:0/1639710712 reap_dead start 2024-04-14T23:48:23.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.921+0000 7fa123e00640 1 -- 172.21.15.60:0/1639710712 shutdown_connections 2024-04-14T23:48:23.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.921+0000 7fa123e00640 1 -- 172.21.15.60:0/1639710712 >> 172.21.15.60:0/1639710712 conn(0x7fa124066f10 msgr2=0x7fa124067320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:23.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.921+0000 7fa123e00640 1 -- 172.21.15.60:0/1639710712 shutdown_connections 2024-04-14T23:48:23.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.921+0000 7fa123e00640 1 -- 172.21.15.60:0/1639710712 wait complete. 2024-04-14T23:48:23.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.923+0000 7fa123e00640 1 Processor -- start 2024-04-14T23:48:23.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.923+0000 7fa123e00640 1 -- start start 2024-04-14T23:48:23.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.924+0000 7fa123e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa108004d40 0x7fa1241f0910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:23.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.924+0000 7fa123e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa1240c7c10 0x7fa124156eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:23.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.924+0000 7fa123e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1241f3e80 0x7fa1240cc700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:23.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.924+0000 7fa123e00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fa12415ec40 con 0x7fa1241f3e80 2024-04-14T23:48:23.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.924+0000 7fa123e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa12415e960 con 0x7fa108004d40 2024-04-14T23:48:23.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.924+0000 7fa123e00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fa124155260 con 0x7fa1240c7c10 2024-04-14T23:48:23.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.924+0000 7fa122a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa108004d40 0x7fa1241f0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:23.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.924+0000 7fa122a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa108004d40 0x7fa1241f0910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:49494/0 (socket says 172.21.15.60:49494) 2024-04-14T23:48:23.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.924+0000 7fa122a00640 1 -- 172.21.15.60:0/673827560 learned_addr learned my addr 172.21.15.60:0/673827560 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:23.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.924+0000 7fa11be00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa1240c7c10 0x7fa124156eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:23.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.924+0000 7fa123400640 1 --2- 172.21.15.60:0/673827560 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1241f3e80 0x7fa1240cc700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:23.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.925+0000 7fa122a00640 1 -- 172.21.15.60:0/673827560 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa1240c7c10 msgr2=0x7fa124156eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:23.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.925+0000 7fa122a00640 1 --2- 172.21.15.60:0/673827560 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa1240c7c10 0x7fa124156eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:23.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.925+0000 7fa122a00640 1 -- 172.21.15.60:0/673827560 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1241f3e80 msgr2=0x7fa1240cc700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:23.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.925+0000 7fa122a00640 1 --2- 172.21.15.60:0/673827560 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1241f3e80 0x7fa1240cc700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:23.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.925+0000 7fa122a00640 1 -- 172.21.15.60:0/673827560 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1241f1030 con 0x7fa108004d40 2024-04-14T23:48:23.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.925+0000 7fa123400640 1 --2- 172.21.15.60:0/673827560 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1241f3e80 0x7fa1240cc700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-14T23:48:23.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.925+0000 7fa122a00640 1 --2- 172.21.15.60:0/673827560 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa108004d40 0x7fa1241f0910 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa11400f8e0 tx=0x7fa11400fda0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:23.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.925+0000 7fa11aa00640 1 -- 172.21.15.60:0/673827560 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1140053a0 con 0x7fa108004d40 2024-04-14T23:48:23.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.925+0000 7fa11aa00640 1 -- 172.21.15.60:0/673827560 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa114018070 con 0x7fa108004d40 2024-04-14T23:48:23.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.925+0000 7fa11aa00640 1 -- 172.21.15.60:0/673827560 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa11400d970 con 0x7fa108004d40 2024-04-14T23:48:23.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.925+0000 7fa123e00640 1 -- 172.21.15.60:0/673827560 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1241f1310 con 0x7fa108004d40 2024-04-14T23:48:23.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.925+0000 7fa123e00640 1 -- 172.21.15.60:0/673827560 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1241566b0 con 0x7fa108004d40 2024-04-14T23:48:23.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.925+0000 7fa11aa00640 1 -- 172.21.15.60:0/673827560 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa11400db10 con 0x7fa108004d40 2024-04-14T23:48:23.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.926+0000 7fa11aa00640 1 -- 172.21.15.60:0/673827560 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fa11401d5d0 con 0x7fa108004d40 2024-04-14T23:48:23.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.927+0000 7fa123e00640 1 -- 172.21.15.60:0/673827560 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1241f1830 con 0x7fa108004d40 2024-04-14T23:48:23.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:23.933+0000 7fa11aa00640 1 -- 172.21.15.60:0/673827560 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fa1241f1830 con 0x7fa108004d40 2024-04-14T23:48:24.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:24.007+0000 7fa123e00640 1 -- 172.21.15.60:0/673827560 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7fa12404f540 con 0x7fa108004d40 2024-04-14T23:48:24.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:24.014+0000 7fa11aa00640 1 -- 172.21.15.60:0/673827560 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7fa1140035f0 con 0x7fa108004d40 2024-04-14T23:48:24.020 INFO:teuthology.orchestra.run.smithi060.stderr:adjusted tunables profile to default 2024-04-14T23:48:24.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:24.016+0000 7fa123e00640 1 -- 172.21.15.60:0/673827560 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa108004d40 msgr2=0x7fa1241f0910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:24.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:24.016+0000 7fa123e00640 1 --2- 172.21.15.60:0/673827560 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa108004d40 0x7fa1241f0910 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa11400f8e0 tx=0x7fa11400fda0 comp rx=0 tx=0).stop 2024-04-14T23:48:24.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:24.016+0000 7fa123e00640 1 -- 172.21.15.60:0/673827560 shutdown_connections 2024-04-14T23:48:24.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:24.016+0000 7fa123e00640 1 --2- 172.21.15.60:0/673827560 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1241f3e80 0x7fa1240cc700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:24.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:24.016+0000 7fa123e00640 1 --2- 172.21.15.60:0/673827560 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa1240c7c10 0x7fa124156eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:24.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:24.016+0000 7fa123e00640 1 --2- 172.21.15.60:0/673827560 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa108004d40 0x7fa1241f0910 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:24.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:24.016+0000 7fa123e00640 1 -- 172.21.15.60:0/673827560 >> 172.21.15.60:0/673827560 conn(0x7fa124066f10 msgr2=0x7fa12415d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:24.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:24.016+0000 7fa123e00640 1 -- 172.21.15.60:0/673827560 shutdown_connections 2024-04-14T23:48:24.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:24.016+0000 7fa123e00640 1 -- 172.21.15.60:0/673827560 wait complete. 2024-04-14T23:48:24.040 INFO:tasks.ceph:check_enable_crimson: False 2024-04-14T23:48:24.040 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-14T23:48:24.040 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-04-14T23:48:24.040 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-14T23:48:24.126 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-04-14T23:48:24.127 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-14T23:48:24.262 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-04-14T23:48:24.262 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-14T23:48:24.359 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-04-14T23:48:24.359 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-04-14T23:48:24.378 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-14T23:48:24.373+0000 7ff99ec45040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-14T23:48:24.430 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-14T23:48:24.425+0000 7ff99ec45040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-14T23:48:24.454 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-14T23:48:24.454 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-04-14T23:48:24.480 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-14T23:48:24.481 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-04-14T23:48:24.547 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-14T23:48:24.547 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-04-14T23:48:24.548 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-04-14T23:48:24.543+0000 7ffaf7042040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-14T23:48:24.614 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-04-14T23:48:24.608+0000 7ffaf7042040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-14T23:48:24.615 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-14T23:48:24.615 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-04-14T23:48:24.681 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd new b2b2b755-7f42-4f9b-af9a-a178f7cc5a52 0 2024-04-14T23:48:24.823 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.818+0000 7faa5be05640 1 Processor -- start 2024-04-14T23:48:24.823 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.818+0000 7faa5be05640 1 -- start start 2024-04-14T23:48:24.824 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.819+0000 7faa5be05640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7faa5406ff00 0x7faa5406f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:24.825 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.819+0000 7faa5be05640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa540ce7a0 0x7faa540cebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:24.825 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.819+0000 7faa5be05640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa540c9130 con 0x7faa54152c30 2024-04-14T23:48:24.825 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.819+0000 7faa5be05640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7faa54159380 con 0x7faa5406ff00 2024-04-14T23:48:24.825 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.819+0000 7faa5be05640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7faa54159810 con 0x7faa540ce7a0 2024-04-14T23:48:24.825 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.819+0000 7faa593a0640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7faa5406ff00 0x7faa5406f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:24.825 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.819+0000 7faa593a0640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7faa5406ff00 0x7faa5406f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:59454/0 (socket says 172.21.15.186:59454) 2024-04-14T23:48:24.825 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.819+0000 7faa5a3a2640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa540ce7a0 0x7faa540cebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:24.825 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.819+0000 7faa593a0640 1 -- 172.21.15.186:0/1601108705 learned_addr learned my addr 172.21.15.186:0/1601108705 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-04-14T23:48:24.825 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.819+0000 7faa593a0640 1 -- 172.21.15.186:0/1601108705 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa540ce7a0 msgr2=0x7faa540cebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:24.825 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.819+0000 7faa593a0640 1 --2- 172.21.15.186:0/1601108705 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa540ce7a0 0x7faa540cebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:24.825 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.819+0000 7faa593a0640 1 -- 172.21.15.186:0/1601108705 >> v1:172.21.15.60:6789/0 conn(0x7faa54152c30 legacy=0x7faa5415be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:24.825 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.819+0000 7faa593a0640 1 -- 172.21.15.186:0/1601108705 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa54153010 con 0x7faa5406ff00 2024-04-14T23:48:24.825 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.819+0000 7faa5a3a2640 1 --2- 172.21.15.186:0/1601108705 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa540ce7a0 0x7faa540cebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:48:24.827 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.820+0000 7faa593a0640 1 --2- 172.21.15.186:0/1601108705 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa5406ff00 0x7faa5406f8b0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7faa50009870 tx=0x7faa5002b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e88f4f770f1a9e15 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:24.827 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.820+0000 7faa58b9f640 1 -- 172.21.15.186:0/1601108705 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa50035020 con 0x7faa5406ff00 2024-04-14T23:48:24.827 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.820+0000 7faa58b9f640 1 -- 172.21.15.186:0/1601108705 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa50031640 con 0x7faa5406ff00 2024-04-14T23:48:24.827 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.820+0000 7faa58b9f640 1 -- 172.21.15.186:0/1601108705 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa5002d040 con 0x7faa5406ff00 2024-04-14T23:48:24.827 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.821+0000 7faa5be05640 1 -- 172.21.15.186:0/1601108705 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa5406ff00 msgr2=0x7faa5406f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:24.827 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.821+0000 7faa5be05640 1 --2- 172.21.15.186:0/1601108705 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa5406ff00 0x7faa5406f8b0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7faa50009870 tx=0x7faa5002b8f0 comp rx=0 tx=0).stop 2024-04-14T23:48:24.827 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.821+0000 7faa5be05640 1 -- 172.21.15.186:0/1601108705 shutdown_connections 2024-04-14T23:48:24.827 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.821+0000 7faa5be05640 1 --2- 172.21.15.186:0/1601108705 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa5406ff00 0x7faa5406f8b0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:24.828 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.821+0000 7faa5be05640 1 --2- 172.21.15.186:0/1601108705 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa540ce7a0 0x7faa540cebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:24.828 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.821+0000 7faa5be05640 1 -- 172.21.15.186:0/1601108705 >> 172.21.15.186:0/1601108705 conn(0x7faa54067330 msgr2=0x7faa54067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:24.828 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.821+0000 7faa5be05640 1 -- 172.21.15.186:0/1601108705 shutdown_connections 2024-04-14T23:48:24.828 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.821+0000 7faa5be05640 1 -- 172.21.15.186:0/1601108705 wait complete. 2024-04-14T23:48:24.828 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.822+0000 7faa5be05640 1 Processor -- start 2024-04-14T23:48:24.828 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.822+0000 7faa5be05640 1 -- start start 2024-04-14T23:48:24.830 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.823+0000 7faa5be05640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa5406ff00 0x7faa541f2780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:24.830 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.823+0000 7faa5be05640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa540ce7a0 0x7faa54154c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:24.830 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.823+0000 7faa5be05640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa54152c30 0x7faa5414f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:24.830 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.823+0000 7faa5be05640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7faa54158d30 con 0x7faa5406ff00 2024-04-14T23:48:24.830 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.823+0000 7faa5be05640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7faa54158a50 con 0x7faa54152c30 2024-04-14T23:48:24.830 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.823+0000 7faa5be05640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7faa54159010 con 0x7faa540ce7a0 2024-04-14T23:48:24.830 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.823+0000 7faa593a0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa540ce7a0 0x7faa54154c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:24.830 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.823+0000 7faa5a3a2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa54152c30 0x7faa5414f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:24.831 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.823+0000 7faa593a0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa540ce7a0 0x7faa54154c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.186:58170/0 (socket says 172.21.15.186:58170) 2024-04-14T23:48:24.831 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.823+0000 7faa593a0640 1 -- 172.21.15.186:0/3482273917 learned_addr learned my addr 172.21.15.186:0/3482273917 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-04-14T23:48:24.831 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.823+0000 7faa593a0640 1 -- 172.21.15.186:0/3482273917 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa54152c30 msgr2=0x7faa5414f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:24.831 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.823+0000 7faa593a0640 1 --2- 172.21.15.186:0/3482273917 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa54152c30 0x7faa5414f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:24.831 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.823+0000 7faa593a0640 1 -- 172.21.15.186:0/3482273917 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa5406ff00 msgr2=0x7faa541f2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:24.831 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.823+0000 7faa593a0640 1 --2- 172.21.15.186:0/3482273917 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa5406ff00 0x7faa541f2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:24.831 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.823+0000 7faa593a0640 1 -- 172.21.15.186:0/3482273917 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa54155180 con 0x7faa540ce7a0 2024-04-14T23:48:24.831 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.823+0000 7faa593a0640 1 --2- 172.21.15.186:0/3482273917 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa540ce7a0 0x7faa54154c40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7faa5002b930 tx=0x7faa500046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:24.831 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.824+0000 7faa4a7fc640 1 -- 172.21.15.186:0/3482273917 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa50040070 con 0x7faa540ce7a0 2024-04-14T23:48:24.831 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.824+0000 7faa4a7fc640 1 -- 172.21.15.186:0/3482273917 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa500399f0 con 0x7faa540ce7a0 2024-04-14T23:48:24.831 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.824+0000 7faa5be05640 1 -- 172.21.15.186:0/3482273917 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa541504c0 con 0x7faa540ce7a0 2024-04-14T23:48:24.831 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.824+0000 7faa4a7fc640 1 -- 172.21.15.186:0/3482273917 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa50035040 con 0x7faa540ce7a0 2024-04-14T23:48:24.831 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.824+0000 7faa5be05640 1 -- 172.21.15.186:0/3482273917 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7faa540c7240 con 0x7faa540ce7a0 2024-04-14T23:48:24.831 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.824+0000 7faa4a7fc640 1 -- 172.21.15.186:0/3482273917 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7faa5004b060 con 0x7faa540ce7a0 2024-04-14T23:48:24.831 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.824+0000 7faa4a7fc640 1 -- 172.21.15.186:0/3482273917 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7faa5003a420 con 0x7faa540ce7a0 2024-04-14T23:48:24.832 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.825+0000 7faa5be05640 1 -- 172.21.15.186:0/3482273917 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa540684b0 con 0x7faa540ce7a0 2024-04-14T23:48:24.832 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.827+0000 7faa4a7fc640 1 -- 172.21.15.186:0/3482273917 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7faa5004d920 con 0x7faa540ce7a0 2024-04-14T23:48:24.905 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.900+0000 7faa5be05640 1 -- 172.21.15.186:0/3482273917 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "osd new", "uuid": "b2b2b755-7f42-4f9b-af9a-a178f7cc5a52", "id": 0} v 0) -- 0x7faa540c3740 con 0x7faa540ce7a0 2024-04-14T23:48:24.909 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.904+0000 7faa4a7fc640 1 -- 172.21.15.186:0/3482273917 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b2b2b755-7f42-4f9b-af9a-a178f7cc5a52", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7faa5004db00 con 0x7faa540ce7a0 2024-04-14T23:48:24.909 INFO:teuthology.orchestra.run.smithi186.stdout:0 2024-04-14T23:48:24.910 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.905+0000 7faa5be05640 1 -- 172.21.15.186:0/3482273917 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa540ce7a0 msgr2=0x7faa54154c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:24.911 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.905+0000 7faa5be05640 1 --2- 172.21.15.186:0/3482273917 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa540ce7a0 0x7faa54154c40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7faa5002b930 tx=0x7faa500046c0 comp rx=0 tx=0).stop 2024-04-14T23:48:24.911 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.905+0000 7faa5be05640 1 -- 172.21.15.186:0/3482273917 shutdown_connections 2024-04-14T23:48:24.911 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.905+0000 7faa5be05640 1 --2- 172.21.15.186:0/3482273917 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa54152c30 0x7faa5414f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:24.911 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.905+0000 7faa5be05640 1 --2- 172.21.15.186:0/3482273917 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa540ce7a0 0x7faa54154c40 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:24.911 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.905+0000 7faa5be05640 1 --2- 172.21.15.186:0/3482273917 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa5406ff00 0x7faa541f2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:24.911 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.905+0000 7faa5be05640 1 -- 172.21.15.186:0/3482273917 >> 172.21.15.186:0/3482273917 conn(0x7faa54067330 msgr2=0x7faa541566c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:24.911 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.905+0000 7faa5be05640 1 -- 172.21.15.186:0/3482273917 shutdown_connections 2024-04-14T23:48:24.911 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:24.905+0000 7faa5be05640 1 -- 172.21.15.186:0/3482273917 wait complete. 2024-04-14T23:48:24.921 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd new 321418cd-5ab7-41a5-b9c8-b3fc12f68ebc 1 2024-04-14T23:48:24.964 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-14T23:48:24.959+0000 7ff99ec45040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-14T23:48:25.021 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-14T23:48:25.016+0000 7ff99ec45040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-14T23:48:25.023 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.018+0000 7f6b9ea0f640 1 Processor -- start 2024-04-14T23:48:25.023 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.018+0000 7f6b9ea0f640 1 -- start start 2024-04-14T23:48:25.026 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.018+0000 7f6b9ea0f640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6b9815c140 0x7f6b98159740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:25.026 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.018+0000 7f6b9ea0f640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b980c7ea0 con 0x7f6b98152d00 2024-04-14T23:48:25.026 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.019+0000 7f6b9ea0f640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b980c9dc0 con 0x7f6b980c4300 2024-04-14T23:48:25.026 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.019+0000 7f6b9ea0f640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f6b981592b0 con 0x7f6b9815c140 2024-04-14T23:48:25.026 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.019+0000 7f6b9cfac640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6b9815c140 0x7f6b98159740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:25.026 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.019+0000 7f6b977fe640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f6b980c4300 0x7f6b980c6b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:42496/0 (socket says 172.21.15.186:42496) 2024-04-14T23:48:25.026 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.019+0000 7f6b977fe640 1 -- 172.21.15.186:0/2765809597 learned_addr learned my addr 172.21.15.186:0/2765809597 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-04-14T23:48:25.026 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.019+0000 7f6b96ffd640 1 -- 172.21.15.186:0/2765809597 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2119836330 0 0) 0x7f6b980c9dc0 con 0x7f6b980c4300 2024-04-14T23:48:25.026 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.019+0000 7f6b9cfac640 1 -- 172.21.15.186:0/2765809597 >> v1:172.21.15.186:6789/0 conn(0x7f6b980c4300 legacy=0x7f6b980c6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:25.026 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.019+0000 7f6b9cfac640 1 -- 172.21.15.186:0/2765809597 >> v1:172.21.15.60:6789/0 conn(0x7f6b98152d00 legacy=0x7f6b9806b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:25.027 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.019+0000 7f6b9cfac640 1 -- 172.21.15.186:0/2765809597 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b980c8140 con 0x7f6b9815c140 2024-04-14T23:48:25.027 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.020+0000 7f6b9cfac640 1 --2- 172.21.15.186:0/2765809597 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6b9815c140 0x7f6b98159740 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6b90009dc0 tx=0x7f6b9002bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b510666abb2922d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:25.027 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.020+0000 7f6b96ffd640 1 -- 172.21.15.186:0/2765809597 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b90039070 con 0x7f6b9815c140 2024-04-14T23:48:25.027 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.020+0000 7f6b96ffd640 1 -- 172.21.15.186:0/2765809597 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b9002ccd0 con 0x7f6b9815c140 2024-04-14T23:48:25.027 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.020+0000 7f6b96ffd640 1 -- 172.21.15.186:0/2765809597 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b90034660 con 0x7f6b9815c140 2024-04-14T23:48:25.027 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.020+0000 7f6b9ea0f640 1 -- 172.21.15.186:0/2765809597 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6b9815c140 msgr2=0x7f6b98159740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:25.027 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.020+0000 7f6b9ea0f640 1 --2- 172.21.15.186:0/2765809597 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6b9815c140 0x7f6b98159740 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6b90009dc0 tx=0x7f6b9002bf20 comp rx=0 tx=0).stop 2024-04-14T23:48:25.027 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.020+0000 7f6b9ea0f640 1 -- 172.21.15.186:0/2765809597 shutdown_connections 2024-04-14T23:48:25.027 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.020+0000 7f6b9ea0f640 1 --2- 172.21.15.186:0/2765809597 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6b9815c140 0x7f6b98159740 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:25.027 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.020+0000 7f6b9ea0f640 1 -- 172.21.15.186:0/2765809597 >> 172.21.15.186:0/2765809597 conn(0x7f6b9806c2d0 msgr2=0x7f6b9806c6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:25.027 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.021+0000 7f6b9ea0f640 1 -- 172.21.15.186:0/2765809597 shutdown_connections 2024-04-14T23:48:25.027 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.021+0000 7f6b9ea0f640 1 -- 172.21.15.186:0/2765809597 wait complete. 2024-04-14T23:48:25.030 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.022+0000 7f6b9ea0f640 1 Processor -- start 2024-04-14T23:48:25.030 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.022+0000 7f6b9ea0f640 1 -- start start 2024-04-14T23:48:25.030 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.022+0000 7f6b9ea0f640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6b980c4300 0x7f6b9814c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:25.030 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.022+0000 7f6b9ea0f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b98152d00 0x7f6b9814b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:25.030 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.022+0000 7f6b9ea0f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b9815c140 0x7f6b981f18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:25.030 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.022+0000 7f6b9ea0f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f6b980c8420 con 0x7f6b9815c140 2024-04-14T23:48:25.030 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.022+0000 7f6b9ea0f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6b98158f40 con 0x7f6b98152d00 2024-04-14T23:48:25.030 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.022+0000 7f6b9ea0f640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f6b980c85a0 con 0x7f6b980c4300 2024-04-14T23:48:25.030 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.023+0000 7f6b97fff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6b980c4300 0x7f6b9814c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:25.030 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.023+0000 7f6b977fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b98152d00 0x7f6b9814b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:25.030 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.023+0000 7f6b97fff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6b980c4300 0x7f6b9814c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.186:58202/0 (socket says 172.21.15.186:58202) 2024-04-14T23:48:25.030 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.023+0000 7f6b97fff640 1 -- 172.21.15.186:0/2443123588 learned_addr learned my addr 172.21.15.186:0/2443123588 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-04-14T23:48:25.031 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.023+0000 7f6b977fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b98152d00 0x7f6b9814b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:59472/0 (socket says 172.21.15.186:59472) 2024-04-14T23:48:25.031 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.023+0000 7f6b9cfac640 1 --2- 172.21.15.186:0/2443123588 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b9815c140 0x7f6b981f18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:25.031 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.023+0000 7f6b97fff640 1 -- 172.21.15.186:0/2443123588 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b98152d00 msgr2=0x7f6b9814b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:25.031 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.023+0000 7f6b97fff640 1 --2- 172.21.15.186:0/2443123588 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b98152d00 0x7f6b9814b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:25.031 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.023+0000 7f6b97fff640 1 -- 172.21.15.186:0/2443123588 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b9815c140 msgr2=0x7f6b981f18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:25.031 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.023+0000 7f6b97fff640 1 --2- 172.21.15.186:0/2443123588 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b9815c140 0x7f6b981f18e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:25.031 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.023+0000 7f6b97fff640 1 -- 172.21.15.186:0/2443123588 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b980cb2c0 con 0x7f6b980c4300 2024-04-14T23:48:25.031 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.023+0000 7f6b97fff640 1 --2- 172.21.15.186:0/2443123588 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6b980c4300 0x7f6b9814c680 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6b8800c440 tx=0x7f6b8800c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:25.031 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.023+0000 7f6b94ff9640 1 -- 172.21.15.186:0/2443123588 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b88017070 con 0x7f6b980c4300 2024-04-14T23:48:25.031 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.023+0000 7f6b94ff9640 1 -- 172.21.15.186:0/2443123588 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b880035f0 con 0x7f6b980c4300 2024-04-14T23:48:25.031 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.023+0000 7f6b9ea0f640 1 -- 172.21.15.186:0/2443123588 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b980cb560 con 0x7f6b980c4300 2024-04-14T23:48:25.031 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.023+0000 7f6b94ff9640 1 -- 172.21.15.186:0/2443123588 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b880039e0 con 0x7f6b980c4300 2024-04-14T23:48:25.031 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.023+0000 7f6b9ea0f640 1 -- 172.21.15.186:0/2443123588 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b980cd4b0 con 0x7f6b980c4300 2024-04-14T23:48:25.031 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.023+0000 7f6b94ff9640 1 -- 172.21.15.186:0/2443123588 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6b8801a070 con 0x7f6b980c4300 2024-04-14T23:48:25.032 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.023+0000 7f6b94ff9640 1 -- 172.21.15.186:0/2443123588 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7f6b880127b0 con 0x7f6b980c4300 2024-04-14T23:48:25.032 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.024+0000 7f6b9ea0f640 1 -- 172.21.15.186:0/2443123588 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b980684b0 con 0x7f6b980c4300 2024-04-14T23:48:25.032 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.026+0000 7f6b94ff9640 1 -- 172.21.15.186:0/2443123588 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f6b88021070 con 0x7f6b980c4300 2024-04-14T23:48:25.069 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-14T23:48:25.063+0000 7ff99ec45040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-14T23:48:25.104 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.099+0000 7f6b9ea0f640 1 -- 172.21.15.186:0/2443123588 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "osd new", "uuid": "321418cd-5ab7-41a5-b9c8-b3fc12f68ebc", "id": 1} v 0) -- 0x7f6b9806b290 con 0x7f6b980c4300 2024-04-14T23:48:25.109 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.104+0000 7f6b94ff9640 1 -- 172.21.15.186:0/2443123588 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "321418cd-5ab7-41a5-b9c8-b3fc12f68ebc", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7f6b8801c440 con 0x7f6b980c4300 2024-04-14T23:48:25.109 INFO:teuthology.orchestra.run.smithi186.stdout:1 2024-04-14T23:48:25.110 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.105+0000 7f6b9ea0f640 1 -- 172.21.15.186:0/2443123588 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6b980c4300 msgr2=0x7f6b9814c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:25.110 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.105+0000 7f6b9ea0f640 1 --2- 172.21.15.186:0/2443123588 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6b980c4300 0x7f6b9814c680 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6b8800c440 tx=0x7f6b8800c900 comp rx=0 tx=0).stop 2024-04-14T23:48:25.110 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.105+0000 7f6b9ea0f640 1 -- 172.21.15.186:0/2443123588 shutdown_connections 2024-04-14T23:48:25.110 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.105+0000 7f6b9ea0f640 1 --2- 172.21.15.186:0/2443123588 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b9815c140 0x7f6b981f18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:25.110 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.105+0000 7f6b9ea0f640 1 --2- 172.21.15.186:0/2443123588 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b98152d00 0x7f6b9814b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:25.110 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.105+0000 7f6b9ea0f640 1 --2- 172.21.15.186:0/2443123588 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6b980c4300 0x7f6b9814c680 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:25.111 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.105+0000 7f6b9ea0f640 1 -- 172.21.15.186:0/2443123588 >> 172.21.15.186:0/2443123588 conn(0x7f6b9806c2d0 msgr2=0x7f6b980c9b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:25.111 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.105+0000 7f6b9ea0f640 1 -- 172.21.15.186:0/2443123588 shutdown_connections 2024-04-14T23:48:25.111 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.105+0000 7f6b9ea0f640 1 -- 172.21.15.186:0/2443123588 wait complete. 2024-04-14T23:48:25.120 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd new 07874e9f-3a38-41dc-be9f-2b75e84e53d9 2 2024-04-14T23:48:25.125 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-14T23:48:25.120+0000 7ff99ec45040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-14T23:48:25.171 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-14T23:48:25.166+0000 7ff99ec45040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-14T23:48:25.226 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.221+0000 7ff9028de640 1 Processor -- start 2024-04-14T23:48:25.226 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.221+0000 7ff9028de640 1 -- start start 2024-04-14T23:48:25.227 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.222+0000 7ff9028de640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff8fc0cd370 0x7ff8fc0cd7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:25.227 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.222+0000 7ff9028de640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff8fc15e350 0x7ff8fc15e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:25.227 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.222+0000 7ff9028de640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8fc0707e0 con 0x7ff8fc155050 2024-04-14T23:48:25.227 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.222+0000 7ff9028de640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7ff8fc0c89d0 con 0x7ff8fc0cd370 2024-04-14T23:48:25.227 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.222+0000 7ff9028de640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7ff8fc0c8cb0 con 0x7ff8fc15e350 2024-04-14T23:48:25.228 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.222+0000 7ff900e7b640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff8fc15e350 0x7ff8fc15e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:25.228 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.222+0000 7ff900e7b640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff8fc15e350 0x7ff8fc15e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:59476/0 (socket says 172.21.15.186:59476) 2024-04-14T23:48:25.228 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.222+0000 7ff900e7b640 1 -- 172.21.15.186:0/3379252344 learned_addr learned my addr 172.21.15.186:0/3379252344 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-04-14T23:48:25.228 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.222+0000 7ff900e7b640 1 -- 172.21.15.186:0/3379252344 >> v2:172.21.15.60:3300/0 conn(0x7ff8fc0cd370 msgr2=0x7ff8fc0cd7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:25.228 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.222+0000 7ff900e7b640 1 --2- 172.21.15.186:0/3379252344 >> v2:172.21.15.60:3300/0 conn(0x7ff8fc0cd370 0x7ff8fc0cd7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:25.228 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.222+0000 7ff900e7b640 1 -- 172.21.15.186:0/3379252344 >> v1:172.21.15.60:6789/0 conn(0x7ff8fc155050 legacy=0x7ff8fc06b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:25.228 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.222+0000 7ff900e7b640 1 -- 172.21.15.186:0/3379252344 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8fc0c8f90 con 0x7ff8fc15e350 2024-04-14T23:48:25.229 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.223+0000 7ff900e7b640 1 --2- 172.21.15.186:0/3379252344 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff8fc15e350 0x7ff8fc15e730 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff8fc06b870 tx=0x7ff8f402bf80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=695037e97dce5a44 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:25.229 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.223+0000 7ff8faffd640 1 -- 172.21.15.186:0/3379252344 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8f402cb20 con 0x7ff8fc15e350 2024-04-14T23:48:25.229 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.223+0000 7ff8faffd640 1 -- 172.21.15.186:0/3379252344 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8f402ccc0 con 0x7ff8fc15e350 2024-04-14T23:48:25.229 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.223+0000 7ff8faffd640 1 -- 172.21.15.186:0/3379252344 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8f4034610 con 0x7ff8fc15e350 2024-04-14T23:48:25.230 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.224+0000 7ff9028de640 1 -- 172.21.15.186:0/3379252344 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff8fc15e350 msgr2=0x7ff8fc15e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:25.230 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.224+0000 7ff9028de640 1 --2- 172.21.15.186:0/3379252344 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff8fc15e350 0x7ff8fc15e730 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff8fc06b870 tx=0x7ff8f402bf80 comp rx=0 tx=0).stop 2024-04-14T23:48:25.230 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.224+0000 7ff9028de640 1 -- 172.21.15.186:0/3379252344 shutdown_connections 2024-04-14T23:48:25.230 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.224+0000 7ff9028de640 1 --2- 172.21.15.186:0/3379252344 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff8fc15e350 0x7ff8fc15e730 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:25.230 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.224+0000 7ff9028de640 1 --2- 172.21.15.186:0/3379252344 >> v2:172.21.15.60:3300/0 conn(0x7ff8fc0cd370 0x7ff8fc0cd7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:25.230 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.224+0000 7ff9028de640 1 -- 172.21.15.186:0/3379252344 >> 172.21.15.186:0/3379252344 conn(0x7ff8fc072080 msgr2=0x7ff8fc072490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:25.230 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.224+0000 7ff9028de640 1 -- 172.21.15.186:0/3379252344 shutdown_connections 2024-04-14T23:48:25.230 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.224+0000 7ff9028de640 1 -- 172.21.15.186:0/3379252344 wait complete. 2024-04-14T23:48:25.233 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.225+0000 7ff9028de640 1 Processor -- start 2024-04-14T23:48:25.233 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.225+0000 7ff9028de640 1 -- start start 2024-04-14T23:48:25.233 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.226+0000 7ff9028de640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff8fc0cd370 0x7ff8fc0cb490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:25.233 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.226+0000 7ff9028de640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff8fc155050 0x7ff8fc1f4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:25.233 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.226+0000 7ff9028de640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8fc15e350 0x7ff8fc1f4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:25.233 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.226+0000 7ff9028de640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff8fc15ec70 con 0x7ff8fc15e350 2024-04-14T23:48:25.234 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.226+0000 7ff9028de640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff8fc06c020 con 0x7ff8fc0cd370 2024-04-14T23:48:25.234 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.226+0000 7ff9028de640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7ff8fc155430 con 0x7ff8fc155050 2024-04-14T23:48:25.234 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.226+0000 7ff8fbfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff8fc0cd370 0x7ff8fc0cb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:25.234 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.226+0000 7ff8fb7fe640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff8fc155050 0x7ff8fc1f4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:25.234 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.226+0000 7ff8fb7fe640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff8fc155050 0x7ff8fc1f4ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.186:58204/0 (socket says 172.21.15.186:58204) 2024-04-14T23:48:25.234 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.226+0000 7ff8fbfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff8fc0cd370 0x7ff8fc0cb490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:59482/0 (socket says 172.21.15.186:59482) 2024-04-14T23:48:25.234 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.226+0000 7ff8fb7fe640 1 -- 172.21.15.186:0/1787482555 learned_addr learned my addr 172.21.15.186:0/1787482555 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-04-14T23:48:25.234 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.226+0000 7ff8fbfff640 1 -- 172.21.15.186:0/1787482555 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff8fc155050 msgr2=0x7ff8fc1f4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:25.234 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.226+0000 7ff8fbfff640 1 --2- 172.21.15.186:0/1787482555 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff8fc155050 0x7ff8fc1f4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:25.234 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.226+0000 7ff8fbfff640 1 -- 172.21.15.186:0/1787482555 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8fc15e350 msgr2=0x7ff8fc1f4ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:25.234 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.226+0000 7ff900e7b640 1 --2- 172.21.15.186:0/1787482555 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8fc15e350 0x7ff8fc1f4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:25.234 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.226+0000 7ff8fbfff640 1 --2- 172.21.15.186:0/1787482555 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8fc15e350 0x7ff8fc1f4ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:25.234 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.226+0000 7ff8fbfff640 1 -- 172.21.15.186:0/1787482555 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8fc1474a0 con 0x7ff8fc0cd370 2024-04-14T23:48:25.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.226+0000 7ff8fb7fe640 1 --2- 172.21.15.186:0/1787482555 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff8fc155050 0x7ff8fc1f4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:48:25.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.226+0000 7ff8fbfff640 1 --2- 172.21.15.186:0/1787482555 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff8fc0cd370 0x7ff8fc0cb490 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec00c470 tx=0x7ff8ec00c870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:25.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.227+0000 7ff8f8ff9640 1 -- 172.21.15.186:0/1787482555 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8ec017070 con 0x7ff8fc0cd370 2024-04-14T23:48:25.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.227+0000 7ff8f8ff9640 1 -- 172.21.15.186:0/1787482555 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8ec0035f0 con 0x7ff8fc0cd370 2024-04-14T23:48:25.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.227+0000 7ff8f8ff9640 1 -- 172.21.15.186:0/1787482555 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8ec0039e0 con 0x7ff8fc0cd370 2024-04-14T23:48:25.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.227+0000 7ff9028de640 1 -- 172.21.15.186:0/1787482555 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8fc1e7190 con 0x7ff8fc0cd370 2024-04-14T23:48:25.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.227+0000 7ff9028de640 1 -- 172.21.15.186:0/1787482555 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8fc1e7750 con 0x7ff8fc0cd370 2024-04-14T23:48:25.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.227+0000 7ff8f8ff9640 1 -- 172.21.15.186:0/1787482555 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff8ec01a070 con 0x7ff8fc0cd370 2024-04-14T23:48:25.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.227+0000 7ff8f8ff9640 1 -- 172.21.15.186:0/1787482555 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7ff8ec012820 con 0x7ff8fc0cd370 2024-04-14T23:48:25.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.228+0000 7ff9028de640 1 -- 172.21.15.186:0/1787482555 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8fc0684b0 con 0x7ff8fc0cd370 2024-04-14T23:48:25.236 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.230+0000 7ff8f8ff9640 1 -- 172.21.15.186:0/1787482555 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7ff8ec021070 con 0x7ff8fc0cd370 2024-04-14T23:48:25.281 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-04-14T23:48:25.276+0000 7ffaf7042040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-14T23:48:25.292 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-14T23:48:25.288+0000 7ff99ec45040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-14T23:48:25.307 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.302+0000 7ff9028de640 1 -- 172.21.15.186:0/1787482555 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd new", "uuid": "07874e9f-3a38-41dc-be9f-2b75e84e53d9", "id": 2} v 0) -- 0x7ff8fc15bc20 con 0x7ff8fc0cd370 2024-04-14T23:48:25.311 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.306+0000 7ff8f8ff9640 1 -- 172.21.15.186:0/1787482555 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "07874e9f-3a38-41dc-be9f-2b75e84e53d9", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7ff8ec012400 con 0x7ff8fc0cd370 2024-04-14T23:48:25.312 INFO:teuthology.orchestra.run.smithi186.stdout:2 2024-04-14T23:48:25.314 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.307+0000 7ff9028de640 1 -- 172.21.15.186:0/1787482555 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff8fc0cd370 msgr2=0x7ff8fc0cb490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:25.314 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.307+0000 7ff9028de640 1 --2- 172.21.15.186:0/1787482555 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff8fc0cd370 0x7ff8fc0cb490 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec00c470 tx=0x7ff8ec00c870 comp rx=0 tx=0).stop 2024-04-14T23:48:25.314 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.307+0000 7ff9028de640 1 -- 172.21.15.186:0/1787482555 shutdown_connections 2024-04-14T23:48:25.314 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.307+0000 7ff9028de640 1 --2- 172.21.15.186:0/1787482555 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8fc15e350 0x7ff8fc1f4ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:25.314 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.307+0000 7ff9028de640 1 --2- 172.21.15.186:0/1787482555 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff8fc155050 0x7ff8fc1f4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:25.314 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.307+0000 7ff9028de640 1 --2- 172.21.15.186:0/1787482555 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff8fc0cd370 0x7ff8fc0cb490 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:25.315 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.307+0000 7ff9028de640 1 -- 172.21.15.186:0/1787482555 >> 172.21.15.186:0/1787482555 conn(0x7ff8fc072080 msgr2=0x7ff8fc151eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:25.315 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.308+0000 7ff9028de640 1 -- 172.21.15.186:0/1787482555 shutdown_connections 2024-04-14T23:48:25.315 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.308+0000 7ff9028de640 1 -- 172.21.15.186:0/1787482555 wait complete. 2024-04-14T23:48:25.324 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd new 7e4ef6c9-b17a-47c6-a320-f5d16a6fe05e 3 2024-04-14T23:48:25.350 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-04-14T23:48:25.344+0000 7ffaf7042040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-14T23:48:25.368 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-14T23:48:25.363+0000 7ff99ec45040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-14T23:48:25.403 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-04-14T23:48:25.398+0000 7ffaf7042040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-14T23:48:25.416 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-14T23:48:25.411+0000 7ff99ec45040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-14T23:48:25.425 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.420+0000 7ff85362d640 1 Processor -- start 2024-04-14T23:48:25.425 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.420+0000 7ff85362d640 1 -- start start 2024-04-14T23:48:25.427 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.421+0000 7ff85362d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff84c154dc0 0x7ff84c06fd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:25.427 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.421+0000 7ff85362d640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff84c06e170 0x7ff84c0c32f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:25.428 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.421+0000 7ff85362d640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff84c06fa90 con 0x7ff84c0c3830 2024-04-14T23:48:25.428 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.421+0000 7ff85362d640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7ff84c15ba70 con 0x7ff84c06e170 2024-04-14T23:48:25.428 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.421+0000 7ff85362d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7ff84c15acb0 con 0x7ff84c154dc0 2024-04-14T23:48:25.428 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.421+0000 7ff8513c9640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff84c154dc0 0x7ff84c06fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:25.428 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.421+0000 7ff850bc8640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff84c06e170 0x7ff84c0c32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:25.428 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.421+0000 7ff8513c9640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff84c154dc0 0x7ff84c06fd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.186:58214/0 (socket says 172.21.15.186:58214) 2024-04-14T23:48:25.428 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.421+0000 7ff8513c9640 1 -- 172.21.15.186:0/2533876186 learned_addr learned my addr 172.21.15.186:0/2533876186 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-04-14T23:48:25.428 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.422+0000 7ff8513c9640 1 -- 172.21.15.186:0/2533876186 >> v2:172.21.15.186:3300/0 conn(0x7ff84c06e170 msgr2=0x7ff84c0c32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:25.428 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.422+0000 7ff8513c9640 1 --2- 172.21.15.186:0/2533876186 >> v2:172.21.15.186:3300/0 conn(0x7ff84c06e170 0x7ff84c0c32f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:25.428 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.422+0000 7ff8513c9640 1 -- 172.21.15.186:0/2533876186 >> v1:172.21.15.60:6789/0 conn(0x7ff84c0c3830 legacy=0x7ff84c06c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:25.428 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.422+0000 7ff8513c9640 1 -- 172.21.15.186:0/2533876186 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff84c15af90 con 0x7ff84c154dc0 2024-04-14T23:48:25.429 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.422+0000 7ff8513c9640 1 --2- 172.21.15.186:0/2533876186 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff84c154dc0 0x7ff84c06fd70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff840009870 tx=0x7ff84002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90a7f975c537de09 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:25.429 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.423+0000 7ff83ffff640 1 -- 172.21.15.186:0/2533876186 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff840034070 con 0x7ff84c154dc0 2024-04-14T23:48:25.429 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.423+0000 7ff83ffff640 1 -- 172.21.15.186:0/2533876186 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff840030840 con 0x7ff84c154dc0 2024-04-14T23:48:25.429 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.423+0000 7ff83ffff640 1 -- 172.21.15.186:0/2533876186 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff840030c10 con 0x7ff84c154dc0 2024-04-14T23:48:25.433 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.423+0000 7ff85362d640 1 -- 172.21.15.186:0/2533876186 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff84c154dc0 msgr2=0x7ff84c06fd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:25.434 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.423+0000 7ff85362d640 1 --2- 172.21.15.186:0/2533876186 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff84c154dc0 0x7ff84c06fd70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff840009870 tx=0x7ff84002b980 comp rx=0 tx=0).stop 2024-04-14T23:48:25.434 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.423+0000 7ff85362d640 1 -- 172.21.15.186:0/2533876186 shutdown_connections 2024-04-14T23:48:25.434 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.423+0000 7ff85362d640 1 --2- 172.21.15.186:0/2533876186 >> v2:172.21.15.186:3300/0 conn(0x7ff84c06e170 0x7ff84c0c32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:25.434 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.423+0000 7ff85362d640 1 --2- 172.21.15.186:0/2533876186 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff84c154dc0 0x7ff84c06fd70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:25.434 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.423+0000 7ff85362d640 1 -- 172.21.15.186:0/2533876186 >> 172.21.15.186:0/2533876186 conn(0x7ff84c067330 msgr2=0x7ff84c067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:25.434 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.423+0000 7ff85362d640 1 -- 172.21.15.186:0/2533876186 shutdown_connections 2024-04-14T23:48:25.435 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.423+0000 7ff85362d640 1 -- 172.21.15.186:0/2533876186 wait complete. 2024-04-14T23:48:25.435 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.424+0000 7ff85362d640 1 Processor -- start 2024-04-14T23:48:25.435 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.424+0000 7ff85362d640 1 -- start start 2024-04-14T23:48:25.435 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.425+0000 7ff85362d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff84c06e170 0x7ff84c149810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:25.435 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.425+0000 7ff85362d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff84c0c3830 0x7ff84c147b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:25.435 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.425+0000 7ff85362d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff84c154dc0 0x7ff84c155570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:25.435 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.425+0000 7ff85362d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff84c1551a0 con 0x7ff84c154dc0 2024-04-14T23:48:25.435 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.425+0000 7ff85362d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff84c15b550 con 0x7ff84c06e170 2024-04-14T23:48:25.435 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.425+0000 7ff85362d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7ff84c06f750 con 0x7ff84c0c3830 2024-04-14T23:48:25.435 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.425+0000 7ff8513c9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff84c06e170 0x7ff84c149810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:25.435 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.425+0000 7ff850bc8640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff84c0c3830 0x7ff84c147b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:25.435 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.425+0000 7ff8513c9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff84c06e170 0x7ff84c149810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:59500/0 (socket says 172.21.15.186:59500) 2024-04-14T23:48:25.435 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.425+0000 7ff850bc8640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff84c0c3830 0x7ff84c147b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.186:58226/0 (socket says 172.21.15.186:58226) 2024-04-14T23:48:25.435 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.425+0000 7ff8513c9640 1 -- 172.21.15.186:0/791989223 learned_addr learned my addr 172.21.15.186:0/791989223 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-04-14T23:48:25.436 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.425+0000 7ff850bc8640 1 -- 172.21.15.186:0/791989223 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff84c06e170 msgr2=0x7ff84c149810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:25.436 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.425+0000 7ff850bc8640 1 --2- 172.21.15.186:0/791989223 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff84c06e170 0x7ff84c149810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:25.436 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.425+0000 7ff851bca640 1 --2- 172.21.15.186:0/791989223 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff84c154dc0 0x7ff84c155570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:25.436 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.425+0000 7ff850bc8640 1 -- 172.21.15.186:0/791989223 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff84c154dc0 msgr2=0x7ff84c155570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:25.436 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.426+0000 7ff850bc8640 1 --2- 172.21.15.186:0/791989223 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff84c154dc0 0x7ff84c155570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:25.436 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.426+0000 7ff850bc8640 1 -- 172.21.15.186:0/791989223 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff84c14a4d0 con 0x7ff84c0c3830 2024-04-14T23:48:25.436 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.426+0000 7ff850bc8640 1 --2- 172.21.15.186:0/791989223 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff84c0c3830 0x7ff84c147b30 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff84800e880 tx=0x7ff84800ed40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:25.436 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.426+0000 7ff83dffb640 1 -- 172.21.15.186:0/791989223 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff848017070 con 0x7ff84c0c3830 2024-04-14T23:48:25.436 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.426+0000 7ff83dffb640 1 -- 172.21.15.186:0/791989223 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff84800c940 con 0x7ff84c0c3830 2024-04-14T23:48:25.436 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.426+0000 7ff85362d640 1 -- 172.21.15.186:0/791989223 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff84c14a790 con 0x7ff84c0c3830 2024-04-14T23:48:25.436 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.426+0000 7ff83dffb640 1 -- 172.21.15.186:0/791989223 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff84800cd30 con 0x7ff84c0c3830 2024-04-14T23:48:25.436 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.426+0000 7ff85362d640 1 -- 172.21.15.186:0/791989223 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff84c1f10e0 con 0x7ff84c0c3830 2024-04-14T23:48:25.437 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.426+0000 7ff83dffb640 1 -- 172.21.15.186:0/791989223 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff84801a070 con 0x7ff84c0c3830 2024-04-14T23:48:25.437 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.426+0000 7ff83dffb640 1 -- 172.21.15.186:0/791989223 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7ff848013470 con 0x7ff84c0c3830 2024-04-14T23:48:25.437 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.427+0000 7ff85362d640 1 -- 172.21.15.186:0/791989223 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff84c0684b0 con 0x7ff84c0c3830 2024-04-14T23:48:25.437 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.429+0000 7ff83dffb640 1 -- 172.21.15.186:0/791989223 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7ff848006020 con 0x7ff84c0c3830 2024-04-14T23:48:25.469 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-04-14T23:48:25.464+0000 7ffaf7042040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-14T23:48:25.471 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-14T23:48:25.466+0000 7ff99ec45040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-14T23:48:25.506 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.501+0000 7ff85362d640 1 -- 172.21.15.186:0/791989223 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "osd new", "uuid": "7e4ef6c9-b17a-47c6-a320-f5d16a6fe05e", "id": 3} v 0) -- 0x7ff84c0c3610 con 0x7ff84c0c3830 2024-04-14T23:48:25.510 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.505+0000 7ff83dffb640 1 -- 172.21.15.186:0/791989223 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7e4ef6c9-b17a-47c6-a320-f5d16a6fe05e", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7ff848006520 con 0x7ff84c0c3830 2024-04-14T23:48:25.510 INFO:teuthology.orchestra.run.smithi186.stdout:3 2024-04-14T23:48:25.512 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.506+0000 7ff85362d640 1 -- 172.21.15.186:0/791989223 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff84c0c3830 msgr2=0x7ff84c147b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:25.512 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.506+0000 7ff85362d640 1 --2- 172.21.15.186:0/791989223 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff84c0c3830 0x7ff84c147b30 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff84800e880 tx=0x7ff84800ed40 comp rx=0 tx=0).stop 2024-04-14T23:48:25.512 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.506+0000 7ff85362d640 1 -- 172.21.15.186:0/791989223 shutdown_connections 2024-04-14T23:48:25.512 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.506+0000 7ff85362d640 1 --2- 172.21.15.186:0/791989223 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff84c154dc0 0x7ff84c155570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:25.512 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.506+0000 7ff85362d640 1 --2- 172.21.15.186:0/791989223 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff84c0c3830 0x7ff84c147b30 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:25.512 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.506+0000 7ff85362d640 1 --2- 172.21.15.186:0/791989223 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff84c06e170 0x7ff84c149810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:25.512 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.506+0000 7ff85362d640 1 -- 172.21.15.186:0/791989223 >> 172.21.15.186:0/791989223 conn(0x7ff84c067330 msgr2=0x7ff84c06c240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:25.512 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.506+0000 7ff85362d640 1 -- 172.21.15.186:0/791989223 shutdown_connections 2024-04-14T23:48:25.512 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.506+0000 7ff85362d640 1 -- 172.21.15.186:0/791989223 wait complete. 2024-04-14T23:48:25.520 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-14T23:48:25.515+0000 7ff99ec45040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-14T23:48:25.522 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd new 759a3efc-dade-44c5-b57a-1623499aba50 4 2024-04-14T23:48:25.523 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-04-14T23:48:25.517+0000 7ffaf7042040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-14T23:48:25.621 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.616+0000 7fcbbc715640 1 Processor -- start 2024-04-14T23:48:25.622 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.616+0000 7fcbbc715640 1 -- start start 2024-04-14T23:48:25.624 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.617+0000 7fcbbc715640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcbb4152c30 0x7fcbb406ff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:25.624 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.617+0000 7fcbbc715640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fcbb40ce7a0 0x7fcbb40cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:25.624 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.617+0000 7fcbbc715640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbb40c9130 con 0x7fcbb406f8b0 2024-04-14T23:48:25.625 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.617+0000 7fcbbc715640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fcbb4159380 con 0x7fcbb40ce7a0 2024-04-14T23:48:25.625 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.617+0000 7fcbbc715640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fcbb4159810 con 0x7fcbb4152c30 2024-04-14T23:48:25.625 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.617+0000 7fcbba4b1640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcbb4152c30 0x7fcbb406ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:25.625 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.617+0000 7fcbba4b1640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcbb4152c30 0x7fcbb406ff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.186:58234/0 (socket says 172.21.15.186:58234) 2024-04-14T23:48:25.625 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.617+0000 7fcbba4b1640 1 -- 172.21.15.186:0/272113476 learned_addr learned my addr 172.21.15.186:0/272113476 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-04-14T23:48:25.625 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.618+0000 7fcbba4b1640 1 -- 172.21.15.186:0/272113476 >> v2:172.21.15.60:3300/0 conn(0x7fcbb40ce7a0 msgr2=0x7fcbb40cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:25.625 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.618+0000 7fcbba4b1640 1 --2- 172.21.15.186:0/272113476 >> v2:172.21.15.60:3300/0 conn(0x7fcbb40ce7a0 0x7fcbb40cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:25.625 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.618+0000 7fcbba4b1640 1 -- 172.21.15.186:0/272113476 >> v1:172.21.15.60:6789/0 conn(0x7fcbb406f8b0 legacy=0x7fcbb415be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:25.625 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.618+0000 7fcbba4b1640 1 -- 172.21.15.186:0/272113476 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbb406fcf0 con 0x7fcbb4152c30 2024-04-14T23:48:25.625 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.618+0000 7fcbba4b1640 1 --2- 172.21.15.186:0/272113476 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcbb4152c30 0x7fcbb406ff00 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fcba8009870 tx=0x7fcba802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e45b7b89d2acce9e server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:25.625 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.618+0000 7fcbb94af640 1 -- 172.21.15.186:0/272113476 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcba8035020 con 0x7fcbb4152c30 2024-04-14T23:48:25.625 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.619+0000 7fcbb94af640 1 -- 172.21.15.186:0/272113476 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcba8031640 con 0x7fcbb4152c30 2024-04-14T23:48:25.626 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.619+0000 7fcbb94af640 1 -- 172.21.15.186:0/272113476 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcba802d040 con 0x7fcbb4152c30 2024-04-14T23:48:25.626 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.619+0000 7fcbbc715640 1 -- 172.21.15.186:0/272113476 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcbb4152c30 msgr2=0x7fcbb406ff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:25.626 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.619+0000 7fcbbc715640 1 --2- 172.21.15.186:0/272113476 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcbb4152c30 0x7fcbb406ff00 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fcba8009870 tx=0x7fcba802b8f0 comp rx=0 tx=0).stop 2024-04-14T23:48:25.626 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.619+0000 7fcbbc715640 1 -- 172.21.15.186:0/272113476 shutdown_connections 2024-04-14T23:48:25.626 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.619+0000 7fcbbc715640 1 --2- 172.21.15.186:0/272113476 >> v2:172.21.15.60:3300/0 conn(0x7fcbb40ce7a0 0x7fcbb40cebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:25.626 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.619+0000 7fcbbc715640 1 --2- 172.21.15.186:0/272113476 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcbb4152c30 0x7fcbb406ff00 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:25.626 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.619+0000 7fcbbc715640 1 -- 172.21.15.186:0/272113476 >> 172.21.15.186:0/272113476 conn(0x7fcbb4067330 msgr2=0x7fcbb4067740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:25.626 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.619+0000 7fcbbc715640 1 -- 172.21.15.186:0/272113476 shutdown_connections 2024-04-14T23:48:25.626 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.619+0000 7fcbbc715640 1 -- 172.21.15.186:0/272113476 wait complete. 2024-04-14T23:48:25.629 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.620+0000 7fcbbc715640 1 Processor -- start 2024-04-14T23:48:25.629 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.620+0000 7fcbbc715640 1 -- start start 2024-04-14T23:48:25.629 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.621+0000 7fcbbc715640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbb406f8b0 0x7fcbb4155010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:25.629 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.621+0000 7fcbbc715640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcbb40ce7a0 0x7fcbb414b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:25.629 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.621+0000 7fcbbc715640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcbb4152c30 0x7fcbb41eebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:25.629 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.621+0000 7fcbbc715640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb4158a50 con 0x7fcbb406f8b0 2024-04-14T23:48:25.629 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.621+0000 7fcbbc715640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb40c8e50 con 0x7fcbb40ce7a0 2024-04-14T23:48:25.629 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.621+0000 7fcbbc715640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fcbb4158d30 con 0x7fcbb4152c30 2024-04-14T23:48:25.630 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.621+0000 7fcbb9cb0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcbb40ce7a0 0x7fcbb414b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:25.630 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.621+0000 7fcbbacb2640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcbb4152c30 0x7fcbb41eebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:25.630 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.621+0000 7fcbb9cb0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcbb40ce7a0 0x7fcbb414b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:59512/0 (socket says 172.21.15.186:59512) 2024-04-14T23:48:25.630 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.621+0000 7fcbb9cb0640 1 -- 172.21.15.186:0/1863832029 learned_addr learned my addr 172.21.15.186:0/1863832029 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-04-14T23:48:25.630 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.621+0000 7fcbba4b1640 1 --2- 172.21.15.186:0/1863832029 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbb406f8b0 0x7fcbb4155010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:25.630 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.621+0000 7fcbb9cb0640 1 -- 172.21.15.186:0/1863832029 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcbb4152c30 msgr2=0x7fcbb41eebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:25.630 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.621+0000 7fcbb9cb0640 1 --2- 172.21.15.186:0/1863832029 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcbb4152c30 0x7fcbb41eebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:25.630 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.621+0000 7fcbb9cb0640 1 -- 172.21.15.186:0/1863832029 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbb406f8b0 msgr2=0x7fcbb4155010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:25.630 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.621+0000 7fcbb9cb0640 1 --2- 172.21.15.186:0/1863832029 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbb406f8b0 0x7fcbb4155010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:25.630 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.621+0000 7fcbb9cb0640 1 -- 172.21.15.186:0/1863832029 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbb41f2930 con 0x7fcbb40ce7a0 2024-04-14T23:48:25.631 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.621+0000 7fcbba4b1640 1 --2- 172.21.15.186:0/1863832029 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbb406f8b0 0x7fcbb4155010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-14T23:48:25.631 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.621+0000 7fcbbacb2640 1 --2- 172.21.15.186:0/1863832029 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcbb4152c30 0x7fcbb41eebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:48:25.631 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.622+0000 7fcbb9cb0640 1 --2- 172.21.15.186:0/1863832029 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcbb40ce7a0 0x7fcbb414b7c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fcbb000c410 tx=0x7fcbb000c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:25.631 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.622+0000 7fcba6ffd640 1 -- 172.21.15.186:0/1863832029 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb0017070 con 0x7fcbb40ce7a0 2024-04-14T23:48:25.631 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.622+0000 7fcba6ffd640 1 -- 172.21.15.186:0/1863832029 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbb00035f0 con 0x7fcbb40ce7a0 2024-04-14T23:48:25.631 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.622+0000 7fcba6ffd640 1 -- 172.21.15.186:0/1863832029 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb00039e0 con 0x7fcbb40ce7a0 2024-04-14T23:48:25.631 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.622+0000 7fcbbc715640 1 -- 172.21.15.186:0/1863832029 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbb4155ed0 con 0x7fcbb40ce7a0 2024-04-14T23:48:25.631 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.622+0000 7fcbbc715640 1 -- 172.21.15.186:0/1863832029 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbb4154910 con 0x7fcbb40ce7a0 2024-04-14T23:48:25.631 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.622+0000 7fcba6ffd640 1 -- 172.21.15.186:0/1863832029 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fcbb001a070 con 0x7fcbb40ce7a0 2024-04-14T23:48:25.631 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.622+0000 7fcba6ffd640 1 -- 172.21.15.186:0/1863832029 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7fcbb0012910 con 0x7fcbb40ce7a0 2024-04-14T23:48:25.631 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.623+0000 7fcbbc715640 1 -- 172.21.15.186:0/1863832029 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbb40684b0 con 0x7fcbb40ce7a0 2024-04-14T23:48:25.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.625+0000 7fcba6ffd640 1 -- 172.21.15.186:0/1863832029 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fcbb0021070 con 0x7fcbb40ce7a0 2024-04-14T23:48:25.660 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-04-14T23:48:25.654+0000 7ffaf7042040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-14T23:48:25.701 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.696+0000 7fcbbc715640 1 -- 172.21.15.186:0/1863832029 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd new", "uuid": "759a3efc-dade-44c5-b57a-1623499aba50", "id": 4} v 0) -- 0x7fcbb4159000 con 0x7fcbb40ce7a0 2024-04-14T23:48:25.706 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.701+0000 7fcba6ffd640 1 -- 172.21.15.186:0/1863832029 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "759a3efc-dade-44c5-b57a-1623499aba50", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7fcbb0003d60 con 0x7fcbb40ce7a0 2024-04-14T23:48:25.706 INFO:teuthology.orchestra.run.smithi186.stdout:4 2024-04-14T23:48:25.707 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.702+0000 7fcbbc715640 1 -- 172.21.15.186:0/1863832029 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcbb40ce7a0 msgr2=0x7fcbb414b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:25.707 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.702+0000 7fcbbc715640 1 --2- 172.21.15.186:0/1863832029 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcbb40ce7a0 0x7fcbb414b7c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fcbb000c410 tx=0x7fcbb000c8d0 comp rx=0 tx=0).stop 2024-04-14T23:48:25.707 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.702+0000 7fcbbc715640 1 -- 172.21.15.186:0/1863832029 shutdown_connections 2024-04-14T23:48:25.707 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.702+0000 7fcbbc715640 1 --2- 172.21.15.186:0/1863832029 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcbb4152c30 0x7fcbb41eebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:25.707 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.702+0000 7fcbbc715640 1 --2- 172.21.15.186:0/1863832029 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcbb40ce7a0 0x7fcbb414b7c0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:25.707 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.702+0000 7fcbbc715640 1 --2- 172.21.15.186:0/1863832029 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbb406f8b0 0x7fcbb4155010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:25.708 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.702+0000 7fcbbc715640 1 -- 172.21.15.186:0/1863832029 >> 172.21.15.186:0/1863832029 conn(0x7fcbb4067330 msgr2=0x7fcbb40c84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:25.708 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.702+0000 7fcbbc715640 1 -- 172.21.15.186:0/1863832029 shutdown_connections 2024-04-14T23:48:25.708 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.702+0000 7fcbbc715640 1 -- 172.21.15.186:0/1863832029 wait complete. 2024-04-14T23:48:25.717 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd new 24cd58c1-0cf1-4077-ab33-84c39c6907ea 5 2024-04-14T23:48:25.746 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-04-14T23:48:25.740+0000 7ffaf7042040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-14T23:48:25.801 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-04-14T23:48:25.796+0000 7ffaf7042040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-14T23:48:25.819 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.814+0000 7f94394c8640 1 Processor -- start 2024-04-14T23:48:25.820 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.814+0000 7f94394c8640 1 -- start start 2024-04-14T23:48:25.822 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.815+0000 7f94394c8640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f943415ea60 0x7f94340cd7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:25.822 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.815+0000 7f94394c8640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94340707e0 con 0x7f9434155050 2024-04-14T23:48:25.822 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.815+0000 7f94394c8640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94340c89d0 con 0x7f94340cd370 2024-04-14T23:48:25.822 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.815+0000 7f94394c8640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f94340c8cb0 con 0x7f943415ea60 2024-04-14T23:48:25.822 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.815+0000 7f94327fc640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f94340cd370 0x7f943415e350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:42512/0 (socket says 172.21.15.186:42512) 2024-04-14T23:48:25.823 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.815+0000 7f94327fc640 1 -- 172.21.15.186:0/1799496621 learned_addr learned my addr 172.21.15.186:0/1799496621 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-04-14T23:48:25.823 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.815+0000 7f9431ffb640 1 -- 172.21.15.186:0/1799496621 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1745871186 0 0) 0x7f94340c89d0 con 0x7f94340cd370 2024-04-14T23:48:25.823 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.815+0000 7f9431ffb640 1 -- 172.21.15.186:0/1799496621 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9420003520 con 0x7f94340cd370 2024-04-14T23:48:25.823 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.815+0000 7f94337fe640 1 --2- 172.21.15.186:0/1799496621 >> v2:172.21.15.60:3300/0 conn(0x7f943415ea60 0x7f94340cd7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:25.823 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.816+0000 7f9431ffb640 1 -- 172.21.15.186:0/1799496621 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2954658271 0 0) 0x7f9420003520 con 0x7f94340cd370 2024-04-14T23:48:25.823 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.816+0000 7f9431ffb640 1 -- 172.21.15.186:0/1799496621 >> v2:172.21.15.60:3300/0 conn(0x7f943415ea60 msgr2=0x7f94340cd7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:25.823 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.816+0000 7f9431ffb640 1 --2- 172.21.15.186:0/1799496621 >> v2:172.21.15.60:3300/0 conn(0x7f943415ea60 0x7f94340cd7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:25.823 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.816+0000 7f9431ffb640 1 -- 172.21.15.186:0/1799496621 >> v1:172.21.15.60:6789/0 conn(0x7f9434155050 legacy=0x7f943406b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:25.823 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.816+0000 7f9431ffb640 1 -- 172.21.15.186:0/1799496621 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94340c8f90 con 0x7f94340cd370 2024-04-14T23:48:25.823 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.816+0000 7f9431ffb640 1 -- 172.21.15.186:0/1799496621 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2814038680 0 0) 0x7f9424001670 con 0x7f94340cd370 2024-04-14T23:48:25.823 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.816+0000 7f9431ffb640 1 -- 172.21.15.186:0/1799496621 >> v1:172.21.15.186:6789/0 conn(0x7f94340cd370 legacy=0x7f943415e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:25.823 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.816+0000 7f9431ffb640 1 --2- 172.21.15.186:0/1799496621 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f94200039b0 0x7f9420003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:25.823 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.816+0000 7f9431ffb640 1 --2- 172.21.15.186:0/1799496621 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9420004800 0x7f9420004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:25.823 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.816+0000 7f9431ffb640 1 --2- 172.21.15.186:0/1799496621 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94200051c0 0x7f942000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:25.824 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.816+0000 7f9431ffb640 1 -- 172.21.15.186:0/1799496621 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f94340c8cb0 con 0x7f94200051c0 2024-04-14T23:48:25.824 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.816+0000 7f9431ffb640 1 -- 172.21.15.186:0/1799496621 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f942000c0d0 con 0x7f9420004800 2024-04-14T23:48:25.824 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.816+0000 7f9431ffb640 1 -- 172.21.15.186:0/1799496621 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f942000c2d0 con 0x7f94200039b0 2024-04-14T23:48:25.824 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.816+0000 7f94327fc640 1 --2- 172.21.15.186:0/1799496621 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9420004800 0x7f9420004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:25.824 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.816+0000 7f9432ffd640 1 --2- 172.21.15.186:0/1799496621 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f94200039b0 0x7f9420003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:25.824 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.816+0000 7f94327fc640 1 -- 172.21.15.186:0/1799496621 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f94200039b0 msgr2=0x7f9420003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:25.824 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.816+0000 7f94337fe640 1 --2- 172.21.15.186:0/1799496621 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94200051c0 0x7f942000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:25.824 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.816+0000 7f94327fc640 1 --2- 172.21.15.186:0/1799496621 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f94200039b0 0x7f9420003db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:25.824 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.816+0000 7f94327fc640 1 -- 172.21.15.186:0/1799496621 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94200051c0 msgr2=0x7f942000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:25.824 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.816+0000 7f94327fc640 1 --2- 172.21.15.186:0/1799496621 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94200051c0 0x7f942000b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:25.824 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.816+0000 7f94327fc640 1 -- 172.21.15.186:0/1799496621 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9420003520 con 0x7f9420004800 2024-04-14T23:48:25.824 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.816+0000 7f94337fe640 1 -- 172.21.15.186:0/1799496621 reap_dead start 2024-04-14T23:48:25.824 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.817+0000 7f94327fc640 1 --2- 172.21.15.186:0/1799496621 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9420004800 0x7f9420004c80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f942400c240 tx=0x7f9420005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38056f9b79af9236 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:25.824 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.817+0000 7f9431ffb640 1 -- 172.21.15.186:0/1799496621 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9424007d10 con 0x7f9420004800 2024-04-14T23:48:25.824 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.817+0000 7f9431ffb640 1 -- 172.21.15.186:0/1799496621 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9424030070 con 0x7f9420004800 2024-04-14T23:48:25.825 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.817+0000 7f9431ffb640 1 -- 172.21.15.186:0/1799496621 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9424005d70 con 0x7f9420004800 2024-04-14T23:48:25.827 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.818+0000 7f94394c8640 1 -- 172.21.15.186:0/1799496621 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9420004800 msgr2=0x7f9420004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:25.827 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.818+0000 7f94394c8640 1 --2- 172.21.15.186:0/1799496621 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9420004800 0x7f9420004c80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f942400c240 tx=0x7f9420005830 comp rx=0 tx=0).stop 2024-04-14T23:48:25.828 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.818+0000 7f94394c8640 1 -- 172.21.15.186:0/1799496621 shutdown_connections 2024-04-14T23:48:25.828 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.818+0000 7f94394c8640 1 --2- 172.21.15.186:0/1799496621 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9420004800 0x7f9420004c80 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:25.828 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.818+0000 7f94394c8640 1 -- 172.21.15.186:0/1799496621 >> 172.21.15.186:0/1799496621 conn(0x7f9434072080 msgr2=0x7f9434072490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:25.828 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.818+0000 7f94394c8640 1 -- 172.21.15.186:0/1799496621 shutdown_connections 2024-04-14T23:48:25.828 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.818+0000 7f94394c8640 1 -- 172.21.15.186:0/1799496621 wait complete. 2024-04-14T23:48:25.828 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.820+0000 7f94394c8640 1 Processor -- start 2024-04-14T23:48:25.828 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.820+0000 7f94394c8640 1 -- start start 2024-04-14T23:48:25.828 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.821+0000 7f94394c8640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9420004800 0x7f943415b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:25.828 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.821+0000 7f94394c8640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f94341f43e0 0x7f943415aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:25.828 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.821+0000 7f94394c8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9434146110 0x7f94341465c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:25.828 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.821+0000 7f94394c8640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f94340c8480 con 0x7f9434146110 2024-04-14T23:48:25.828 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.821+0000 7f94394c8640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9434155430 con 0x7f9420004800 2024-04-14T23:48:25.828 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.821+0000 7f94394c8640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f94340703c0 con 0x7f94341f43e0 2024-04-14T23:48:25.828 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.821+0000 7f9432ffd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9420004800 0x7f943415b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:25.829 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.821+0000 7f94327fc640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f94341f43e0 0x7f943415aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:25.829 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.821+0000 7f94327fc640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f94341f43e0 0x7f943415aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.186:58260/0 (socket says 172.21.15.186:58260) 2024-04-14T23:48:25.829 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.821+0000 7f94327fc640 1 -- 172.21.15.186:0/1097782341 learned_addr learned my addr 172.21.15.186:0/1097782341 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-04-14T23:48:25.829 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.821+0000 7f9432ffd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9420004800 0x7f943415b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:59532/0 (socket says 172.21.15.186:59532) 2024-04-14T23:48:25.829 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.821+0000 7f94337fe640 1 --2- 172.21.15.186:0/1097782341 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9434146110 0x7f94341465c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:25.829 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.821+0000 7f94327fc640 1 -- 172.21.15.186:0/1097782341 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9420004800 msgr2=0x7f943415b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:25.829 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.821+0000 7f94327fc640 1 --2- 172.21.15.186:0/1097782341 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9420004800 0x7f943415b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:25.829 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.821+0000 7f94327fc640 1 -- 172.21.15.186:0/1097782341 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9434146110 msgr2=0x7f94341465c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:25.829 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.821+0000 7f94327fc640 1 --2- 172.21.15.186:0/1097782341 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9434146110 0x7f94341465c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:25.829 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.821+0000 7f94327fc640 1 -- 172.21.15.186:0/1097782341 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94341f4860 con 0x7f94341f43e0 2024-04-14T23:48:25.829 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.821+0000 7f9432ffd640 1 --2- 172.21.15.186:0/1097782341 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9420004800 0x7f943415b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:48:25.829 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.821+0000 7f94337fe640 1 --2- 172.21.15.186:0/1097782341 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9434146110 0x7f94341465c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-14T23:48:25.829 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.822+0000 7f94327fc640 1 --2- 172.21.15.186:0/1097782341 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f94341f43e0 0x7f943415aad0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f942400c360 tx=0x7f94340c4c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:25.829 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.822+0000 7f942b7fe640 1 -- 172.21.15.186:0/1097782341 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9424004bf0 con 0x7f94341f43e0 2024-04-14T23:48:25.830 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.822+0000 7f942b7fe640 1 -- 172.21.15.186:0/1097782341 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9424004d90 con 0x7f94341f43e0 2024-04-14T23:48:25.830 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.822+0000 7f94394c8640 1 -- 172.21.15.186:0/1097782341 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94341f0800 con 0x7f94341f43e0 2024-04-14T23:48:25.830 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.822+0000 7f942b7fe640 1 -- 172.21.15.186:0/1097782341 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9424007270 con 0x7f94341f43e0 2024-04-14T23:48:25.830 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.822+0000 7f94394c8640 1 -- 172.21.15.186:0/1097782341 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f94341f0dc0 con 0x7f94341f43e0 2024-04-14T23:48:25.830 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.822+0000 7f942b7fe640 1 -- 172.21.15.186:0/1097782341 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f9424048020 con 0x7f94341f43e0 2024-04-14T23:48:25.830 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.822+0000 7f942b7fe640 1 -- 172.21.15.186:0/1097782341 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7f9424035d30 con 0x7f94341f43e0 2024-04-14T23:48:25.830 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.823+0000 7f94394c8640 1 -- 172.21.15.186:0/1097782341 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94340684b0 con 0x7f94341f43e0 2024-04-14T23:48:25.830 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.825+0000 7f942b7fe640 1 -- 172.21.15.186:0/1097782341 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f942404f020 con 0x7f94341f43e0 2024-04-14T23:48:25.866 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-04-14T23:48:25.861+0000 7ffaf7042040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-14T23:48:25.926 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-04-14T23:48:25.920+0000 7ffaf7042040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-14T23:48:25.950 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.945+0000 7f94394c8640 1 -- 172.21.15.186:0/1097782341 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "osd new", "uuid": "24cd58c1-0cf1-4077-ab33-84c39c6907ea", "id": 5} v 0) -- 0x7f943406d7d0 con 0x7f94341f43e0 2024-04-14T23:48:25.954 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.949+0000 7f942b7fe640 1 -- 172.21.15.186:0/1097782341 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "24cd58c1-0cf1-4077-ab33-84c39c6907ea", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7f94341f4860 con 0x7f94341f43e0 2024-04-14T23:48:25.954 INFO:teuthology.orchestra.run.smithi186.stdout:5 2024-04-14T23:48:25.956 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.950+0000 7f94394c8640 1 -- 172.21.15.186:0/1097782341 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f94341f43e0 msgr2=0x7f943415aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:25.956 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.950+0000 7f94394c8640 1 --2- 172.21.15.186:0/1097782341 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f94341f43e0 0x7f943415aad0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f942400c360 tx=0x7f94340c4c30 comp rx=0 tx=0).stop 2024-04-14T23:48:25.956 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.950+0000 7f94394c8640 1 -- 172.21.15.186:0/1097782341 shutdown_connections 2024-04-14T23:48:25.956 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.950+0000 7f94394c8640 1 --2- 172.21.15.186:0/1097782341 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9434146110 0x7f94341465c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:25.957 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.950+0000 7f94394c8640 1 --2- 172.21.15.186:0/1097782341 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f94341f43e0 0x7f943415aad0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:25.957 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.950+0000 7f94394c8640 1 --2- 172.21.15.186:0/1097782341 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9420004800 0x7f943415b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:25.957 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.950+0000 7f94394c8640 1 -- 172.21.15.186:0/1097782341 >> 172.21.15.186:0/1097782341 conn(0x7f9434072080 msgr2=0x7f9434072460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:25.957 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.950+0000 7f94394c8640 1 -- 172.21.15.186:0/1097782341 shutdown_connections 2024-04-14T23:48:25.957 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:25.950+0000 7f94394c8640 1 -- 172.21.15.186:0/1097782341 wait complete. 2024-04-14T23:48:25.960 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-14T23:48:25.955+0000 7ff99ec45040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-14T23:48:25.966 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd new 6add38f1-65e9-4b0d-927a-4ab77f5e2323 6 2024-04-14T23:48:26.067 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.062+0000 7f0564b8c640 1 Processor -- start 2024-04-14T23:48:26.067 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.062+0000 7f0564b8c640 1 -- start start 2024-04-14T23:48:26.068 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.063+0000 7f0564b8c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0560152c30 0x7f05600c8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:26.068 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.063+0000 7f0564b8c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f05600c32f0 0x7f056015c520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:26.068 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.063+0000 7f0564b8c640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05600c3d80 con 0x7f05600c3710 2024-04-14T23:48:26.068 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.063+0000 7f0564b8c640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f056015c090 con 0x7f05600c32f0 2024-04-14T23:48:26.069 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.063+0000 7f0564b8c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f05600c4890 con 0x7f0560152c30 2024-04-14T23:48:26.069 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.063+0000 7f055e59c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0560152c30 0x7f05600c8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:26.069 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.063+0000 7f055ed9d640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f05600c3710 0x7f05600ce240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:42520/0 (socket says 172.21.15.186:42520) 2024-04-14T23:48:26.069 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.063+0000 7f055ed9d640 1 -- 172.21.15.186:0/2085584459 learned_addr learned my addr 172.21.15.186:0/2085584459 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-04-14T23:48:26.069 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.063+0000 7f055d59a640 1 -- 172.21.15.186:0/2085584459 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3474160386 0 0) 0x7f05600c3d80 con 0x7f05600c3710 2024-04-14T23:48:26.069 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.063+0000 7f055d59a640 1 -- 172.21.15.186:0/2085584459 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0548003520 con 0x7f05600c3710 2024-04-14T23:48:26.069 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.063+0000 7f055e59c640 1 -- 172.21.15.186:0/2085584459 >> v2:172.21.15.60:3300/0 conn(0x7f05600c32f0 msgr2=0x7f056015c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:26.069 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.063+0000 7f055e59c640 1 --2- 172.21.15.186:0/2085584459 >> v2:172.21.15.60:3300/0 conn(0x7f05600c32f0 0x7f056015c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:26.069 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.063+0000 7f055e59c640 1 -- 172.21.15.186:0/2085584459 >> v1:172.21.15.186:6789/0 conn(0x7f05600c3710 legacy=0x7f05600ce240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:26.069 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.063+0000 7f055e59c640 1 -- 172.21.15.186:0/2085584459 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0560153010 con 0x7f0560152c30 2024-04-14T23:48:26.071 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.064+0000 7f055e59c640 1 --2- 172.21.15.186:0/2085584459 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0560152c30 0x7f05600c8cd0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f05500097a0 tx=0x7f055002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab9625336e72d895 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:26.071 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.064+0000 7f055d59a640 1 -- 172.21.15.186:0/2085584459 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0550035020 con 0x7f0560152c30 2024-04-14T23:48:26.071 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.064+0000 7f055d59a640 1 -- 172.21.15.186:0/2085584459 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0550031830 con 0x7f0560152c30 2024-04-14T23:48:26.071 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.064+0000 7f055d59a640 1 -- 172.21.15.186:0/2085584459 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0550031c00 con 0x7f0560152c30 2024-04-14T23:48:26.071 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.065+0000 7f0564b8c640 1 -- 172.21.15.186:0/2085584459 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0560152c30 msgr2=0x7f05600c8cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:26.072 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.065+0000 7f0564b8c640 1 --2- 172.21.15.186:0/2085584459 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0560152c30 0x7f05600c8cd0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f05500097a0 tx=0x7f055002b8d0 comp rx=0 tx=0).stop 2024-04-14T23:48:26.072 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.065+0000 7f0564b8c640 1 -- 172.21.15.186:0/2085584459 shutdown_connections 2024-04-14T23:48:26.072 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.065+0000 7f0564b8c640 1 --2- 172.21.15.186:0/2085584459 >> v2:172.21.15.60:3300/0 conn(0x7f05600c32f0 0x7f056015c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:26.072 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.065+0000 7f0564b8c640 1 --2- 172.21.15.186:0/2085584459 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0560152c30 0x7f05600c8cd0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:26.072 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.065+0000 7f0564b8c640 1 -- 172.21.15.186:0/2085584459 >> 172.21.15.186:0/2085584459 conn(0x7f056006f330 msgr2=0x7f0560066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:26.072 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.065+0000 7f0564b8c640 1 -- 172.21.15.186:0/2085584459 shutdown_connections 2024-04-14T23:48:26.072 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.065+0000 7f0564b8c640 1 -- 172.21.15.186:0/2085584459 wait complete. 2024-04-14T23:48:26.072 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.066+0000 7f0564b8c640 1 Processor -- start 2024-04-14T23:48:26.072 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.066+0000 7f0564b8c640 1 -- start start 2024-04-14T23:48:26.074 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.067+0000 7f0564b8c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f05600c32f0 0x7f05600cc4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:26.074 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.067+0000 7f0564b8c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f05600c3710 0x7f056015b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:26.074 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.067+0000 7f0564b8c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0560152c30 0x7f05601440b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:26.075 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.067+0000 7f0564b8c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f05600c45b0 con 0x7f05600c32f0 2024-04-14T23:48:26.075 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.067+0000 7f0564b8c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f05600c42d0 con 0x7f0560152c30 2024-04-14T23:48:26.075 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.067+0000 7f0564b8c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f05600c9210 con 0x7f05600c3710 2024-04-14T23:48:26.075 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.067+0000 7f055dd9b640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f05600c3710 0x7f056015b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:26.075 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.067+0000 7f055ed9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0560152c30 0x7f05601440b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:26.075 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.067+0000 7f055dd9b640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f05600c3710 0x7f056015b050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.186:58280/0 (socket says 172.21.15.186:58280) 2024-04-14T23:48:26.075 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.067+0000 7f055ed9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0560152c30 0x7f05601440b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:59538/0 (socket says 172.21.15.186:59538) 2024-04-14T23:48:26.075 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.067+0000 7f055dd9b640 1 -- 172.21.15.186:0/1661227291 learned_addr learned my addr 172.21.15.186:0/1661227291 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-04-14T23:48:26.075 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.067+0000 7f055e59c640 1 --2- 172.21.15.186:0/1661227291 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f05600c32f0 0x7f05600cc4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:26.075 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.067+0000 7f055ed9d640 1 -- 172.21.15.186:0/1661227291 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f05600c3710 msgr2=0x7f056015b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:26.075 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.067+0000 7f055ed9d640 1 --2- 172.21.15.186:0/1661227291 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f05600c3710 0x7f056015b050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:26.075 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.067+0000 7f055ed9d640 1 -- 172.21.15.186:0/1661227291 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f05600c32f0 msgr2=0x7f05600cc4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:26.075 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.067+0000 7f055ed9d640 1 --2- 172.21.15.186:0/1661227291 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f05600c32f0 0x7f05600cc4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:26.075 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.067+0000 7f055ed9d640 1 -- 172.21.15.186:0/1661227291 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05601ef330 con 0x7f0560152c30 2024-04-14T23:48:26.076 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.067+0000 7f055dd9b640 1 --2- 172.21.15.186:0/1661227291 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f05600c3710 0x7f056015b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:48:26.076 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.067+0000 7f055e59c640 1 --2- 172.21.15.186:0/1661227291 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f05600c32f0 0x7f05600cc4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-14T23:48:26.076 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.067+0000 7f055ed9d640 1 --2- 172.21.15.186:0/1661227291 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0560152c30 0x7f05601440b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f056006e720 tx=0x7f055800cee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:26.076 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.067+0000 7f054effd640 1 -- 172.21.15.186:0/1661227291 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0558017070 con 0x7f0560152c30 2024-04-14T23:48:26.076 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.068+0000 7f054effd640 1 -- 172.21.15.186:0/1661227291 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f055800abd0 con 0x7f0560152c30 2024-04-14T23:48:26.076 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.068+0000 7f0564b8c640 1 -- 172.21.15.186:0/1661227291 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0560146240 con 0x7f0560152c30 2024-04-14T23:48:26.076 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.068+0000 7f054effd640 1 -- 172.21.15.186:0/1661227291 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0558012400 con 0x7f0560152c30 2024-04-14T23:48:26.076 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.068+0000 7f0564b8c640 1 -- 172.21.15.186:0/1661227291 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05601438b0 con 0x7f0560152c30 2024-04-14T23:48:26.076 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.068+0000 7f054effd640 1 -- 172.21.15.186:0/1661227291 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f055801a070 con 0x7f0560152c30 2024-04-14T23:48:26.076 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.068+0000 7f054effd640 1 -- 172.21.15.186:0/1661227291 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f055801cd50 con 0x7f0560152c30 2024-04-14T23:48:26.076 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.068+0000 7f0564b8c640 1 -- 172.21.15.186:0/1661227291 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0560146760 con 0x7f0560152c30 2024-04-14T23:48:26.077 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.070+0000 7f054effd640 1 -- 172.21.15.186:0/1661227291 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f0560146760 con 0x7f0560152c30 2024-04-14T23:48:26.103 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-14T23:48:26.098+0000 7ff99ec45040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-14T23:48:26.149 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.144+0000 7f0564b8c640 1 -- 172.21.15.186:0/1661227291 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd new", "uuid": "6add38f1-65e9-4b0d-927a-4ab77f5e2323", "id": 6} v 0) -- 0x7f05600c70d0 con 0x7f0560152c30 2024-04-14T23:48:26.154 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.149+0000 7f054effd640 1 -- 172.21.15.186:0/1661227291 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6add38f1-65e9-4b0d-927a-4ab77f5e2323", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7f05600c70d0 con 0x7f0560152c30 2024-04-14T23:48:26.154 INFO:teuthology.orchestra.run.smithi186.stdout:6 2024-04-14T23:48:26.156 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.150+0000 7f0564b8c640 1 -- 172.21.15.186:0/1661227291 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0560152c30 msgr2=0x7f05601440b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:26.156 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.150+0000 7f0564b8c640 1 --2- 172.21.15.186:0/1661227291 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0560152c30 0x7f05601440b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f056006e720 tx=0x7f055800cee0 comp rx=0 tx=0).stop 2024-04-14T23:48:26.156 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.150+0000 7f0564b8c640 1 -- 172.21.15.186:0/1661227291 shutdown_connections 2024-04-14T23:48:26.157 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.150+0000 7f0564b8c640 1 --2- 172.21.15.186:0/1661227291 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0560152c30 0x7f05601440b0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:26.157 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.150+0000 7f0564b8c640 1 --2- 172.21.15.186:0/1661227291 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f05600c3710 0x7f056015b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:26.157 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.150+0000 7f0564b8c640 1 --2- 172.21.15.186:0/1661227291 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f05600c32f0 0x7f05600cc4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:26.157 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.150+0000 7f0564b8c640 1 -- 172.21.15.186:0/1661227291 >> 172.21.15.186:0/1661227291 conn(0x7f056006f330 msgr2=0x7f05600cd540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:26.157 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.150+0000 7f0564b8c640 1 -- 172.21.15.186:0/1661227291 shutdown_connections 2024-04-14T23:48:26.157 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.150+0000 7f0564b8c640 1 -- 172.21.15.186:0/1661227291 wait complete. 2024-04-14T23:48:26.166 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd new a2326b6f-7601-4dda-a456-67c443ea5929 7 2024-04-14T23:48:26.231 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-14T23:48:26.226+0000 7ff99ec45040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-14T23:48:26.266 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.261+0000 7f8e7fca2640 1 Processor -- start 2024-04-14T23:48:26.267 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.262+0000 7f8e7fca2640 1 -- start start 2024-04-14T23:48:26.269 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.262+0000 7f8e7fca2640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f8e78154aa0 0x7f8e780c32f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:26.269 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.262+0000 7f8e7fca2640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8e7806f7b0 0x7f8e78165510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:26.269 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.262+0000 7f8e7fca2640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e7815aca0 con 0x7f8e780c8cd0 2024-04-14T23:48:26.269 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.262+0000 7f8e7fca2640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f8e7806fd60 con 0x7f8e7806f7b0 2024-04-14T23:48:26.269 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.262+0000 7f8e7fca2640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f8e7815af10 con 0x7f8e78154aa0 2024-04-14T23:48:26.269 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.263+0000 7f8e7da3e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f8e78154aa0 0x7f8e780c32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:26.269 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.263+0000 7f8e7da3e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f8e78154aa0 0x7f8e780c32f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:59544/0 (socket says 172.21.15.186:59544) 2024-04-14T23:48:26.269 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.263+0000 7f8e7da3e640 1 -- 172.21.15.186:0/1505420390 learned_addr learned my addr 172.21.15.186:0/1505420390 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-04-14T23:48:26.269 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.263+0000 7f8e7da3e640 1 -- 172.21.15.186:0/1505420390 >> v2:172.21.15.60:3300/0 conn(0x7f8e7806f7b0 msgr2=0x7f8e78165510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:26.269 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.263+0000 7f8e7da3e640 1 --2- 172.21.15.186:0/1505420390 >> v2:172.21.15.60:3300/0 conn(0x7f8e7806f7b0 0x7f8e78165510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:26.269 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.263+0000 7f8e7da3e640 1 -- 172.21.15.186:0/1505420390 >> v1:172.21.15.60:6789/0 conn(0x7f8e780c8cd0 legacy=0x7f8e780ce220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:26.270 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.263+0000 7f8e7da3e640 1 -- 172.21.15.186:0/1505420390 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e7815b1f0 con 0x7f8e78154aa0 2024-04-14T23:48:26.270 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.264+0000 7f8e7da3e640 1 --2- 172.21.15.186:0/1505420390 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e78154aa0 0x7f8e780c32f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8e78067c40 tx=0x7f8e7402b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1be37922f6be4b60 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:26.270 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.264+0000 7f8e7ca3c640 1 -- 172.21.15.186:0/1505420390 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e7402e070 con 0x7f8e78154aa0 2024-04-14T23:48:26.270 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.264+0000 7f8e7ca3c640 1 -- 172.21.15.186:0/1505420390 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e74037070 con 0x7f8e78154aa0 2024-04-14T23:48:26.270 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.264+0000 7f8e7ca3c640 1 -- 172.21.15.186:0/1505420390 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e74031a00 con 0x7f8e78154aa0 2024-04-14T23:48:26.273 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.264+0000 7f8e7fca2640 1 -- 172.21.15.186:0/1505420390 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e78154aa0 msgr2=0x7f8e780c32f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:26.273 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.264+0000 7f8e7fca2640 1 --2- 172.21.15.186:0/1505420390 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e78154aa0 0x7f8e780c32f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8e78067c40 tx=0x7f8e7402b880 comp rx=0 tx=0).stop 2024-04-14T23:48:26.273 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.264+0000 7f8e7fca2640 1 -- 172.21.15.186:0/1505420390 shutdown_connections 2024-04-14T23:48:26.274 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.264+0000 7f8e7fca2640 1 --2- 172.21.15.186:0/1505420390 >> v2:172.21.15.60:3300/0 conn(0x7f8e7806f7b0 0x7f8e78165510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:26.274 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.264+0000 7f8e7fca2640 1 --2- 172.21.15.186:0/1505420390 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e78154aa0 0x7f8e780c32f0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:26.274 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.264+0000 7f8e7fca2640 1 -- 172.21.15.186:0/1505420390 >> 172.21.15.186:0/1505420390 conn(0x7f8e7806b850 msgr2=0x7f8e78066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:26.274 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.264+0000 7f8e7fca2640 1 -- 172.21.15.186:0/1505420390 shutdown_connections 2024-04-14T23:48:26.274 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.264+0000 7f8e7fca2640 1 -- 172.21.15.186:0/1505420390 wait complete. 2024-04-14T23:48:26.274 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.265+0000 7f8e7fca2640 1 Processor -- start 2024-04-14T23:48:26.274 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.266+0000 7f8e7fca2640 1 -- start start 2024-04-14T23:48:26.274 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.266+0000 7f8e7fca2640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8e7806f7b0 0x7f8e780cb4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:26.274 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.266+0000 7f8e7fca2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e780c8cd0 0x7f8e781e4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:26.274 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.266+0000 7f8e7fca2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e78154aa0 0x7f8e780c9dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:26.274 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.266+0000 7f8e7fca2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f8e7815b9f0 con 0x7f8e78154aa0 2024-04-14T23:48:26.274 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.266+0000 7f8e7fca2640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8e78070040 con 0x7f8e780c8cd0 2024-04-14T23:48:26.274 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.266+0000 7f8e7fca2640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f8e780cb9f0 con 0x7f8e7806f7b0 2024-04-14T23:48:26.275 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.266+0000 7f8e7da3e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8e7806f7b0 0x7f8e780cb4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:26.275 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.266+0000 7f8e7d23d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e780c8cd0 0x7f8e781e4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:26.275 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.266+0000 7f8e7d23d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e780c8cd0 0x7f8e781e4230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:59558/0 (socket says 172.21.15.186:59558) 2024-04-14T23:48:26.275 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.266+0000 7f8e7d23d640 1 -- 172.21.15.186:0/2773025547 learned_addr learned my addr 172.21.15.186:0/2773025547 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-04-14T23:48:26.275 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.267+0000 7f8e7e23f640 1 --2- 172.21.15.186:0/2773025547 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e78154aa0 0x7f8e780c9dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:26.275 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.267+0000 7f8e7d23d640 1 -- 172.21.15.186:0/2773025547 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8e7806f7b0 msgr2=0x7f8e780cb4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:26.275 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.267+0000 7f8e7d23d640 1 --2- 172.21.15.186:0/2773025547 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8e7806f7b0 0x7f8e780cb4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:26.275 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.267+0000 7f8e7d23d640 1 -- 172.21.15.186:0/2773025547 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e78154aa0 msgr2=0x7f8e780c9dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:26.275 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.267+0000 7f8e7d23d640 1 --2- 172.21.15.186:0/2773025547 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e78154aa0 0x7f8e780c9dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:26.275 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.267+0000 7f8e7d23d640 1 -- 172.21.15.186:0/2773025547 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e781f10c0 con 0x7f8e780c8cd0 2024-04-14T23:48:26.275 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.267+0000 7f8e7da3e640 1 --2- 172.21.15.186:0/2773025547 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8e7806f7b0 0x7f8e780cb4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:48:26.275 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.267+0000 7f8e7e23f640 1 --2- 172.21.15.186:0/2773025547 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e78154aa0 0x7f8e780c9dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-14T23:48:26.275 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.267+0000 7f8e7d23d640 1 --2- 172.21.15.186:0/2773025547 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e780c8cd0 0x7f8e781e4230 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8e6800b410 tx=0x7f8e6800b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:26.275 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.267+0000 7f8e6e7fc640 1 -- 172.21.15.186:0/2773025547 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e68015020 con 0x7f8e780c8cd0 2024-04-14T23:48:26.276 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.267+0000 7f8e6e7fc640 1 -- 172.21.15.186:0/2773025547 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e68011830 con 0x7f8e780c8cd0 2024-04-14T23:48:26.276 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.267+0000 7f8e7fca2640 1 -- 172.21.15.186:0/2773025547 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e781f1360 con 0x7f8e780c8cd0 2024-04-14T23:48:26.276 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.267+0000 7f8e6e7fc640 1 -- 172.21.15.186:0/2773025547 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e68011c20 con 0x7f8e780c8cd0 2024-04-14T23:48:26.276 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.267+0000 7f8e7fca2640 1 -- 172.21.15.186:0/2773025547 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e781f1920 con 0x7f8e780c8cd0 2024-04-14T23:48:26.276 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.267+0000 7f8e6e7fc640 1 -- 172.21.15.186:0/2773025547 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f8e68018020 con 0x7f8e780c8cd0 2024-04-14T23:48:26.276 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.267+0000 7f8e6e7fc640 1 -- 172.21.15.186:0/2773025547 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7f8e68019dd0 con 0x7f8e780c8cd0 2024-04-14T23:48:26.276 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.268+0000 7f8e7fca2640 1 -- 172.21.15.186:0/2773025547 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e780684b0 con 0x7f8e780c8cd0 2024-04-14T23:48:26.276 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.270+0000 7f8e6e7fc640 1 -- 172.21.15.186:0/2773025547 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f8e6801f020 con 0x7f8e780c8cd0 2024-04-14T23:48:26.280 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-14T23:48:26.275+0000 7ff99ec45040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-14T23:48:26.348 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.343+0000 7f8e7fca2640 1 -- 172.21.15.186:0/2773025547 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd new", "uuid": "a2326b6f-7601-4dda-a456-67c443ea5929", "id": 7} v 0) -- 0x7f8e78153900 con 0x7f8e780c8cd0 2024-04-14T23:48:26.353 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.348+0000 7f8e6e7fc640 1 -- 172.21.15.186:0/2773025547 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a2326b6f-7601-4dda-a456-67c443ea5929", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7f8e68011dc0 con 0x7f8e780c8cd0 2024-04-14T23:48:26.353 INFO:teuthology.orchestra.run.smithi186.stdout:7 2024-04-14T23:48:26.354 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.349+0000 7f8e7fca2640 1 -- 172.21.15.186:0/2773025547 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e780c8cd0 msgr2=0x7f8e781e4230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:26.354 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.349+0000 7f8e7fca2640 1 --2- 172.21.15.186:0/2773025547 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e780c8cd0 0x7f8e781e4230 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8e6800b410 tx=0x7f8e6800b8d0 comp rx=0 tx=0).stop 2024-04-14T23:48:26.355 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.349+0000 7f8e7fca2640 1 -- 172.21.15.186:0/2773025547 shutdown_connections 2024-04-14T23:48:26.355 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.349+0000 7f8e7fca2640 1 --2- 172.21.15.186:0/2773025547 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e78154aa0 0x7f8e780c9dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:26.355 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.349+0000 7f8e7fca2640 1 --2- 172.21.15.186:0/2773025547 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e780c8cd0 0x7f8e781e4230 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:26.355 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.349+0000 7f8e7fca2640 1 --2- 172.21.15.186:0/2773025547 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8e7806f7b0 0x7f8e780cb4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:26.355 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.349+0000 7f8e7fca2640 1 -- 172.21.15.186:0/2773025547 >> 172.21.15.186:0/2773025547 conn(0x7f8e7806b850 msgr2=0x7f8e7814efb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:26.355 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.349+0000 7f8e7fca2640 1 -- 172.21.15.186:0/2773025547 shutdown_connections 2024-04-14T23:48:26.355 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-14T23:48:26.349+0000 7f8e7fca2640 1 -- 172.21.15.186:0/2773025547 wait complete. 2024-04-14T23:48:26.364 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-14T23:48:26.364 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-04-14T23:48:26.372 INFO:tasks.ceph.osd.0:Started 2024-04-14T23:48:26.372 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-14T23:48:26.372 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-04-14T23:48:26.378 INFO:tasks.ceph.osd.1:Started 2024-04-14T23:48:26.378 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-14T23:48:26.378 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-04-14T23:48:26.383 INFO:tasks.ceph.osd.2:Started 2024-04-14T23:48:26.384 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-14T23:48:26.384 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-04-14T23:48:26.388 INFO:tasks.ceph.osd.3:Started 2024-04-14T23:48:26.388 INFO:tasks.ceph.osd.4:Restarting daemon 2024-04-14T23:48:26.389 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-04-14T23:48:26.391 INFO:tasks.ceph.osd.4:Started 2024-04-14T23:48:26.391 INFO:tasks.ceph.osd.5:Restarting daemon 2024-04-14T23:48:26.391 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-04-14T23:48:26.393 INFO:tasks.ceph.osd.5:Started 2024-04-14T23:48:26.393 INFO:tasks.ceph.osd.6:Restarting daemon 2024-04-14T23:48:26.393 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-04-14T23:48:26.395 INFO:tasks.ceph.osd.6:Started 2024-04-14T23:48:26.395 INFO:tasks.ceph.osd.7:Restarting daemon 2024-04-14T23:48:26.395 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-04-14T23:48:26.397 INFO:tasks.ceph.osd.7:Started 2024-04-14T23:48:26.398 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-14T23:48:26.399 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-14T23:48:26.394+0000 7ff99ec45040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-14T23:48:26.415 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-04-14T23:48:26.410+0000 7ffaf7042040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-14T23:48:26.527 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-14T23:48:26.522+0000 7ff99ec45040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-14T23:48:26.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.616+0000 7f49ae000640 1 Processor -- start 2024-04-14T23:48:26.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.616+0000 7f49ae000640 1 -- start start 2024-04-14T23:48:26.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.618+0000 7f49ae000640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f49a80c8360 0x7f49a815ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:26.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.618+0000 7f49ae000640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f49a80c8740 0x7f49a806f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:26.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.618+0000 7f49ae000640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f49a80c75c0 0x7f49a8158e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:26.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.618+0000 7f49ae000640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f49a80c6b70 con 0x7f49a80c8740 2024-04-14T23:48:26.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.618+0000 7f49ae000640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f49a815f7c0 con 0x7f49a80c75c0 2024-04-14T23:48:26.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.618+0000 7f49ae000640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f49a815faa0 con 0x7f49a80c8360 2024-04-14T23:48:26.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.618+0000 7f49a7e00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f49a80c75c0 0x7f49a8158e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:26.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.618+0000 7f499fe00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f49a80c8740 0x7f49a806f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:26.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.618+0000 7f49a7e00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f49a80c75c0 0x7f49a8158e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:49534/0 (socket says 172.21.15.60:49534) 2024-04-14T23:48:26.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.618+0000 7f49a7e00640 1 -- 172.21.15.60:0/2228657298 learned_addr learned my addr 172.21.15.60:0/2228657298 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:26.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.618+0000 7f49a7400640 1 --2- 172.21.15.60:0/2228657298 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f49a80c8360 0x7f49a815ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:26.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.618+0000 7f49a7e00640 1 -- 172.21.15.60:0/2228657298 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f49a80c8360 msgr2=0x7f49a815ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:26.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.618+0000 7f49a7e00640 1 --2- 172.21.15.60:0/2228657298 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f49a80c8360 0x7f49a815ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:26.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.618+0000 7f49a7e00640 1 -- 172.21.15.60:0/2228657298 >> v2:172.21.15.60:3300/0 conn(0x7f49a80c8740 msgr2=0x7f49a806f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:26.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.618+0000 7f49a7e00640 1 --2- 172.21.15.60:0/2228657298 >> v2:172.21.15.60:3300/0 conn(0x7f49a80c8740 0x7f49a806f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:26.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.618+0000 7f49a7e00640 1 -- 172.21.15.60:0/2228657298 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49a815fd80 con 0x7f49a80c75c0 2024-04-14T23:48:26.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.620+0000 7f49a7e00640 1 --2- 172.21.15.60:0/2228657298 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f49a80c75c0 0x7f49a8158e30 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f49a80cd1f0 tx=0x7f49a002bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63cb09a63ea26a97 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:26.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.620+0000 7f49a6a00640 1 -- 172.21.15.60:0/2228657298 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49a002cb00 con 0x7f49a80c75c0 2024-04-14T23:48:26.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.620+0000 7f49a6a00640 1 -- 172.21.15.60:0/2228657298 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49a002cca0 con 0x7f49a80c75c0 2024-04-14T23:48:26.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.620+0000 7f49a6a00640 1 -- 172.21.15.60:0/2228657298 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49a0034610 con 0x7f49a80c75c0 2024-04-14T23:48:26.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.620+0000 7f49ae000640 1 -- 172.21.15.60:0/2228657298 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f49a80c75c0 msgr2=0x7f49a8158e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:26.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.620+0000 7f49ae000640 1 --2- 172.21.15.60:0/2228657298 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f49a80c75c0 0x7f49a8158e30 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f49a80cd1f0 tx=0x7f49a002bf60 comp rx=0 tx=0).stop 2024-04-14T23:48:26.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.621+0000 7f49ae000640 1 -- 172.21.15.60:0/2228657298 shutdown_connections 2024-04-14T23:48:26.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.621+0000 7f49ae000640 1 --2- 172.21.15.60:0/2228657298 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f49a80c75c0 0x7f49a8158e30 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:26.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.621+0000 7f49ae000640 1 --2- 172.21.15.60:0/2228657298 >> v2:172.21.15.60:3300/0 conn(0x7f49a80c8740 0x7f49a806f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:26.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.621+0000 7f49ae000640 1 --2- 172.21.15.60:0/2228657298 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f49a80c8360 0x7f49a815ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:26.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.621+0000 7f49ae000640 1 -- 172.21.15.60:0/2228657298 >> 172.21.15.60:0/2228657298 conn(0x7f49a80cc170 msgr2=0x7f49a80cc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:26.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.621+0000 7f49ae000640 1 -- 172.21.15.60:0/2228657298 shutdown_connections 2024-04-14T23:48:26.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.621+0000 7f49ae000640 1 -- 172.21.15.60:0/2228657298 wait complete. 2024-04-14T23:48:26.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.622+0000 7f49ae000640 1 Processor -- start 2024-04-14T23:48:26.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.622+0000 7f49ae000640 1 -- start start 2024-04-14T23:48:26.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.623+0000 7f49ae000640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f49a80c75c0 0x7f49a80cb5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:26.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.623+0000 7f49ae000640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f49a80c8360 0x7f49a81f8750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:26.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.623+0000 7f49ae000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f49a80c8740 0x7f49a80c4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:26.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.623+0000 7f49ae000640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f49a815f4c0 con 0x7f49a80c75c0 2024-04-14T23:48:26.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.623+0000 7f49ae000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f49a80c70a0 con 0x7f49a80c8740 2024-04-14T23:48:26.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.623+0000 7f49ae000640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f49a8071030 con 0x7f49a80c8360 2024-04-14T23:48:26.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.624+0000 7f499fe00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f49a80c8360 0x7f49a81f8750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:26.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.624+0000 7f499fe00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f49a80c8360 0x7f49a81f8750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.60:36548/0 (socket says 172.21.15.60:36548) 2024-04-14T23:48:26.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.624+0000 7f499fe00640 1 -- 172.21.15.60:0/298774407 learned_addr learned my addr 172.21.15.60:0/298774407 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:26.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.624+0000 7f49a7e00640 1 --2- 172.21.15.60:0/298774407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f49a80c8740 0x7f49a80c4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:26.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.624+0000 7f49a7400640 1 --2- 172.21.15.60:0/298774407 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f49a80c75c0 0x7f49a80cb5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:26.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.624+0000 7f499fe00640 1 -- 172.21.15.60:0/298774407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f49a80c8740 msgr2=0x7f49a80c4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:26.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.624+0000 7f499fe00640 1 --2- 172.21.15.60:0/298774407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f49a80c8740 0x7f49a80c4dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:26.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.624+0000 7f499fe00640 1 -- 172.21.15.60:0/298774407 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f49a80c75c0 msgr2=0x7f49a80cb5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:26.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.624+0000 7f499fe00640 1 --2- 172.21.15.60:0/298774407 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f49a80c75c0 0x7f49a80cb5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:26.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.624+0000 7f499fe00640 1 -- 172.21.15.60:0/298774407 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49a815c240 con 0x7f49a80c8360 2024-04-14T23:48:26.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.624+0000 7f499fe00640 1 --2- 172.21.15.60:0/298774407 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f49a80c8360 0x7f49a81f8750 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f499000e9e0 tx=0x7f499000eea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:26.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.625+0000 7f499f400640 1 -- 172.21.15.60:0/298774407 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4990017070 con 0x7f49a80c8360 2024-04-14T23:48:26.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.625+0000 7f499f400640 1 -- 172.21.15.60:0/298774407 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f499000c9d0 con 0x7f49a80c8360 2024-04-14T23:48:26.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.625+0000 7f499f400640 1 -- 172.21.15.60:0/298774407 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f499000cdc0 con 0x7f49a80c8360 2024-04-14T23:48:26.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.625+0000 7f49ae000640 1 -- 172.21.15.60:0/298774407 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49a81f7f70 con 0x7f49a80c8360 2024-04-14T23:48:26.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.625+0000 7f49ae000640 1 -- 172.21.15.60:0/298774407 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f49a81f8c90 con 0x7f49a80c8360 2024-04-14T23:48:26.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.625+0000 7f499f400640 1 -- 172.21.15.60:0/298774407 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f499001a070 con 0x7f49a80c8360 2024-04-14T23:48:26.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.625+0000 7f499f400640 1 -- 172.21.15.60:0/298774407 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f4990006660 con 0x7f49a80c8360 2024-04-14T23:48:26.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.626+0000 7f49ae000640 1 -- 172.21.15.60:0/298774407 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49a8076970 con 0x7f49a80c8360 2024-04-14T23:48:26.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.628+0000 7f499f400640 1 -- 172.21.15.60:0/298774407 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f4990005990 con 0x7f49a80c8360 2024-04-14T23:48:26.641 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-04-14T23:48:26.636+0000 7ffaf7042040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-14T23:48:26.683 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-14T23:48:26.678+0000 7ff99ec45040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-14T23:48:26.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.695+0000 7f49ae000640 1 -- 172.21.15.60:0/298774407 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f49a80cb3d0 con 0x7f49a80c8360 2024-04-14T23:48:26.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.696+0000 7f499f400640 1 -- 172.21.15.60:0/298774407 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f4990005b70 con 0x7f49a80c8360 2024-04-14T23:48:26.701 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:48:26.702 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":10,"fsid":"5847db56-a76d-4486-b563-d83f1186af92","created":"2024-04-14T23:48:23.900003+0000","modified":"2024-04-14T23:48:26.345499+0000","last_up_change":"0.000000","last_in_change":"2024-04-14T23:48:26.345499+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b2b2b755-7f42-4f9b-af9a-a178f7cc5a52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"321418cd-5ab7-41a5-b9c8-b3fc12f68ebc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"07874e9f-3a38-41dc-be9f-2b75e84e53d9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"7e4ef6c9-b17a-47c6-a320-f5d16a6fe05e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"759a3efc-dade-44c5-b57a-1623499aba50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"24cd58c1-0cf1-4077-ab33-84c39c6907ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6add38f1-65e9-4b0d-927a-4ab77f5e2323","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a2326b6f-7601-4dda-a456-67c443ea5929","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-14T23:48:26.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.697+0000 7f49ae000640 1 -- 172.21.15.60:0/298774407 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f49a80c8360 msgr2=0x7f49a81f8750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:26.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.697+0000 7f49ae000640 1 --2- 172.21.15.60:0/298774407 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f49a80c8360 0x7f49a81f8750 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f499000e9e0 tx=0x7f499000eea0 comp rx=0 tx=0).stop 2024-04-14T23:48:26.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.697+0000 7f49ae000640 1 -- 172.21.15.60:0/298774407 shutdown_connections 2024-04-14T23:48:26.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.697+0000 7f49ae000640 1 --2- 172.21.15.60:0/298774407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f49a80c8740 0x7f49a80c4dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:26.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.697+0000 7f49ae000640 1 --2- 172.21.15.60:0/298774407 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f49a80c8360 0x7f49a81f8750 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:26.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.697+0000 7f49ae000640 1 --2- 172.21.15.60:0/298774407 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f49a80c75c0 0x7f49a80cb5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:26.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.697+0000 7f49ae000640 1 -- 172.21.15.60:0/298774407 >> 172.21.15.60:0/298774407 conn(0x7f49a80cc170 msgr2=0x7f49a8071cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:26.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.697+0000 7f49ae000640 1 -- 172.21.15.60:0/298774407 shutdown_connections 2024-04-14T23:48:26.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:26.697+0000 7f49ae000640 1 -- 172.21.15.60:0/298774407 wait complete. 2024-04-14T23:48:26.719 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-14T23:48:26.720 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-14T23:48:26.726 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-14T23:48:26.721+0000 7ff99ec45040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-14T23:48:26.800 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-04-14T23:48:26.795+0000 7ffaf7042040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-14T23:48:26.856 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-04-14T23:48:26.851+0000 7ffaf7042040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-14T23:48:26.882 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-14T23:48:26.877+0000 7ff99ec45040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-14T23:48:26.990 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-04-14T23:48:26.985+0000 7f2eaf7bd640 -1 Falling back to public interface 2024-04-14T23:48:26.991 INFO:tasks.ceph.osd.5.smithi186.stderr:2024-04-14T23:48:26.986+0000 7f680ab93640 -1 Falling back to public interface 2024-04-14T23:48:26.991 INFO:tasks.ceph.osd.6.smithi186.stderr:2024-04-14T23:48:26.986+0000 7f8fabcbd640 -1 Falling back to public interface 2024-04-14T23:48:26.992 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-04-14T23:48:26.986+0000 7f2eae3fd640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6801/0: (98) Address already in use 2024-04-14T23:48:26.992 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-04-14T23:48:26.986+0000 7f2eae3fd640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6802/0: (98) Address already in use 2024-04-14T23:48:26.992 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-04-14T23:48:26.986+0000 7f2eae3fd640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6803/0: (98) Address already in use 2024-04-14T23:48:26.992 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-04-14T23:48:26.986+0000 7f2eae3fd640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6804/0: (98) Address already in use 2024-04-14T23:48:26.992 INFO:tasks.ceph.osd.5.smithi186.stderr:2024-04-14T23:48:26.987+0000 7f68097fd640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6817/0: (98) Address already in use 2024-04-14T23:48:26.992 INFO:tasks.ceph.osd.6.smithi186.stderr:2024-04-14T23:48:26.987+0000 7f8faa9fd640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6820/0: (98) Address already in use 2024-04-14T23:48:26.993 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-04-14T23:48:26.986+0000 7ffaf7042040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-14T23:48:27.023 INFO:tasks.ceph.osd.7.smithi186.stderr:2024-04-14T23:48:27.019+0000 7f01a4b73640 -1 Falling back to public interface 2024-04-14T23:48:27.094 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-04-14T23:48:27.089+0000 7fa5de373640 -1 Falling back to public interface 2024-04-14T23:48:27.108 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-04-14T23:48:27.102+0000 7ffaf7042040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-14T23:48:27.113 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-04-14T23:48:27.108+0000 7f65b8468640 -1 Falling back to public interface 2024-04-14T23:48:27.122 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-04-14T23:48:27.117+0000 7f2de868b640 -1 Falling back to public interface 2024-04-14T23:48:27.122 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-04-14T23:48:27.117+0000 7fab46a24640 -1 Falling back to public interface 2024-04-14T23:48:27.298 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-04-14T23:48:27.292+0000 7ffaf7042040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-14T23:48:27.350 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-04-14T23:48:27.344+0000 7ffaf7042040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-14T23:48:27.524 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-14T23:48:27.534 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-04-14T23:48:27.528+0000 7ffaf7042040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-14T23:48:27.687 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.681+0000 7fb833200640 1 Processor -- start 2024-04-14T23:48:27.687 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.681+0000 7fb833200640 1 -- start start 2024-04-14T23:48:27.688 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.683+0000 7fb833200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb82c15ca40 0x7fb82c0cb600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:27.688 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.683+0000 7fb833200640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb82c0ccda0 con 0x7fb82c154f20 2024-04-14T23:48:27.688 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.683+0000 7fb833200640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb82c0cd080 con 0x7fb82c15bb50 2024-04-14T23:48:27.689 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.683+0000 7fb833200640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fb82c0cd360 con 0x7fb82c15ca40 2024-04-14T23:48:27.689 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.683+0000 7fb82be00640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fb82c154f20 0x7fb82c0d0580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:39468/0 (socket says 172.21.15.60:39468) 2024-04-14T23:48:27.689 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.683+0000 7fb82be00640 1 -- 172.21.15.60:0/1327570599 learned_addr learned my addr 172.21.15.60:0/1327570599 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:48:27.689 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.683+0000 7fb831200640 1 --2- 172.21.15.60:0/1327570599 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb82c15ca40 0x7fb82c0cb600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:27.690 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.683+0000 7fb82aa00640 1 -- 172.21.15.60:0/1327570599 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3323594163 0 0) 0x7fb82c0ccda0 con 0x7fb82c154f20 2024-04-14T23:48:27.690 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.683+0000 7fb82aa00640 1 -- 172.21.15.60:0/1327570599 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb810003520 con 0x7fb82c154f20 2024-04-14T23:48:27.690 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.683+0000 7fb82aa00640 1 -- 172.21.15.60:0/1327570599 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3835108449 0 0) 0x7fb82c0cd080 con 0x7fb82c15bb50 2024-04-14T23:48:27.690 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.683+0000 7fb82aa00640 1 -- 172.21.15.60:0/1327570599 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb82c0ccda0 con 0x7fb82c15bb50 2024-04-14T23:48:27.690 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.683+0000 7fb831200640 1 -- 172.21.15.60:0/1327570599 >> v1:172.21.15.186:6789/0 conn(0x7fb82c15bb50 legacy=0x7fb82c15c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:27.690 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.683+0000 7fb831200640 1 -- 172.21.15.60:0/1327570599 >> v1:172.21.15.60:6789/0 conn(0x7fb82c154f20 legacy=0x7fb82c0d0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:27.690 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.683+0000 7fb831200640 1 -- 172.21.15.60:0/1327570599 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb82c0cd640 con 0x7fb82c15ca40 2024-04-14T23:48:27.691 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.685+0000 7fb831200640 1 --2- 172.21.15.60:0/1327570599 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb82c15ca40 0x7fb82c0cb600 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb824009dc0 tx=0x7fb82402bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e10ec7739e69612 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:27.691 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.685+0000 7fb82aa00640 1 -- 172.21.15.60:0/1327570599 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb824039070 con 0x7fb82c15ca40 2024-04-14T23:48:27.691 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.685+0000 7fb82aa00640 1 -- 172.21.15.60:0/1327570599 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb82402ccd0 con 0x7fb82c15ca40 2024-04-14T23:48:27.691 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.686+0000 7fb82aa00640 1 -- 172.21.15.60:0/1327570599 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb824034660 con 0x7fb82c15ca40 2024-04-14T23:48:27.692 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.686+0000 7fb833200640 1 -- 172.21.15.60:0/1327570599 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb82c15ca40 msgr2=0x7fb82c0cb600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:27.692 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.686+0000 7fb833200640 1 --2- 172.21.15.60:0/1327570599 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb82c15ca40 0x7fb82c0cb600 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb824009dc0 tx=0x7fb82402bf20 comp rx=0 tx=0).stop 2024-04-14T23:48:27.692 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.686+0000 7fb833200640 1 -- 172.21.15.60:0/1327570599 shutdown_connections 2024-04-14T23:48:27.692 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.686+0000 7fb833200640 1 --2- 172.21.15.60:0/1327570599 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb82c15ca40 0x7fb82c0cb600 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:27.692 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.686+0000 7fb833200640 1 -- 172.21.15.60:0/1327570599 >> 172.21.15.60:0/1327570599 conn(0x7fb82c06d1e0 msgr2=0x7fb82c067060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:27.692 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.686+0000 7fb833200640 1 -- 172.21.15.60:0/1327570599 shutdown_connections 2024-04-14T23:48:27.692 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.686+0000 7fb833200640 1 -- 172.21.15.60:0/1327570599 wait complete. 2024-04-14T23:48:27.694 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.689+0000 7fb833200640 1 Processor -- start 2024-04-14T23:48:27.694 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.689+0000 7fb833200640 1 -- start start 2024-04-14T23:48:27.697 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.690+0000 7fb833200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb82c154f20 0x7fb82c1f4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:27.697 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.690+0000 7fb833200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb82c15bb50 0x7fb82c1f1140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:27.697 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.690+0000 7fb833200640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb82c15ca40 0x7fb82c155ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:27.698 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.690+0000 7fb833200640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb82c0d0c90 con 0x7fb82c15ca40 2024-04-14T23:48:27.698 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.690+0000 7fb833200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb82c0cdc00 con 0x7fb82c15bb50 2024-04-14T23:48:27.698 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.690+0000 7fb833200640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fb82c0d0e10 con 0x7fb82c154f20 2024-04-14T23:48:27.698 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.690+0000 7fb82be00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb82c154f20 0x7fb82c1f4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:27.698 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.690+0000 7fb82be00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb82c154f20 0x7fb82c1f4720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.60:36564/0 (socket says 172.21.15.60:36564) 2024-04-14T23:48:27.698 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.690+0000 7fb82be00640 1 -- 172.21.15.60:0/1943066835 learned_addr learned my addr 172.21.15.60:0/1943066835 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:27.698 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.690+0000 7fb82b400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb82c15bb50 0x7fb82c1f1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:27.698 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.690+0000 7fb831200640 1 --2- 172.21.15.60:0/1943066835 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb82c15ca40 0x7fb82c155ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:27.698 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.691+0000 7fb82be00640 1 -- 172.21.15.60:0/1943066835 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb82c15bb50 msgr2=0x7fb82c1f1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:27.698 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.691+0000 7fb82be00640 1 --2- 172.21.15.60:0/1943066835 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb82c15bb50 0x7fb82c1f1140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:27.698 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.691+0000 7fb82be00640 1 -- 172.21.15.60:0/1943066835 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb82c15ca40 msgr2=0x7fb82c155ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:27.699 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.691+0000 7fb82be00640 1 --2- 172.21.15.60:0/1943066835 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb82c15ca40 0x7fb82c155ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:27.699 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.691+0000 7fb82be00640 1 -- 172.21.15.60:0/1943066835 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb82c159b70 con 0x7fb82c154f20 2024-04-14T23:48:27.699 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.691+0000 7fb82be00640 1 --2- 172.21.15.60:0/1943066835 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb82c154f20 0x7fb82c1f4720 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb81c00c500 tx=0x7fb81c00c9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:27.699 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.691+0000 7fb823e00640 1 -- 172.21.15.60:0/1943066835 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb81c017070 con 0x7fb82c154f20 2024-04-14T23:48:27.699 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.691+0000 7fb823e00640 1 -- 172.21.15.60:0/1943066835 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb81c005d00 con 0x7fb82c154f20 2024-04-14T23:48:27.699 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.691+0000 7fb833200640 1 -- 172.21.15.60:0/1943066835 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb82c159e10 con 0x7fb82c154f20 2024-04-14T23:48:27.699 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.691+0000 7fb823e00640 1 -- 172.21.15.60:0/1943066835 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb81c007330 con 0x7fb82c154f20 2024-04-14T23:48:27.699 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.691+0000 7fb833200640 1 -- 172.21.15.60:0/1943066835 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb82c1f0940 con 0x7fb82c154f20 2024-04-14T23:48:27.700 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.691+0000 7fb823e00640 1 -- 172.21.15.60:0/1943066835 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fb81c01a070 con 0x7fb82c154f20 2024-04-14T23:48:27.700 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.692+0000 7fb823e00640 1 -- 172.21.15.60:0/1943066835 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7fb81c0074d0 con 0x7fb82c154f20 2024-04-14T23:48:27.700 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-14T23:48:27.693+0000 7ff99ec45040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-14T23:48:27.701 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.692+0000 7fb833200640 1 -- 172.21.15.60:0/1943066835 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb82c068980 con 0x7fb82c154f20 2024-04-14T23:48:27.701 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.695+0000 7fb823e00640 1 -- 172.21.15.60:0/1943066835 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fb81c021020 con 0x7fb82c154f20 2024-04-14T23:48:27.768 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.763+0000 7fb833200640 1 -- 172.21.15.60:0/1943066835 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb82c075e30 con 0x7fb82c154f20 2024-04-14T23:48:27.769 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.763+0000 7fb823e00640 1 -- 172.21.15.60:0/1943066835 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7fb81c01c440 con 0x7fb82c154f20 2024-04-14T23:48:27.769 INFO:teuthology.misc.health.smithi060.stdout: 2024-04-14T23:48:27.769 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"5847db56-a76d-4486-b563-d83f1186af92","created":"2024-04-14T23:48:23.900003+0000","modified":"2024-04-14T23:48:26.345499+0000","last_up_change":"0.000000","last_in_change":"2024-04-14T23:48:26.345499+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b2b2b755-7f42-4f9b-af9a-a178f7cc5a52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"321418cd-5ab7-41a5-b9c8-b3fc12f68ebc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"07874e9f-3a38-41dc-be9f-2b75e84e53d9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"7e4ef6c9-b17a-47c6-a320-f5d16a6fe05e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"759a3efc-dade-44c5-b57a-1623499aba50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"24cd58c1-0cf1-4077-ab33-84c39c6907ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6add38f1-65e9-4b0d-927a-4ab77f5e2323","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a2326b6f-7601-4dda-a456-67c443ea5929","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-14T23:48:27.771 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.765+0000 7fb833200640 1 -- 172.21.15.60:0/1943066835 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb82c154f20 msgr2=0x7fb82c1f4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:27.771 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.765+0000 7fb833200640 1 --2- 172.21.15.60:0/1943066835 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb82c154f20 0x7fb82c1f4720 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb81c00c500 tx=0x7fb81c00c9c0 comp rx=0 tx=0).stop 2024-04-14T23:48:27.771 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.765+0000 7fb833200640 1 -- 172.21.15.60:0/1943066835 shutdown_connections 2024-04-14T23:48:27.771 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.765+0000 7fb833200640 1 --2- 172.21.15.60:0/1943066835 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb82c15ca40 0x7fb82c155ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:27.771 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.765+0000 7fb833200640 1 --2- 172.21.15.60:0/1943066835 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb82c15bb50 0x7fb82c1f1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:27.771 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.765+0000 7fb833200640 1 --2- 172.21.15.60:0/1943066835 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb82c154f20 0x7fb82c1f4720 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:27.771 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.765+0000 7fb833200640 1 -- 172.21.15.60:0/1943066835 >> 172.21.15.60:0/1943066835 conn(0x7fb82c06d1e0 msgr2=0x7fb82c0cb590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:27.771 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.765+0000 7fb833200640 1 -- 172.21.15.60:0/1943066835 shutdown_connections 2024-04-14T23:48:27.771 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:27.765+0000 7fb833200640 1 -- 172.21.15.60:0/1943066835 wait complete. 2024-04-14T23:48:27.783 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-04-14T23:48:27.976 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-14T23:48:27.972+0000 7ff99ec45040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-14T23:48:28.028 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-14T23:48:28.023+0000 7ff99ec45040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-14T23:48:28.145 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-14T23:48:28.140+0000 7ff99ec45040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-14T23:48:28.694 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-04-14T23:48:28.688+0000 7ffaf7042040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-14T23:48:29.024 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-04-14T23:48:29.019+0000 7ffaf7042040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-14T23:48:29.088 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-04-14T23:48:29.082+0000 7ffaf7042040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-14T23:48:29.179 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-14T23:48:29.174+0000 7ff93f64c640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-14T23:48:29.180 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-14T23:48:29.175+0000 7ff93f64c640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-14T23:48:29.224 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-04-14T23:48:29.219+0000 7ffaf7042040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-14T23:48:30.494 INFO:tasks.ceph.osd.5.smithi186.stderr:2024-04-14T23:48:30.489+0000 7f680ab93640 -1 osd.5 0 log_to_monitors true 2024-04-14T23:48:30.520 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-04-14T23:48:30.515+0000 7f2eaf7bd640 -1 osd.4 0 log_to_monitors true 2024-04-14T23:48:30.527 INFO:tasks.ceph.osd.6.smithi186.stderr:2024-04-14T23:48:30.522+0000 7f8fabcbd640 -1 osd.6 0 log_to_monitors true 2024-04-14T23:48:30.535 INFO:tasks.ceph.osd.7.smithi186.stderr:2024-04-14T23:48:30.530+0000 7f01a4b73640 -1 osd.7 0 log_to_monitors true 2024-04-14T23:48:30.719 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-04-14T23:48:30.714+0000 7f65b8468640 -1 osd.3 0 log_to_monitors true 2024-04-14T23:48:30.739 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-04-14T23:48:30.734+0000 7f2de868b640 -1 osd.2 0 log_to_monitors true 2024-04-14T23:48:30.937 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-04-14T23:48:30.932+0000 7fa5de373640 -1 osd.1 0 log_to_monitors true 2024-04-14T23:48:31.002 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-04-14T23:48:30.997+0000 7fab46a24640 -1 osd.0 0 log_to_monitors true 2024-04-14T23:48:33.183 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-04-14T23:48:33.178+0000 7f2eab3f7640 -1 osd.4 0 waiting for initial osdmap 2024-04-14T23:48:33.194 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-04-14T23:48:33.189+0000 7f2ea3280640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-14T23:48:33.197 INFO:tasks.ceph.osd.5.smithi186.stderr:2024-04-14T23:48:33.191+0000 7f68067f7640 -1 osd.5 0 waiting for initial osdmap 2024-04-14T23:48:33.204 INFO:tasks.ceph.osd.6.smithi186.stderr:2024-04-14T23:48:33.199+0000 7f8fa79f7640 -1 osd.6 0 waiting for initial osdmap 2024-04-14T23:48:33.208 INFO:tasks.ceph.osd.5.smithi186.stderr:2024-04-14T23:48:33.204+0000 7f67fe680640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-14T23:48:33.212 INFO:tasks.ceph.osd.7.smithi186.stderr:2024-04-14T23:48:33.207+0000 7f01a0ff8640 -1 osd.7 0 waiting for initial osdmap 2024-04-14T23:48:33.215 INFO:tasks.ceph.osd.6.smithi186.stderr:2024-04-14T23:48:33.210+0000 7f8f9f86f640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-14T23:48:33.226 INFO:tasks.ceph.osd.7.smithi186.stderr:2024-04-14T23:48:33.222+0000 7f0198680640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-14T23:48:33.681 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-04-14T23:48:33.675+0000 7fab41400640 -1 osd.0 0 waiting for initial osdmap 2024-04-14T23:48:33.691 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-04-14T23:48:33.685+0000 7fab37400640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-14T23:48:33.784 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-04-14T23:48:33.778+0000 7fa5d8e00640 -1 osd.1 0 waiting for initial osdmap 2024-04-14T23:48:33.790 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-04-14T23:48:33.785+0000 7fa5cee00640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-14T23:48:33.796 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-04-14T23:48:33.790+0000 7f2de3200640 -1 osd.2 0 waiting for initial osdmap 2024-04-14T23:48:33.803 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-04-14T23:48:33.798+0000 7f65b3000640 -1 osd.3 0 waiting for initial osdmap 2024-04-14T23:48:33.809 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-04-14T23:48:33.804+0000 7f65a9000640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-14T23:48:33.810 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-04-14T23:48:33.805+0000 7f2dd9200640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-14T23:48:34.590 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-14T23:48:34.777 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.772+0000 7fab22600640 1 Processor -- start 2024-04-14T23:48:34.778 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.772+0000 7fab22600640 1 -- start start 2024-04-14T23:48:34.779 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.774+0000 7fab22600640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fab1c152900 0x7fab1c1592e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:34.779 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.774+0000 7fab22600640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fab1c0c6a10 0x7fab1c0c5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:34.779 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.774+0000 7fab22600640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab1c15bff0 con 0x7fab1c0c8530 2024-04-14T23:48:34.779 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.774+0000 7fab22600640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fab1c152ce0 con 0x7fab1c0c6a10 2024-04-14T23:48:34.779 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.774+0000 7fab22600640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fab1c15c2d0 con 0x7fab1c152900 2024-04-14T23:48:34.780 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.774+0000 7fab21c00640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fab1c0c8530 0x7fab1c06cf80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.60:39508/0 (socket says 172.21.15.60:39508) 2024-04-14T23:48:34.780 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.774+0000 7fab21c00640 1 -- 172.21.15.60:0/4186389911 learned_addr learned my addr 172.21.15.60:0/4186389911 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:48:34.781 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.774+0000 7fab1be00640 1 --2- 172.21.15.60:0/4186389911 >> v2:172.21.15.60:3300/0 conn(0x7fab1c0c6a10 0x7fab1c0c5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:34.781 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.774+0000 7fab21200640 1 --2- 172.21.15.60:0/4186389911 >> v2:172.21.15.186:3300/0 conn(0x7fab1c152900 0x7fab1c1592e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:34.781 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.775+0000 7fab1b400640 1 -- 172.21.15.60:0/4186389911 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2017166002 0 0) 0x7fab1c15bff0 con 0x7fab1c0c8530 2024-04-14T23:48:34.781 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.775+0000 7fab1b400640 1 -- 172.21.15.60:0/4186389911 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fab04003520 con 0x7fab1c0c8530 2024-04-14T23:48:34.781 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.775+0000 7fab1be00640 1 -- 172.21.15.60:0/4186389911 >> v2:172.21.15.186:3300/0 conn(0x7fab1c152900 msgr2=0x7fab1c1592e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:34.781 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.775+0000 7fab1be00640 1 --2- 172.21.15.60:0/4186389911 >> v2:172.21.15.186:3300/0 conn(0x7fab1c152900 0x7fab1c1592e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:34.781 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.775+0000 7fab1be00640 1 -- 172.21.15.60:0/4186389911 >> v1:172.21.15.186:6789/0 conn(0x7fab1c0c8530 legacy=0x7fab1c06cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:34.781 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.775+0000 7fab1be00640 1 -- 172.21.15.60:0/4186389911 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab1c15c5b0 con 0x7fab1c0c6a10 2024-04-14T23:48:34.782 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.776+0000 7fab1be00640 1 --2- 172.21.15.60:0/4186389911 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab1c0c6a10 0x7fab1c0c5720 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fab1c06c1b0 tx=0x7fab0802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7a73a7cc947fac5 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:34.782 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.776+0000 7fab1b400640 1 -- 172.21.15.60:0/4186389911 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab08034070 con 0x7fab1c0c6a10 2024-04-14T23:48:34.782 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.776+0000 7fab1b400640 1 -- 172.21.15.60:0/4186389911 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab08030610 con 0x7fab1c0c6a10 2024-04-14T23:48:34.782 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.776+0000 7fab1b400640 1 -- 172.21.15.60:0/4186389911 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab080309e0 con 0x7fab1c0c6a10 2024-04-14T23:48:34.783 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.777+0000 7fab22600640 1 -- 172.21.15.60:0/4186389911 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab1c0c6a10 msgr2=0x7fab1c0c5720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:34.783 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.777+0000 7fab22600640 1 --2- 172.21.15.60:0/4186389911 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab1c0c6a10 0x7fab1c0c5720 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fab1c06c1b0 tx=0x7fab0802b960 comp rx=0 tx=0).stop 2024-04-14T23:48:34.783 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.777+0000 7fab22600640 1 -- 172.21.15.60:0/4186389911 shutdown_connections 2024-04-14T23:48:34.783 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.777+0000 7fab22600640 1 --2- 172.21.15.60:0/4186389911 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab1c0c6a10 0x7fab1c0c5720 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:34.783 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.777+0000 7fab22600640 1 --2- 172.21.15.60:0/4186389911 >> v2:172.21.15.186:3300/0 conn(0x7fab1c152900 0x7fab1c1592e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:34.783 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.777+0000 7fab22600640 1 -- 172.21.15.60:0/4186389911 >> 172.21.15.60:0/4186389911 conn(0x7fab1c06ea00 msgr2=0x7fab1c06ee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:34.783 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.777+0000 7fab22600640 1 -- 172.21.15.60:0/4186389911 shutdown_connections 2024-04-14T23:48:34.783 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.777+0000 7fab22600640 1 -- 172.21.15.60:0/4186389911 wait complete. 2024-04-14T23:48:34.784 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.779+0000 7fab22600640 1 Processor -- start 2024-04-14T23:48:34.784 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.779+0000 7fab22600640 1 -- start start 2024-04-14T23:48:34.786 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.780+0000 7fab22600640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab1c0c6a10 0x7fab1c1f26b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:34.786 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.780+0000 7fab22600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab1c0c8530 0x7fab1c1491d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:34.786 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.780+0000 7fab22600640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab1c152900 0x7fab1c14af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:34.786 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.780+0000 7fab22600640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fab1c06d690 con 0x7fab1c0c6a10 2024-04-14T23:48:34.786 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.780+0000 7fab22600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fab1c15cb70 con 0x7fab1c0c8530 2024-04-14T23:48:34.787 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.780+0000 7fab22600640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fab1c06d810 con 0x7fab1c152900 2024-04-14T23:48:34.787 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.780+0000 7fab21200640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab1c0c6a10 0x7fab1c1f26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:34.787 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.780+0000 7fab21200640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab1c0c6a10 0x7fab1c1f26b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37534/0 (socket says 172.21.15.60:37534) 2024-04-14T23:48:34.787 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.780+0000 7fab21200640 1 -- 172.21.15.60:0/1053016249 learned_addr learned my addr 172.21.15.60:0/1053016249 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:34.787 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.780+0000 7fab1be00640 1 --2- 172.21.15.60:0/1053016249 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab1c0c8530 0x7fab1c1491d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:34.787 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.780+0000 7fab21c00640 1 --2- 172.21.15.60:0/1053016249 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab1c152900 0x7fab1c14af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:34.787 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.780+0000 7fab21200640 1 -- 172.21.15.60:0/1053016249 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab1c152900 msgr2=0x7fab1c14af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:34.787 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.780+0000 7fab21200640 1 --2- 172.21.15.60:0/1053016249 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab1c152900 0x7fab1c14af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:34.787 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.780+0000 7fab21200640 1 -- 172.21.15.60:0/1053016249 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab1c0c8530 msgr2=0x7fab1c1491d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:34.787 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.780+0000 7fab21200640 1 --2- 172.21.15.60:0/1053016249 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab1c0c8530 0x7fab1c1491d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:34.787 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.780+0000 7fab21200640 1 -- 172.21.15.60:0/1053016249 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab1c0cac20 con 0x7fab1c0c6a10 2024-04-14T23:48:34.787 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.780+0000 7fab21c00640 1 --2- 172.21.15.60:0/1053016249 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab1c152900 0x7fab1c14af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-14T23:48:34.787 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.781+0000 7fab21200640 1 --2- 172.21.15.60:0/1053016249 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab1c0c6a10 0x7fab1c1f26b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fab1000e9b0 tx=0x7fab1000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:34.788 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.781+0000 7fab18c00640 1 -- 172.21.15.60:0/1053016249 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab10017070 con 0x7fab1c0c6a10 2024-04-14T23:48:34.788 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.781+0000 7fab18c00640 1 -- 172.21.15.60:0/1053016249 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab1000c9a0 con 0x7fab1c0c6a10 2024-04-14T23:48:34.788 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.781+0000 7fab18c00640 1 -- 172.21.15.60:0/1053016249 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab1000cd90 con 0x7fab1c0c6a10 2024-04-14T23:48:34.788 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.781+0000 7fab22600640 1 -- 172.21.15.60:0/1053016249 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab1c1f2bf0 con 0x7fab1c0c6a10 2024-04-14T23:48:34.789 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.781+0000 7fab22600640 1 -- 172.21.15.60:0/1053016249 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab1c0cb1c0 con 0x7fab1c0c6a10 2024-04-14T23:48:34.789 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.782+0000 7fab22600640 1 -- 172.21.15.60:0/1053016249 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab1c068980 con 0x7fab1c0c6a10 2024-04-14T23:48:34.789 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.783+0000 7fab18c00640 1 -- 172.21.15.60:0/1053016249 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) ==== 90098+0+0 (secure 0 0 0) 0x7fab1001a070 con 0x7fab1c0c6a10 2024-04-14T23:48:34.789 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.784+0000 7fab18c00640 1 --2- 172.21.15.60:0/1053016249 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7faafc06b900 0x7faafc06ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:34.789 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.784+0000 7fab18c00640 1 -- 172.21.15.60:0/1053016249 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(13..13 src has 1..13) ==== 4471+0+0 (secure 0 0 0) 0x7fab1008a320 con 0x7fab1c0c6a10 2024-04-14T23:48:34.790 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.784+0000 7fab1be00640 1 --2- 172.21.15.60:0/1053016249 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7faafc06b900 0x7faafc06ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:34.790 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.785+0000 7fab1be00640 1 --2- 172.21.15.60:0/1053016249 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7faafc06b900 0x7faafc06ddc0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fab08036000 tx=0x7fab0803b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:34.800 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.794+0000 7fab18c00640 1 -- 172.21.15.60:0/1053016249 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fab10058df0 con 0x7fab1c0c6a10 2024-04-14T23:48:34.992 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.987+0000 7fab22600640 1 -- 172.21.15.60:0/1053016249 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fab1c075180 con 0x7fab1c0c6a10 2024-04-14T23:48:34.993 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.988+0000 7fab18c00640 1 -- 172.21.15.60:0/1053016249 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) ==== 74+0+10909 (secure 0 0 0) 0x7fab1005ce90 con 0x7fab1c0c6a10 2024-04-14T23:48:34.994 INFO:teuthology.misc.health.smithi060.stdout: 2024-04-14T23:48:34.994 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":13,"fsid":"5847db56-a76d-4486-b563-d83f1186af92","created":"2024-04-14T23:48:23.900003+0000","modified":"2024-04-14T23:48:34.203573+0000","last_up_change":"2024-04-14T23:48:34.203573+0000","last_in_change":"2024-04-14T23:48:26.345499+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b2b2b755-7f42-4f9b-af9a-a178f7cc5a52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6817","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6818","nonce":2536528895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6822","nonce":2536528895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6827","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6828","nonce":2536528895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6825","nonce":2536528895}]},"public_addr":"172.21.15.60:6818/2536528895","cluster_addr":"172.21.15.60:6822/2536528895","heartbeat_back_addr":"172.21.15.60:6828/2536528895","heartbeat_front_addr":"172.21.15.60:6825/2536528895","state":["exists","up"]},{"osd":1,"uuid":"321418cd-5ab7-41a5-b9c8-b3fc12f68ebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6801","nonce":3115695676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6803","nonce":3115695676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6807","nonce":3115695676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6805","nonce":3115695676}]},"public_addr":"172.21.15.60:6801/3115695676","cluster_addr":"172.21.15.60:6803/3115695676","heartbeat_back_addr":"172.21.15.60:6807/3115695676","heartbeat_front_addr":"172.21.15.60:6805/3115695676","state":["exists","up"]},{"osd":2,"uuid":"07874e9f-3a38-41dc-be9f-2b75e84e53d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6819","nonce":3475168698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6824","nonce":3475168698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6831","nonce":3475168698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6829","nonce":3475168698}]},"public_addr":"172.21.15.60:6819/3475168698","cluster_addr":"172.21.15.60:6824/3475168698","heartbeat_back_addr":"172.21.15.60:6831/3475168698","heartbeat_front_addr":"172.21.15.60:6829/3475168698","state":["exists","up"]},{"osd":3,"uuid":"7e4ef6c9-b17a-47c6-a320-f5d16a6fe05e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6809","nonce":1730813674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6811","nonce":1730813674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6815","nonce":1730813674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6813","nonce":1730813674}]},"public_addr":"172.21.15.60:6809/1730813674","cluster_addr":"172.21.15.60:6811/1730813674","heartbeat_back_addr":"172.21.15.60:6815/1730813674","heartbeat_front_addr":"172.21.15.60:6813/1730813674","state":["exists","up"]},{"osd":4,"uuid":"759a3efc-dade-44c5-b57a-1623499aba50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6805","nonce":1464587038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6810","nonce":1464587038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6819","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6822","nonce":1464587038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6813","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6816","nonce":1464587038}]},"public_addr":"172.21.15.186:6805/1464587038","cluster_addr":"172.21.15.186:6810/1464587038","heartbeat_back_addr":"172.21.15.186:6822/1464587038","heartbeat_front_addr":"172.21.15.186:6816/1464587038","state":["exists","up"]},{"osd":5,"uuid":"24cd58c1-0cf1-4077-ab33-84c39c6907ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6801","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6803","nonce":1697098067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6809","nonce":1697098067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6820","nonce":1697098067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6815","nonce":1697098067}]},"public_addr":"172.21.15.186:6803/1697098067","cluster_addr":"172.21.15.186:6809/1697098067","heartbeat_back_addr":"172.21.15.186:6820/1697098067","heartbeat_front_addr":"172.21.15.186:6815/1697098067","state":["exists","up"]},{"osd":6,"uuid":"6add38f1-65e9-4b0d-927a-4ab77f5e2323","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6804","nonce":1837108559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6807","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6811","nonce":1837108559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6823","nonce":1837108559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6817","nonce":1837108559}]},"public_addr":"172.21.15.186:6804/1837108559","cluster_addr":"172.21.15.186:6811/1837108559","heartbeat_back_addr":"172.21.15.186:6823/1837108559","heartbeat_front_addr":"172.21.15.186:6817/1837108559","state":["exists","up"]},{"osd":7,"uuid":"a2326b6f-7601-4dda-a456-67c443ea5929","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6825","nonce":842537875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6827","nonce":842537875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6831","nonce":842537875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6829","nonce":842537875}]},"public_addr":"172.21.15.186:6825/842537875","cluster_addr":"172.21.15.186:6827/842537875","heartbeat_back_addr":"172.21.15.186:6831/842537875","heartbeat_front_addr":"172.21.15.186:6829/842537875","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-14T23:48:34.997 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.991+0000 7fab22600640 1 -- 172.21.15.60:0/1053016249 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7faafc06b900 msgr2=0x7faafc06ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:34.997 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.991+0000 7fab22600640 1 --2- 172.21.15.60:0/1053016249 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7faafc06b900 0x7faafc06ddc0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fab08036000 tx=0x7fab0803b040 comp rx=0 tx=0).stop 2024-04-14T23:48:34.997 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.991+0000 7fab22600640 1 -- 172.21.15.60:0/1053016249 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab1c0c6a10 msgr2=0x7fab1c1f26b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:34.997 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.991+0000 7fab22600640 1 --2- 172.21.15.60:0/1053016249 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab1c0c6a10 0x7fab1c1f26b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fab1000e9b0 tx=0x7fab1000ee70 comp rx=0 tx=0).stop 2024-04-14T23:48:34.997 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.991+0000 7fab22600640 1 -- 172.21.15.60:0/1053016249 shutdown_connections 2024-04-14T23:48:34.997 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.991+0000 7fab22600640 1 --2- 172.21.15.60:0/1053016249 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7faafc06b900 0x7faafc06ddc0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:34.997 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.991+0000 7fab22600640 1 --2- 172.21.15.60:0/1053016249 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab1c152900 0x7fab1c14af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:34.997 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.991+0000 7fab22600640 1 --2- 172.21.15.60:0/1053016249 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab1c0c8530 0x7fab1c1491d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:34.997 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.991+0000 7fab22600640 1 --2- 172.21.15.60:0/1053016249 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab1c0c6a10 0x7fab1c1f26b0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:34.997 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.991+0000 7fab22600640 1 -- 172.21.15.60:0/1053016249 >> 172.21.15.60:0/1053016249 conn(0x7fab1c06ea00 msgr2=0x7fab1c0c5720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:34.997 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.991+0000 7fab22600640 1 -- 172.21.15.60:0/1053016249 shutdown_connections 2024-04-14T23:48:34.997 INFO:teuthology.misc.health.smithi060.stderr:2024-04-14T23:48:34.991+0000 7fab22600640 1 -- 172.21.15.60:0/1053016249 wait complete. 2024-04-14T23:48:35.010 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-04-14T23:48:35.010 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-14T23:48:35.011 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-14T23:48:35.011 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-14T23:48:35.018 INFO:tasks.ceph.mds.a:Started 2024-04-14T23:48:35.018 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-14T23:48:35.018 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-14T23:48:35.025 INFO:tasks.ceph.mds.c:Started 2024-04-14T23:48:35.025 INFO:tasks.ceph.mds.e:Restarting daemon 2024-04-14T23:48:35.025 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2024-04-14T23:48:35.032 INFO:tasks.ceph.mds.e:Started 2024-04-14T23:48:35.032 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-14T23:48:35.032 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-14T23:48:35.035 INFO:tasks.ceph.mds.b:Started 2024-04-14T23:48:35.035 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-14T23:48:35.036 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-14T23:48:35.038 INFO:tasks.ceph.mds.d:Started 2024-04-14T23:48:35.038 INFO:tasks.ceph.mds.f:Restarting daemon 2024-04-14T23:48:35.038 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2024-04-14T23:48:35.040 INFO:tasks.ceph.mds.f:Started 2024-04-14T23:48:35.041 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-04-14T23:48:35.041 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-14T23:48:35.123 INFO:tasks.ceph.mds.d.smithi186.stdout:starting mds.d at 2024-04-14T23:48:35.123 INFO:tasks.ceph.mds.b.smithi186.stdout:starting mds.b at 2024-04-14T23:48:35.123 INFO:tasks.ceph.mds.f.smithi186.stdout:starting mds.f at 2024-04-14T23:48:35.124 INFO:tasks.ceph.mds.b.smithi186.stderr:2024-04-14T23:48:35.119+0000 7f9f483fd640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2024-04-14T23:48:35.124 INFO:tasks.ceph.mds.b.smithi186.stderr:2024-04-14T23:48:35.119+0000 7f9f483fd640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6837/0: (98) Address already in use 2024-04-14T23:48:35.168 INFO:tasks.ceph.mds.a.smithi060.stdout:starting mds.a at 2024-04-14T23:48:35.178 INFO:tasks.ceph.mds.c.smithi060.stdout:starting mds.c at 2024-04-14T23:48:35.181 INFO:tasks.ceph.mds.e.smithi060.stdout:starting mds.e at 2024-04-14T23:48:35.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.215+0000 7ff3e1800640 1 Processor -- start 2024-04-14T23:48:35.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.216+0000 7ff3e1800640 1 -- start start 2024-04-14T23:48:35.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.217+0000 7ff3e1800640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff3dc0cd540 0x7ff3dc15ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:35.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.217+0000 7ff3e1800640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff3dc0cb750 0x7ff3dc154b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:35.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.217+0000 7ff3e1800640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3dc0b3f50 con 0x7ff3dc0cd160 2024-04-14T23:48:35.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.217+0000 7ff3e1800640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7ff3dc15b370 con 0x7ff3dc0cb750 2024-04-14T23:48:35.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.217+0000 7ff3e1800640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7ff3dc0b39b0 con 0x7ff3dc0cd540 2024-04-14T23:48:35.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.217+0000 7ff3daa00640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7ff3dc0cd160 0x7ff3dc0b3190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:39978/0 (socket says 172.21.15.60:39978) 2024-04-14T23:48:35.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.217+0000 7ff3daa00640 1 -- 172.21.15.60:0/204948988 learned_addr learned my addr 172.21.15.60:0/204948988 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:48:35.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.217+0000 7ff3db400640 1 --2- 172.21.15.60:0/204948988 >> v2:172.21.15.186:3300/0 conn(0x7ff3dc0cb750 0x7ff3dc154b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:35.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.217+0000 7ff3da000640 1 --2- 172.21.15.60:0/204948988 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff3dc0cd540 0x7ff3dc15ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:35.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.217+0000 7ff3d9600640 1 -- 172.21.15.60:0/204948988 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 293551101 0 0) 0x7ff3dc0b3f50 con 0x7ff3dc0cd160 2024-04-14T23:48:35.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.217+0000 7ff3db400640 1 -- 172.21.15.60:0/204948988 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff3dc0cd540 msgr2=0x7ff3dc15ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:35.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.217+0000 7ff3db400640 1 --2- 172.21.15.60:0/204948988 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff3dc0cd540 0x7ff3dc15ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:35.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.217+0000 7ff3db400640 1 -- 172.21.15.60:0/204948988 >> v1:172.21.15.60:6789/0 conn(0x7ff3dc0cd160 legacy=0x7ff3dc0b3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:35.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.217+0000 7ff3db400640 1 -- 172.21.15.60:0/204948988 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3dc0b3c70 con 0x7ff3dc0cb750 2024-04-14T23:48:35.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.218+0000 7ff3db400640 1 --2- 172.21.15.60:0/204948988 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff3dc0cb750 0x7ff3dc154b40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff3d4009d90 tx=0x7ff3d402bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6151fc2e95c05d03 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:35.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.219+0000 7ff3d9600640 1 -- 172.21.15.60:0/204948988 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3d4039070 con 0x7ff3dc0cb750 2024-04-14T23:48:35.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.219+0000 7ff3d9600640 1 -- 172.21.15.60:0/204948988 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3d402cc50 con 0x7ff3dc0cb750 2024-04-14T23:48:35.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.219+0000 7ff3d9600640 1 -- 172.21.15.60:0/204948988 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3d4034440 con 0x7ff3dc0cb750 2024-04-14T23:48:35.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.219+0000 7ff3e1800640 1 -- 172.21.15.60:0/204948988 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff3dc0cb750 msgr2=0x7ff3dc154b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:35.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.219+0000 7ff3e1800640 1 --2- 172.21.15.60:0/204948988 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff3dc0cb750 0x7ff3dc154b40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff3d4009d90 tx=0x7ff3d402bea0 comp rx=0 tx=0).stop 2024-04-14T23:48:35.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.219+0000 7ff3e1800640 1 -- 172.21.15.60:0/204948988 shutdown_connections 2024-04-14T23:48:35.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.219+0000 7ff3e1800640 1 --2- 172.21.15.60:0/204948988 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff3dc0cb750 0x7ff3dc154b40 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:35.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.219+0000 7ff3e1800640 1 --2- 172.21.15.60:0/204948988 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff3dc0cd540 0x7ff3dc15ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:35.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.219+0000 7ff3e1800640 1 -- 172.21.15.60:0/204948988 >> 172.21.15.60:0/204948988 conn(0x7ff3dc066f10 msgr2=0x7ff3dc0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:35.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.219+0000 7ff3e1800640 1 -- 172.21.15.60:0/204948988 shutdown_connections 2024-04-14T23:48:35.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.219+0000 7ff3e1800640 1 -- 172.21.15.60:0/204948988 wait complete. 2024-04-14T23:48:35.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.221+0000 7ff3e1800640 1 Processor -- start 2024-04-14T23:48:35.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.221+0000 7ff3e1800640 1 -- start start 2024-04-14T23:48:35.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.222+0000 7ff3e1800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff3dc0cb750 0x7ff3dc1f0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:35.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.222+0000 7ff3e1800640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff3dc0cd160 0x7ff3dc14e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:35.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.222+0000 7ff3e1800640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3dc0cd540 0x7ff3dc14e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:35.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.222+0000 7ff3e1800640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff3dc15ea70 con 0x7ff3dc0cd540 2024-04-14T23:48:35.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.222+0000 7ff3e1800640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff3dc15e790 con 0x7ff3dc0cb750 2024-04-14T23:48:35.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.222+0000 7ff3e1800640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7ff3dc0ccb20 con 0x7ff3dc0cd160 2024-04-14T23:48:35.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.222+0000 7ff3da000640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff3dc0cd160 0x7ff3dc14e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:35.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.222+0000 7ff3daa00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff3dc0cb750 0x7ff3dc1f0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:35.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.222+0000 7ff3da000640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff3dc0cd160 0x7ff3dc14e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.60:49508/0 (socket says 172.21.15.60:49508) 2024-04-14T23:48:35.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.222+0000 7ff3da000640 1 -- 172.21.15.60:0/2863152743 learned_addr learned my addr 172.21.15.60:0/2863152743 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:35.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.222+0000 7ff3db400640 1 --2- 172.21.15.60:0/2863152743 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3dc0cd540 0x7ff3dc14e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:35.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.223+0000 7ff3da000640 1 -- 172.21.15.60:0/2863152743 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff3dc0cb750 msgr2=0x7ff3dc1f0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:35.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.223+0000 7ff3da000640 1 --2- 172.21.15.60:0/2863152743 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff3dc0cb750 0x7ff3dc1f0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:35.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.223+0000 7ff3da000640 1 -- 172.21.15.60:0/2863152743 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3dc0cd540 msgr2=0x7ff3dc14e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:35.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.223+0000 7ff3da000640 1 --2- 172.21.15.60:0/2863152743 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3dc0cd540 0x7ff3dc14e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:35.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.223+0000 7ff3da000640 1 -- 172.21.15.60:0/2863152743 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3dc14df20 con 0x7ff3dc0cd160 2024-04-14T23:48:35.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.223+0000 7ff3daa00640 1 --2- 172.21.15.60:0/2863152743 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff3dc0cb750 0x7ff3dc1f0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:48:35.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.223+0000 7ff3da000640 1 --2- 172.21.15.60:0/2863152743 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff3dc0cd160 0x7ff3dc14e400 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff3c800e9b0 tx=0x7ff3c800ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:35.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.223+0000 7ff3d2a00640 1 -- 172.21.15.60:0/2863152743 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c8017070 con 0x7ff3dc0cd160 2024-04-14T23:48:35.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.223+0000 7ff3d2a00640 1 -- 172.21.15.60:0/2863152743 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3c800c9a0 con 0x7ff3dc0cd160 2024-04-14T23:48:35.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.223+0000 7ff3d2a00640 1 -- 172.21.15.60:0/2863152743 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c800cd90 con 0x7ff3dc0cd160 2024-04-14T23:48:35.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.223+0000 7ff3e1800640 1 -- 172.21.15.60:0/2863152743 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3dc14cd50 con 0x7ff3dc0cd160 2024-04-14T23:48:35.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.224+0000 7ff3e1800640 1 -- 172.21.15.60:0/2863152743 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3dc14d310 con 0x7ff3dc0cd160 2024-04-14T23:48:35.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.225+0000 7ff3e1800640 1 -- 172.21.15.60:0/2863152743 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3dc076970 con 0x7ff3dc0cd160 2024-04-14T23:48:35.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.225+0000 7ff3d2a00640 1 -- 172.21.15.60:0/2863152743 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 4) ==== 90098+0+0 (secure 0 0 0) 0x7ff3c801a070 con 0x7ff3dc0cd160 2024-04-14T23:48:35.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.226+0000 7ff3d2a00640 1 --2- 172.21.15.60:0/2863152743 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff3bc06b900 0x7ff3bc06ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:35.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.226+0000 7ff3d2a00640 1 -- 172.21.15.60:0/2863152743 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(13..13 src has 1..13) ==== 4471+0+0 (secure 0 0 0) 0x7ff3c808a260 con 0x7ff3dc0cd160 2024-04-14T23:48:35.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.226+0000 7ff3daa00640 1 --2- 172.21.15.60:0/2863152743 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff3bc06b900 0x7ff3bc06ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:35.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.227+0000 7ff3daa00640 1 --2- 172.21.15.60:0/2863152743 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff3bc06b900 0x7ff3bc06ddc0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff3cc007fd0 tx=0x7ff3cc007de0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:35.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.231+0000 7ff3d2a00640 1 -- 172.21.15.60:0/2863152743 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff3c8058dc0 con 0x7ff3dc0cd160 2024-04-14T23:48:35.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.428+0000 7ff3e1800640 1 -- 172.21.15.60:0/2863152743 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff3dc0b4bd0 con 0x7ff3dc0cd160 2024-04-14T23:48:35.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.429+0000 7ff3d2a00640 1 -- 172.21.15.60:0/2863152743 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) ==== 74+0+11093 (secure 0 0 0) 0x7ff3c805ce60 con 0x7ff3dc0cd160 2024-04-14T23:48:35.434 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:48:35.435 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":14,"fsid":"5847db56-a76d-4486-b563-d83f1186af92","created":"2024-04-14T23:48:23.900003+0000","modified":"2024-04-14T23:48:35.229834+0000","last_up_change":"2024-04-14T23:48:34.203573+0000","last_in_change":"2024-04-14T23:48:26.345499+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b2b2b755-7f42-4f9b-af9a-a178f7cc5a52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6817","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6818","nonce":2536528895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6822","nonce":2536528895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6827","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6828","nonce":2536528895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6825","nonce":2536528895}]},"public_addr":"172.21.15.60:6818/2536528895","cluster_addr":"172.21.15.60:6822/2536528895","heartbeat_back_addr":"172.21.15.60:6828/2536528895","heartbeat_front_addr":"172.21.15.60:6825/2536528895","state":["exists","up"]},{"osd":1,"uuid":"321418cd-5ab7-41a5-b9c8-b3fc12f68ebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6801","nonce":3115695676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6803","nonce":3115695676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6807","nonce":3115695676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6805","nonce":3115695676}]},"public_addr":"172.21.15.60:6801/3115695676","cluster_addr":"172.21.15.60:6803/3115695676","heartbeat_back_addr":"172.21.15.60:6807/3115695676","heartbeat_front_addr":"172.21.15.60:6805/3115695676","state":["exists","up"]},{"osd":2,"uuid":"07874e9f-3a38-41dc-be9f-2b75e84e53d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6819","nonce":3475168698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6824","nonce":3475168698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6831","nonce":3475168698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6829","nonce":3475168698}]},"public_addr":"172.21.15.60:6819/3475168698","cluster_addr":"172.21.15.60:6824/3475168698","heartbeat_back_addr":"172.21.15.60:6831/3475168698","heartbeat_front_addr":"172.21.15.60:6829/3475168698","state":["exists","up"]},{"osd":3,"uuid":"7e4ef6c9-b17a-47c6-a320-f5d16a6fe05e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6809","nonce":1730813674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6811","nonce":1730813674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6815","nonce":1730813674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6813","nonce":1730813674}]},"public_addr":"172.21.15.60:6809/1730813674","cluster_addr":"172.21.15.60:6811/1730813674","heartbeat_back_addr":"172.21.15.60:6815/1730813674","heartbeat_front_addr":"172.21.15.60:6813/1730813674","state":["exists","up"]},{"osd":4,"uuid":"759a3efc-dade-44c5-b57a-1623499aba50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6805","nonce":1464587038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6810","nonce":1464587038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6819","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6822","nonce":1464587038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6813","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6816","nonce":1464587038}]},"public_addr":"172.21.15.186:6805/1464587038","cluster_addr":"172.21.15.186:6810/1464587038","heartbeat_back_addr":"172.21.15.186:6822/1464587038","heartbeat_front_addr":"172.21.15.186:6816/1464587038","state":["exists","up"]},{"osd":5,"uuid":"24cd58c1-0cf1-4077-ab33-84c39c6907ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6801","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6803","nonce":1697098067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6809","nonce":1697098067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6820","nonce":1697098067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6815","nonce":1697098067}]},"public_addr":"172.21.15.186:6803/1697098067","cluster_addr":"172.21.15.186:6809/1697098067","heartbeat_back_addr":"172.21.15.186:6820/1697098067","heartbeat_front_addr":"172.21.15.186:6815/1697098067","state":["exists","up"]},{"osd":6,"uuid":"6add38f1-65e9-4b0d-927a-4ab77f5e2323","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6804","nonce":1837108559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6807","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6811","nonce":1837108559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6823","nonce":1837108559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6817","nonce":1837108559}]},"public_addr":"172.21.15.186:6804/1837108559","cluster_addr":"172.21.15.186:6811/1837108559","heartbeat_back_addr":"172.21.15.186:6823/1837108559","heartbeat_front_addr":"172.21.15.186:6817/1837108559","state":["exists","up"]},{"osd":7,"uuid":"a2326b6f-7601-4dda-a456-67c443ea5929","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6825","nonce":842537875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6827","nonce":842537875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6831","nonce":842537875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6829","nonce":842537875}]},"public_addr":"172.21.15.186:6825/842537875","cluster_addr":"172.21.15.186:6827/842537875","heartbeat_back_addr":"172.21.15.186:6831/842537875","heartbeat_front_addr":"172.21.15.186:6829/842537875","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.975583+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.976806+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.722725+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.763705+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.502140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.528972+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.517961+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.513350+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-14T23:48:35.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.431+0000 7ff3e1800640 1 -- 172.21.15.60:0/2863152743 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff3bc06b900 msgr2=0x7ff3bc06ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:35.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.431+0000 7ff3e1800640 1 --2- 172.21.15.60:0/2863152743 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff3bc06b900 0x7ff3bc06ddc0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff3cc007fd0 tx=0x7ff3cc007de0 comp rx=0 tx=0).stop 2024-04-14T23:48:35.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.431+0000 7ff3e1800640 1 -- 172.21.15.60:0/2863152743 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff3dc0cd160 msgr2=0x7ff3dc14e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:35.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.431+0000 7ff3e1800640 1 --2- 172.21.15.60:0/2863152743 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff3dc0cd160 0x7ff3dc14e400 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff3c800e9b0 tx=0x7ff3c800ee70 comp rx=0 tx=0).stop 2024-04-14T23:48:35.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.432+0000 7ff3e1800640 1 -- 172.21.15.60:0/2863152743 shutdown_connections 2024-04-14T23:48:35.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.432+0000 7ff3e1800640 1 --2- 172.21.15.60:0/2863152743 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff3bc06b900 0x7ff3bc06ddc0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:35.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.432+0000 7ff3e1800640 1 --2- 172.21.15.60:0/2863152743 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3dc0cd540 0x7ff3dc14e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:35.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.432+0000 7ff3e1800640 1 --2- 172.21.15.60:0/2863152743 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff3dc0cd160 0x7ff3dc14e400 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:35.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.432+0000 7ff3e1800640 1 --2- 172.21.15.60:0/2863152743 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff3dc0cb750 0x7ff3dc1f0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:35.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.432+0000 7ff3e1800640 1 -- 172.21.15.60:0/2863152743 >> 172.21.15.60:0/2863152743 conn(0x7ff3dc066f10 msgr2=0x7ff3dc0b50f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:35.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.432+0000 7ff3e1800640 1 -- 172.21.15.60:0/2863152743 shutdown_connections 2024-04-14T23:48:35.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.432+0000 7ff3e1800640 1 -- 172.21.15.60:0/2863152743 wait complete. 2024-04-14T23:48:35.453 INFO:tasks.cephfs.filesystem.ceph_manager:[] 2024-04-14T23:48:35.454 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:48:35.666 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.661+0000 7fc5a9200640 1 Processor -- start 2024-04-14T23:48:35.667 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.661+0000 7fc5a9200640 1 -- start start 2024-04-14T23:48:35.668 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.662+0000 7fc5a9200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc5a4151ae0 0x7fc5a40c3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:35.668 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.663+0000 7fc5a9200640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5a40c5710 con 0x7fc5a415ae00 2024-04-14T23:48:35.668 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.663+0000 7fc5a9200640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5a40ccf80 con 0x7fc5a40c52b0 2024-04-14T23:48:35.668 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.663+0000 7fc5a9200640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fc5a406e920 con 0x7fc5a4151ae0 2024-04-14T23:48:35.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.663+0000 7fc5a2800640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fc5a415ae00 0x7fc5a415b250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:39986/0 (socket says 172.21.15.60:39986) 2024-04-14T23:48:35.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.663+0000 7fc5a2800640 1 -- 172.21.15.60:0/3105865459 learned_addr learned my addr 172.21.15.60:0/3105865459 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:48:35.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.663+0000 7fc5a1e00640 1 --2- 172.21.15.60:0/3105865459 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc5a4151ae0 0x7fc5a40c3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:35.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.663+0000 7fc5a1e00640 1 -- 172.21.15.60:0/3105865459 >> v1:172.21.15.186:6789/0 conn(0x7fc5a40c52b0 legacy=0x7fc5a406e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:35.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.663+0000 7fc5a1e00640 1 -- 172.21.15.60:0/3105865459 >> v1:172.21.15.60:6789/0 conn(0x7fc5a415ae00 legacy=0x7fc5a415b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:35.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.663+0000 7fc5a0a00640 1 -- 172.21.15.60:0/3105865459 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2466237460 0 0) 0x7fc5a40ccf80 con 0x7fc5a40c52b0 2024-04-14T23:48:35.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.663+0000 7fc5a1e00640 1 -- 172.21.15.60:0/3105865459 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5a4072dc0 con 0x7fc5a4151ae0 2024-04-14T23:48:35.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.664+0000 7fc5a1e00640 1 --2- 172.21.15.60:0/3105865459 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc5a4151ae0 0x7fc5a40c3910 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc5940097d0 tx=0x7fc59402da10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=819422f6c2b88f36 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:35.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.665+0000 7fc5a0a00640 1 -- 172.21.15.60:0/3105865459 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc594036070 con 0x7fc5a4151ae0 2024-04-14T23:48:35.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.665+0000 7fc5a0a00640 1 -- 172.21.15.60:0/3105865459 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc594032850 con 0x7fc5a4151ae0 2024-04-14T23:48:35.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.665+0000 7fc5a0a00640 1 -- 172.21.15.60:0/3105865459 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc594032c20 con 0x7fc5a4151ae0 2024-04-14T23:48:35.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.665+0000 7fc5a9200640 1 -- 172.21.15.60:0/3105865459 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc5a4151ae0 msgr2=0x7fc5a40c3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:35.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.665+0000 7fc5a9200640 1 --2- 172.21.15.60:0/3105865459 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc5a4151ae0 0x7fc5a40c3910 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc5940097d0 tx=0x7fc59402da10 comp rx=0 tx=0).stop 2024-04-14T23:48:35.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.665+0000 7fc5a9200640 1 -- 172.21.15.60:0/3105865459 shutdown_connections 2024-04-14T23:48:35.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.665+0000 7fc5a9200640 1 --2- 172.21.15.60:0/3105865459 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc5a4151ae0 0x7fc5a40c3910 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:35.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.665+0000 7fc5a9200640 1 -- 172.21.15.60:0/3105865459 >> 172.21.15.60:0/3105865459 conn(0x7fc5a40cc170 msgr2=0x7fc5a40cc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:35.672 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.665+0000 7fc5a9200640 1 -- 172.21.15.60:0/3105865459 shutdown_connections 2024-04-14T23:48:35.672 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.665+0000 7fc5a9200640 1 -- 172.21.15.60:0/3105865459 wait complete. 2024-04-14T23:48:35.672 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.667+0000 7fc5a9200640 1 Processor -- start 2024-04-14T23:48:35.672 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.667+0000 7fc5a9200640 1 -- start start 2024-04-14T23:48:35.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.668+0000 7fc5a9200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc5a40c52b0 0x7fc5a41f0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:35.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.668+0000 7fc5a9200640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5a4151ae0 0x7fc5a41500a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:35.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.668+0000 7fc5a9200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc5a415ae00 0x7fc5a41ecf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:35.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.668+0000 7fc5a9200640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fc5a40726c0 con 0x7fc5a4151ae0 2024-04-14T23:48:35.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.668+0000 7fc5a9200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc5a40723e0 con 0x7fc5a40c52b0 2024-04-14T23:48:35.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.668+0000 7fc5a9200640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fc5a415b960 con 0x7fc5a415ae00 2024-04-14T23:48:35.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.668+0000 7fc5a1e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc5a40c52b0 0x7fc5a41f0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:35.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.668+0000 7fc5a2800640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc5a415ae00 0x7fc5a41ecf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:35.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.668+0000 7fc5a1e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc5a40c52b0 0x7fc5a41f0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:46232/0 (socket says 172.21.15.60:46232) 2024-04-14T23:48:35.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.668+0000 7fc5a1e00640 1 -- 172.21.15.60:0/1187306151 learned_addr learned my addr 172.21.15.60:0/1187306151 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:35.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.668+0000 7fc5a1400640 1 --2- 172.21.15.60:0/1187306151 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5a4151ae0 0x7fc5a41500a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:35.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.669+0000 7fc5a1e00640 1 -- 172.21.15.60:0/1187306151 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc5a415ae00 msgr2=0x7fc5a41ecf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:35.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.669+0000 7fc5a1e00640 1 --2- 172.21.15.60:0/1187306151 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc5a415ae00 0x7fc5a41ecf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:35.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.669+0000 7fc5a1e00640 1 -- 172.21.15.60:0/1187306151 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5a4151ae0 msgr2=0x7fc5a41500a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:35.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.669+0000 7fc5a1e00640 1 --2- 172.21.15.60:0/1187306151 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5a4151ae0 0x7fc5a41500a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:35.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.669+0000 7fc5a1e00640 1 -- 172.21.15.60:0/1187306151 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5a414c950 con 0x7fc5a40c52b0 2024-04-14T23:48:35.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.669+0000 7fc5a2800640 1 --2- 172.21.15.60:0/1187306151 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc5a415ae00 0x7fc5a41ecf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:48:35.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.669+0000 7fc5a1400640 1 --2- 172.21.15.60:0/1187306151 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5a4151ae0 0x7fc5a41500a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:48:35.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.669+0000 7fc5a1e00640 1 --2- 172.21.15.60:0/1187306151 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc5a40c52b0 0x7fc5a41f0820 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc5940098f0 tx=0x7fc5940046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:35.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.669+0000 7fc59a000640 1 -- 172.21.15.60:0/1187306151 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc594048070 con 0x7fc5a40c52b0 2024-04-14T23:48:35.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.669+0000 7fc59a000640 1 -- 172.21.15.60:0/1187306151 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc59403bd70 con 0x7fc5a40c52b0 2024-04-14T23:48:35.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.669+0000 7fc59a000640 1 -- 172.21.15.60:0/1187306151 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc594036040 con 0x7fc5a40c52b0 2024-04-14T23:48:35.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.669+0000 7fc5a9200640 1 -- 172.21.15.60:0/1187306151 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5a414cbf0 con 0x7fc5a40c52b0 2024-04-14T23:48:35.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.670+0000 7fc5a9200640 1 -- 172.21.15.60:0/1187306151 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5a414d1b0 con 0x7fc5a40c52b0 2024-04-14T23:48:35.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.671+0000 7fc5a9200640 1 -- 172.21.15.60:0/1187306151 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5a40ca7b0 con 0x7fc5a40c52b0 2024-04-14T23:48:35.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.672+0000 7fc59a000640 1 -- 172.21.15.60:0/1187306151 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 4) ==== 90098+0+0 (secure 0 0 0) 0x7fc594039070 con 0x7fc5a40c52b0 2024-04-14T23:48:35.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.672+0000 7fc59a000640 1 --2- 172.21.15.60:0/1187306151 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fc58406b900 0x7fc58406ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:35.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.673+0000 7fc59a000640 1 -- 172.21.15.60:0/1187306151 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 4471+0+0 (secure 0 0 0) 0x7fc5940ac0e0 con 0x7fc5a40c52b0 2024-04-14T23:48:35.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.673+0000 7fc5a1400640 1 --2- 172.21.15.60:0/1187306151 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fc58406b900 0x7fc58406ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:35.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.673+0000 7fc5a1400640 1 --2- 172.21.15.60:0/1187306151 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fc58406b900 0x7fc58406ddc0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc5a4158af0 tx=0x7fc590009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:35.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.677+0000 7fc59a000640 1 -- 172.21.15.60:0/1187306151 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc5a40ca7b0 con 0x7fc5a40c52b0 2024-04-14T23:48:35.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.921+0000 7fc5a9200640 1 -- 172.21.15.60:0/1187306151 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc5a4076970 con 0x7fc5a40c52b0 2024-04-14T23:48:35.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.921+0000 7fc59a000640 1 -- 172.21.15.60:0/1187306151 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1395 (secure 0 0 0) 0x7fc594040070 con 0x7fc5a40c52b0 2024-04-14T23:48:35.927 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:48:35.927 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-14T23:48:35.928 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 2 2024-04-14T23:48:35.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.924+0000 7fc5a9200640 1 -- 172.21.15.60:0/1187306151 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fc58406b900 msgr2=0x7fc58406ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:35.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.924+0000 7fc5a9200640 1 --2- 172.21.15.60:0/1187306151 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fc58406b900 0x7fc58406ddc0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc5a4158af0 tx=0x7fc590009040 comp rx=0 tx=0).stop 2024-04-14T23:48:35.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.924+0000 7fc5a9200640 1 -- 172.21.15.60:0/1187306151 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc5a40c52b0 msgr2=0x7fc5a41f0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:35.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.924+0000 7fc5a9200640 1 --2- 172.21.15.60:0/1187306151 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc5a40c52b0 0x7fc5a41f0820 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc5940098f0 tx=0x7fc5940046c0 comp rx=0 tx=0).stop 2024-04-14T23:48:35.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.924+0000 7fc5a9200640 1 -- 172.21.15.60:0/1187306151 shutdown_connections 2024-04-14T23:48:35.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.924+0000 7fc5a9200640 1 --2- 172.21.15.60:0/1187306151 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fc58406b900 0x7fc58406ddc0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:35.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.924+0000 7fc5a9200640 1 --2- 172.21.15.60:0/1187306151 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc5a415ae00 0x7fc5a41ecf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:35.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.924+0000 7fc5a9200640 1 --2- 172.21.15.60:0/1187306151 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5a4151ae0 0x7fc5a41500a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:35.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.924+0000 7fc5a9200640 1 --2- 172.21.15.60:0/1187306151 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc5a40c52b0 0x7fc5a41f0820 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:35.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.924+0000 7fc5a9200640 1 -- 172.21.15.60:0/1187306151 >> 172.21.15.60:0/1187306151 conn(0x7fc5a40cc170 msgr2=0x7fc5a40717c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:35.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.924+0000 7fc5a9200640 1 -- 172.21.15.60:0/1187306151 shutdown_connections 2024-04-14T23:48:35.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:35.925+0000 7fc5a9200640 1 -- 172.21.15.60:0/1187306151 wait complete. 2024-04-14T23:48:37.949 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:48:38.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.186+0000 7fcb16a00640 1 Processor -- start 2024-04-14T23:48:38.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.186+0000 7fcb16a00640 1 -- start start 2024-04-14T23:48:38.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.187+0000 7fcb16a00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fcb10073320 0x7fcb100c6950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:38.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.187+0000 7fcb16a00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb10152fb0 con 0x7fcb100b1e00 2024-04-14T23:48:38.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.187+0000 7fcb16a00640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb10159310 con 0x7fcb100c3d90 2024-04-14T23:48:38.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.187+0000 7fcb16a00640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fcb100c6540 con 0x7fcb10073320 2024-04-14T23:48:38.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.188+0000 7fcb14c00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fcb10073320 0x7fcb100c6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:38.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.188+0000 7fcb16000640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fcb100c3d90 0x7fcb100c41c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.60:39532/0 (socket says 172.21.15.60:39532) 2024-04-14T23:48:38.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.188+0000 7fcb16000640 1 -- 172.21.15.60:0/2849887752 learned_addr learned my addr 172.21.15.60:0/2849887752 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:48:38.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.188+0000 7fcb0fe00640 1 -- 172.21.15.60:0/2849887752 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2141874790 0 0) 0x7fcb10159310 con 0x7fcb100c3d90 2024-04-14T23:48:38.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.188+0000 7fcb0fe00640 1 -- 172.21.15.60:0/2849887752 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcaf8003520 con 0x7fcb100c3d90 2024-04-14T23:48:38.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.188+0000 7fcb14c00640 1 -- 172.21.15.60:0/2849887752 >> v1:172.21.15.186:6789/0 conn(0x7fcb100c3d90 legacy=0x7fcb100c41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:38.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.188+0000 7fcb14c00640 1 -- 172.21.15.60:0/2849887752 >> v1:172.21.15.60:6789/0 conn(0x7fcb100b1e00 legacy=0x7fcb100c7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:38.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.188+0000 7fcb0fe00640 1 -- 172.21.15.60:0/2849887752 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1696448852 0 0) 0x7fcb10152fb0 con 0x7fcb100b1e00 2024-04-14T23:48:38.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.189+0000 7fcb14c00640 1 -- 172.21.15.60:0/2849887752 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb100c38b0 con 0x7fcb10073320 2024-04-14T23:48:38.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.190+0000 7fcb14c00640 1 --2- 172.21.15.60:0/2849887752 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb10073320 0x7fcb100c6950 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fcb101594e0 tx=0x7fcafc02b9d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=90d2ef56cd75143f server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:38.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.191+0000 7fcb0fe00640 1 -- 172.21.15.60:0/2849887752 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcafc034070 con 0x7fcb10073320 2024-04-14T23:48:38.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.191+0000 7fcb0fe00640 1 -- 172.21.15.60:0/2849887752 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcafc030890 con 0x7fcb10073320 2024-04-14T23:48:38.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.191+0000 7fcb0fe00640 1 -- 172.21.15.60:0/2849887752 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcafc030c60 con 0x7fcb10073320 2024-04-14T23:48:38.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.192+0000 7fcb16a00640 1 -- 172.21.15.60:0/2849887752 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb10073320 msgr2=0x7fcb100c6950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:38.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.192+0000 7fcb16a00640 1 --2- 172.21.15.60:0/2849887752 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb10073320 0x7fcb100c6950 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fcb101594e0 tx=0x7fcafc02b9d0 comp rx=0 tx=0).stop 2024-04-14T23:48:38.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.192+0000 7fcb16a00640 1 -- 172.21.15.60:0/2849887752 shutdown_connections 2024-04-14T23:48:38.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.192+0000 7fcb16a00640 1 --2- 172.21.15.60:0/2849887752 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb10073320 0x7fcb100c6950 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:38.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.192+0000 7fcb16a00640 1 -- 172.21.15.60:0/2849887752 >> 172.21.15.60:0/2849887752 conn(0x7fcb100c8b50 msgr2=0x7fcb100c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:38.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.192+0000 7fcb16a00640 1 -- 172.21.15.60:0/2849887752 shutdown_connections 2024-04-14T23:48:38.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.192+0000 7fcb16a00640 1 -- 172.21.15.60:0/2849887752 wait complete. 2024-04-14T23:48:38.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.194+0000 7fcb16a00640 1 Processor -- start 2024-04-14T23:48:38.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.194+0000 7fcb16a00640 1 -- start start 2024-04-14T23:48:38.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.195+0000 7fcb16a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb10073320 0x7fcb10159a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:38.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.195+0000 7fcb16a00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb100b1e00 0x7fcb101e7220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:38.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.195+0000 7fcb16a00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcb100c3d90 0x7fcb101567e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:38.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.195+0000 7fcb16a00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fcb10159890 con 0x7fcb100b1e00 2024-04-14T23:48:38.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.195+0000 7fcb16a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcb100c5f80 con 0x7fcb10073320 2024-04-14T23:48:38.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.195+0000 7fcb16a00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fcb100c6260 con 0x7fcb100c3d90 2024-04-14T23:48:38.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.195+0000 7fcb15600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb10073320 0x7fcb10159a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:38.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.195+0000 7fcb14c00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb100b1e00 0x7fcb101e7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:38.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.195+0000 7fcb15600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb10073320 0x7fcb10159a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:46240/0 (socket says 172.21.15.60:46240) 2024-04-14T23:48:38.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.195+0000 7fcb15600640 1 -- 172.21.15.60:0/3298549960 learned_addr learned my addr 172.21.15.60:0/3298549960 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:38.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.195+0000 7fcb16000640 1 --2- 172.21.15.60:0/3298549960 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcb100c3d90 0x7fcb101567e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:38.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.196+0000 7fcb15600640 1 -- 172.21.15.60:0/3298549960 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcb100c3d90 msgr2=0x7fcb101567e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:38.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.196+0000 7fcb15600640 1 --2- 172.21.15.60:0/3298549960 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcb100c3d90 0x7fcb101567e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:38.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.196+0000 7fcb15600640 1 -- 172.21.15.60:0/3298549960 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb100b1e00 msgr2=0x7fcb101e7220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:38.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.196+0000 7fcb15600640 1 --2- 172.21.15.60:0/3298549960 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb100b1e00 0x7fcb101e7220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:38.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.196+0000 7fcb15600640 1 -- 172.21.15.60:0/3298549960 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb100c2b60 con 0x7fcb10073320 2024-04-14T23:48:38.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.196+0000 7fcb16000640 1 --2- 172.21.15.60:0/3298549960 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcb100c3d90 0x7fcb101567e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-14T23:48:38.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.196+0000 7fcb15600640 1 --2- 172.21.15.60:0/3298549960 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb10073320 0x7fcb10159a50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fcb0400c4e0 tx=0x7fcb0400c9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:38.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.197+0000 7fcb0d600640 1 -- 172.21.15.60:0/3298549960 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb04017070 con 0x7fcb10073320 2024-04-14T23:48:38.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.197+0000 7fcb0d600640 1 -- 172.21.15.60:0/3298549960 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb040035f0 con 0x7fcb10073320 2024-04-14T23:48:38.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.197+0000 7fcb0d600640 1 -- 172.21.15.60:0/3298549960 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb040039e0 con 0x7fcb10073320 2024-04-14T23:48:38.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.197+0000 7fcb16a00640 1 -- 172.21.15.60:0/3298549960 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb101f50d0 con 0x7fcb10073320 2024-04-14T23:48:38.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.197+0000 7fcb16a00640 1 -- 172.21.15.60:0/3298549960 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb101f10a0 con 0x7fcb10073320 2024-04-14T23:48:38.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.198+0000 7fcb16a00640 1 -- 172.21.15.60:0/3298549960 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb1004f540 con 0x7fcb10073320 2024-04-14T23:48:38.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.199+0000 7fcb0d600640 1 -- 172.21.15.60:0/3298549960 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 4) ==== 90098+0+0 (secure 0 0 0) 0x7fcb0401a070 con 0x7fcb10073320 2024-04-14T23:48:38.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.200+0000 7fcb0d600640 1 --2- 172.21.15.60:0/3298549960 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fcaf006b900 0x7fcaf006ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:38.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.200+0000 7fcb0d600640 1 -- 172.21.15.60:0/3298549960 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7fcb04089770 con 0x7fcb10073320 2024-04-14T23:48:38.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.201+0000 7fcb14c00640 1 --2- 172.21.15.60:0/3298549960 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fcaf006b900 0x7fcaf006ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:38.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.201+0000 7fcb14c00640 1 --2- 172.21.15.60:0/3298549960 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fcaf006b900 0x7fcaf006ddc0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fcafc036000 tx=0x7fcafc03c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:38.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.206+0000 7fcb0d600640 1 -- 172.21.15.60:0/3298549960 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fcb100c2b60 con 0x7fcb10073320 2024-04-14T23:48:38.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.458+0000 7fcb16a00640 1 -- 172.21.15.60:0/3298549960 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcb100b21e0 con 0x7fcb10073320 2024-04-14T23:48:38.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.459+0000 7fcb0d600640 1 -- 172.21.15.60:0/3298549960 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+5515 (secure 0 0 0) 0x7fcb04058040 con 0x7fcb10073320 2024-04-14T23:48:38.465 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:48:38.465 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/78303888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":78303888},{"type":"v1","addr":"172.21.15.60:6833","nonce":78303888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4279,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/2677133122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2677133122},{"type":"v1","addr":"172.21.15.60:6835","nonce":2677133122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-14T23:48:38.466 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 3 2024-04-14T23:48:38.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.462+0000 7fcb16a00640 1 -- 172.21.15.60:0/3298549960 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fcaf006b900 msgr2=0x7fcaf006ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:38.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.462+0000 7fcb16a00640 1 --2- 172.21.15.60:0/3298549960 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fcaf006b900 0x7fcaf006ddc0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fcafc036000 tx=0x7fcafc03c040 comp rx=0 tx=0).stop 2024-04-14T23:48:38.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.462+0000 7fcb16a00640 1 -- 172.21.15.60:0/3298549960 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb10073320 msgr2=0x7fcb10159a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:38.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.462+0000 7fcb16a00640 1 --2- 172.21.15.60:0/3298549960 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb10073320 0x7fcb10159a50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fcb0400c4e0 tx=0x7fcb0400c9a0 comp rx=0 tx=0).stop 2024-04-14T23:48:38.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.463+0000 7fcb16a00640 1 -- 172.21.15.60:0/3298549960 shutdown_connections 2024-04-14T23:48:38.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.463+0000 7fcb16a00640 1 --2- 172.21.15.60:0/3298549960 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fcaf006b900 0x7fcaf006ddc0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:38.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.463+0000 7fcb16a00640 1 --2- 172.21.15.60:0/3298549960 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcb100c3d90 0x7fcb101567e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:38.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.463+0000 7fcb16a00640 1 --2- 172.21.15.60:0/3298549960 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb100b1e00 0x7fcb101e7220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:38.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.463+0000 7fcb16a00640 1 --2- 172.21.15.60:0/3298549960 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb10073320 0x7fcb10159a50 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:38.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.463+0000 7fcb16a00640 1 -- 172.21.15.60:0/3298549960 >> 172.21.15.60:0/3298549960 conn(0x7fcb100c8b50 msgr2=0x7fcb100c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:38.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.463+0000 7fcb16a00640 1 -- 172.21.15.60:0/3298549960 shutdown_connections 2024-04-14T23:48:38.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.463+0000 7fcb16a00640 1 -- 172.21.15.60:0/3298549960 wait complete. 2024-04-14T23:48:38.486 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-14T23:48:38.674 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.668+0000 7fc6d2c00640 1 Processor -- start 2024-04-14T23:48:38.674 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.668+0000 7fc6d2c00640 1 -- start start 2024-04-14T23:48:38.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.670+0000 7fc6d2c00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc6cc06f1f0 0x7fc6cc0c72a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:38.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.670+0000 7fc6d2c00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc6cc0731e0 0x7fc6cc0c77e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:38.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.670+0000 7fc6d2c00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6cc15b0d0 con 0x7fc6cc06f5d0 2024-04-14T23:48:38.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.670+0000 7fc6d2c00640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fc6cc0c6e10 con 0x7fc6cc06f1f0 2024-04-14T23:48:38.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.670+0000 7fc6d2c00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fc6cc15b370 con 0x7fc6cc0731e0 2024-04-14T23:48:38.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.670+0000 7fc6cbe00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc6cc06f1f0 0x7fc6cc0c72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:38.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.670+0000 7fc6cbe00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc6cc06f1f0 0x7fc6cc0c72a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37632/0 (socket says 172.21.15.60:37632) 2024-04-14T23:48:38.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.670+0000 7fc6cbe00640 1 -- 172.21.15.60:0/2542791585 learned_addr learned my addr 172.21.15.60:0/2542791585 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:38.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.670+0000 7fc6d0c00640 1 --2- 172.21.15.60:0/2542791585 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc6cc0731e0 0x7fc6cc0c77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:38.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.670+0000 7fc6cbe00640 1 -- 172.21.15.60:0/2542791585 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc6cc0731e0 msgr2=0x7fc6cc0c77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:38.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.670+0000 7fc6cbe00640 1 --2- 172.21.15.60:0/2542791585 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc6cc0731e0 0x7fc6cc0c77e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:38.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.670+0000 7fc6cbe00640 1 -- 172.21.15.60:0/2542791585 >> v1:172.21.15.60:6789/0 conn(0x7fc6cc06f5d0 legacy=0x7fc6cc154980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:38.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.670+0000 7fc6cbe00640 1 -- 172.21.15.60:0/2542791585 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6cc15b650 con 0x7fc6cc06f1f0 2024-04-14T23:48:38.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.670+0000 7fc6cb400640 1 -- 172.21.15.60:0/2542791585 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1014649701 0 0) 0x7fc6cc15b0d0 con 0x7fc6cc06f5d0 2024-04-14T23:48:38.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.671+0000 7fc6d0c00640 1 --2- 172.21.15.60:0/2542791585 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc6cc0731e0 0x7fc6cc0c77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-14T23:48:38.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.672+0000 7fc6cbe00640 1 --2- 172.21.15.60:0/2542791585 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6cc06f1f0 0x7fc6cc0c72a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc6bc009870 tx=0x7fc6bc02ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70598d29cd2107a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:38.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.672+0000 7fc6cb400640 1 -- 172.21.15.60:0/2542791585 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6bc02f830 con 0x7fc6cc06f1f0 2024-04-14T23:48:38.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.672+0000 7fc6cb400640 1 -- 172.21.15.60:0/2542791585 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6bc02f9d0 con 0x7fc6cc06f1f0 2024-04-14T23:48:38.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.672+0000 7fc6cb400640 1 -- 172.21.15.60:0/2542791585 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6bc02fda0 con 0x7fc6cc06f1f0 2024-04-14T23:48:38.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.672+0000 7fc6d2c00640 1 -- 172.21.15.60:0/2542791585 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6cc06f1f0 msgr2=0x7fc6cc0c72a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:38.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.672+0000 7fc6d2c00640 1 --2- 172.21.15.60:0/2542791585 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6cc06f1f0 0x7fc6cc0c72a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc6bc009870 tx=0x7fc6bc02ba10 comp rx=0 tx=0).stop 2024-04-14T23:48:38.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.673+0000 7fc6d2c00640 1 -- 172.21.15.60:0/2542791585 shutdown_connections 2024-04-14T23:48:38.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.673+0000 7fc6d2c00640 1 --2- 172.21.15.60:0/2542791585 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc6cc0731e0 0x7fc6cc0c77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:38.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.673+0000 7fc6d2c00640 1 --2- 172.21.15.60:0/2542791585 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6cc06f1f0 0x7fc6cc0c72a0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:38.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.673+0000 7fc6d2c00640 1 -- 172.21.15.60:0/2542791585 >> 172.21.15.60:0/2542791585 conn(0x7fc6cc0cc170 msgr2=0x7fc6cc0cc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:38.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.673+0000 7fc6d2c00640 1 -- 172.21.15.60:0/2542791585 shutdown_connections 2024-04-14T23:48:38.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.673+0000 7fc6d2c00640 1 -- 172.21.15.60:0/2542791585 wait complete. 2024-04-14T23:48:38.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.674+0000 7fc6d2c00640 1 Processor -- start 2024-04-14T23:48:38.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.674+0000 7fc6d2c00640 1 -- start start 2024-04-14T23:48:38.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.675+0000 7fc6d2c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6cc06f1f0 0x7fc6cc1f1300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:38.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.675+0000 7fc6d2c00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6cc06f5d0 0x7fc6cc157b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:38.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.675+0000 7fc6d2c00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc6cc0731e0 0x7fc6cc156180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:38.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.675+0000 7fc6d2c00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fc6cc155090 con 0x7fc6cc06f5d0 2024-04-14T23:48:38.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.675+0000 7fc6d2c00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc6cc071180 con 0x7fc6cc06f1f0 2024-04-14T23:48:38.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.675+0000 7fc6d2c00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fc6cc155210 con 0x7fc6cc0731e0 2024-04-14T23:48:38.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.676+0000 7fc6cbe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6cc06f1f0 0x7fc6cc1f1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:38.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.676+0000 7fc6cbe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6cc06f1f0 0x7fc6cc1f1300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:46248/0 (socket says 172.21.15.60:46248) 2024-04-14T23:48:38.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.676+0000 7fc6cbe00640 1 -- 172.21.15.60:0/3322341190 learned_addr learned my addr 172.21.15.60:0/3322341190 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:38.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.676+0000 7fc6c3e00640 1 --2- 172.21.15.60:0/3322341190 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6cc06f5d0 0x7fc6cc157b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:38.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.676+0000 7fc6d0c00640 1 --2- 172.21.15.60:0/3322341190 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc6cc0731e0 0x7fc6cc156180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:38.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.676+0000 7fc6cbe00640 1 -- 172.21.15.60:0/3322341190 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc6cc0731e0 msgr2=0x7fc6cc156180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:38.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.676+0000 7fc6cbe00640 1 --2- 172.21.15.60:0/3322341190 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc6cc0731e0 0x7fc6cc156180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:38.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.676+0000 7fc6cbe00640 1 -- 172.21.15.60:0/3322341190 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6cc06f5d0 msgr2=0x7fc6cc157b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:38.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.676+0000 7fc6cbe00640 1 --2- 172.21.15.60:0/3322341190 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6cc06f5d0 0x7fc6cc157b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:38.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.676+0000 7fc6cbe00640 1 -- 172.21.15.60:0/3322341190 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6cc150fc0 con 0x7fc6cc06f1f0 2024-04-14T23:48:38.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.676+0000 7fc6c3e00640 1 --2- 172.21.15.60:0/3322341190 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6cc06f5d0 0x7fc6cc157b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:48:38.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.676+0000 7fc6cbe00640 1 --2- 172.21.15.60:0/3322341190 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6cc06f1f0 0x7fc6cc1f1300 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc6bc0090b0 tx=0x7fc6bc004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:38.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.677+0000 7fc6c8c00640 1 -- 172.21.15.60:0/3322341190 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6bc047070 con 0x7fc6cc06f1f0 2024-04-14T23:48:38.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.677+0000 7fc6c8c00640 1 -- 172.21.15.60:0/3322341190 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6bc037bc0 con 0x7fc6cc06f1f0 2024-04-14T23:48:38.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.677+0000 7fc6d2c00640 1 -- 172.21.15.60:0/3322341190 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6cc1512a0 con 0x7fc6cc06f1f0 2024-04-14T23:48:38.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.677+0000 7fc6c8c00640 1 -- 172.21.15.60:0/3322341190 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6bc004030 con 0x7fc6cc06f1f0 2024-04-14T23:48:38.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.677+0000 7fc6d2c00640 1 -- 172.21.15.60:0/3322341190 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6cc155980 con 0x7fc6cc06f1f0 2024-04-14T23:48:38.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.678+0000 7fc6d2c00640 1 -- 172.21.15.60:0/3322341190 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6cc0c4890 con 0x7fc6cc06f1f0 2024-04-14T23:48:38.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.679+0000 7fc6c8c00640 1 -- 172.21.15.60:0/3322341190 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 4) ==== 90098+0+0 (secure 0 0 0) 0x7fc6bc04a070 con 0x7fc6cc06f1f0 2024-04-14T23:48:38.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.680+0000 7fc6c8c00640 1 --2- 172.21.15.60:0/3322341190 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fc6a806b900 0x7fc6a806ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:38.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.680+0000 7fc6c8c00640 1 -- 172.21.15.60:0/3322341190 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7fc6bc0ac520 con 0x7fc6cc06f1f0 2024-04-14T23:48:38.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.680+0000 7fc6c3e00640 1 --2- 172.21.15.60:0/3322341190 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fc6a806b900 0x7fc6a806ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:38.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.681+0000 7fc6c3e00640 1 --2- 172.21.15.60:0/3322341190 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fc6a806b900 0x7fc6a806ddc0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc071fd0 tx=0x7fc6b4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:38.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.684+0000 7fc6c8c00640 1 -- 172.21.15.60:0/3322341190 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc6cc0c4890 con 0x7fc6cc06f1f0 2024-04-14T23:48:38.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.882+0000 7fc6d2c00640 1 -- 172.21.15.60:0/3322341190 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc6cc076970 con 0x7fc6cc06f1f0 2024-04-14T23:48:38.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.883+0000 7fc6c8c00640 1 -- 172.21.15.60:0/3322341190 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+12897 (secure 0 0 0) 0x7fc6bc033030 con 0x7fc6cc06f1f0 2024-04-14T23:48:38.888 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:48:38.889 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":16,"fsid":"5847db56-a76d-4486-b563-d83f1186af92","created":"2024-04-14T23:48:23.900003+0000","modified":"2024-04-14T23:48:38.137208+0000","last_up_change":"2024-04-14T23:48:34.203573+0000","last_in_change":"2024-04-14T23:48:26.345499+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-14T23:48:36.247372+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b2b2b755-7f42-4f9b-af9a-a178f7cc5a52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6817","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6818","nonce":2536528895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6822","nonce":2536528895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6827","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6828","nonce":2536528895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6825","nonce":2536528895}]},"public_addr":"172.21.15.60:6818/2536528895","cluster_addr":"172.21.15.60:6822/2536528895","heartbeat_back_addr":"172.21.15.60:6828/2536528895","heartbeat_front_addr":"172.21.15.60:6825/2536528895","state":["exists","up"]},{"osd":1,"uuid":"321418cd-5ab7-41a5-b9c8-b3fc12f68ebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6801","nonce":3115695676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6803","nonce":3115695676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6807","nonce":3115695676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6805","nonce":3115695676}]},"public_addr":"172.21.15.60:6801/3115695676","cluster_addr":"172.21.15.60:6803/3115695676","heartbeat_back_addr":"172.21.15.60:6807/3115695676","heartbeat_front_addr":"172.21.15.60:6805/3115695676","state":["exists","up"]},{"osd":2,"uuid":"07874e9f-3a38-41dc-be9f-2b75e84e53d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6819","nonce":3475168698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6824","nonce":3475168698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6831","nonce":3475168698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6829","nonce":3475168698}]},"public_addr":"172.21.15.60:6819/3475168698","cluster_addr":"172.21.15.60:6824/3475168698","heartbeat_back_addr":"172.21.15.60:6831/3475168698","heartbeat_front_addr":"172.21.15.60:6829/3475168698","state":["exists","up"]},{"osd":3,"uuid":"7e4ef6c9-b17a-47c6-a320-f5d16a6fe05e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6809","nonce":1730813674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6811","nonce":1730813674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6815","nonce":1730813674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6813","nonce":1730813674}]},"public_addr":"172.21.15.60:6809/1730813674","cluster_addr":"172.21.15.60:6811/1730813674","heartbeat_back_addr":"172.21.15.60:6815/1730813674","heartbeat_front_addr":"172.21.15.60:6813/1730813674","state":["exists","up"]},{"osd":4,"uuid":"759a3efc-dade-44c5-b57a-1623499aba50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6805","nonce":1464587038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6810","nonce":1464587038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6819","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6822","nonce":1464587038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6813","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6816","nonce":1464587038}]},"public_addr":"172.21.15.186:6805/1464587038","cluster_addr":"172.21.15.186:6810/1464587038","heartbeat_back_addr":"172.21.15.186:6822/1464587038","heartbeat_front_addr":"172.21.15.186:6816/1464587038","state":["exists","up"]},{"osd":5,"uuid":"24cd58c1-0cf1-4077-ab33-84c39c6907ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6801","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6803","nonce":1697098067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6809","nonce":1697098067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6820","nonce":1697098067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6815","nonce":1697098067}]},"public_addr":"172.21.15.186:6803/1697098067","cluster_addr":"172.21.15.186:6809/1697098067","heartbeat_back_addr":"172.21.15.186:6820/1697098067","heartbeat_front_addr":"172.21.15.186:6815/1697098067","state":["exists","up"]},{"osd":6,"uuid":"6add38f1-65e9-4b0d-927a-4ab77f5e2323","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6804","nonce":1837108559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6807","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6811","nonce":1837108559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6823","nonce":1837108559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6817","nonce":1837108559}]},"public_addr":"172.21.15.186:6804/1837108559","cluster_addr":"172.21.15.186:6811/1837108559","heartbeat_back_addr":"172.21.15.186:6823/1837108559","heartbeat_front_addr":"172.21.15.186:6817/1837108559","state":["exists","up"]},{"osd":7,"uuid":"a2326b6f-7601-4dda-a456-67c443ea5929","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6825","nonce":842537875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6827","nonce":842537875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6831","nonce":842537875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6829","nonce":842537875}]},"public_addr":"172.21.15.186:6825/842537875","cluster_addr":"172.21.15.186:6827/842537875","heartbeat_back_addr":"172.21.15.186:6831/842537875","heartbeat_front_addr":"172.21.15.186:6829/842537875","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.975583+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.976806+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.722725+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.763705+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.502140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.528972+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.517961+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.513350+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-14T23:48:38.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.885+0000 7fc6d2c00640 1 -- 172.21.15.60:0/3322341190 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fc6a806b900 msgr2=0x7fc6a806ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:38.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.885+0000 7fc6d2c00640 1 --2- 172.21.15.60:0/3322341190 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fc6a806b900 0x7fc6a806ddc0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc071fd0 tx=0x7fc6b4009040 comp rx=0 tx=0).stop 2024-04-14T23:48:38.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.885+0000 7fc6d2c00640 1 -- 172.21.15.60:0/3322341190 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6cc06f1f0 msgr2=0x7fc6cc1f1300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:38.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.885+0000 7fc6d2c00640 1 --2- 172.21.15.60:0/3322341190 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6cc06f1f0 0x7fc6cc1f1300 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc6bc0090b0 tx=0x7fc6bc004730 comp rx=0 tx=0).stop 2024-04-14T23:48:38.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.886+0000 7fc6d2c00640 1 -- 172.21.15.60:0/3322341190 shutdown_connections 2024-04-14T23:48:38.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.886+0000 7fc6d2c00640 1 --2- 172.21.15.60:0/3322341190 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fc6a806b900 0x7fc6a806ddc0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:38.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.886+0000 7fc6d2c00640 1 --2- 172.21.15.60:0/3322341190 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc6cc0731e0 0x7fc6cc156180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:38.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.886+0000 7fc6d2c00640 1 --2- 172.21.15.60:0/3322341190 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6cc06f5d0 0x7fc6cc157b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:38.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.886+0000 7fc6d2c00640 1 --2- 172.21.15.60:0/3322341190 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6cc06f1f0 0x7fc6cc1f1300 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:38.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.886+0000 7fc6d2c00640 1 -- 172.21.15.60:0/3322341190 >> 172.21.15.60:0/3322341190 conn(0x7fc6cc0cc170 msgr2=0x7fc6cc0c4e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:38.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.886+0000 7fc6d2c00640 1 -- 172.21.15.60:0/3322341190 shutdown_connections 2024-04-14T23:48:38.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:38.886+0000 7fc6d2c00640 1 -- 172.21.15.60:0/3322341190 wait complete. 2024-04-14T23:48:38.908 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-14T23:48:36.247372+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-14T23:48:38.908 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-14T23:48:39.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.082+0000 7fb395000640 1 Processor -- start 2024-04-14T23:48:39.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.083+0000 7fb395000640 1 -- start start 2024-04-14T23:48:39.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.084+0000 7fb395000640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb390152cf0 0x7fb39015c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:39.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.084+0000 7fb395000640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3901530d0 con 0x7fb390159450 2024-04-14T23:48:39.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.084+0000 7fb395000640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3900c4da0 con 0x7fb3900c63c0 2024-04-14T23:48:39.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.084+0000 7fb395000640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fb3900751f0 con 0x7fb390152cf0 2024-04-14T23:48:39.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.084+0000 7fb38de00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb390152cf0 0x7fb39015c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:39.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.084+0000 7fb38de00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb390152cf0 0x7fb39015c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37640/0 (socket says 172.21.15.60:37640) 2024-04-14T23:48:39.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.084+0000 7fb38e800640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fb390159450 0x7fb3900c4690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:40012/0 (socket says 172.21.15.60:40012) 2024-04-14T23:48:39.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.084+0000 7fb38de00640 1 -- 172.21.15.60:0/371037757 learned_addr learned my addr 172.21.15.60:0/371037757 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:39.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.085+0000 7fb38de00640 1 -- 172.21.15.60:0/371037757 >> v1:172.21.15.186:6789/0 conn(0x7fb3900c63c0 legacy=0x7fb3900c3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:39.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.085+0000 7fb38de00640 1 -- 172.21.15.60:0/371037757 >> v1:172.21.15.60:6789/0 conn(0x7fb390159450 legacy=0x7fb3900c4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:39.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.085+0000 7fb38de00640 1 -- 172.21.15.60:0/371037757 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3900c8370 con 0x7fb390152cf0 2024-04-14T23:48:39.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.085+0000 7fb38ca00640 1 -- 172.21.15.60:0/371037757 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3322076319 0 0) 0x7fb3901530d0 con 0x7fb390159450 2024-04-14T23:48:39.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.086+0000 7fb38de00640 1 --2- 172.21.15.60:0/371037757 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb390152cf0 0x7fb39015c2a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb380009870 tx=0x7fb38002bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6c1ac802f450b6c server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:39.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.086+0000 7fb38ca00640 1 -- 172.21.15.60:0/371037757 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb38002d870 con 0x7fb390152cf0 2024-04-14T23:48:39.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.086+0000 7fb38ca00640 1 -- 172.21.15.60:0/371037757 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb38002da10 con 0x7fb390152cf0 2024-04-14T23:48:39.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.086+0000 7fb38ca00640 1 -- 172.21.15.60:0/371037757 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb38002dde0 con 0x7fb390152cf0 2024-04-14T23:48:39.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.087+0000 7fb395000640 1 -- 172.21.15.60:0/371037757 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb390152cf0 msgr2=0x7fb39015c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:39.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.087+0000 7fb395000640 1 --2- 172.21.15.60:0/371037757 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb390152cf0 0x7fb39015c2a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb380009870 tx=0x7fb38002bab0 comp rx=0 tx=0).stop 2024-04-14T23:48:39.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.087+0000 7fb395000640 1 -- 172.21.15.60:0/371037757 shutdown_connections 2024-04-14T23:48:39.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.087+0000 7fb395000640 1 --2- 172.21.15.60:0/371037757 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb390152cf0 0x7fb39015c2a0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:39.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.087+0000 7fb395000640 1 -- 172.21.15.60:0/371037757 >> 172.21.15.60:0/371037757 conn(0x7fb3900cc170 msgr2=0x7fb3900cc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:39.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.087+0000 7fb395000640 1 -- 172.21.15.60:0/371037757 shutdown_connections 2024-04-14T23:48:39.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.087+0000 7fb395000640 1 -- 172.21.15.60:0/371037757 wait complete. 2024-04-14T23:48:39.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.088+0000 7fb395000640 1 Processor -- start 2024-04-14T23:48:39.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.089+0000 7fb395000640 1 -- start start 2024-04-14T23:48:39.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.089+0000 7fb395000640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3900c63c0 0x7fb3900cba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:39.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.089+0000 7fb395000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb390152cf0 0x7fb390145d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:39.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.089+0000 7fb395000640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb390159450 0x7fb3901f1530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:39.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.090+0000 7fb395000640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb3900c8610 con 0x7fb3900c63c0 2024-04-14T23:48:39.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.090+0000 7fb395000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb3900c3af0 con 0x7fb390152cf0 2024-04-14T23:48:39.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.090+0000 7fb395000640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fb3900c88f0 con 0x7fb390159450 2024-04-14T23:48:39.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.090+0000 7fb38de00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3900c63c0 0x7fb3900cba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:39.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.090+0000 7fb38de00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3900c63c0 0x7fb3900cba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37650/0 (socket says 172.21.15.60:37650) 2024-04-14T23:48:39.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.090+0000 7fb38de00640 1 -- 172.21.15.60:0/362714557 learned_addr learned my addr 172.21.15.60:0/362714557 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:39.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.090+0000 7fb38e800640 1 --2- 172.21.15.60:0/362714557 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb390159450 0x7fb3901f1530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:39.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.090+0000 7fb38d400640 1 --2- 172.21.15.60:0/362714557 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb390152cf0 0x7fb390145d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:39.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.090+0000 7fb38de00640 1 -- 172.21.15.60:0/362714557 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb390159450 msgr2=0x7fb3901f1530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:39.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.090+0000 7fb38de00640 1 --2- 172.21.15.60:0/362714557 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb390159450 0x7fb3901f1530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:39.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.090+0000 7fb38de00640 1 -- 172.21.15.60:0/362714557 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb390152cf0 msgr2=0x7fb390145d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:39.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.090+0000 7fb38de00640 1 --2- 172.21.15.60:0/362714557 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb390152cf0 0x7fb390145d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:39.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.090+0000 7fb38de00640 1 -- 172.21.15.60:0/362714557 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3900cb7a0 con 0x7fb3900c63c0 2024-04-14T23:48:39.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.091+0000 7fb38de00640 1 --2- 172.21.15.60:0/362714557 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3900c63c0 0x7fb3900cba60 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb380009990 tx=0x7fb3800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:39.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.091+0000 7fb386000640 1 -- 172.21.15.60:0/362714557 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb380037c40 con 0x7fb3900c63c0 2024-04-14T23:48:39.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.091+0000 7fb386000640 1 -- 172.21.15.60:0/362714557 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb380046070 con 0x7fb3900c63c0 2024-04-14T23:48:39.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.091+0000 7fb386000640 1 -- 172.21.15.60:0/362714557 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb38002d830 con 0x7fb3900c63c0 2024-04-14T23:48:39.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.091+0000 7fb395000640 1 -- 172.21.15.60:0/362714557 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb390159cf0 con 0x7fb3900c63c0 2024-04-14T23:48:39.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.091+0000 7fb395000640 1 -- 172.21.15.60:0/362714557 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb39015a2b0 con 0x7fb3900c63c0 2024-04-14T23:48:39.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.092+0000 7fb395000640 1 -- 172.21.15.60:0/362714557 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb390076970 con 0x7fb3900c63c0 2024-04-14T23:48:39.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.094+0000 7fb386000640 1 -- 172.21.15.60:0/362714557 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) ==== 90098+0+0 (secure 0 0 0) 0x7fb38004a410 con 0x7fb3900c63c0 2024-04-14T23:48:39.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.095+0000 7fb386000640 1 --2- 172.21.15.60:0/362714557 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fb37006b930 0x7fb37006ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:39.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.095+0000 7fb386000640 1 -- 172.21.15.60:0/362714557 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7fb3800ac9a0 con 0x7fb3900c63c0 2024-04-14T23:48:39.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.095+0000 7fb38d400640 1 --2- 172.21.15.60:0/362714557 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fb37006b930 0x7fb37006ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:39.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.096+0000 7fb38d400640 1 --2- 172.21.15.60:0/362714557 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fb37006b930 0x7fb37006ddf0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb39014ede0 tx=0x7fb37c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:39.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.099+0000 7fb386000640 1 -- 172.21.15.60:0/362714557 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb38004f020 con 0x7fb3900c63c0 2024-04-14T23:48:39.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.312+0000 7fb395000640 1 -- 172.21.15.60:0/362714557 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb390156a70 con 0x7fb3900c63c0 2024-04-14T23:48:39.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.312+0000 7fb386000640 1 -- 172.21.15.60:0/362714557 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) ==== 93+0+10 (secure 0 0 0) 0x7fb38003e070 con 0x7fb3900c63c0 2024-04-14T23:48:39.318 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2024-04-14T23:48:39.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.315+0000 7fb395000640 1 -- 172.21.15.60:0/362714557 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fb37006b930 msgr2=0x7fb37006ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:39.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.315+0000 7fb395000640 1 --2- 172.21.15.60:0/362714557 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fb37006b930 0x7fb37006ddf0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb39014ede0 tx=0x7fb37c009040 comp rx=0 tx=0).stop 2024-04-14T23:48:39.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.315+0000 7fb395000640 1 -- 172.21.15.60:0/362714557 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3900c63c0 msgr2=0x7fb3900cba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:39.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.315+0000 7fb395000640 1 --2- 172.21.15.60:0/362714557 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3900c63c0 0x7fb3900cba60 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb380009990 tx=0x7fb3800046c0 comp rx=0 tx=0).stop 2024-04-14T23:48:39.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.315+0000 7fb395000640 1 -- 172.21.15.60:0/362714557 shutdown_connections 2024-04-14T23:48:39.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.315+0000 7fb395000640 1 --2- 172.21.15.60:0/362714557 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fb37006b930 0x7fb37006ddf0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:39.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.315+0000 7fb395000640 1 --2- 172.21.15.60:0/362714557 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb390159450 0x7fb3901f1530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:39.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.315+0000 7fb395000640 1 --2- 172.21.15.60:0/362714557 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb390152cf0 0x7fb390145d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:39.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.315+0000 7fb395000640 1 --2- 172.21.15.60:0/362714557 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3900c63c0 0x7fb3900cba60 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:39.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.315+0000 7fb395000640 1 -- 172.21.15.60:0/362714557 >> 172.21.15.60:0/362714557 conn(0x7fb3900cc170 msgr2=0x7fb39006edb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:39.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.315+0000 7fb395000640 1 -- 172.21.15.60:0/362714557 shutdown_connections 2024-04-14T23:48:39.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.315+0000 7fb395000640 1 -- 172.21.15.60:0/362714557 wait complete. 2024-04-14T23:48:39.338 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-14T23:48:39.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.536+0000 7f5e3ea00640 1 Processor -- start 2024-04-14T23:48:39.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.536+0000 7f5e3ea00640 1 -- start start 2024-04-14T23:48:39.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.537+0000 7f5e3ea00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5e380c5380 0x7f5e3815a980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:39.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.537+0000 7f5e3ea00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e3806b4b0 con 0x7f5e380c4fa0 2024-04-14T23:48:39.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.537+0000 7f5e3ea00640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e3806b790 con 0x7f5e38070d50 2024-04-14T23:48:39.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.537+0000 7f5e3ea00640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f5e380c69f0 con 0x7f5e380c5380 2024-04-14T23:48:39.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.538+0000 7f5e3e000640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5e380c5380 0x7f5e3815a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:39.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.538+0000 7f5e3d600640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f5e380c4fa0 0x7f5e380c77b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:40016/0 (socket says 172.21.15.60:40016) 2024-04-14T23:48:39.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.538+0000 7f5e3cc00640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f5e38070d50 0x7f5e3815a270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.60:39544/0 (socket says 172.21.15.60:39544) 2024-04-14T23:48:39.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.538+0000 7f5e3d600640 1 -- 172.21.15.60:0/476327341 learned_addr learned my addr 172.21.15.60:0/476327341 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:48:39.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.538+0000 7f5e37e00640 1 -- 172.21.15.60:0/476327341 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3572003081 0 0) 0x7f5e3806b790 con 0x7f5e38070d50 2024-04-14T23:48:39.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.538+0000 7f5e37e00640 1 -- 172.21.15.60:0/476327341 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5e20005730 con 0x7f5e38070d50 2024-04-14T23:48:39.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.538+0000 7f5e3e000640 1 -- 172.21.15.60:0/476327341 >> v1:172.21.15.186:6789/0 conn(0x7f5e38070d50 legacy=0x7f5e3815a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:39.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.538+0000 7f5e3e000640 1 -- 172.21.15.60:0/476327341 >> v1:172.21.15.60:6789/0 conn(0x7f5e380c4fa0 legacy=0x7f5e380c77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:39.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.538+0000 7f5e37e00640 1 -- 172.21.15.60:0/476327341 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2053399994 0 0) 0x7f5e3806b4b0 con 0x7f5e380c4fa0 2024-04-14T23:48:39.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.538+0000 7f5e3e000640 1 -- 172.21.15.60:0/476327341 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e380c6cd0 con 0x7f5e380c5380 2024-04-14T23:48:39.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.539+0000 7f5e3e000640 1 --2- 172.21.15.60:0/476327341 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e380c5380 0x7f5e3815a980 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5e300097a0 tx=0x7f5e3002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0ea6d13f9973462 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:39.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.540+0000 7f5e37e00640 1 -- 172.21.15.60:0/476327341 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e30034070 con 0x7f5e380c5380 2024-04-14T23:48:39.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.540+0000 7f5e37e00640 1 -- 172.21.15.60:0/476327341 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e30030610 con 0x7f5e380c5380 2024-04-14T23:48:39.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.540+0000 7f5e37e00640 1 -- 172.21.15.60:0/476327341 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e300309e0 con 0x7f5e380c5380 2024-04-14T23:48:39.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.540+0000 7f5e3ea00640 1 -- 172.21.15.60:0/476327341 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e380c5380 msgr2=0x7f5e3815a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:39.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.540+0000 7f5e3ea00640 1 --2- 172.21.15.60:0/476327341 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e380c5380 0x7f5e3815a980 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5e300097a0 tx=0x7f5e3002b960 comp rx=0 tx=0).stop 2024-04-14T23:48:39.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.540+0000 7f5e3ea00640 1 -- 172.21.15.60:0/476327341 shutdown_connections 2024-04-14T23:48:39.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.540+0000 7f5e3ea00640 1 --2- 172.21.15.60:0/476327341 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e380c5380 0x7f5e3815a980 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:39.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.540+0000 7f5e3ea00640 1 -- 172.21.15.60:0/476327341 >> 172.21.15.60:0/476327341 conn(0x7f5e380c8b50 msgr2=0x7f5e380c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:39.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.540+0000 7f5e3ea00640 1 -- 172.21.15.60:0/476327341 shutdown_connections 2024-04-14T23:48:39.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.540+0000 7f5e3ea00640 1 -- 172.21.15.60:0/476327341 wait complete. 2024-04-14T23:48:39.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.542+0000 7f5e3ea00640 1 Processor -- start 2024-04-14T23:48:39.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.542+0000 7f5e3ea00640 1 -- start start 2024-04-14T23:48:39.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.543+0000 7f5e3ea00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5e38070d50 0x7f5e380ca160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:39.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.543+0000 7f5e3ea00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5e380c4fa0 0x7f5e380c1500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:39.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.543+0000 7f5e3ea00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e380c5380 0x7f5e380c1a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:39.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.543+0000 7f5e3ea00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f5e3806b1c0 con 0x7f5e380c5380 2024-04-14T23:48:39.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.543+0000 7f5e3ea00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5e380c7290 con 0x7f5e380c4fa0 2024-04-14T23:48:39.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.543+0000 7f5e3ea00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f5e380c99d0 con 0x7f5e38070d50 2024-04-14T23:48:39.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.543+0000 7f5e3cc00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5e380c4fa0 0x7f5e380c1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:39.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.543+0000 7f5e3d600640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5e38070d50 0x7f5e380ca160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:39.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.543+0000 7f5e3cc00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5e380c4fa0 0x7f5e380c1500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:46276/0 (socket says 172.21.15.60:46276) 2024-04-14T23:48:39.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.543+0000 7f5e3cc00640 1 -- 172.21.15.60:0/1942652580 learned_addr learned my addr 172.21.15.60:0/1942652580 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:39.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.543+0000 7f5e3e000640 1 --2- 172.21.15.60:0/1942652580 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e380c5380 0x7f5e380c1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:39.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.544+0000 7f5e3cc00640 1 -- 172.21.15.60:0/1942652580 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5e38070d50 msgr2=0x7f5e380ca160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:39.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.544+0000 7f5e3cc00640 1 --2- 172.21.15.60:0/1942652580 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5e38070d50 0x7f5e380ca160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:39.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.544+0000 7f5e3cc00640 1 -- 172.21.15.60:0/1942652580 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e380c5380 msgr2=0x7f5e380c1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:39.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.544+0000 7f5e3cc00640 1 --2- 172.21.15.60:0/1942652580 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e380c5380 0x7f5e380c1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:39.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.544+0000 7f5e3cc00640 1 -- 172.21.15.60:0/1942652580 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e380c0290 con 0x7f5e380c4fa0 2024-04-14T23:48:39.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.544+0000 7f5e3cc00640 1 --2- 172.21.15.60:0/1942652580 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5e380c4fa0 0x7f5e380c1500 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c00c700 tx=0x7f5e2c00cbc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:39.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.544+0000 7f5e35600640 1 -- 172.21.15.60:0/1942652580 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e2c017070 con 0x7f5e380c4fa0 2024-04-14T23:48:39.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.544+0000 7f5e35600640 1 -- 172.21.15.60:0/1942652580 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e2c00ce10 con 0x7f5e380c4fa0 2024-04-14T23:48:39.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.544+0000 7f5e35600640 1 -- 172.21.15.60:0/1942652580 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e2c00a800 con 0x7f5e380c4fa0 2024-04-14T23:48:39.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.544+0000 7f5e3ea00640 1 -- 172.21.15.60:0/1942652580 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e380c2c50 con 0x7f5e380c4fa0 2024-04-14T23:48:39.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.544+0000 7f5e3ea00640 1 -- 172.21.15.60:0/1942652580 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e380c0fe0 con 0x7f5e380c4fa0 2024-04-14T23:48:39.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.546+0000 7f5e3ea00640 1 -- 172.21.15.60:0/1942652580 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e3804f540 con 0x7f5e380c4fa0 2024-04-14T23:48:39.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.546+0000 7f5e35600640 1 -- 172.21.15.60:0/1942652580 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 4) ==== 90098+0+0 (secure 0 0 0) 0x7f5e2c01a070 con 0x7f5e380c4fa0 2024-04-14T23:48:39.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.548+0000 7f5e35600640 1 --2- 172.21.15.60:0/1942652580 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f5e1c06b930 0x7f5e1c06ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:39.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.548+0000 7f5e35600640 1 -- 172.21.15.60:0/1942652580 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f5e2c0899b0 con 0x7f5e380c4fa0 2024-04-14T23:48:39.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.548+0000 7f5e3d600640 1 --2- 172.21.15.60:0/1942652580 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f5e1c06b930 0x7f5e1c06ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:39.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.548+0000 7f5e3d600640 1 --2- 172.21.15.60:0/1942652580 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f5e1c06b930 0x7f5e1c06ddf0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5e28000f60 tx=0x7f5e28007d50 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:39.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.552+0000 7f5e35600640 1 -- 172.21.15.60:0/1942652580 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f5e2c058270 con 0x7f5e380c4fa0 2024-04-14T23:48:39.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.754+0000 7f5e3ea00640 1 -- 172.21.15.60:0/1942652580 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f5e380b1c20 con 0x7f5e380c4fa0 2024-04-14T23:48:39.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.755+0000 7f5e35600640 1 -- 172.21.15.60:0/1942652580 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v17) ==== 84+0+65 (secure 0 0 0) 0x7f5e2c05c310 con 0x7f5e380c4fa0 2024-04-14T23:48:39.760 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:48:39.760 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2024-04-14T23:48:39.760 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-14T23:48:39.760 INFO:teuthology.orchestra.run.smithi060.stdout: "poolnum": 1, 2024-04-14T23:48:39.760 INFO:teuthology.orchestra.run.smithi060.stdout: "poolname": ".mgr" 2024-04-14T23:48:39.761 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-14T23:48:39.761 INFO:teuthology.orchestra.run.smithi060.stdout:] 2024-04-14T23:48:39.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.757+0000 7f5e3ea00640 1 -- 172.21.15.60:0/1942652580 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f5e1c06b930 msgr2=0x7f5e1c06ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:39.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.757+0000 7f5e3ea00640 1 --2- 172.21.15.60:0/1942652580 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f5e1c06b930 0x7f5e1c06ddf0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5e28000f60 tx=0x7f5e28007d50 comp rx=0 tx=0).stop 2024-04-14T23:48:39.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.757+0000 7f5e3ea00640 1 -- 172.21.15.60:0/1942652580 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5e380c4fa0 msgr2=0x7f5e380c1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:39.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.757+0000 7f5e3ea00640 1 --2- 172.21.15.60:0/1942652580 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5e380c4fa0 0x7f5e380c1500 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c00c700 tx=0x7f5e2c00cbc0 comp rx=0 tx=0).stop 2024-04-14T23:48:39.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.757+0000 7f5e3ea00640 1 -- 172.21.15.60:0/1942652580 shutdown_connections 2024-04-14T23:48:39.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.757+0000 7f5e3ea00640 1 --2- 172.21.15.60:0/1942652580 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f5e1c06b930 0x7f5e1c06ddf0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:39.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.757+0000 7f5e3ea00640 1 --2- 172.21.15.60:0/1942652580 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e380c5380 0x7f5e380c1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:39.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.757+0000 7f5e3ea00640 1 --2- 172.21.15.60:0/1942652580 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5e380c4fa0 0x7f5e380c1500 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:39.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.757+0000 7f5e3ea00640 1 --2- 172.21.15.60:0/1942652580 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5e38070d50 0x7f5e380ca160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:39.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.757+0000 7f5e3ea00640 1 -- 172.21.15.60:0/1942652580 >> 172.21.15.60:0/1942652580 conn(0x7f5e380c8b50 msgr2=0x7f5e3815a040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:39.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.757+0000 7f5e3ea00640 1 -- 172.21.15.60:0/1942652580 shutdown_connections 2024-04-14T23:48:39.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.757+0000 7f5e3ea00640 1 -- 172.21.15.60:0/1942652580 wait complete. 2024-04-14T23:48:39.783 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-14T23:48:39.783 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-14T23:48:39.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.986+0000 7f3b68e00640 1 Processor -- start 2024-04-14T23:48:39.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.986+0000 7f3b68e00640 1 -- start start 2024-04-14T23:48:39.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.987+0000 7f3b68e00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3b64154e30 0x7f3b6415ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:39.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.987+0000 7f3b68e00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f3b6415b0c0 0x7f3b640c7ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:39.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.988+0000 7f3b68e00640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b640c9270 con 0x7f3b64070fc0 2024-04-14T23:48:39.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.988+0000 7f3b68e00640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f3b640c9550 con 0x7f3b64154e30 2024-04-14T23:48:39.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.988+0000 7f3b68e00640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f3b6415e210 con 0x7f3b6415b0c0 2024-04-14T23:48:39.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.988+0000 7f3b5be00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f3b6415b0c0 0x7f3b640c7ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:39.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.988+0000 7f3b62800640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f3b64070fc0 0x7f3b640c86e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.60:39554/0 (socket says 172.21.15.60:39554) 2024-04-14T23:48:39.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.988+0000 7f3b62800640 1 -- 172.21.15.60:0/3612867560 learned_addr learned my addr 172.21.15.60:0/3612867560 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:48:39.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.988+0000 7f3b61e00640 1 --2- 172.21.15.60:0/3612867560 >> v2:172.21.15.60:3300/0 conn(0x7f3b64154e30 0x7f3b6415ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:39.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.988+0000 7f3b5be00640 1 -- 172.21.15.60:0/3612867560 >> v2:172.21.15.60:3300/0 conn(0x7f3b64154e30 msgr2=0x7f3b6415ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:39.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.988+0000 7f3b5be00640 1 --2- 172.21.15.60:0/3612867560 >> v2:172.21.15.60:3300/0 conn(0x7f3b64154e30 0x7f3b6415ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:39.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.988+0000 7f3b5be00640 1 -- 172.21.15.60:0/3612867560 >> v1:172.21.15.186:6789/0 conn(0x7f3b64070fc0 legacy=0x7f3b640c86e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:39.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.989+0000 7f3b5be00640 1 -- 172.21.15.60:0/3612867560 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b6415e4f0 con 0x7f3b6415b0c0 2024-04-14T23:48:39.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.989+0000 7f3b61400640 1 -- 172.21.15.60:0/3612867560 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4056550016 0 0) 0x7f3b640c9270 con 0x7f3b64070fc0 2024-04-14T23:48:39.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.990+0000 7f3b5be00640 1 --2- 172.21.15.60:0/3612867560 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b6415b0c0 0x7f3b640c7ba0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3b640c6d90 tx=0x7f3b4c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25e9670765310735 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:39.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.990+0000 7f3b61400640 1 -- 172.21.15.60:0/3612867560 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b4c035020 con 0x7f3b6415b0c0 2024-04-14T23:48:39.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.990+0000 7f3b61400640 1 -- 172.21.15.60:0/3612867560 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b4c031830 con 0x7f3b6415b0c0 2024-04-14T23:48:39.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.990+0000 7f3b61400640 1 -- 172.21.15.60:0/3612867560 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b4c031c00 con 0x7f3b6415b0c0 2024-04-14T23:48:39.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.991+0000 7f3b68e00640 1 -- 172.21.15.60:0/3612867560 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b6415b0c0 msgr2=0x7f3b640c7ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:39.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.991+0000 7f3b68e00640 1 --2- 172.21.15.60:0/3612867560 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b6415b0c0 0x7f3b640c7ba0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3b640c6d90 tx=0x7f3b4c02b8d0 comp rx=0 tx=0).stop 2024-04-14T23:48:39.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.991+0000 7f3b68e00640 1 -- 172.21.15.60:0/3612867560 shutdown_connections 2024-04-14T23:48:39.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.991+0000 7f3b68e00640 1 --2- 172.21.15.60:0/3612867560 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b6415b0c0 0x7f3b640c7ba0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:39.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.991+0000 7f3b68e00640 1 --2- 172.21.15.60:0/3612867560 >> v2:172.21.15.60:3300/0 conn(0x7f3b64154e30 0x7f3b6415ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:39.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.991+0000 7f3b68e00640 1 -- 172.21.15.60:0/3612867560 >> 172.21.15.60:0/3612867560 conn(0x7f3b64067540 msgr2=0x7f3b6406b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:39.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.991+0000 7f3b68e00640 1 -- 172.21.15.60:0/3612867560 shutdown_connections 2024-04-14T23:48:39.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.991+0000 7f3b68e00640 1 -- 172.21.15.60:0/3612867560 wait complete. 2024-04-14T23:48:39.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.993+0000 7f3b68e00640 1 Processor -- start 2024-04-14T23:48:39.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.993+0000 7f3b68e00640 1 -- start start 2024-04-14T23:48:40.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.994+0000 7f3b68e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3b64070fc0 0x7f3b641f0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:40.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.994+0000 7f3b68e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3b64154e30 0x7f3b64152c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:40.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.994+0000 7f3b68e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b6415b0c0 0x7f3b6414f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:40.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.994+0000 7f3b68e00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f3b64155210 con 0x7f3b64154e30 2024-04-14T23:48:40.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.994+0000 7f3b68e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3b6415eab0 con 0x7f3b6415b0c0 2024-04-14T23:48:40.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.994+0000 7f3b68e00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f3b6415efd0 con 0x7f3b64070fc0 2024-04-14T23:48:40.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.994+0000 7f3b5be00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3b64154e30 0x7f3b64152c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:40.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.994+0000 7f3b5be00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3b64154e30 0x7f3b64152c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37682/0 (socket says 172.21.15.60:37682) 2024-04-14T23:48:40.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.994+0000 7f3b5be00640 1 -- 172.21.15.60:0/3553053953 learned_addr learned my addr 172.21.15.60:0/3553053953 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:40.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.994+0000 7f3b61e00640 1 --2- 172.21.15.60:0/3553053953 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3b64070fc0 0x7f3b641f0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:40.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.994+0000 7f3b62800640 1 --2- 172.21.15.60:0/3553053953 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b6415b0c0 0x7f3b6414f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:40.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.994+0000 7f3b62800640 1 -- 172.21.15.60:0/3553053953 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3b64070fc0 msgr2=0x7f3b641f0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:40.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.994+0000 7f3b62800640 1 --2- 172.21.15.60:0/3553053953 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3b64070fc0 0x7f3b641f0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:40.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.995+0000 7f3b62800640 1 -- 172.21.15.60:0/3553053953 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3b64154e30 msgr2=0x7f3b64152c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:40.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.995+0000 7f3b62800640 1 --2- 172.21.15.60:0/3553053953 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3b64154e30 0x7f3b64152c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:40.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.995+0000 7f3b62800640 1 -- 172.21.15.60:0/3553053953 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b64153290 con 0x7f3b6415b0c0 2024-04-14T23:48:40.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.995+0000 7f3b62800640 1 --2- 172.21.15.60:0/3553053953 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b6415b0c0 0x7f3b6414f390 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c00c6d0 tx=0x7f3b5c00cb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:40.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.995+0000 7f3b5a000640 1 -- 172.21.15.60:0/3553053953 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b5c018070 con 0x7f3b6415b0c0 2024-04-14T23:48:40.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.995+0000 7f3b5a000640 1 -- 172.21.15.60:0/3553053953 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b5c004d60 con 0x7f3b6415b0c0 2024-04-14T23:48:40.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.995+0000 7f3b5a000640 1 -- 172.21.15.60:0/3553053953 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b5c00a880 con 0x7f3b6415b0c0 2024-04-14T23:48:40.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.995+0000 7f3b68e00640 1 -- 172.21.15.60:0/3553053953 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b64153570 con 0x7f3b6415b0c0 2024-04-14T23:48:40.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.995+0000 7f3b68e00640 1 -- 172.21.15.60:0/3553053953 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b6414eb90 con 0x7f3b6415b0c0 2024-04-14T23:48:40.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.996+0000 7f3b68e00640 1 -- 172.21.15.60:0/3553053953 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b640729a0 con 0x7f3b6415b0c0 2024-04-14T23:48:40.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.997+0000 7f3b5a000640 1 -- 172.21.15.60:0/3553053953 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 4) ==== 90098+0+0 (secure 0 0 0) 0x7f3b5c00ac30 con 0x7f3b6415b0c0 2024-04-14T23:48:40.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.998+0000 7f3b5a000640 1 --2- 172.21.15.60:0/3553053953 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f3b4006b930 0x7f3b4006ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:40.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.998+0000 7f3b5a000640 1 -- 172.21.15.60:0/3553053953 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f3b5c089ad0 con 0x7f3b6415b0c0 2024-04-14T23:48:40.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.998+0000 7f3b61e00640 1 --2- 172.21.15.60:0/3553053953 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f3b4006b930 0x7f3b4006ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:40.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:39.999+0000 7f3b61e00640 1 --2- 172.21.15.60:0/3553053953 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f3b4006b930 0x7f3b4006ddf0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3b5400aac0 tx=0x7f3b54008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:40.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:40.002+0000 7f3b5a000640 1 -- 172.21.15.60:0/3553053953 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3b640729a0 con 0x7f3b6415b0c0 2024-04-14T23:48:40.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:40.219+0000 7f3b68e00640 1 -- 172.21.15.60:0/3553053953 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f3b641e6d90 con 0x7f3b6415b0c0 2024-04-14T23:48:41.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.202+0000 7f3b5a000640 1 -- 172.21.15.60:0/3553053953 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f3b5c054a90 con 0x7f3b6415b0c0 2024-04-14T23:48:41.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.203+0000 7f3b5a000640 1 -- 172.21.15.60:0/3553053953 <== mon.1 v2:172.21.15.186:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v18) ==== 138+0+0 (secure 0 0 0) 0x7f3b5c08be20 con 0x7f3b6415b0c0 2024-04-14T23:48:41.211 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' created 2024-04-14T23:48:41.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.209+0000 7f3b68e00640 1 -- 172.21.15.60:0/3553053953 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f3b4006b930 msgr2=0x7f3b4006ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:41.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.209+0000 7f3b68e00640 1 --2- 172.21.15.60:0/3553053953 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f3b4006b930 0x7f3b4006ddf0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3b5400aac0 tx=0x7f3b54008040 comp rx=0 tx=0).stop 2024-04-14T23:48:41.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.209+0000 7f3b68e00640 1 -- 172.21.15.60:0/3553053953 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b6415b0c0 msgr2=0x7f3b6414f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:41.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.209+0000 7f3b68e00640 1 --2- 172.21.15.60:0/3553053953 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b6415b0c0 0x7f3b6414f390 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c00c6d0 tx=0x7f3b5c00cb90 comp rx=0 tx=0).stop 2024-04-14T23:48:41.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.212+0000 7f3b68e00640 1 -- 172.21.15.60:0/3553053953 shutdown_connections 2024-04-14T23:48:41.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.212+0000 7f3b68e00640 1 --2- 172.21.15.60:0/3553053953 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f3b4006b930 0x7f3b4006ddf0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:41.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.212+0000 7f3b68e00640 1 --2- 172.21.15.60:0/3553053953 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b6415b0c0 0x7f3b6414f390 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:41.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.212+0000 7f3b68e00640 1 --2- 172.21.15.60:0/3553053953 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3b64154e30 0x7f3b64152c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:41.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.212+0000 7f3b68e00640 1 --2- 172.21.15.60:0/3553053953 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3b64070fc0 0x7f3b641f0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:41.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.212+0000 7f3b68e00640 1 -- 172.21.15.60:0/3553053953 >> 172.21.15.60:0/3553053953 conn(0x7f3b64067540 msgr2=0x7f3b6406b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:41.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.212+0000 7f3b68e00640 1 -- 172.21.15.60:0/3553053953 shutdown_connections 2024-04-14T23:48:41.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.212+0000 7f3b68e00640 1 -- 172.21.15.60:0/3553053953 wait complete. 2024-04-14T23:48:41.241 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-14T23:48:41.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.443+0000 7f28d7800640 1 Processor -- start 2024-04-14T23:48:41.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.444+0000 7f28d7800640 1 -- start start 2024-04-14T23:48:41.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.445+0000 7f28d7800640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f28d00ceb70 0x7f28d00c3e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:41.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.445+0000 7f28d7800640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28d015afd0 con 0x7f28d00705d0 2024-04-14T23:48:41.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.445+0000 7f28d7800640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28d015b270 con 0x7f28d00cef50 2024-04-14T23:48:41.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.445+0000 7f28d7800640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f28d00c93f0 con 0x7f28d00ceb70 2024-04-14T23:48:41.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.446+0000 7f28d4e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f28d00ceb70 0x7f28d00c3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:41.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.446+0000 7f28cfe00640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f28d00705d0 0x7f28d0157e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:40032/0 (socket says 172.21.15.60:40032) 2024-04-14T23:48:41.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.446+0000 7f28cfe00640 1 -- 172.21.15.60:0/444505239 learned_addr learned my addr 172.21.15.60:0/444505239 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:48:41.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.446+0000 7f28d4e00640 1 -- 172.21.15.60:0/444505239 >> v1:172.21.15.186:6789/0 conn(0x7f28d00cef50 legacy=0x7f28d0165660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:41.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.446+0000 7f28d4e00640 1 -- 172.21.15.60:0/444505239 >> v1:172.21.15.60:6789/0 conn(0x7f28d00705d0 legacy=0x7f28d0157e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:41.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.446+0000 7f28cf400640 1 -- 172.21.15.60:0/444505239 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3613322060 0 0) 0x7f28d015b270 con 0x7f28d00cef50 2024-04-14T23:48:41.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.446+0000 7f28d4e00640 1 -- 172.21.15.60:0/444505239 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28d015b550 con 0x7f28d00ceb70 2024-04-14T23:48:41.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.447+0000 7f28d4e00640 1 --2- 172.21.15.60:0/444505239 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f28d00ceb70 0x7f28d00c3e00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f28c40097d0 tx=0x7f28c402b9d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dae0d7e7d37d9211 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:41.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.448+0000 7f28cf400640 1 -- 172.21.15.60:0/444505239 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c4034070 con 0x7f28d00ceb70 2024-04-14T23:48:41.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.448+0000 7f28cf400640 1 -- 172.21.15.60:0/444505239 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28c4030890 con 0x7f28d00ceb70 2024-04-14T23:48:41.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.448+0000 7f28cf400640 1 -- 172.21.15.60:0/444505239 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c4030c60 con 0x7f28d00ceb70 2024-04-14T23:48:41.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.448+0000 7f28d7800640 1 -- 172.21.15.60:0/444505239 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f28d00ceb70 msgr2=0x7f28d00c3e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:41.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.448+0000 7f28d7800640 1 --2- 172.21.15.60:0/444505239 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f28d00ceb70 0x7f28d00c3e00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f28c40097d0 tx=0x7f28c402b9d0 comp rx=0 tx=0).stop 2024-04-14T23:48:41.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.449+0000 7f28d7800640 1 -- 172.21.15.60:0/444505239 shutdown_connections 2024-04-14T23:48:41.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.449+0000 7f28d7800640 1 --2- 172.21.15.60:0/444505239 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f28d00ceb70 0x7f28d00c3e00 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:41.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.449+0000 7f28d7800640 1 -- 172.21.15.60:0/444505239 >> 172.21.15.60:0/444505239 conn(0x7f28d0067360 msgr2=0x7f28d0067770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:41.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.449+0000 7f28d7800640 1 -- 172.21.15.60:0/444505239 shutdown_connections 2024-04-14T23:48:41.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.449+0000 7f28d7800640 1 -- 172.21.15.60:0/444505239 wait complete. 2024-04-14T23:48:41.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.450+0000 7f28d7800640 1 Processor -- start 2024-04-14T23:48:41.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.451+0000 7f28d7800640 1 -- start start 2024-04-14T23:48:41.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.451+0000 7f28d7800640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f28d00705d0 0x7f28d01533f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:41.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.451+0000 7f28d7800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28d00ceb70 0x7f28d00ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:41.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.451+0000 7f28d7800640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28d00cef50 0x7f28d00ca720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:41.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.451+0000 7f28d7800640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f28d006d720 con 0x7f28d00cef50 2024-04-14T23:48:41.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.451+0000 7f28d7800640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f28d015bb10 con 0x7f28d00ceb70 2024-04-14T23:48:41.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.451+0000 7f28d7800640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f28d006c2d0 con 0x7f28d00705d0 2024-04-14T23:48:41.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.452+0000 7f28d5800640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28d00cef50 0x7f28d00ca720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:41.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.452+0000 7f28cfe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28d00ceb70 0x7f28d00ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:41.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.452+0000 7f28d4e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f28d00705d0 0x7f28d01533f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:41.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.452+0000 7f28d5800640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28d00cef50 0x7f28d00ca720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37696/0 (socket says 172.21.15.60:37696) 2024-04-14T23:48:41.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.452+0000 7f28d5800640 1 -- 172.21.15.60:0/2142964090 learned_addr learned my addr 172.21.15.60:0/2142964090 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:41.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.452+0000 7f28d4e00640 1 -- 172.21.15.60:0/2142964090 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28d00ceb70 msgr2=0x7f28d00ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:41.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.452+0000 7f28d4e00640 1 --2- 172.21.15.60:0/2142964090 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28d00ceb70 0x7f28d00ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:41.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.452+0000 7f28d4e00640 1 -- 172.21.15.60:0/2142964090 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28d00cef50 msgr2=0x7f28d00ca720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:41.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.452+0000 7f28d4e00640 1 --2- 172.21.15.60:0/2142964090 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28d00cef50 0x7f28d00ca720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:41.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.452+0000 7f28d4e00640 1 -- 172.21.15.60:0/2142964090 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28d01f16f0 con 0x7f28d00705d0 2024-04-14T23:48:41.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.453+0000 7f28d4e00640 1 --2- 172.21.15.60:0/2142964090 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f28d00705d0 0x7f28d01533f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f28d015b1a0 tx=0x7f28c40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:41.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.453+0000 7f28ccc00640 1 -- 172.21.15.60:0/2142964090 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c4034040 con 0x7f28d00705d0 2024-04-14T23:48:41.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.453+0000 7f28ccc00640 1 -- 172.21.15.60:0/2142964090 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28c4039d10 con 0x7f28d00705d0 2024-04-14T23:48:41.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.453+0000 7f28ccc00640 1 -- 172.21.15.60:0/2142964090 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c4042630 con 0x7f28d00705d0 2024-04-14T23:48:41.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.453+0000 7f28d7800640 1 -- 172.21.15.60:0/2142964090 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28d00c5970 con 0x7f28d00705d0 2024-04-14T23:48:41.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.453+0000 7f28d7800640 1 -- 172.21.15.60:0/2142964090 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f28d00c5c10 con 0x7f28d00705d0 2024-04-14T23:48:41.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.454+0000 7f28d7800640 1 -- 172.21.15.60:0/2142964090 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28d0156060 con 0x7f28d00705d0 2024-04-14T23:48:41.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.455+0000 7f28ccc00640 1 -- 172.21.15.60:0/2142964090 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f28c4037070 con 0x7f28d00705d0 2024-04-14T23:48:41.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.456+0000 7f28ccc00640 1 --2- 172.21.15.60:0/2142964090 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f28b006b990 0x7f28b006de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:41.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.456+0000 7f28ccc00640 1 -- 172.21.15.60:0/2142964090 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 5227+0+0 (secure 0 0 0) 0x7f28c40aca70 con 0x7f28d00705d0 2024-04-14T23:48:41.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.456+0000 7f28cfe00640 1 --2- 172.21.15.60:0/2142964090 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f28b006b990 0x7f28b006de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:41.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.457+0000 7f28cfe00640 1 --2- 172.21.15.60:0/2142964090 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f28b006b990 0x7f28b006de50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f28d0157180 tx=0x7f28bc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:41.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.461+0000 7f28ccc00640 1 -- 172.21.15.60:0/2142964090 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f28d0156060 con 0x7f28d00705d0 2024-04-14T23:48:41.668 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:41.663+0000 7f28d7800640 1 -- 172.21.15.60:0/2142964090 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f28d0076970 con 0x7f28d00705d0 2024-04-14T23:48:42.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.204+0000 7f28ccc00640 1 -- 172.21.15.60:0/2142964090 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v19) ==== 170+0+0 (secure 0 0 0) 0x7f28c407b200 con 0x7f28d00705d0 2024-04-14T23:48:42.210 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' created 2024-04-14T23:48:42.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.212+0000 7f28d7800640 1 -- 172.21.15.60:0/2142964090 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f28b006b990 msgr2=0x7f28b006de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:42.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.212+0000 7f28d7800640 1 --2- 172.21.15.60:0/2142964090 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f28b006b990 0x7f28b006de50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f28d0157180 tx=0x7f28bc009040 comp rx=0 tx=0).stop 2024-04-14T23:48:42.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.212+0000 7f28d7800640 1 -- 172.21.15.60:0/2142964090 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f28d00705d0 msgr2=0x7f28d01533f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:42.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.212+0000 7f28d7800640 1 --2- 172.21.15.60:0/2142964090 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f28d00705d0 0x7f28d01533f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f28d015b1a0 tx=0x7f28c40046c0 comp rx=0 tx=0).stop 2024-04-14T23:48:42.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.213+0000 7f28d7800640 1 -- 172.21.15.60:0/2142964090 shutdown_connections 2024-04-14T23:48:42.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.213+0000 7f28d7800640 1 --2- 172.21.15.60:0/2142964090 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f28b006b990 0x7f28b006de50 secure :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f28d0157180 tx=0x7f28bc009040 comp rx=0 tx=0).stop 2024-04-14T23:48:42.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.213+0000 7f28d7800640 1 --2- 172.21.15.60:0/2142964090 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28d00cef50 0x7f28d00ca720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:42.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.213+0000 7f28d7800640 1 --2- 172.21.15.60:0/2142964090 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28d00ceb70 0x7f28d00ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:42.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.213+0000 7f28d7800640 1 --2- 172.21.15.60:0/2142964090 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f28d00705d0 0x7f28d01533f0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:42.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.213+0000 7f28d7800640 1 -- 172.21.15.60:0/2142964090 >> 172.21.15.60:0/2142964090 conn(0x7f28d0067360 msgr2=0x7f28d0158c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:42.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.214+0000 7f28d7800640 1 -- 172.21.15.60:0/2142964090 shutdown_connections 2024-04-14T23:48:42.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.215+0000 7f28d7800640 1 -- 172.21.15.60:0/2142964090 wait complete. 2024-04-14T23:48:42.255 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-14T23:48:42.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.436+0000 7f260fe00640 1 Processor -- start 2024-04-14T23:48:42.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.436+0000 7f260fe00640 1 -- start start 2024-04-14T23:48:42.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.438+0000 7f260fe00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f261006c7d0 0x7f261015dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:42.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.438+0000 7f260fe00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26100cb370 con 0x7f2610154bc0 2024-04-14T23:48:42.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.438+0000 7f260fe00640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26100cb650 con 0x7f2610154fa0 2024-04-14T23:48:42.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.438+0000 7f260fe00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f261006ce20 con 0x7f261006c7d0 2024-04-14T23:48:42.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.438+0000 7f260e000640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f2610154fa0 0x7f26100c6610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.60:56740/0 (socket says 172.21.15.60:56740) 2024-04-14T23:48:42.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.438+0000 7f260e000640 1 -- 172.21.15.60:0/2476900169 learned_addr learned my addr 172.21.15.60:0/2476900169 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:48:42.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.438+0000 7f260ea00640 1 --1- 172.21.15.60:0/2476900169 >> v1:172.21.15.60:6789/0 conn(0x7f2610154bc0 0x7f26100c5590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:43620/0 (socket says 172.21.15.60:43620) 2024-04-14T23:48:42.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.438+0000 7f260f400640 1 --2- 172.21.15.60:0/2476900169 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f261006c7d0 0x7f261015dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:42.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.439+0000 7f260d600640 1 -- 172.21.15.60:0/2476900169 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3624391468 0 0) 0x7f26100cb650 con 0x7f2610154fa0 2024-04-14T23:48:42.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.439+0000 7f260d600640 1 -- 172.21.15.60:0/2476900169 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f25f4003520 con 0x7f2610154fa0 2024-04-14T23:48:42.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.439+0000 7f260f400640 1 -- 172.21.15.60:0/2476900169 >> v1:172.21.15.186:6789/0 conn(0x7f2610154fa0 legacy=0x7f26100c6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:42.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.439+0000 7f260f400640 1 -- 172.21.15.60:0/2476900169 >> v1:172.21.15.60:6789/0 conn(0x7f2610154bc0 legacy=0x7f26100c5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:42.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.439+0000 7f260d600640 1 -- 172.21.15.60:0/2476900169 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 534722070 0 0) 0x7f26100cb370 con 0x7f2610154bc0 2024-04-14T23:48:42.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.439+0000 7f260f400640 1 -- 172.21.15.60:0/2476900169 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26100cb930 con 0x7f261006c7d0 2024-04-14T23:48:42.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.440+0000 7f260f400640 1 --2- 172.21.15.60:0/2476900169 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f261006c7d0 0x7f261015dfb0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2608009dc0 tx=0x7f260802bfa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec6cf34a1800f60a server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:42.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.440+0000 7f260d600640 1 -- 172.21.15.60:0/2476900169 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2608039070 con 0x7f261006c7d0 2024-04-14T23:48:42.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.440+0000 7f260d600640 1 -- 172.21.15.60:0/2476900169 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f260802cd50 con 0x7f261006c7d0 2024-04-14T23:48:42.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.440+0000 7f260d600640 1 -- 172.21.15.60:0/2476900169 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2608034660 con 0x7f261006c7d0 2024-04-14T23:48:42.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.441+0000 7f260fe00640 1 -- 172.21.15.60:0/2476900169 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f261006c7d0 msgr2=0x7f261015dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:42.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.441+0000 7f260fe00640 1 --2- 172.21.15.60:0/2476900169 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f261006c7d0 0x7f261015dfb0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2608009dc0 tx=0x7f260802bfa0 comp rx=0 tx=0).stop 2024-04-14T23:48:42.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.441+0000 7f260fe00640 1 -- 172.21.15.60:0/2476900169 shutdown_connections 2024-04-14T23:48:42.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.441+0000 7f260fe00640 1 --2- 172.21.15.60:0/2476900169 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f261006c7d0 0x7f261015dfb0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:42.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.441+0000 7f260fe00640 1 -- 172.21.15.60:0/2476900169 >> 172.21.15.60:0/2476900169 conn(0x7f2610066f10 msgr2=0x7f2610067320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:42.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.441+0000 7f260fe00640 1 -- 172.21.15.60:0/2476900169 shutdown_connections 2024-04-14T23:48:42.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.441+0000 7f260fe00640 1 -- 172.21.15.60:0/2476900169 wait complete. 2024-04-14T23:48:42.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.443+0000 7f260fe00640 1 Processor -- start 2024-04-14T23:48:42.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.443+0000 7f260fe00640 1 -- start start 2024-04-14T23:48:42.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.444+0000 7f260fe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f261006c7d0 0x7f261014a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:42.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.444+0000 7f260fe00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2610154bc0 0x7f26100b54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:42.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.444+0000 7f260fe00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2610154fa0 0x7f26101f3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:42.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.444+0000 7f260fe00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f261015ec40 con 0x7f2610154fa0 2024-04-14T23:48:42.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.444+0000 7f260fe00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f26100c60f0 con 0x7f261006c7d0 2024-04-14T23:48:42.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.444+0000 7f260fe00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f26100c6d20 con 0x7f2610154bc0 2024-04-14T23:48:42.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.444+0000 7f260f400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2610154fa0 0x7f26101f3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:42.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.444+0000 7f260f400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2610154fa0 0x7f26101f3ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53826/0 (socket says 172.21.15.60:53826) 2024-04-14T23:48:42.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.444+0000 7f260f400640 1 -- 172.21.15.60:0/1569738850 learned_addr learned my addr 172.21.15.60:0/1569738850 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:42.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.444+0000 7f260e000640 1 --2- 172.21.15.60:0/1569738850 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2610154bc0 0x7f26100b54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:42.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.444+0000 7f260ea00640 1 --2- 172.21.15.60:0/1569738850 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f261006c7d0 0x7f261014a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:42.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.445+0000 7f260f400640 1 -- 172.21.15.60:0/1569738850 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2610154bc0 msgr2=0x7f26100b54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:42.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.445+0000 7f260f400640 1 --2- 172.21.15.60:0/1569738850 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2610154bc0 0x7f26100b54b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:42.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.445+0000 7f260f400640 1 -- 172.21.15.60:0/1569738850 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f261006c7d0 msgr2=0x7f261014a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:42.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.445+0000 7f260f400640 1 --2- 172.21.15.60:0/1569738850 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f261006c7d0 0x7f261014a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:42.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.445+0000 7f260f400640 1 -- 172.21.15.60:0/1569738850 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f261014a860 con 0x7f2610154fa0 2024-04-14T23:48:42.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.445+0000 7f260e000640 1 --2- 172.21.15.60:0/1569738850 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2610154bc0 0x7f26100b54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:48:42.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.445+0000 7f260ea00640 1 --2- 172.21.15.60:0/1569738850 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f261006c7d0 0x7f261014a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:48:42.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.445+0000 7f260f400640 1 --2- 172.21.15.60:0/1569738850 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2610154fa0 0x7f26101f3ac0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2608037040 tx=0x7f2608004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:42.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.445+0000 7f2606a00640 1 -- 172.21.15.60:0/1569738850 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2608039050 con 0x7f2610154fa0 2024-04-14T23:48:42.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.445+0000 7f2606a00640 1 -- 172.21.15.60:0/1569738850 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2608045070 con 0x7f2610154fa0 2024-04-14T23:48:42.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.445+0000 7f260fe00640 1 -- 172.21.15.60:0/1569738850 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f261014aaf0 con 0x7f2610154fa0 2024-04-14T23:48:42.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.445+0000 7f2606a00640 1 -- 172.21.15.60:0/1569738850 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2608009450 con 0x7f2610154fa0 2024-04-14T23:48:42.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.446+0000 7f260fe00640 1 -- 172.21.15.60:0/1569738850 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26100b46f0 con 0x7f2610154fa0 2024-04-14T23:48:42.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.447+0000 7f260fe00640 1 -- 172.21.15.60:0/1569738850 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f261004f540 con 0x7f2610154fa0 2024-04-14T23:48:42.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.448+0000 7f2606a00640 1 -- 172.21.15.60:0/1569738850 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f260803e830 con 0x7f2610154fa0 2024-04-14T23:48:42.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.449+0000 7f2606a00640 1 --2- 172.21.15.60:0/1569738850 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f25f006b990 0x7f25f006de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:42.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.449+0000 7f2606a00640 1 -- 172.21.15.60:0/1569738850 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5609+0+0 (secure 0 0 0) 0x7f26080ad4a0 con 0x7f2610154fa0 2024-04-14T23:48:42.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.449+0000 7f260ea00640 1 --2- 172.21.15.60:0/1569738850 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f25f006b990 0x7f25f006de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:42.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.450+0000 7f260ea00640 1 --2- 172.21.15.60:0/1569738850 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f25f006b990 0x7f25f006de50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2600000f60 tx=0x7f2600007d50 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:42.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.455+0000 7f2606a00640 1 -- 172.21.15.60:0/1569738850 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f260807bbe0 con 0x7f2610154fa0 2024-04-14T23:48:42.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:42.697+0000 7f260fe00640 1 -- 172.21.15.60:0/1569738850 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f26100cdde0 con 0x7f2610154fa0 2024-04-14T23:48:43.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.203+0000 7f2606a00640 1 -- 172.21.15.60:0/1569738850 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) ==== 172+0+0 (secure 0 0 0) 0x7f260807fc80 con 0x7f2610154fa0 2024-04-14T23:48:43.210 INFO:teuthology.orchestra.run.smithi060.stderr:new fs with metadata pool 2 and data pool 3 2024-04-14T23:48:43.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.209+0000 7f260fe00640 1 -- 172.21.15.60:0/1569738850 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f25f006b990 msgr2=0x7f25f006de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:43.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.209+0000 7f260fe00640 1 --2- 172.21.15.60:0/1569738850 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f25f006b990 0x7f25f006de50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2600000f60 tx=0x7f2600007d50 comp rx=0 tx=0).stop 2024-04-14T23:48:43.217 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.209+0000 7f260fe00640 1 -- 172.21.15.60:0/1569738850 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2610154fa0 msgr2=0x7f26101f3ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:43.217 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.209+0000 7f260fe00640 1 --2- 172.21.15.60:0/1569738850 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2610154fa0 0x7f26101f3ac0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2608037040 tx=0x7f2608004a40 comp rx=0 tx=0).stop 2024-04-14T23:48:43.217 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.210+0000 7f260fe00640 1 -- 172.21.15.60:0/1569738850 shutdown_connections 2024-04-14T23:48:43.217 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.210+0000 7f260fe00640 1 --2- 172.21.15.60:0/1569738850 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f25f006b990 0x7f25f006de50 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:43.217 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.210+0000 7f260fe00640 1 --2- 172.21.15.60:0/1569738850 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2610154fa0 0x7f26101f3ac0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:43.217 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.210+0000 7f260fe00640 1 --2- 172.21.15.60:0/1569738850 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2610154bc0 0x7f26100b54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:43.217 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.210+0000 7f260fe00640 1 --2- 172.21.15.60:0/1569738850 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f261006c7d0 0x7f261014a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:43.217 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.210+0000 7f260fe00640 1 -- 172.21.15.60:0/1569738850 >> 172.21.15.60:0/1569738850 conn(0x7f2610066f10 msgr2=0x7f261014f390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:43.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.212+0000 7f260fe00640 1 -- 172.21.15.60:0/1569738850 shutdown_connections 2024-04-14T23:48:43.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.213+0000 7f260fe00640 1 -- 172.21.15.60:0/1569738850 wait complete. 2024-04-14T23:48:43.249 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-14T23:48:43.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.423+0000 7f2a5a800640 1 Processor -- start 2024-04-14T23:48:43.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.424+0000 7f2a5a800640 1 -- start start 2024-04-14T23:48:43.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.425+0000 7f2a5a800640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2a540c4fa0 0x7f2a540c77b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:43.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.425+0000 7f2a5a800640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2a54070d50 0x7f2a540c7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:43.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.425+0000 7f2a5a800640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a540c5380 0x7f2a5415a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:43.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.425+0000 7f2a5a800640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f2a5406b4b0 con 0x7f2a54070d50 2024-04-14T23:48:43.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.425+0000 7f2a5a800640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f2a5406b790 con 0x7f2a540c4fa0 2024-04-14T23:48:43.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.425+0000 7f2a5a800640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f2a540c69f0 con 0x7f2a540c5380 2024-04-14T23:48:43.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.425+0000 7f2a59400640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2a540c4fa0 0x7f2a540c77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:43.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.425+0000 7f2a58a00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2a54070d50 0x7f2a540c7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:43.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.426+0000 7f2a59400640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2a540c4fa0 0x7f2a540c77b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:43238/0 (socket says 172.21.15.60:43238) 2024-04-14T23:48:43.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.426+0000 7f2a59e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a540c5380 0x7f2a5415a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:43.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.426+0000 7f2a59400640 1 -- 172.21.15.60:0/3064317018 learned_addr learned my addr 172.21.15.60:0/3064317018 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:43.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.426+0000 7f2a59400640 1 -- 172.21.15.60:0/3064317018 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a540c5380 msgr2=0x7f2a5415a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:43.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.426+0000 7f2a59400640 1 --2- 172.21.15.60:0/3064317018 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a540c5380 0x7f2a5415a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:43.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.426+0000 7f2a59400640 1 -- 172.21.15.60:0/3064317018 >> v2:172.21.15.60:3300/0 conn(0x7f2a54070d50 msgr2=0x7f2a540c7cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:43.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.426+0000 7f2a59400640 1 --2- 172.21.15.60:0/3064317018 >> v2:172.21.15.60:3300/0 conn(0x7f2a54070d50 0x7f2a540c7cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:43.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.426+0000 7f2a59400640 1 -- 172.21.15.60:0/3064317018 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a540c6cd0 con 0x7f2a540c4fa0 2024-04-14T23:48:43.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.427+0000 7f2a59400640 1 --2- 172.21.15.60:0/3064317018 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a540c4fa0 0x7f2a540c77b0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2a540b1ff0 tx=0x7f2a4802b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=50aab74f2d27c32d server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:43.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.427+0000 7f2a53e00640 1 -- 172.21.15.60:0/3064317018 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a48035020 con 0x7f2a540c4fa0 2024-04-14T23:48:43.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.427+0000 7f2a53e00640 1 -- 172.21.15.60:0/3064317018 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a48031870 con 0x7f2a540c4fa0 2024-04-14T23:48:43.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.427+0000 7f2a53e00640 1 -- 172.21.15.60:0/3064317018 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a48031c40 con 0x7f2a540c4fa0 2024-04-14T23:48:43.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.428+0000 7f2a5a800640 1 -- 172.21.15.60:0/3064317018 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a540c4fa0 msgr2=0x7f2a540c77b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:43.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.428+0000 7f2a5a800640 1 --2- 172.21.15.60:0/3064317018 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a540c4fa0 0x7f2a540c77b0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2a540b1ff0 tx=0x7f2a4802b910 comp rx=0 tx=0).stop 2024-04-14T23:48:43.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.428+0000 7f2a5a800640 1 -- 172.21.15.60:0/3064317018 shutdown_connections 2024-04-14T23:48:43.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.428+0000 7f2a5a800640 1 --2- 172.21.15.60:0/3064317018 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a540c5380 0x7f2a5415a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:43.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.428+0000 7f2a5a800640 1 --2- 172.21.15.60:0/3064317018 >> v2:172.21.15.60:3300/0 conn(0x7f2a54070d50 0x7f2a540c7cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:43.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.428+0000 7f2a5a800640 1 --2- 172.21.15.60:0/3064317018 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a540c4fa0 0x7f2a540c77b0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:43.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.428+0000 7f2a5a800640 1 -- 172.21.15.60:0/3064317018 >> 172.21.15.60:0/3064317018 conn(0x7f2a540c8b50 msgr2=0x7f2a540c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:43.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.428+0000 7f2a5a800640 1 -- 172.21.15.60:0/3064317018 shutdown_connections 2024-04-14T23:48:43.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.428+0000 7f2a5a800640 1 -- 172.21.15.60:0/3064317018 wait complete. 2024-04-14T23:48:43.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.430+0000 7f2a5a800640 1 Processor -- start 2024-04-14T23:48:43.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.430+0000 7f2a5a800640 1 -- start start 2024-04-14T23:48:43.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.431+0000 7f2a5a800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a54070d50 0x7f2a540c2960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:43.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.431+0000 7f2a5a800640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a540c4fa0 0x7f2a540aebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:43.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.431+0000 7f2a5a800640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a540c5380 0x7f2a540af0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:43.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.431+0000 7f2a5a800640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f2a5406b1c0 con 0x7f2a540c5380 2024-04-14T23:48:43.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.431+0000 7f2a5a800640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2a540c7290 con 0x7f2a54070d50 2024-04-14T23:48:43.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.431+0000 7f2a5a800640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f2a540c99d0 con 0x7f2a540c4fa0 2024-04-14T23:48:43.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.431+0000 7f2a59e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a540c5380 0x7f2a540af0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:43.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.431+0000 7f2a59e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a540c5380 0x7f2a540af0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53834/0 (socket says 172.21.15.60:53834) 2024-04-14T23:48:43.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.431+0000 7f2a59e00640 1 -- 172.21.15.60:0/522800305 learned_addr learned my addr 172.21.15.60:0/522800305 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:43.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.431+0000 7f2a59400640 1 --2- 172.21.15.60:0/522800305 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a54070d50 0x7f2a540c2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:43.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.431+0000 7f2a58a00640 1 --2- 172.21.15.60:0/522800305 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a540c4fa0 0x7f2a540aebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:43.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.432+0000 7f2a59e00640 1 -- 172.21.15.60:0/522800305 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a540c4fa0 msgr2=0x7f2a540aebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:43.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.432+0000 7f2a59e00640 1 --2- 172.21.15.60:0/522800305 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a540c4fa0 0x7f2a540aebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:43.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.432+0000 7f2a59e00640 1 -- 172.21.15.60:0/522800305 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a54070d50 msgr2=0x7f2a540c2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:43.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.432+0000 7f2a59e00640 1 --2- 172.21.15.60:0/522800305 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a54070d50 0x7f2a540c2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:43.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.432+0000 7f2a59e00640 1 -- 172.21.15.60:0/522800305 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a541543b0 con 0x7f2a540c5380 2024-04-14T23:48:43.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.432+0000 7f2a59400640 1 --2- 172.21.15.60:0/522800305 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a54070d50 0x7f2a540c2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:48:43.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.432+0000 7f2a59e00640 1 --2- 172.21.15.60:0/522800305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a540c5380 0x7f2a540af0f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c00efc0 tx=0x7f2a4c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:43.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.432+0000 7f2a51600640 1 -- 172.21.15.60:0/522800305 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a4c018020 con 0x7f2a540c5380 2024-04-14T23:48:43.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.432+0000 7f2a51600640 1 -- 172.21.15.60:0/522800305 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a4c014480 con 0x7f2a540c5380 2024-04-14T23:48:43.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.432+0000 7f2a5a800640 1 -- 172.21.15.60:0/522800305 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a54154690 con 0x7f2a540c5380 2024-04-14T23:48:43.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.432+0000 7f2a51600640 1 -- 172.21.15.60:0/522800305 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a4c009e60 con 0x7f2a540c5380 2024-04-14T23:48:43.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.433+0000 7f2a5a800640 1 -- 172.21.15.60:0/522800305 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a54150520 con 0x7f2a540c5380 2024-04-14T23:48:43.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.434+0000 7f2a5a800640 1 -- 172.21.15.60:0/522800305 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a5404f540 con 0x7f2a540c5380 2024-04-14T23:48:43.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.435+0000 7f2a51600640 1 -- 172.21.15.60:0/522800305 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f2a4c005d10 con 0x7f2a540c5380 2024-04-14T23:48:43.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.436+0000 7f2a51600640 1 --2- 172.21.15.60:0/522800305 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f2a3406b9c0 0x7f2a3406de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:43.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.436+0000 7f2a51600640 1 -- 172.21.15.60:0/522800305 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f2a4c08b410 con 0x7f2a540c5380 2024-04-14T23:48:43.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.436+0000 7f2a59400640 1 --2- 172.21.15.60:0/522800305 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f2a3406b9c0 0x7f2a3406de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:43.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.437+0000 7f2a59400640 1 --2- 172.21.15.60:0/522800305 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f2a3406b9c0 0x7f2a3406de80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2a48036000 tx=0x7f2a4803c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:43.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.441+0000 7f2a51600640 1 -- 172.21.15.60:0/522800305 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2a4c022030 con 0x7f2a540c5380 2024-04-14T23:48:43.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.637+0000 7f2a5a800640 1 -- 172.21.15.60:0/522800305 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2a54157400 con 0x7f2a540c5380 2024-04-14T23:48:43.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.639+0000 7f2a51600640 1 -- 172.21.15.60:0/522800305 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16618 (secure 0 0 0) 0x7f2a4c010040 con 0x7f2a540c5380 2024-04-14T23:48:43.645 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:48:43.645 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":20,"fsid":"5847db56-a76d-4486-b563-d83f1186af92","created":"2024-04-14T23:48:23.900003+0000","modified":"2024-04-14T23:48:42.699710+0000","last_up_change":"2024-04-14T23:48:34.203573+0000","last_in_change":"2024-04-14T23:48:26.345499+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-14T23:48:36.247372+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-14T23:48:40.274195+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-14T23:48:41.722183+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b2b2b755-7f42-4f9b-af9a-a178f7cc5a52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6817","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6818","nonce":2536528895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6822","nonce":2536528895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6827","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6828","nonce":2536528895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6825","nonce":2536528895}]},"public_addr":"172.21.15.60:6818/2536528895","cluster_addr":"172.21.15.60:6822/2536528895","heartbeat_back_addr":"172.21.15.60:6828/2536528895","heartbeat_front_addr":"172.21.15.60:6825/2536528895","state":["exists","up"]},{"osd":1,"uuid":"321418cd-5ab7-41a5-b9c8-b3fc12f68ebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6801","nonce":3115695676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6803","nonce":3115695676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6807","nonce":3115695676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6805","nonce":3115695676}]},"public_addr":"172.21.15.60:6801/3115695676","cluster_addr":"172.21.15.60:6803/3115695676","heartbeat_back_addr":"172.21.15.60:6807/3115695676","heartbeat_front_addr":"172.21.15.60:6805/3115695676","state":["exists","up"]},{"osd":2,"uuid":"07874e9f-3a38-41dc-be9f-2b75e84e53d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6819","nonce":3475168698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6824","nonce":3475168698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6831","nonce":3475168698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6829","nonce":3475168698}]},"public_addr":"172.21.15.60:6819/3475168698","cluster_addr":"172.21.15.60:6824/3475168698","heartbeat_back_addr":"172.21.15.60:6831/3475168698","heartbeat_front_addr":"172.21.15.60:6829/3475168698","state":["exists","up"]},{"osd":3,"uuid":"7e4ef6c9-b17a-47c6-a320-f5d16a6fe05e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6809","nonce":1730813674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6811","nonce":1730813674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6815","nonce":1730813674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6813","nonce":1730813674}]},"public_addr":"172.21.15.60:6809/1730813674","cluster_addr":"172.21.15.60:6811/1730813674","heartbeat_back_addr":"172.21.15.60:6815/1730813674","heartbeat_front_addr":"172.21.15.60:6813/1730813674","state":["exists","up"]},{"osd":4,"uuid":"759a3efc-dade-44c5-b57a-1623499aba50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6805","nonce":1464587038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6810","nonce":1464587038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6819","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6822","nonce":1464587038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6813","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6816","nonce":1464587038}]},"public_addr":"172.21.15.186:6805/1464587038","cluster_addr":"172.21.15.186:6810/1464587038","heartbeat_back_addr":"172.21.15.186:6822/1464587038","heartbeat_front_addr":"172.21.15.186:6816/1464587038","state":["exists","up"]},{"osd":5,"uuid":"24cd58c1-0cf1-4077-ab33-84c39c6907ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6801","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6803","nonce":1697098067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6809","nonce":1697098067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6820","nonce":1697098067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6815","nonce":1697098067}]},"public_addr":"172.21.15.186:6803/1697098067","cluster_addr":"172.21.15.186:6809/1697098067","heartbeat_back_addr":"172.21.15.186:6820/1697098067","heartbeat_front_addr":"172.21.15.186:6815/1697098067","state":["exists","up"]},{"osd":6,"uuid":"6add38f1-65e9-4b0d-927a-4ab77f5e2323","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6804","nonce":1837108559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6807","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6811","nonce":1837108559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6823","nonce":1837108559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6817","nonce":1837108559}]},"public_addr":"172.21.15.186:6804/1837108559","cluster_addr":"172.21.15.186:6811/1837108559","heartbeat_back_addr":"172.21.15.186:6823/1837108559","heartbeat_front_addr":"172.21.15.186:6817/1837108559","state":["exists","up"]},{"osd":7,"uuid":"a2326b6f-7601-4dda-a456-67c443ea5929","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6825","nonce":842537875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6827","nonce":842537875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6831","nonce":842537875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6829","nonce":842537875}]},"public_addr":"172.21.15.186:6825/842537875","cluster_addr":"172.21.15.186:6827/842537875","heartbeat_back_addr":"172.21.15.186:6831/842537875","heartbeat_front_addr":"172.21.15.186:6829/842537875","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.975583+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.976806+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.722725+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.763705+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.502140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.528972+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.517961+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.513350+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-14T23:48:43.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.642+0000 7f2a5a800640 1 -- 172.21.15.60:0/522800305 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f2a3406b9c0 msgr2=0x7f2a3406de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:43.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.642+0000 7f2a5a800640 1 --2- 172.21.15.60:0/522800305 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f2a3406b9c0 0x7f2a3406de80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2a48036000 tx=0x7f2a4803c040 comp rx=0 tx=0).stop 2024-04-14T23:48:43.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.642+0000 7f2a5a800640 1 -- 172.21.15.60:0/522800305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a540c5380 msgr2=0x7f2a540af0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:43.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.642+0000 7f2a5a800640 1 --2- 172.21.15.60:0/522800305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a540c5380 0x7f2a540af0f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c00efc0 tx=0x7f2a4c00c490 comp rx=0 tx=0).stop 2024-04-14T23:48:43.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.642+0000 7f2a5a800640 1 -- 172.21.15.60:0/522800305 shutdown_connections 2024-04-14T23:48:43.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.642+0000 7f2a5a800640 1 --2- 172.21.15.60:0/522800305 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f2a3406b9c0 0x7f2a3406de80 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:43.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.642+0000 7f2a5a800640 1 --2- 172.21.15.60:0/522800305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a540c5380 0x7f2a540af0f0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:43.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.642+0000 7f2a5a800640 1 --2- 172.21.15.60:0/522800305 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a540c4fa0 0x7f2a540aebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:43.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.642+0000 7f2a5a800640 1 --2- 172.21.15.60:0/522800305 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a54070d50 0x7f2a540c2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:43.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.642+0000 7f2a5a800640 1 -- 172.21.15.60:0/522800305 >> 172.21.15.60:0/522800305 conn(0x7f2a540c8b50 msgr2=0x7f2a540c58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:43.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.642+0000 7f2a5a800640 1 -- 172.21.15.60:0/522800305 shutdown_connections 2024-04-14T23:48:43.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.642+0000 7f2a5a800640 1 -- 172.21.15.60:0/522800305 wait complete. 2024-04-14T23:48:43.666 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-14T23:48:43.751 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-14T23:48:43.759 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-14T23:48:43.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.877+0000 7f08b4000640 1 Processor -- start 2024-04-14T23:48:43.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.878+0000 7f08b4000640 1 -- start start 2024-04-14T23:48:43.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.879+0000 7f08b4000640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f08ac0cd160 0x7f08ac15ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:43.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.879+0000 7f08b4000640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f08ac0cd540 0x7f08ac0b3190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:43.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.879+0000 7f08b4000640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08ac0b3f50 con 0x7f08ac0cb750 2024-04-14T23:48:43.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.879+0000 7f08b4000640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f08ac15b370 con 0x7f08ac0cd540 2024-04-14T23:48:43.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.879+0000 7f08b4000640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f08ac0b39b0 con 0x7f08ac0cd160 2024-04-14T23:48:43.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.880+0000 7f08b0c00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f08ac0cd540 0x7f08ac0b3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:43.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.880+0000 7f08b0c00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f08ac0cd540 0x7f08ac0b3190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53842/0 (socket says 172.21.15.60:53842) 2024-04-14T23:48:43.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.880+0000 7f08b0c00640 1 -- 172.21.15.60:0/1891949366 learned_addr learned my addr 172.21.15.60:0/1891949366 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:43.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.880+0000 7f08b1600640 1 --2- 172.21.15.60:0/1891949366 >> v2:172.21.15.186:3300/0 conn(0x7f08ac0cd160 0x7f08ac15ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:43.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.880+0000 7f08b0c00640 1 -- 172.21.15.60:0/1891949366 >> v2:172.21.15.186:3300/0 conn(0x7f08ac0cd160 msgr2=0x7f08ac15ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:43.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.880+0000 7f08b0c00640 1 --2- 172.21.15.60:0/1891949366 >> v2:172.21.15.186:3300/0 conn(0x7f08ac0cd160 0x7f08ac15ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:43.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.880+0000 7f08b0c00640 1 -- 172.21.15.60:0/1891949366 >> v1:172.21.15.186:6789/0 conn(0x7f08ac0cb750 legacy=0x7f08ac154b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:43.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.880+0000 7f08b0c00640 1 -- 172.21.15.60:0/1891949366 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08ac0b3c70 con 0x7f08ac0cd540 2024-04-14T23:48:43.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.881+0000 7f08b0c00640 1 --2- 172.21.15.60:0/1891949366 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f08ac0cd540 0x7f08ac0b3190 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f08ac067ef0 tx=0x7f089802ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a51f3b4cb662002d server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:43.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.881+0000 7f08abe00640 1 -- 172.21.15.60:0/1891949366 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f089802f850 con 0x7f08ac0cd540 2024-04-14T23:48:43.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.882+0000 7f08abe00640 1 -- 172.21.15.60:0/1891949366 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f089802f9f0 con 0x7f08ac0cd540 2024-04-14T23:48:43.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.882+0000 7f08abe00640 1 -- 172.21.15.60:0/1891949366 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f089802fdc0 con 0x7f08ac0cd540 2024-04-14T23:48:43.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.882+0000 7f08b4000640 1 -- 172.21.15.60:0/1891949366 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f08ac0cd540 msgr2=0x7f08ac0b3190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:43.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.882+0000 7f08b4000640 1 --2- 172.21.15.60:0/1891949366 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f08ac0cd540 0x7f08ac0b3190 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f08ac067ef0 tx=0x7f089802ba30 comp rx=0 tx=0).stop 2024-04-14T23:48:43.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.882+0000 7f08b4000640 1 -- 172.21.15.60:0/1891949366 shutdown_connections 2024-04-14T23:48:43.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.882+0000 7f08b4000640 1 --2- 172.21.15.60:0/1891949366 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f08ac0cd540 0x7f08ac0b3190 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:43.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.882+0000 7f08b4000640 1 --2- 172.21.15.60:0/1891949366 >> v2:172.21.15.186:3300/0 conn(0x7f08ac0cd160 0x7f08ac15ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:43.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.882+0000 7f08b4000640 1 -- 172.21.15.60:0/1891949366 >> 172.21.15.60:0/1891949366 conn(0x7f08ac066f10 msgr2=0x7f08ac0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:43.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.882+0000 7f08b4000640 1 -- 172.21.15.60:0/1891949366 shutdown_connections 2024-04-14T23:48:43.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.882+0000 7f08b4000640 1 -- 172.21.15.60:0/1891949366 wait complete. 2024-04-14T23:48:43.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.884+0000 7f08b4000640 1 Processor -- start 2024-04-14T23:48:43.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.884+0000 7f08b4000640 1 -- start start 2024-04-14T23:48:43.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.885+0000 7f08b4000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f08ac0cb750 0x7f08ac15ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:43.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.885+0000 7f08b4000640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f08ac0cd160 0x7f08ac15d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:43.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.885+0000 7f08b4000640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f08ac0cd540 0x7f08ac1e40b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:43.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.885+0000 7f08b4000640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f08ac15ea70 con 0x7f08ac0cd540 2024-04-14T23:48:43.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.885+0000 7f08b4000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f08ac15e790 con 0x7f08ac0cb750 2024-04-14T23:48:43.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.885+0000 7f08b4000640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f08ac0b36d0 con 0x7f08ac0cd160 2024-04-14T23:48:43.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.886+0000 7f08b1600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f08ac0cb750 0x7f08ac15ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:43.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.886+0000 7f08b1600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f08ac0cb750 0x7f08ac15ac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:43270/0 (socket says 172.21.15.60:43270) 2024-04-14T23:48:43.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.886+0000 7f08b1600640 1 -- 172.21.15.60:0/1156296788 learned_addr learned my addr 172.21.15.60:0/1156296788 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:43.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.886+0000 7f08b0c00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f08ac0cd160 0x7f08ac15d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:43.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.886+0000 7f08b2000640 1 --2- 172.21.15.60:0/1156296788 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f08ac0cd540 0x7f08ac1e40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:43.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.886+0000 7f08b1600640 1 -- 172.21.15.60:0/1156296788 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f08ac0cd160 msgr2=0x7f08ac15d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:43.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.886+0000 7f08b1600640 1 --2- 172.21.15.60:0/1156296788 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f08ac0cd160 0x7f08ac15d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:43.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.886+0000 7f08b1600640 1 -- 172.21.15.60:0/1156296788 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f08ac0cd540 msgr2=0x7f08ac1e40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:43.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.886+0000 7f08b1600640 1 --2- 172.21.15.60:0/1156296788 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f08ac0cd540 0x7f08ac1e40b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:43.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.886+0000 7f08b1600640 1 -- 172.21.15.60:0/1156296788 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08ac1e4b60 con 0x7f08ac0cb750 2024-04-14T23:48:43.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.886+0000 7f08b2000640 1 --2- 172.21.15.60:0/1156296788 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f08ac0cd540 0x7f08ac1e40b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-14T23:48:43.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.886+0000 7f08b1600640 1 --2- 172.21.15.60:0/1156296788 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f08ac0cb750 0x7f08ac15ac50 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f08a000e9b0 tx=0x7f08a000ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:43.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.887+0000 7f08a9600640 1 -- 172.21.15.60:0/1156296788 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08a0017070 con 0x7f08ac0cb750 2024-04-14T23:48:43.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.887+0000 7f08a9600640 1 -- 172.21.15.60:0/1156296788 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08a000c9a0 con 0x7f08ac0cb750 2024-04-14T23:48:43.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.887+0000 7f08a9600640 1 -- 172.21.15.60:0/1156296788 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08a000cd90 con 0x7f08ac0cb750 2024-04-14T23:48:43.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.887+0000 7f08b4000640 1 -- 172.21.15.60:0/1156296788 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08ac1f49c0 con 0x7f08ac0cb750 2024-04-14T23:48:43.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.887+0000 7f08b4000640 1 -- 172.21.15.60:0/1156296788 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08ac1f4f80 con 0x7f08ac0cb750 2024-04-14T23:48:43.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.888+0000 7f08b4000640 1 -- 172.21.15.60:0/1156296788 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08ac076970 con 0x7f08ac0cb750 2024-04-14T23:48:43.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.889+0000 7f08a9600640 1 -- 172.21.15.60:0/1156296788 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f08a001a070 con 0x7f08ac0cb750 2024-04-14T23:48:43.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.890+0000 7f08a9600640 1 --2- 172.21.15.60:0/1156296788 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f088c06b990 0x7f088c06de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:43.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.890+0000 7f08a9600640 1 -- 172.21.15.60:0/1156296788 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f08a008abe0 con 0x7f08ac0cb750 2024-04-14T23:48:43.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.890+0000 7f08b0c00640 1 --2- 172.21.15.60:0/1156296788 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f088c06b990 0x7f088c06de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:43.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.891+0000 7f08b0c00640 1 --2- 172.21.15.60:0/1156296788 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f088c06b990 0x7f088c06de50 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0898036000 tx=0x7f0898008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:43.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:43.895+0000 7f08a9600640 1 -- 172.21.15.60:0/1156296788 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f08a0059100 con 0x7f08ac0cb750 2024-04-14T23:48:44.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.096+0000 7f08b4000640 1 -- 172.21.15.60:0/1156296788 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f08ac06c7d0 con 0x7f08ac0cb750 2024-04-14T23:48:44.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.098+0000 7f08a9600640 1 -- 172.21.15.60:0/1156296788 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16618 (secure 0 0 0) 0x7f08a005d1a0 con 0x7f08ac0cb750 2024-04-14T23:48:44.103 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:48:44.103 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":20,"fsid":"5847db56-a76d-4486-b563-d83f1186af92","created":"2024-04-14T23:48:23.900003+0000","modified":"2024-04-14T23:48:42.699710+0000","last_up_change":"2024-04-14T23:48:34.203573+0000","last_in_change":"2024-04-14T23:48:26.345499+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-14T23:48:36.247372+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-14T23:48:40.274195+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-14T23:48:41.722183+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b2b2b755-7f42-4f9b-af9a-a178f7cc5a52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6817","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6818","nonce":2536528895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6822","nonce":2536528895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6827","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6828","nonce":2536528895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6825","nonce":2536528895}]},"public_addr":"172.21.15.60:6818/2536528895","cluster_addr":"172.21.15.60:6822/2536528895","heartbeat_back_addr":"172.21.15.60:6828/2536528895","heartbeat_front_addr":"172.21.15.60:6825/2536528895","state":["exists","up"]},{"osd":1,"uuid":"321418cd-5ab7-41a5-b9c8-b3fc12f68ebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6801","nonce":3115695676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6803","nonce":3115695676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6807","nonce":3115695676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6805","nonce":3115695676}]},"public_addr":"172.21.15.60:6801/3115695676","cluster_addr":"172.21.15.60:6803/3115695676","heartbeat_back_addr":"172.21.15.60:6807/3115695676","heartbeat_front_addr":"172.21.15.60:6805/3115695676","state":["exists","up"]},{"osd":2,"uuid":"07874e9f-3a38-41dc-be9f-2b75e84e53d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6819","nonce":3475168698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6824","nonce":3475168698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6831","nonce":3475168698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6829","nonce":3475168698}]},"public_addr":"172.21.15.60:6819/3475168698","cluster_addr":"172.21.15.60:6824/3475168698","heartbeat_back_addr":"172.21.15.60:6831/3475168698","heartbeat_front_addr":"172.21.15.60:6829/3475168698","state":["exists","up"]},{"osd":3,"uuid":"7e4ef6c9-b17a-47c6-a320-f5d16a6fe05e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6809","nonce":1730813674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6811","nonce":1730813674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6815","nonce":1730813674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6813","nonce":1730813674}]},"public_addr":"172.21.15.60:6809/1730813674","cluster_addr":"172.21.15.60:6811/1730813674","heartbeat_back_addr":"172.21.15.60:6815/1730813674","heartbeat_front_addr":"172.21.15.60:6813/1730813674","state":["exists","up"]},{"osd":4,"uuid":"759a3efc-dade-44c5-b57a-1623499aba50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6805","nonce":1464587038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6810","nonce":1464587038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6819","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6822","nonce":1464587038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6813","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6816","nonce":1464587038}]},"public_addr":"172.21.15.186:6805/1464587038","cluster_addr":"172.21.15.186:6810/1464587038","heartbeat_back_addr":"172.21.15.186:6822/1464587038","heartbeat_front_addr":"172.21.15.186:6816/1464587038","state":["exists","up"]},{"osd":5,"uuid":"24cd58c1-0cf1-4077-ab33-84c39c6907ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6801","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6803","nonce":1697098067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6809","nonce":1697098067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6820","nonce":1697098067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6815","nonce":1697098067}]},"public_addr":"172.21.15.186:6803/1697098067","cluster_addr":"172.21.15.186:6809/1697098067","heartbeat_back_addr":"172.21.15.186:6820/1697098067","heartbeat_front_addr":"172.21.15.186:6815/1697098067","state":["exists","up"]},{"osd":6,"uuid":"6add38f1-65e9-4b0d-927a-4ab77f5e2323","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6804","nonce":1837108559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6807","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6811","nonce":1837108559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6823","nonce":1837108559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6817","nonce":1837108559}]},"public_addr":"172.21.15.186:6804/1837108559","cluster_addr":"172.21.15.186:6811/1837108559","heartbeat_back_addr":"172.21.15.186:6823/1837108559","heartbeat_front_addr":"172.21.15.186:6817/1837108559","state":["exists","up"]},{"osd":7,"uuid":"a2326b6f-7601-4dda-a456-67c443ea5929","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6825","nonce":842537875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6827","nonce":842537875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6831","nonce":842537875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6829","nonce":842537875}]},"public_addr":"172.21.15.186:6825/842537875","cluster_addr":"172.21.15.186:6827/842537875","heartbeat_back_addr":"172.21.15.186:6831/842537875","heartbeat_front_addr":"172.21.15.186:6829/842537875","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.975583+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.976806+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.722725+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.763705+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.502140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.528972+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.517961+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.513350+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-14T23:48:44.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.100+0000 7f08b4000640 1 -- 172.21.15.60:0/1156296788 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f088c06b990 msgr2=0x7f088c06de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:44.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.100+0000 7f08b4000640 1 --2- 172.21.15.60:0/1156296788 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f088c06b990 0x7f088c06de50 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0898036000 tx=0x7f0898008030 comp rx=0 tx=0).stop 2024-04-14T23:48:44.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.100+0000 7f08b4000640 1 -- 172.21.15.60:0/1156296788 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f08ac0cb750 msgr2=0x7f08ac15ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:44.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.100+0000 7f08b4000640 1 --2- 172.21.15.60:0/1156296788 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f08ac0cb750 0x7f08ac15ac50 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f08a000e9b0 tx=0x7f08a000ee70 comp rx=0 tx=0).stop 2024-04-14T23:48:44.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.100+0000 7f08b4000640 1 -- 172.21.15.60:0/1156296788 shutdown_connections 2024-04-14T23:48:44.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.100+0000 7f08b4000640 1 --2- 172.21.15.60:0/1156296788 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f088c06b990 0x7f088c06de50 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:44.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.100+0000 7f08b4000640 1 --2- 172.21.15.60:0/1156296788 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f08ac0cd540 0x7f08ac1e40b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:44.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.100+0000 7f08b4000640 1 --2- 172.21.15.60:0/1156296788 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f08ac0cd160 0x7f08ac15d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:44.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.100+0000 7f08b4000640 1 --2- 172.21.15.60:0/1156296788 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f08ac0cb750 0x7f08ac15ac50 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:44.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.100+0000 7f08b4000640 1 -- 172.21.15.60:0/1156296788 >> 172.21.15.60:0/1156296788 conn(0x7f08ac066f10 msgr2=0x7f08ac150210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:44.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.100+0000 7f08b4000640 1 -- 172.21.15.60:0/1156296788 shutdown_connections 2024-04-14T23:48:44.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.100+0000 7f08b4000640 1 -- 172.21.15.60:0/1156296788 wait complete. 2024-04-14T23:48:44.126 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-14T23:48:44.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.393+0000 7f2357800640 1 Processor -- start 2024-04-14T23:48:44.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.393+0000 7f2357800640 1 -- start start 2024-04-14T23:48:44.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.395+0000 7f2357800640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f23500c63c0 0x7f23500c6b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:44.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.395+0000 7f2357800640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f235015c9b0 0x7f23500c70b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:44.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.395+0000 7f2357800640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23501530d0 con 0x7f2350152cf0 2024-04-14T23:48:44.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.395+0000 7f2357800640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f23500751f0 con 0x7f235015c9b0 2024-04-14T23:48:44.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.395+0000 7f2357800640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f23500c8370 con 0x7f23500c63c0 2024-04-14T23:48:44.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.395+0000 7f2354e00640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f2350152cf0 0x7f235015c2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:43626/0 (socket says 172.21.15.60:43626) 2024-04-14T23:48:44.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.395+0000 7f2354e00640 1 -- 172.21.15.60:0/941190603 learned_addr learned my addr 172.21.15.60:0/941190603 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:48:44.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.395+0000 7f2355800640 1 --2- 172.21.15.60:0/941190603 >> v2:172.21.15.60:3300/0 conn(0x7f235015c9b0 0x7f23500c70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:44.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.396+0000 7f234fe00640 1 --2- 172.21.15.60:0/941190603 >> v2:172.21.15.186:3300/0 conn(0x7f23500c63c0 0x7f23500c6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:44.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.396+0000 7f234f400640 1 -- 172.21.15.60:0/941190603 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1516491551 0 0) 0x7f23501530d0 con 0x7f2350152cf0 2024-04-14T23:48:44.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.396+0000 7f234f400640 1 -- 172.21.15.60:0/941190603 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2338003520 con 0x7f2350152cf0 2024-04-14T23:48:44.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.396+0000 7f2355800640 1 -- 172.21.15.60:0/941190603 >> v2:172.21.15.186:3300/0 conn(0x7f23500c63c0 msgr2=0x7f23500c6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:44.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.396+0000 7f2355800640 1 --2- 172.21.15.60:0/941190603 >> v2:172.21.15.186:3300/0 conn(0x7f23500c63c0 0x7f23500c6b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:44.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.396+0000 7f2355800640 1 -- 172.21.15.60:0/941190603 >> v1:172.21.15.60:6789/0 conn(0x7f2350152cf0 legacy=0x7f235015c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:44.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.396+0000 7f2355800640 1 -- 172.21.15.60:0/941190603 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23500c3810 con 0x7f235015c9b0 2024-04-14T23:48:44.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.397+0000 7f2355800640 1 --2- 172.21.15.60:0/941190603 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f235015c9b0 0x7f23500c70b0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2350070e80 tx=0x7f234802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27905757cc6bb1ac server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:44.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.398+0000 7f234f400640 1 -- 172.21.15.60:0/941190603 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2348039070 con 0x7f235015c9b0 2024-04-14T23:48:44.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.398+0000 7f234f400640 1 -- 172.21.15.60:0/941190603 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f234802cd00 con 0x7f235015c9b0 2024-04-14T23:48:44.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.398+0000 7f234f400640 1 -- 172.21.15.60:0/941190603 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2348034660 con 0x7f235015c9b0 2024-04-14T23:48:44.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.398+0000 7f2357800640 1 -- 172.21.15.60:0/941190603 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f235015c9b0 msgr2=0x7f23500c70b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:44.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.398+0000 7f2357800640 1 --2- 172.21.15.60:0/941190603 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f235015c9b0 0x7f23500c70b0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2350070e80 tx=0x7f234802bf50 comp rx=0 tx=0).stop 2024-04-14T23:48:44.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.399+0000 7f2357800640 1 -- 172.21.15.60:0/941190603 shutdown_connections 2024-04-14T23:48:44.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.399+0000 7f2357800640 1 --2- 172.21.15.60:0/941190603 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f235015c9b0 0x7f23500c70b0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:44.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.399+0000 7f2357800640 1 --2- 172.21.15.60:0/941190603 >> v2:172.21.15.186:3300/0 conn(0x7f23500c63c0 0x7f23500c6b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:44.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.399+0000 7f2357800640 1 -- 172.21.15.60:0/941190603 >> 172.21.15.60:0/941190603 conn(0x7f23500cc170 msgr2=0x7f23500cc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:44.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.399+0000 7f2357800640 1 -- 172.21.15.60:0/941190603 shutdown_connections 2024-04-14T23:48:44.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.399+0000 7f2357800640 1 -- 172.21.15.60:0/941190603 wait complete. 2024-04-14T23:48:44.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.401+0000 7f2357800640 1 Processor -- start 2024-04-14T23:48:44.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.401+0000 7f2357800640 1 -- start start 2024-04-14T23:48:44.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.402+0000 7f2357800640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23500c63c0 0x7f23500cb910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:44.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.402+0000 7f2357800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2350152cf0 0x7f23501f11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:44.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.402+0000 7f2357800640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f235015c9b0 0x7f23501414b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:44.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.402+0000 7f2357800640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f23500c88f0 con 0x7f235015c9b0 2024-04-14T23:48:44.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.402+0000 7f2357800640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f23500c8610 con 0x7f2350152cf0 2024-04-14T23:48:44.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.402+0000 7f2357800640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f2350159450 con 0x7f23500c63c0 2024-04-14T23:48:44.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.402+0000 7f2355800640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f235015c9b0 0x7f23501414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:44.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.402+0000 7f2355800640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f235015c9b0 0x7f23501414b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53862/0 (socket says 172.21.15.60:53862) 2024-04-14T23:48:44.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.402+0000 7f2355800640 1 -- 172.21.15.60:0/581322515 learned_addr learned my addr 172.21.15.60:0/581322515 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:44.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.402+0000 7f2354e00640 1 --2- 172.21.15.60:0/581322515 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23500c63c0 0x7f23500cb910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:44.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.402+0000 7f234fe00640 1 --2- 172.21.15.60:0/581322515 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2350152cf0 0x7f23501f11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:44.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.402+0000 7f2355800640 1 -- 172.21.15.60:0/581322515 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23500c63c0 msgr2=0x7f23500cb910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:44.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.402+0000 7f2355800640 1 --2- 172.21.15.60:0/581322515 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23500c63c0 0x7f23500cb910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:44.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.402+0000 7f2355800640 1 -- 172.21.15.60:0/581322515 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2350152cf0 msgr2=0x7f23501f11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:44.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.402+0000 7f2355800640 1 --2- 172.21.15.60:0/581322515 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2350152cf0 0x7f23501f11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:44.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.402+0000 7f2355800640 1 -- 172.21.15.60:0/581322515 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f235014ee80 con 0x7f235015c9b0 2024-04-14T23:48:44.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.402+0000 7f2354e00640 1 --2- 172.21.15.60:0/581322515 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23500c63c0 0x7f23500cb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-14T23:48:44.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.403+0000 7f2355800640 1 --2- 172.21.15.60:0/581322515 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f235015c9b0 0x7f23501414b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f234802c560 tx=0x7f2348004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:44.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.403+0000 7f234cc00640 1 -- 172.21.15.60:0/581322515 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2348044020 con 0x7f235015c9b0 2024-04-14T23:48:44.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.403+0000 7f234cc00640 1 -- 172.21.15.60:0/581322515 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f234803f470 con 0x7f235015c9b0 2024-04-14T23:48:44.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.403+0000 7f234cc00640 1 -- 172.21.15.60:0/581322515 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2348039040 con 0x7f235015c9b0 2024-04-14T23:48:44.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.403+0000 7f2357800640 1 -- 172.21.15.60:0/581322515 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f235014f160 con 0x7f235015c9b0 2024-04-14T23:48:44.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.403+0000 7f2357800640 1 -- 172.21.15.60:0/581322515 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23501f4c40 con 0x7f235015c9b0 2024-04-14T23:48:44.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.405+0000 7f2357800640 1 -- 172.21.15.60:0/581322515 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23500c7860 con 0x7f235015c9b0 2024-04-14T23:48:44.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.405+0000 7f234cc00640 1 -- 172.21.15.60:0/581322515 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f2348047020 con 0x7f235015c9b0 2024-04-14T23:48:44.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.407+0000 7f234cc00640 1 --2- 172.21.15.60:0/581322515 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f233406b990 0x7f233406de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:44.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.407+0000 7f234cc00640 1 -- 172.21.15.60:0/581322515 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f23480ad260 con 0x7f235015c9b0 2024-04-14T23:48:44.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.407+0000 7f2354e00640 1 --2- 172.21.15.60:0/581322515 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f233406b990 0x7f233406de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:44.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.408+0000 7f2354e00640 1 --2- 172.21.15.60:0/581322515 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f233406b990 0x7f233406de50 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2340007fd0 tx=0x7f2340009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:44.416 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.411+0000 7f234cc00640 1 -- 172.21.15.60:0/581322515 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f23500c7860 con 0x7f235015c9b0 2024-04-14T23:48:44.658 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:44.652+0000 7f2357800640 1 -- 172.21.15.60:0/581322515 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f23501f4750 con 0x7f235015c9b0 2024-04-14T23:48:45.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.230+0000 7f234cc00640 1 -- 172.21.15.60:0/581322515 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v5) ==== 118+0+0 (secure 0 0 0) 0x7f234804e030 con 0x7f235015c9b0 2024-04-14T23:48:45.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.235+0000 7f2357800640 1 -- 172.21.15.60:0/581322515 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f233406b990 msgr2=0x7f233406de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:45.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.235+0000 7f2357800640 1 --2- 172.21.15.60:0/581322515 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f233406b990 0x7f233406de50 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2340007fd0 tx=0x7f2340009040 comp rx=0 tx=0).stop 2024-04-14T23:48:45.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.235+0000 7f2357800640 1 -- 172.21.15.60:0/581322515 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f235015c9b0 msgr2=0x7f23501414b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:45.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.235+0000 7f2357800640 1 --2- 172.21.15.60:0/581322515 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f235015c9b0 0x7f23501414b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f234802c560 tx=0x7f2348004a40 comp rx=0 tx=0).stop 2024-04-14T23:48:45.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.235+0000 7f2357800640 1 -- 172.21.15.60:0/581322515 shutdown_connections 2024-04-14T23:48:45.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.235+0000 7f2357800640 1 --2- 172.21.15.60:0/581322515 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f233406b990 0x7f233406de50 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:45.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.235+0000 7f2357800640 1 --2- 172.21.15.60:0/581322515 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f235015c9b0 0x7f23501414b0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:45.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.235+0000 7f2357800640 1 --2- 172.21.15.60:0/581322515 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2350152cf0 0x7f23501f11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:45.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.235+0000 7f2357800640 1 --2- 172.21.15.60:0/581322515 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23500c63c0 0x7f23500cb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:45.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.235+0000 7f2357800640 1 -- 172.21.15.60:0/581322515 >> 172.21.15.60:0/581322515 conn(0x7f23500cc170 msgr2=0x7f23500c7150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:45.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.235+0000 7f2357800640 1 -- 172.21.15.60:0/581322515 shutdown_connections 2024-04-14T23:48:45.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.235+0000 7f2357800640 1 -- 172.21.15.60:0/581322515 wait complete. 2024-04-14T23:48:45.261 DEBUG:tasks.cephfs.filesystem:fs_config: {'session_timeout': 300} 2024-04-14T23:48:45.262 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-14T23:48:45.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.483+0000 7f6e59600640 1 Processor -- start 2024-04-14T23:48:45.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.484+0000 7f6e59600640 1 -- start start 2024-04-14T23:48:45.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.485+0000 7f6e59600640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6e5415b740 0x7f6e5406eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:45.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.485+0000 7f6e59600640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6e5406f0a0 0x7f6e540c6d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:45.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.485+0000 7f6e59600640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e5415bb20 con 0x7f6e540c72b0 2024-04-14T23:48:45.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.485+0000 7f6e59600640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f6e54070d10 con 0x7f6e5415b740 2024-04-14T23:48:45.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.485+0000 7f6e59600640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f6e540c7740 con 0x7f6e5406f0a0 2024-04-14T23:48:45.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.486+0000 7f6e53e00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6e5415b740 0x7f6e5406eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:45.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.486+0000 7f6e53400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6e5406f0a0 0x7f6e540c6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:45.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.486+0000 7f6e53400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6e5406f0a0 0x7f6e540c6d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.60:48370/0 (socket says 172.21.15.60:48370) 2024-04-14T23:48:45.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.486+0000 7f6e53e00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6e5415b740 0x7f6e5406eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:43296/0 (socket says 172.21.15.60:43296) 2024-04-14T23:48:45.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.486+0000 7f6e53400640 1 -- 172.21.15.60:0/3327201823 learned_addr learned my addr 172.21.15.60:0/3327201823 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:45.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.486+0000 7f6e58c00640 1 --1- 172.21.15.60:0/3327201823 >> v1:172.21.15.60:6789/0 conn(0x7f6e540c72b0 0x7f6e540711a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:43628/0 (socket says 172.21.15.60:43628) 2024-04-14T23:48:45.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.486+0000 7f6e53400640 1 -- 172.21.15.60:0/3327201823 >> v2:172.21.15.186:3300/0 conn(0x7f6e5415b740 msgr2=0x7f6e5406eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:45.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.486+0000 7f6e53400640 1 --2- 172.21.15.60:0/3327201823 >> v2:172.21.15.186:3300/0 conn(0x7f6e5415b740 0x7f6e5406eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:45.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.486+0000 7f6e53400640 1 -- 172.21.15.60:0/3327201823 >> v1:172.21.15.60:6789/0 conn(0x7f6e540c72b0 legacy=0x7f6e540711a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:45.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.486+0000 7f6e53400640 1 -- 172.21.15.60:0/3327201823 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e5415e310 con 0x7f6e5406f0a0 2024-04-14T23:48:45.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.486+0000 7f6e53e00640 1 --2- 172.21.15.60:0/3327201823 >> v2:172.21.15.186:3300/0 conn(0x7f6e5415b740 0x7f6e5406eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:48:45.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.487+0000 7f6e53400640 1 --2- 172.21.15.60:0/3327201823 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6e5406f0a0 0x7f6e540c6d70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6e40009870 tx=0x7f6e4002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97cef4e0492411a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:45.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.488+0000 7f6e52a00640 1 -- 172.21.15.60:0/3327201823 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e40034070 con 0x7f6e5406f0a0 2024-04-14T23:48:45.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.488+0000 7f6e52a00640 1 -- 172.21.15.60:0/3327201823 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e40030840 con 0x7f6e5406f0a0 2024-04-14T23:48:45.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.488+0000 7f6e52a00640 1 -- 172.21.15.60:0/3327201823 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e40030c10 con 0x7f6e5406f0a0 2024-04-14T23:48:45.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.488+0000 7f6e59600640 1 -- 172.21.15.60:0/3327201823 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6e5406f0a0 msgr2=0x7f6e540c6d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:45.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.488+0000 7f6e59600640 1 --2- 172.21.15.60:0/3327201823 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6e5406f0a0 0x7f6e540c6d70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6e40009870 tx=0x7f6e4002b980 comp rx=0 tx=0).stop 2024-04-14T23:48:45.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.488+0000 7f6e59600640 1 -- 172.21.15.60:0/3327201823 shutdown_connections 2024-04-14T23:48:45.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.488+0000 7f6e59600640 1 --2- 172.21.15.60:0/3327201823 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6e5406f0a0 0x7f6e540c6d70 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:45.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.488+0000 7f6e59600640 1 --2- 172.21.15.60:0/3327201823 >> v2:172.21.15.186:3300/0 conn(0x7f6e5415b740 0x7f6e5406eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:45.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.488+0000 7f6e59600640 1 -- 172.21.15.60:0/3327201823 >> 172.21.15.60:0/3327201823 conn(0x7f6e540cc2f0 msgr2=0x7f6e540cc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:45.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.489+0000 7f6e59600640 1 -- 172.21.15.60:0/3327201823 shutdown_connections 2024-04-14T23:48:45.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.489+0000 7f6e59600640 1 -- 172.21.15.60:0/3327201823 wait complete. 2024-04-14T23:48:45.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.490+0000 7f6e59600640 1 Processor -- start 2024-04-14T23:48:45.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.490+0000 7f6e59600640 1 -- start start 2024-04-14T23:48:45.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.491+0000 7f6e59600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6e5406f0a0 0x7f6e541f1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:45.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.491+0000 7f6e59600640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6e540c72b0 0x7f6e541579e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:45.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.491+0000 7f6e59600640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e5415b740 0x7f6e54157f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:45.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.491+0000 7f6e59600640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f6e5415ebb0 con 0x7f6e5415b740 2024-04-14T23:48:45.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.491+0000 7f6e59600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6e5415e8d0 con 0x7f6e5406f0a0 2024-04-14T23:48:45.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.491+0000 7f6e59600640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f6e5415ee90 con 0x7f6e540c72b0 2024-04-14T23:48:45.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.491+0000 7f6e58c00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e5415b740 0x7f6e54157f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:45.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.492+0000 7f6e53e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6e5406f0a0 0x7f6e541f1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:45.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.492+0000 7f6e53400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6e540c72b0 0x7f6e541579e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:45.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.492+0000 7f6e53e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6e5406f0a0 0x7f6e541f1790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:43304/0 (socket says 172.21.15.60:43304) 2024-04-14T23:48:45.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.492+0000 7f6e53e00640 1 -- 172.21.15.60:0/2770909611 learned_addr learned my addr 172.21.15.60:0/2770909611 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:45.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.492+0000 7f6e53400640 1 -- 172.21.15.60:0/2770909611 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6e5406f0a0 msgr2=0x7f6e541f1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:45.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.492+0000 7f6e53400640 1 --2- 172.21.15.60:0/2770909611 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6e5406f0a0 0x7f6e541f1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:45.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.492+0000 7f6e53400640 1 -- 172.21.15.60:0/2770909611 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e5415b740 msgr2=0x7f6e54157f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:45.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.492+0000 7f6e53400640 1 --2- 172.21.15.60:0/2770909611 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e5415b740 0x7f6e54157f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:45.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.492+0000 7f6e53400640 1 -- 172.21.15.60:0/2770909611 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e54158460 con 0x7f6e540c72b0 2024-04-14T23:48:45.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.492+0000 7f6e53e00640 1 --2- 172.21.15.60:0/2770909611 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6e5406f0a0 0x7f6e541f1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:48:45.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.492+0000 7f6e53400640 1 --2- 172.21.15.60:0/2770909611 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6e540c72b0 0x7f6e541579e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6e540c38c0 tx=0x7f6e40004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:45.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.493+0000 7f6e4be00640 1 -- 172.21.15.60:0/2770909611 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e40048070 con 0x7f6e540c72b0 2024-04-14T23:48:45.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.493+0000 7f6e4be00640 1 -- 172.21.15.60:0/2770909611 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e40030640 con 0x7f6e540c72b0 2024-04-14T23:48:45.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.493+0000 7f6e4be00640 1 -- 172.21.15.60:0/2770909611 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e40034040 con 0x7f6e540c72b0 2024-04-14T23:48:45.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.493+0000 7f6e59600640 1 -- 172.21.15.60:0/2770909611 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e541e5aa0 con 0x7f6e540c72b0 2024-04-14T23:48:45.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.493+0000 7f6e59600640 1 -- 172.21.15.60:0/2770909611 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e541e3fa0 con 0x7f6e540c72b0 2024-04-14T23:48:45.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.494+0000 7f6e59600640 1 -- 172.21.15.60:0/2770909611 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e5404f540 con 0x7f6e540c72b0 2024-04-14T23:48:45.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.495+0000 7f6e4be00640 1 -- 172.21.15.60:0/2770909611 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f6e4004b070 con 0x7f6e540c72b0 2024-04-14T23:48:45.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.496+0000 7f6e4be00640 1 --2- 172.21.15.60:0/2770909611 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f6e3406b990 0x7f6e3406de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:45.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.496+0000 7f6e4be00640 1 -- 172.21.15.60:0/2770909611 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f6e400ac880 con 0x7f6e540c72b0 2024-04-14T23:48:45.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.496+0000 7f6e53e00640 1 --2- 172.21.15.60:0/2770909611 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f6e3406b990 0x7f6e3406de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:45.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.497+0000 7f6e53e00640 1 --2- 172.21.15.60:0/2770909611 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f6e3406b990 0x7f6e3406de50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6e440095e0 tx=0x7f6e44009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:45.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.501+0000 7f6e4be00640 1 -- 172.21.15.60:0/2770909611 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6e40030810 con 0x7f6e540c72b0 2024-04-14T23:48:45.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:45.747+0000 7f6e59600640 1 -- 172.21.15.60:0/2770909611 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f6e540c78f0 con 0x7f6e540c72b0 2024-04-14T23:48:46.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.243+0000 7f6e4be00640 1 -- 172.21.15.60:0/2770909611 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v6) ==== 122+0+0 (secure 0 0 0) 0x7f6e4007aec0 con 0x7f6e540c72b0 2024-04-14T23:48:46.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.247+0000 7f6e59600640 1 -- 172.21.15.60:0/2770909611 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f6e3406b990 msgr2=0x7f6e3406de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:46.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.247+0000 7f6e59600640 1 --2- 172.21.15.60:0/2770909611 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f6e3406b990 0x7f6e3406de50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6e440095e0 tx=0x7f6e44009190 comp rx=0 tx=0).stop 2024-04-14T23:48:46.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.247+0000 7f6e59600640 1 -- 172.21.15.60:0/2770909611 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6e540c72b0 msgr2=0x7f6e541579e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:46.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.247+0000 7f6e59600640 1 --2- 172.21.15.60:0/2770909611 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6e540c72b0 0x7f6e541579e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6e540c38c0 tx=0x7f6e40004730 comp rx=0 tx=0).stop 2024-04-14T23:48:46.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.247+0000 7f6e59600640 1 -- 172.21.15.60:0/2770909611 shutdown_connections 2024-04-14T23:48:46.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.247+0000 7f6e59600640 1 --2- 172.21.15.60:0/2770909611 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f6e3406b990 0x7f6e3406de50 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:46.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.247+0000 7f6e59600640 1 --2- 172.21.15.60:0/2770909611 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e5415b740 0x7f6e54157f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:46.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.247+0000 7f6e59600640 1 --2- 172.21.15.60:0/2770909611 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6e540c72b0 0x7f6e541579e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:46.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.247+0000 7f6e59600640 1 --2- 172.21.15.60:0/2770909611 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6e5406f0a0 0x7f6e541f1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:46.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.247+0000 7f6e59600640 1 -- 172.21.15.60:0/2770909611 >> 172.21.15.60:0/2770909611 conn(0x7f6e540cc2f0 msgr2=0x7f6e540c7b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:46.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.248+0000 7f6e59600640 1 -- 172.21.15.60:0/2770909611 shutdown_connections 2024-04-14T23:48:46.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.248+0000 7f6e59600640 1 -- 172.21.15.60:0/2770909611 wait complete. 2024-04-14T23:48:46.277 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-04-14T23:48:46.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.484+0000 7f828be00640 1 Processor -- start 2024-04-14T23:48:46.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.484+0000 7f828be00640 1 -- start start 2024-04-14T23:48:46.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.486+0000 7f828be00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f828c0c5d30 0x7f828c0c6870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:46.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.486+0000 7f828be00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f828c0c6110 0x7f828c158270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:46.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.486+0000 7f828be00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f828c0c6db0 con 0x7f828c073570 2024-04-14T23:48:46.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.486+0000 7f828be00640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f828c0730e0 con 0x7f828c0c5d30 2024-04-14T23:48:46.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.486+0000 7f828be00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f828c06b820 con 0x7f828c0c6110 2024-04-14T23:48:46.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.486+0000 7f828aa00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f828c0c5d30 0x7f828c0c6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:46.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.486+0000 7f828aa00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f828c0c5d30 0x7f828c0c6870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53882/0 (socket says 172.21.15.60:53882) 2024-04-14T23:48:46.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.486+0000 7f828aa00640 1 -- 172.21.15.60:0/3476845804 learned_addr learned my addr 172.21.15.60:0/3476845804 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:46.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.486+0000 7f828a000640 1 --2- 172.21.15.60:0/3476845804 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f828c0c6110 0x7f828c158270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:46.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.487+0000 7f828a000640 1 -- 172.21.15.60:0/3476845804 >> v2:172.21.15.60:3300/0 conn(0x7f828c0c5d30 msgr2=0x7f828c0c6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:46.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.487+0000 7f828a000640 1 --2- 172.21.15.60:0/3476845804 >> v2:172.21.15.60:3300/0 conn(0x7f828c0c5d30 0x7f828c0c6870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:46.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.487+0000 7f828a000640 1 -- 172.21.15.60:0/3476845804 >> v1:172.21.15.60:6789/0 conn(0x7f828c073570 legacy=0x7f828c15afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:46.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.487+0000 7f828a000640 1 -- 172.21.15.60:0/3476845804 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f828c06b260 con 0x7f828c0c6110 2024-04-14T23:48:46.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.487+0000 7f828aa00640 1 --2- 172.21.15.60:0/3476845804 >> v2:172.21.15.60:3300/0 conn(0x7f828c0c5d30 0x7f828c0c6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:48:46.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.488+0000 7f828a000640 1 --2- 172.21.15.60:0/3476845804 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f828c0c6110 0x7f828c158270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f828c073260 tx=0x7f827802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e28f6840dfe24ed server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:46.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.488+0000 7f8289600640 1 -- 172.21.15.60:0/3476845804 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8278035020 con 0x7f828c0c6110 2024-04-14T23:48:46.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.488+0000 7f8289600640 1 -- 172.21.15.60:0/3476845804 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8278031890 con 0x7f828c0c6110 2024-04-14T23:48:46.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.488+0000 7f8289600640 1 -- 172.21.15.60:0/3476845804 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8278031c60 con 0x7f828c0c6110 2024-04-14T23:48:46.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.489+0000 7f828be00640 1 -- 172.21.15.60:0/3476845804 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f828c0c6110 msgr2=0x7f828c158270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:46.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.489+0000 7f828be00640 1 --2- 172.21.15.60:0/3476845804 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f828c0c6110 0x7f828c158270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f828c073260 tx=0x7f827802b930 comp rx=0 tx=0).stop 2024-04-14T23:48:46.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.489+0000 7f828be00640 1 -- 172.21.15.60:0/3476845804 shutdown_connections 2024-04-14T23:48:46.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.489+0000 7f828be00640 1 --2- 172.21.15.60:0/3476845804 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f828c0c6110 0x7f828c158270 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:46.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.489+0000 7f828be00640 1 --2- 172.21.15.60:0/3476845804 >> v2:172.21.15.60:3300/0 conn(0x7f828c0c5d30 0x7f828c0c6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:46.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.489+0000 7f828be00640 1 -- 172.21.15.60:0/3476845804 >> 172.21.15.60:0/3476845804 conn(0x7f828c06e7b0 msgr2=0x7f828c06eb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:46.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.489+0000 7f828be00640 1 -- 172.21.15.60:0/3476845804 shutdown_connections 2024-04-14T23:48:46.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.489+0000 7f828be00640 1 -- 172.21.15.60:0/3476845804 wait complete. 2024-04-14T23:48:46.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.491+0000 7f828be00640 1 Processor -- start 2024-04-14T23:48:46.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.491+0000 7f828be00640 1 -- start start 2024-04-14T23:48:46.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.492+0000 7f828be00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f828c073570 0x7f828c159f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:46.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.492+0000 7f828be00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f828c0c5d30 0x7f828c15a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:46.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.492+0000 7f828be00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f828c0c6110 0x7f828c0c24a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:46.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.492+0000 7f828be00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f828c072d70 con 0x7f828c073570 2024-04-14T23:48:46.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.492+0000 7f828be00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f828c072a90 con 0x7f828c0c5d30 2024-04-14T23:48:46.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.492+0000 7f828be00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f828c0717f0 con 0x7f828c0c6110 2024-04-14T23:48:46.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.492+0000 7f828aa00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f828c073570 0x7f828c159f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:46.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.492+0000 7f828aa00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f828c073570 0x7f828c159f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53890/0 (socket says 172.21.15.60:53890) 2024-04-14T23:48:46.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.492+0000 7f828aa00640 1 -- 172.21.15.60:0/3049936633 learned_addr learned my addr 172.21.15.60:0/3049936633 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:46.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.492+0000 7f828a000640 1 --2- 172.21.15.60:0/3049936633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f828c0c5d30 0x7f828c15a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:46.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.492+0000 7f828b400640 1 --2- 172.21.15.60:0/3049936633 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f828c0c6110 0x7f828c0c24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:46.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.493+0000 7f828aa00640 1 -- 172.21.15.60:0/3049936633 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f828c0c6110 msgr2=0x7f828c0c24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:46.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.493+0000 7f828aa00640 1 --2- 172.21.15.60:0/3049936633 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f828c0c6110 0x7f828c0c24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:46.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.493+0000 7f828aa00640 1 -- 172.21.15.60:0/3049936633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f828c0c5d30 msgr2=0x7f828c15a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:46.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.493+0000 7f828aa00640 1 --2- 172.21.15.60:0/3049936633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f828c0c5d30 0x7f828c15a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:46.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.493+0000 7f828aa00640 1 -- 172.21.15.60:0/3049936633 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f828c0b30f0 con 0x7f828c073570 2024-04-14T23:48:46.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.493+0000 7f828aa00640 1 --2- 172.21.15.60:0/3049936633 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f828c073570 0x7f828c159f80 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f827c00e9d0 tx=0x7f827c00ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:46.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.493+0000 7f8282a00640 1 -- 172.21.15.60:0/3049936633 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f827c017070 con 0x7f828c073570 2024-04-14T23:48:46.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.494+0000 7f8282a00640 1 -- 172.21.15.60:0/3049936633 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f827c00ca60 con 0x7f828c073570 2024-04-14T23:48:46.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.494+0000 7f828be00640 1 -- 172.21.15.60:0/3049936633 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f828c1591c0 con 0x7f828c073570 2024-04-14T23:48:46.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.494+0000 7f8282a00640 1 -- 172.21.15.60:0/3049936633 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f827c00ce10 con 0x7f828c073570 2024-04-14T23:48:46.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.494+0000 7f828be00640 1 -- 172.21.15.60:0/3049936633 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f828c159780 con 0x7f828c073570 2024-04-14T23:48:46.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.495+0000 7f828be00640 1 -- 172.21.15.60:0/3049936633 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f828c04f540 con 0x7f828c073570 2024-04-14T23:48:46.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.496+0000 7f8282a00640 1 -- 172.21.15.60:0/3049936633 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f827c01a070 con 0x7f828c073570 2024-04-14T23:48:46.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.497+0000 7f8282a00640 1 --2- 172.21.15.60:0/3049936633 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f826c06b990 0x7f826c06de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:46.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.497+0000 7f8282a00640 1 -- 172.21.15.60:0/3049936633 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f827c08aba0 con 0x7f828c073570 2024-04-14T23:48:46.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.497+0000 7f828a000640 1 --2- 172.21.15.60:0/3049936633 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f826c06b990 0x7f826c06de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:46.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.498+0000 7f828a000640 1 --2- 172.21.15.60:0/3049936633 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f826c06b990 0x7f826c06de50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8278036000 tx=0x7f827803c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:46.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.502+0000 7f8282a00640 1 -- 172.21.15.60:0/3049936633 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f827c059040 con 0x7f828c073570 2024-04-14T23:48:46.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:46.742+0000 7f828be00640 1 -- 172.21.15.60:0/3049936633 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f828c072070 con 0x7f828c073570 2024-04-14T23:48:47.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.250+0000 7f8282a00640 1 -- 172.21.15.60:0/3049936633 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v7) ==== 115+0+0 (secure 0 0 0) 0x7f827c05d0e0 con 0x7f828c073570 2024-04-14T23:48:47.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.253+0000 7f828be00640 1 -- 172.21.15.60:0/3049936633 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f826c06b990 msgr2=0x7f826c06de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:47.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.253+0000 7f828be00640 1 --2- 172.21.15.60:0/3049936633 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f826c06b990 0x7f826c06de50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8278036000 tx=0x7f827803c040 comp rx=0 tx=0).stop 2024-04-14T23:48:47.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.253+0000 7f828be00640 1 -- 172.21.15.60:0/3049936633 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f828c073570 msgr2=0x7f828c159f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:47.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.253+0000 7f828be00640 1 --2- 172.21.15.60:0/3049936633 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f828c073570 0x7f828c159f80 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f827c00e9d0 tx=0x7f827c00ee90 comp rx=0 tx=0).stop 2024-04-14T23:48:47.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.254+0000 7f828be00640 1 -- 172.21.15.60:0/3049936633 shutdown_connections 2024-04-14T23:48:47.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.254+0000 7f828be00640 1 --2- 172.21.15.60:0/3049936633 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f826c06b990 0x7f826c06de50 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:47.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.254+0000 7f828be00640 1 --2- 172.21.15.60:0/3049936633 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f828c0c6110 0x7f828c0c24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:47.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.254+0000 7f828be00640 1 --2- 172.21.15.60:0/3049936633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f828c0c5d30 0x7f828c15a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:47.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.254+0000 7f828be00640 1 --2- 172.21.15.60:0/3049936633 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f828c073570 0x7f828c159f80 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:47.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.254+0000 7f828be00640 1 -- 172.21.15.60:0/3049936633 >> 172.21.15.60:0/3049936633 conn(0x7f828c06e7b0 msgr2=0x7f828c15b6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:47.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.254+0000 7f828be00640 1 -- 172.21.15.60:0/3049936633 shutdown_connections 2024-04-14T23:48:47.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.254+0000 7f828be00640 1 -- 172.21.15.60:0/3049936633 wait complete. 2024-04-14T23:48:47.282 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-14T23:48:47.283 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:48:47.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.499+0000 7fe686600640 1 Processor -- start 2024-04-14T23:48:47.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.500+0000 7fe686600640 1 -- start start 2024-04-14T23:48:47.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.501+0000 7fe686600640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe6800c6950 0x7fe68006b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:47.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.501+0000 7fe686600640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe68015ba70 0x7fe680070d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:47.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.501+0000 7fe686600640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe68006b850 con 0x7fe6800c7080 2024-04-14T23:48:47.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.501+0000 7fe686600640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fe68006c740 con 0x7fe6800c6950 2024-04-14T23:48:47.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.501+0000 7fe686600640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fe68006bb30 con 0x7fe68015ba70 2024-04-14T23:48:47.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.502+0000 7fe685200640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fe6800c7080 0x7fe68015b360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.60:56770/0 (socket says 172.21.15.60:56770) 2024-04-14T23:48:47.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.502+0000 7fe685200640 1 -- 172.21.15.60:0/2754221969 learned_addr learned my addr 172.21.15.60:0/2754221969 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:48:47.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.502+0000 7fe685c00640 1 --2- 172.21.15.60:0/2754221969 >> v2:172.21.15.186:3300/0 conn(0x7fe68015ba70 0x7fe680070d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:47.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.502+0000 7fe677e00640 1 --2- 172.21.15.60:0/2754221969 >> v2:172.21.15.60:3300/0 conn(0x7fe6800c6950 0x7fe68006b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:47.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.502+0000 7fe67fe00640 1 -- 172.21.15.60:0/2754221969 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3024551049 0 0) 0x7fe68006b850 con 0x7fe6800c7080 2024-04-14T23:48:47.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.502+0000 7fe67fe00640 1 -- 172.21.15.60:0/2754221969 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe660003520 con 0x7fe6800c7080 2024-04-14T23:48:47.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.502+0000 7fe685c00640 1 -- 172.21.15.60:0/2754221969 >> v2:172.21.15.60:3300/0 conn(0x7fe6800c6950 msgr2=0x7fe68006b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:47.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.502+0000 7fe685c00640 1 --2- 172.21.15.60:0/2754221969 >> v2:172.21.15.60:3300/0 conn(0x7fe6800c6950 0x7fe68006b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:47.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.502+0000 7fe685c00640 1 -- 172.21.15.60:0/2754221969 >> v1:172.21.15.186:6789/0 conn(0x7fe6800c7080 legacy=0x7fe68015b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:47.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.502+0000 7fe685c00640 1 -- 172.21.15.60:0/2754221969 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe68006be10 con 0x7fe68015ba70 2024-04-14T23:48:47.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.502+0000 7fe677e00640 1 --2- 172.21.15.60:0/2754221969 >> v2:172.21.15.60:3300/0 conn(0x7fe6800c6950 0x7fe68006b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:48:47.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.503+0000 7fe685c00640 1 --2- 172.21.15.60:0/2754221969 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe68015ba70 0x7fe680070d50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe68006df40 tx=0x7fe67802bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=635a56b7727e640d server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:47.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.504+0000 7fe67fe00640 1 -- 172.21.15.60:0/2754221969 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe678039070 con 0x7fe68015ba70 2024-04-14T23:48:47.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.504+0000 7fe67fe00640 1 -- 172.21.15.60:0/2754221969 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe67802cd10 con 0x7fe68015ba70 2024-04-14T23:48:47.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.504+0000 7fe67fe00640 1 -- 172.21.15.60:0/2754221969 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe678034660 con 0x7fe68015ba70 2024-04-14T23:48:47.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.504+0000 7fe686600640 1 -- 172.21.15.60:0/2754221969 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe68015ba70 msgr2=0x7fe680070d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:47.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.504+0000 7fe686600640 1 --2- 172.21.15.60:0/2754221969 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe68015ba70 0x7fe680070d50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe68006df40 tx=0x7fe67802bf60 comp rx=0 tx=0).stop 2024-04-14T23:48:47.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.504+0000 7fe686600640 1 -- 172.21.15.60:0/2754221969 shutdown_connections 2024-04-14T23:48:47.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.504+0000 7fe686600640 1 --2- 172.21.15.60:0/2754221969 >> v2:172.21.15.60:3300/0 conn(0x7fe6800c6950 0x7fe68006b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:47.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.504+0000 7fe686600640 1 --2- 172.21.15.60:0/2754221969 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe68015ba70 0x7fe680070d50 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:47.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.504+0000 7fe686600640 1 -- 172.21.15.60:0/2754221969 >> 172.21.15.60:0/2754221969 conn(0x7fe6800c8b50 msgr2=0x7fe6800c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:47.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.504+0000 7fe686600640 1 -- 172.21.15.60:0/2754221969 shutdown_connections 2024-04-14T23:48:47.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.504+0000 7fe686600640 1 -- 172.21.15.60:0/2754221969 wait complete. 2024-04-14T23:48:47.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.506+0000 7fe686600640 1 Processor -- start 2024-04-14T23:48:47.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.506+0000 7fe686600640 1 -- start start 2024-04-14T23:48:47.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.507+0000 7fe686600640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe6800c6950 0x7fe6801557a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:47.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.507+0000 7fe686600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe6800c7080 0x7fe6800b1410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:47.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.507+0000 7fe686600640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe68015ba70 0x7fe6801f3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:47.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.507+0000 7fe686600640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fe6800c7460 con 0x7fe68015ba70 2024-04-14T23:48:47.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.507+0000 7fe686600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe68006c3d0 con 0x7fe6800c7080 2024-04-14T23:48:47.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.507+0000 7fe686600640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fe6800c75e0 con 0x7fe6800c6950 2024-04-14T23:48:47.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.507+0000 7fe685c00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe68015ba70 0x7fe6801f3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:47.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.507+0000 7fe685c00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe68015ba70 0x7fe6801f3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53900/0 (socket says 172.21.15.60:53900) 2024-04-14T23:48:47.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.507+0000 7fe685c00640 1 -- 172.21.15.60:0/2133839756 learned_addr learned my addr 172.21.15.60:0/2133839756 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:47.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.507+0000 7fe677e00640 1 --2- 172.21.15.60:0/2133839756 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe6800c7080 0x7fe6800b1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:47.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.507+0000 7fe685200640 1 --2- 172.21.15.60:0/2133839756 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe6800c6950 0x7fe6801557a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:47.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.508+0000 7fe685c00640 1 -- 172.21.15.60:0/2133839756 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe6800c6950 msgr2=0x7fe6801557a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:47.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.508+0000 7fe685c00640 1 --2- 172.21.15.60:0/2133839756 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe6800c6950 0x7fe6801557a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:47.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.508+0000 7fe685c00640 1 -- 172.21.15.60:0/2133839756 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe6800c7080 msgr2=0x7fe6800b1410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:47.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.508+0000 7fe685c00640 1 --2- 172.21.15.60:0/2133839756 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe6800c7080 0x7fe6800b1410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:47.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.508+0000 7fe685c00640 1 -- 172.21.15.60:0/2133839756 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6800bfdc0 con 0x7fe68015ba70 2024-04-14T23:48:47.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.508+0000 7fe677e00640 1 --2- 172.21.15.60:0/2133839756 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe6800c7080 0x7fe6800b1410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:48:47.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.508+0000 7fe685c00640 1 --2- 172.21.15.60:0/2133839756 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe68015ba70 0x7fe6801f3e40 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe678009e80 tx=0x7fe678004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:47.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.508+0000 7fe67d600640 1 -- 172.21.15.60:0/2133839756 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe678039050 con 0x7fe68015ba70 2024-04-14T23:48:47.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.509+0000 7fe67d600640 1 -- 172.21.15.60:0/2133839756 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe678045070 con 0x7fe68015ba70 2024-04-14T23:48:47.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.509+0000 7fe67d600640 1 -- 172.21.15.60:0/2133839756 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe678009450 con 0x7fe68015ba70 2024-04-14T23:48:47.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.509+0000 7fe686600640 1 -- 172.21.15.60:0/2133839756 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6800af900 con 0x7fe68015ba70 2024-04-14T23:48:47.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.509+0000 7fe686600640 1 -- 172.21.15.60:0/2133839756 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6800b0690 con 0x7fe68015ba70 2024-04-14T23:48:47.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.510+0000 7fe686600640 1 -- 172.21.15.60:0/2133839756 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6800af680 con 0x7fe68015ba70 2024-04-14T23:48:47.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.511+0000 7fe67d600640 1 -- 172.21.15.60:0/2133839756 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7fe67803e850 con 0x7fe68015ba70 2024-04-14T23:48:47.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.512+0000 7fe67d600640 1 --2- 172.21.15.60:0/2133839756 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fe65c06b990 0x7fe65c06de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:47.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.512+0000 7fe67d600640 1 -- 172.21.15.60:0/2133839756 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fe6780ad6c0 con 0x7fe68015ba70 2024-04-14T23:48:47.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.512+0000 7fe685200640 1 --2- 172.21.15.60:0/2133839756 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fe65c06b990 0x7fe65c06de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:47.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.513+0000 7fe685200640 1 --2- 172.21.15.60:0/2133839756 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fe65c06b990 0x7fe65c06de50 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe6680011e0 tx=0x7fe668007d30 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:47.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.516+0000 7fe67d600640 1 -- 172.21.15.60:0/2133839756 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe6800af680 con 0x7fe68015ba70 2024-04-14T23:48:47.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.756+0000 7fe686600640 1 -- 172.21.15.60:0/2133839756 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe68004f540 con 0x7fe68015ba70 2024-04-14T23:48:47.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.757+0000 7fe67d600640 1 -- 172.21.15.60:0/2133839756 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+6822 (secure 0 0 0) 0x7fe6800bfdc0 con 0x7fe68015ba70 2024-04-14T23:48:47.763 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:48:47.763 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/78303888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":78303888},{"type":"v1","addr":"172.21.15.60:6833","nonce":78303888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:48:46.744530+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"c","rank":0,"incarnation":4,"state":"up:creating","state_seq":1,"addr":"172.21.15.60:6835/2677133122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2677133122},{"type":"v1","addr":"172.21.15.60:6835","nonce":2677133122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-14T23:48:47.763 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 7 2024-04-14T23:48:47.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.760+0000 7fe686600640 1 -- 172.21.15.60:0/2133839756 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fe65c06b990 msgr2=0x7fe65c06de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:47.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.760+0000 7fe686600640 1 --2- 172.21.15.60:0/2133839756 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fe65c06b990 0x7fe65c06de50 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe6680011e0 tx=0x7fe668007d30 comp rx=0 tx=0).stop 2024-04-14T23:48:47.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.760+0000 7fe686600640 1 -- 172.21.15.60:0/2133839756 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe68015ba70 msgr2=0x7fe6801f3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:47.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.760+0000 7fe686600640 1 --2- 172.21.15.60:0/2133839756 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe68015ba70 0x7fe6801f3e40 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe678009e80 tx=0x7fe678004a40 comp rx=0 tx=0).stop 2024-04-14T23:48:47.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.761+0000 7fe686600640 1 -- 172.21.15.60:0/2133839756 shutdown_connections 2024-04-14T23:48:47.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.761+0000 7fe686600640 1 --2- 172.21.15.60:0/2133839756 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fe65c06b990 0x7fe65c06de50 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:47.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.761+0000 7fe686600640 1 --2- 172.21.15.60:0/2133839756 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe68015ba70 0x7fe6801f3e40 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:47.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.761+0000 7fe686600640 1 --2- 172.21.15.60:0/2133839756 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe6800c7080 0x7fe6800b1410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:47.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.761+0000 7fe686600640 1 --2- 172.21.15.60:0/2133839756 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe6800c6950 0x7fe6801557a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:47.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.761+0000 7fe686600640 1 -- 172.21.15.60:0/2133839756 >> 172.21.15.60:0/2133839756 conn(0x7fe6800c8b50 msgr2=0x7fe6800c8200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:47.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.761+0000 7fe686600640 1 -- 172.21.15.60:0/2133839756 shutdown_connections 2024-04-14T23:48:47.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:47.761+0000 7fe686600640 1 -- 172.21.15.60:0/2133839756 wait complete. 2024-04-14T23:48:47.785 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-14T23:48:48.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.001+0000 7f5e6a200640 1 Processor -- start 2024-04-14T23:48:48.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.001+0000 7f5e6a200640 1 -- start start 2024-04-14T23:48:48.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.003+0000 7f5e6a200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5e640b1e00 0x7f5e640c6950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:48.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.003+0000 7f5e6a200640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f5e64073320 0x7f5e640c7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:48.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.003+0000 7f5e6a200640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e64152fb0 con 0x7f5e640c84e0 2024-04-14T23:48:48.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.003+0000 7f5e6a200640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f5e640c6540 con 0x7f5e64073320 2024-04-14T23:48:48.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.003+0000 7f5e6a200640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f5e640b21e0 con 0x7f5e640b1e00 2024-04-14T23:48:48.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.003+0000 7f5e69800640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f5e640c84e0 0x7f5e640c3d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:43648/0 (socket says 172.21.15.60:43648) 2024-04-14T23:48:48.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.003+0000 7f5e69800640 1 -- 172.21.15.60:0/3322528781 learned_addr learned my addr 172.21.15.60:0/3322528781 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:48:48.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.003+0000 7f5e68e00640 1 --2- 172.21.15.60:0/3322528781 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5e640b1e00 0x7f5e640c6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:48.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.003+0000 7f5e63e00640 1 --2- 172.21.15.60:0/3322528781 >> v2:172.21.15.186:3300/0 conn(0x7f5e64073320 0x7f5e640c7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:48.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.004+0000 7f5e68e00640 1 -- 172.21.15.60:0/3322528781 >> v2:172.21.15.186:3300/0 conn(0x7f5e64073320 msgr2=0x7f5e640c7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:48.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.004+0000 7f5e68e00640 1 --2- 172.21.15.60:0/3322528781 >> v2:172.21.15.186:3300/0 conn(0x7f5e64073320 0x7f5e640c7fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:48.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.004+0000 7f5e68e00640 1 -- 172.21.15.60:0/3322528781 >> v1:172.21.15.60:6789/0 conn(0x7f5e640c84e0 legacy=0x7f5e640c3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:48.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.004+0000 7f5e68e00640 1 -- 172.21.15.60:0/3322528781 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e64159310 con 0x7f5e640b1e00 2024-04-14T23:48:48.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.004+0000 7f5e63400640 1 -- 172.21.15.60:0/3322528781 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3719845459 0 0) 0x7f5e64152fb0 con 0x7f5e640c84e0 2024-04-14T23:48:48.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.005+0000 7f5e68e00640 1 --2- 172.21.15.60:0/3322528781 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5e640b1e00 0x7f5e640c6950 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5e580097a0 tx=0x7f5e5802b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f2d1758d5fc0ed4 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:48.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.005+0000 7f5e63400640 1 -- 172.21.15.60:0/3322528781 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e58034070 con 0x7f5e640b1e00 2024-04-14T23:48:48.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.005+0000 7f5e63400640 1 -- 172.21.15.60:0/3322528781 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e58030610 con 0x7f5e640b1e00 2024-04-14T23:48:48.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.005+0000 7f5e63400640 1 -- 172.21.15.60:0/3322528781 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e580309e0 con 0x7f5e640b1e00 2024-04-14T23:48:48.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.006+0000 7f5e6a200640 1 -- 172.21.15.60:0/3322528781 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5e640b1e00 msgr2=0x7f5e640c6950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:48.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.006+0000 7f5e6a200640 1 --2- 172.21.15.60:0/3322528781 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5e640b1e00 0x7f5e640c6950 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5e580097a0 tx=0x7f5e5802b960 comp rx=0 tx=0).stop 2024-04-14T23:48:48.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.006+0000 7f5e6a200640 1 -- 172.21.15.60:0/3322528781 shutdown_connections 2024-04-14T23:48:48.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.006+0000 7f5e6a200640 1 --2- 172.21.15.60:0/3322528781 >> v2:172.21.15.186:3300/0 conn(0x7f5e64073320 0x7f5e640c7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:48.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.006+0000 7f5e6a200640 1 --2- 172.21.15.60:0/3322528781 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5e640b1e00 0x7f5e640c6950 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:48.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.006+0000 7f5e6a200640 1 -- 172.21.15.60:0/3322528781 >> 172.21.15.60:0/3322528781 conn(0x7f5e640c8b50 msgr2=0x7f5e640c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:48.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.006+0000 7f5e6a200640 1 -- 172.21.15.60:0/3322528781 shutdown_connections 2024-04-14T23:48:48.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.006+0000 7f5e6a200640 1 -- 172.21.15.60:0/3322528781 wait complete. 2024-04-14T23:48:48.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.008+0000 7f5e6a200640 1 Processor -- start 2024-04-14T23:48:48.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.009+0000 7f5e6a200640 1 -- start start 2024-04-14T23:48:48.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.010+0000 7f5e6a200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5e64073320 0x7f5e6414f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:48.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.010+0000 7f5e6a200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5e640b1e00 0x7f5e64155a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:48.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.010+0000 7f5e6a200640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e640c84e0 0x7f5e641eec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:48.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.010+0000 7f5e6a200640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f5e640c38b0 con 0x7f5e640c84e0 2024-04-14T23:48:48.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.010+0000 7f5e6a200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5e64159890 con 0x7f5e64073320 2024-04-14T23:48:48.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.010+0000 7f5e6a200640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f5e640c5f80 con 0x7f5e640b1e00 2024-04-14T23:48:48.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.010+0000 7f5e69800640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e640c84e0 0x7f5e641eec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:48.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.010+0000 7f5e68e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5e64073320 0x7f5e6414f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:48.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.010+0000 7f5e63e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5e640b1e00 0x7f5e64155a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:48.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.010+0000 7f5e69800640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e640c84e0 0x7f5e641eec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53908/0 (socket says 172.21.15.60:53908) 2024-04-14T23:48:48.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.010+0000 7f5e69800640 1 -- 172.21.15.60:0/2249177102 learned_addr learned my addr 172.21.15.60:0/2249177102 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:48.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.011+0000 7f5e68e00640 1 -- 172.21.15.60:0/2249177102 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5e640b1e00 msgr2=0x7f5e64155a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:48.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.011+0000 7f5e68e00640 1 --2- 172.21.15.60:0/2249177102 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5e640b1e00 0x7f5e64155a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:48.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.011+0000 7f5e68e00640 1 -- 172.21.15.60:0/2249177102 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e640c84e0 msgr2=0x7f5e641eec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:48.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.011+0000 7f5e68e00640 1 --2- 172.21.15.60:0/2249177102 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e640c84e0 0x7f5e641eec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:48.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.011+0000 7f5e68e00640 1 -- 172.21.15.60:0/2249177102 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e641f1f00 con 0x7f5e64073320 2024-04-14T23:48:48.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.011+0000 7f5e68e00640 1 --2- 172.21.15.60:0/2249177102 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5e64073320 0x7f5e6414f280 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5e58033000 tx=0x7f5e580046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:48.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.011+0000 7f5e60c00640 1 -- 172.21.15.60:0/2249177102 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e58034040 con 0x7f5e64073320 2024-04-14T23:48:48.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.011+0000 7f5e60c00640 1 -- 172.21.15.60:0/2249177102 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e58030410 con 0x7f5e64073320 2024-04-14T23:48:48.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.011+0000 7f5e60c00640 1 -- 172.21.15.60:0/2249177102 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e58006e10 con 0x7f5e64073320 2024-04-14T23:48:48.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.011+0000 7f5e6a200640 1 -- 172.21.15.60:0/2249177102 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e641f21e0 con 0x7f5e64073320 2024-04-14T23:48:48.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.012+0000 7f5e6a200640 1 -- 172.21.15.60:0/2249177102 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e641f27a0 con 0x7f5e64073320 2024-04-14T23:48:48.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.013+0000 7f5e6a200640 1 -- 172.21.15.60:0/2249177102 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e6404f540 con 0x7f5e64073320 2024-04-14T23:48:48.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.014+0000 7f5e60c00640 1 -- 172.21.15.60:0/2249177102 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f5e58049060 con 0x7f5e64073320 2024-04-14T23:48:48.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.014+0000 7f5e60c00640 1 --2- 172.21.15.60:0/2249177102 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f5e4406b990 0x7f5e4406de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:48.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.015+0000 7f5e60c00640 1 -- 172.21.15.60:0/2249177102 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f5e580abea0 con 0x7f5e64073320 2024-04-14T23:48:48.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.015+0000 7f5e63e00640 1 --2- 172.21.15.60:0/2249177102 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f5e4406b990 0x7f5e4406de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:48.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.015+0000 7f5e63e00640 1 --2- 172.21.15.60:0/2249177102 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f5e4406b990 0x7f5e4406de50 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5e641525d0 tx=0x7f5e50009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:48.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.019+0000 7f5e60c00640 1 -- 172.21.15.60:0/2249177102 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f5e5807aa20 con 0x7f5e64073320 2024-04-14T23:48:48.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.233+0000 7f5e6a200640 1 -- 172.21.15.60:0/2249177102 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5e640c99e0 con 0x7f5e64073320 2024-04-14T23:48:48.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.234+0000 7f5e60c00640 1 -- 172.21.15.60:0/2249177102 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16605 (secure 0 0 0) 0x7f5e5807a320 con 0x7f5e64073320 2024-04-14T23:48:48.241 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:48:48.241 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":21,"fsid":"5847db56-a76d-4486-b563-d83f1186af92","created":"2024-04-14T23:48:23.900003+0000","modified":"2024-04-14T23:48:44.219352+0000","last_up_change":"2024-04-14T23:48:34.203573+0000","last_in_change":"2024-04-14T23:48:26.345499+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-14T23:48:36.247372+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-14T23:48:40.274195+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-14T23:48:41.722183+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b2b2b755-7f42-4f9b-af9a-a178f7cc5a52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6817","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6818","nonce":2536528895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6822","nonce":2536528895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6827","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6828","nonce":2536528895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6825","nonce":2536528895}]},"public_addr":"172.21.15.60:6818/2536528895","cluster_addr":"172.21.15.60:6822/2536528895","heartbeat_back_addr":"172.21.15.60:6828/2536528895","heartbeat_front_addr":"172.21.15.60:6825/2536528895","state":["exists","up"]},{"osd":1,"uuid":"321418cd-5ab7-41a5-b9c8-b3fc12f68ebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6801","nonce":3115695676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6803","nonce":3115695676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6807","nonce":3115695676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6805","nonce":3115695676}]},"public_addr":"172.21.15.60:6801/3115695676","cluster_addr":"172.21.15.60:6803/3115695676","heartbeat_back_addr":"172.21.15.60:6807/3115695676","heartbeat_front_addr":"172.21.15.60:6805/3115695676","state":["exists","up"]},{"osd":2,"uuid":"07874e9f-3a38-41dc-be9f-2b75e84e53d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6819","nonce":3475168698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6824","nonce":3475168698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6831","nonce":3475168698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6829","nonce":3475168698}]},"public_addr":"172.21.15.60:6819/3475168698","cluster_addr":"172.21.15.60:6824/3475168698","heartbeat_back_addr":"172.21.15.60:6831/3475168698","heartbeat_front_addr":"172.21.15.60:6829/3475168698","state":["exists","up"]},{"osd":3,"uuid":"7e4ef6c9-b17a-47c6-a320-f5d16a6fe05e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6809","nonce":1730813674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6811","nonce":1730813674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6815","nonce":1730813674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6813","nonce":1730813674}]},"public_addr":"172.21.15.60:6809/1730813674","cluster_addr":"172.21.15.60:6811/1730813674","heartbeat_back_addr":"172.21.15.60:6815/1730813674","heartbeat_front_addr":"172.21.15.60:6813/1730813674","state":["exists","up"]},{"osd":4,"uuid":"759a3efc-dade-44c5-b57a-1623499aba50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6805","nonce":1464587038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6810","nonce":1464587038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6819","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6822","nonce":1464587038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6813","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6816","nonce":1464587038}]},"public_addr":"172.21.15.186:6805/1464587038","cluster_addr":"172.21.15.186:6810/1464587038","heartbeat_back_addr":"172.21.15.186:6822/1464587038","heartbeat_front_addr":"172.21.15.186:6816/1464587038","state":["exists","up"]},{"osd":5,"uuid":"24cd58c1-0cf1-4077-ab33-84c39c6907ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6801","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6803","nonce":1697098067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6809","nonce":1697098067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6820","nonce":1697098067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6815","nonce":1697098067}]},"public_addr":"172.21.15.186:6803/1697098067","cluster_addr":"172.21.15.186:6809/1697098067","heartbeat_back_addr":"172.21.15.186:6820/1697098067","heartbeat_front_addr":"172.21.15.186:6815/1697098067","state":["exists","up"]},{"osd":6,"uuid":"6add38f1-65e9-4b0d-927a-4ab77f5e2323","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6804","nonce":1837108559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6807","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6811","nonce":1837108559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6823","nonce":1837108559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6817","nonce":1837108559}]},"public_addr":"172.21.15.186:6804/1837108559","cluster_addr":"172.21.15.186:6811/1837108559","heartbeat_back_addr":"172.21.15.186:6823/1837108559","heartbeat_front_addr":"172.21.15.186:6817/1837108559","state":["exists","up"]},{"osd":7,"uuid":"a2326b6f-7601-4dda-a456-67c443ea5929","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6825","nonce":842537875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6827","nonce":842537875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6831","nonce":842537875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6829","nonce":842537875}]},"public_addr":"172.21.15.186:6825/842537875","cluster_addr":"172.21.15.186:6827/842537875","heartbeat_back_addr":"172.21.15.186:6831/842537875","heartbeat_front_addr":"172.21.15.186:6829/842537875","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.975583+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.976806+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.722725+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.763705+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.502140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.528972+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.517961+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.513350+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-14T23:48:48.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.237+0000 7f5e6a200640 1 -- 172.21.15.60:0/2249177102 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f5e4406b990 msgr2=0x7f5e4406de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:48.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.237+0000 7f5e6a200640 1 --2- 172.21.15.60:0/2249177102 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f5e4406b990 0x7f5e4406de50 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5e641525d0 tx=0x7f5e50009190 comp rx=0 tx=0).stop 2024-04-14T23:48:48.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.237+0000 7f5e6a200640 1 -- 172.21.15.60:0/2249177102 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5e64073320 msgr2=0x7f5e6414f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:48.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.237+0000 7f5e6a200640 1 --2- 172.21.15.60:0/2249177102 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5e64073320 0x7f5e6414f280 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5e58033000 tx=0x7f5e580046c0 comp rx=0 tx=0).stop 2024-04-14T23:48:48.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.237+0000 7f5e6a200640 1 -- 172.21.15.60:0/2249177102 shutdown_connections 2024-04-14T23:48:48.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.237+0000 7f5e6a200640 1 --2- 172.21.15.60:0/2249177102 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f5e4406b990 0x7f5e4406de50 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:48.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.237+0000 7f5e6a200640 1 --2- 172.21.15.60:0/2249177102 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e640c84e0 0x7f5e641eec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:48.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.237+0000 7f5e6a200640 1 --2- 172.21.15.60:0/2249177102 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5e640b1e00 0x7f5e64155a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:48.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.237+0000 7f5e6a200640 1 --2- 172.21.15.60:0/2249177102 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5e64073320 0x7f5e6414f280 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:48.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.237+0000 7f5e6a200640 1 -- 172.21.15.60:0/2249177102 >> 172.21.15.60:0/2249177102 conn(0x7f5e640c8b50 msgr2=0x7f5e640cb8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:48.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.237+0000 7f5e6a200640 1 -- 172.21.15.60:0/2249177102 shutdown_connections 2024-04-14T23:48:48.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.237+0000 7f5e6a200640 1 -- 172.21.15.60:0/2249177102 wait complete. 2024-04-14T23:48:48.262 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-14T23:48:48.263 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-14T23:48:48.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.460+0000 7efc8be00640 1 Processor -- start 2024-04-14T23:48:48.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.461+0000 7efc8be00640 1 -- start start 2024-04-14T23:48:48.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.462+0000 7efc8be00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7efc8c0b1e00 0x7efc8c0c6950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:48.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.462+0000 7efc8be00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7efc8c073320 0x7efc8c0c7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:48.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.462+0000 7efc8be00640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc8c152fb0 con 0x7efc8c0c84e0 2024-04-14T23:48:48.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.462+0000 7efc8be00640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7efc8c0c6540 con 0x7efc8c073320 2024-04-14T23:48:48.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.462+0000 7efc8be00640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7efc8c0b21e0 con 0x7efc8c0b1e00 2024-04-14T23:48:48.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.462+0000 7efc82000640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7efc8c073320 0x7efc8c0c7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:48.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.463+0000 7efc82000640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7efc8c073320 0x7efc8c0c7fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53910/0 (socket says 172.21.15.60:53910) 2024-04-14T23:48:48.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.463+0000 7efc82000640 1 -- 172.21.15.60:0/599344281 learned_addr learned my addr 172.21.15.60:0/599344281 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:48.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.463+0000 7efc8aa00640 1 --2- 172.21.15.60:0/599344281 >> v2:172.21.15.186:3300/0 conn(0x7efc8c0b1e00 0x7efc8c0c6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:48.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.463+0000 7efc8aa00640 1 -- 172.21.15.60:0/599344281 >> v2:172.21.15.60:3300/0 conn(0x7efc8c073320 msgr2=0x7efc8c0c7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:48.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.463+0000 7efc8aa00640 1 --2- 172.21.15.60:0/599344281 >> v2:172.21.15.60:3300/0 conn(0x7efc8c073320 0x7efc8c0c7fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:48.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.463+0000 7efc8aa00640 1 -- 172.21.15.60:0/599344281 >> v1:172.21.15.186:6789/0 conn(0x7efc8c0c84e0 legacy=0x7efc8c0c3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:48.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.463+0000 7efc8aa00640 1 -- 172.21.15.60:0/599344281 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc8c159310 con 0x7efc8c0b1e00 2024-04-14T23:48:48.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.463+0000 7efc82000640 1 --2- 172.21.15.60:0/599344281 >> v2:172.21.15.60:3300/0 conn(0x7efc8c073320 0x7efc8c0c7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:48:48.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.463+0000 7efc8a000640 1 -- 172.21.15.60:0/599344281 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2415953378 0 0) 0x7efc8c152fb0 con 0x7efc8c0c84e0 2024-04-14T23:48:48.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.464+0000 7efc8aa00640 1 --2- 172.21.15.60:0/599344281 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efc8c0b1e00 0x7efc8c0c6950 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7efc8c0c36e0 tx=0x7efc7c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=326e0365df8ae483 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:48.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.464+0000 7efc8a000640 1 -- 172.21.15.60:0/599344281 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc7c034070 con 0x7efc8c0b1e00 2024-04-14T23:48:48.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.464+0000 7efc8a000640 1 -- 172.21.15.60:0/599344281 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc7c030610 con 0x7efc8c0b1e00 2024-04-14T23:48:48.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.465+0000 7efc8a000640 1 -- 172.21.15.60:0/599344281 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc7c0309e0 con 0x7efc8c0b1e00 2024-04-14T23:48:48.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.465+0000 7efc8be00640 1 -- 172.21.15.60:0/599344281 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efc8c0b1e00 msgr2=0x7efc8c0c6950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:48.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.465+0000 7efc8be00640 1 --2- 172.21.15.60:0/599344281 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efc8c0b1e00 0x7efc8c0c6950 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7efc8c0c36e0 tx=0x7efc7c02b960 comp rx=0 tx=0).stop 2024-04-14T23:48:48.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.465+0000 7efc8be00640 1 -- 172.21.15.60:0/599344281 shutdown_connections 2024-04-14T23:48:48.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.465+0000 7efc8be00640 1 --2- 172.21.15.60:0/599344281 >> v2:172.21.15.60:3300/0 conn(0x7efc8c073320 0x7efc8c0c7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:48.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.465+0000 7efc8be00640 1 --2- 172.21.15.60:0/599344281 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efc8c0b1e00 0x7efc8c0c6950 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:48.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.465+0000 7efc8be00640 1 -- 172.21.15.60:0/599344281 >> 172.21.15.60:0/599344281 conn(0x7efc8c0c8b50 msgr2=0x7efc8c0c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:48.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.465+0000 7efc8be00640 1 -- 172.21.15.60:0/599344281 shutdown_connections 2024-04-14T23:48:48.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.465+0000 7efc8be00640 1 -- 172.21.15.60:0/599344281 wait complete. 2024-04-14T23:48:48.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.467+0000 7efc8be00640 1 Processor -- start 2024-04-14T23:48:48.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.467+0000 7efc8be00640 1 -- start start 2024-04-14T23:48:48.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.468+0000 7efc8be00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efc8c073320 0x7efc8c14fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:48.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.468+0000 7efc8be00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc8c0b1e00 0x7efc8c0c2e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:48.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.468+0000 7efc8be00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efc8c0c84e0 0x7efc8c06b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:48.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.468+0000 7efc8be00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7efc8c0c38b0 con 0x7efc8c0b1e00 2024-04-14T23:48:48.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.468+0000 7efc8be00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7efc8c159890 con 0x7efc8c0c84e0 2024-04-14T23:48:48.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.468+0000 7efc8be00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7efc8c0c5f80 con 0x7efc8c073320 2024-04-14T23:48:48.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.468+0000 7efc8aa00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efc8c073320 0x7efc8c14fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:48.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.468+0000 7efc82000640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc8c0b1e00 0x7efc8c0c2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:48.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.468+0000 7efc8b400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efc8c0c84e0 0x7efc8c06b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:48.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.468+0000 7efc8aa00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efc8c073320 0x7efc8c14fa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.60:48430/0 (socket says 172.21.15.60:48430) 2024-04-14T23:48:48.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.468+0000 7efc8aa00640 1 -- 172.21.15.60:0/2981591081 learned_addr learned my addr 172.21.15.60:0/2981591081 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:48.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.469+0000 7efc82000640 1 -- 172.21.15.60:0/2981591081 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efc8c073320 msgr2=0x7efc8c14fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:48.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.469+0000 7efc82000640 1 --2- 172.21.15.60:0/2981591081 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efc8c073320 0x7efc8c14fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:48.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.469+0000 7efc82000640 1 -- 172.21.15.60:0/2981591081 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efc8c0c84e0 msgr2=0x7efc8c06b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:48.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.469+0000 7efc82000640 1 --2- 172.21.15.60:0/2981591081 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efc8c0c84e0 0x7efc8c06b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:48.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.469+0000 7efc82000640 1 -- 172.21.15.60:0/2981591081 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc8c0c33c0 con 0x7efc8c0b1e00 2024-04-14T23:48:48.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.469+0000 7efc82000640 1 --2- 172.21.15.60:0/2981591081 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc8c0b1e00 0x7efc8c0c2e80 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7efc7400e890 tx=0x7efc7400ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:48.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.469+0000 7efc83400640 1 -- 172.21.15.60:0/2981591081 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc74004030 con 0x7efc8c0b1e00 2024-04-14T23:48:48.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.469+0000 7efc83400640 1 -- 172.21.15.60:0/2981591081 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc740041d0 con 0x7efc8c0b1e00 2024-04-14T23:48:48.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.469+0000 7efc83400640 1 -- 172.21.15.60:0/2981591081 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc7400c940 con 0x7efc8c0b1e00 2024-04-14T23:48:48.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.470+0000 7efc8be00640 1 -- 172.21.15.60:0/2981591081 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc8c150060 con 0x7efc8c0b1e00 2024-04-14T23:48:48.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.470+0000 7efc8be00640 1 -- 172.21.15.60:0/2981591081 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc8c0c0990 con 0x7efc8c0b1e00 2024-04-14T23:48:48.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.471+0000 7efc8be00640 1 -- 172.21.15.60:0/2981591081 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc8c04f540 con 0x7efc8c0b1e00 2024-04-14T23:48:48.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.472+0000 7efc83400640 1 -- 172.21.15.60:0/2981591081 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7efc7400cae0 con 0x7efc8c0b1e00 2024-04-14T23:48:48.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.473+0000 7efc83400640 1 --2- 172.21.15.60:0/2981591081 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7efc6806b9c0 0x7efc6806de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:48.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.473+0000 7efc83400640 1 -- 172.21.15.60:0/2981591081 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7efc7408ab10 con 0x7efc8c0b1e00 2024-04-14T23:48:48.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.473+0000 7efc8aa00640 1 --2- 172.21.15.60:0/2981591081 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7efc6806b9c0 0x7efc6806de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:48.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.474+0000 7efc8aa00640 1 --2- 172.21.15.60:0/2981591081 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7efc6806b9c0 0x7efc6806de80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7efc7c009560 tx=0x7efc7c03b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:48.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.477+0000 7efc83400640 1 -- 172.21.15.60:0/2981591081 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7efc74017020 con 0x7efc8c0b1e00 2024-04-14T23:48:48.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.676+0000 7efc8be00640 1 -- 172.21.15.60:0/2981591081 --> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7efc8c157290 con 0x7efc6806b9c0 2024-04-14T23:48:48.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.684+0000 7efc83400640 1 -- 172.21.15.60:0/2981591081 <== mgr.4100 v2:172.21.15.186:6832/714104835 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251890 (secure 0 0 0) 0x7efc8c157290 con 0x7efc6806b9c0 2024-04-14T23:48:48.694 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:48:48.696 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-04-14T23:48:48.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.690+0000 7efc8be00640 1 -- 172.21.15.60:0/2981591081 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7efc6806b9c0 msgr2=0x7efc6806de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:48.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.690+0000 7efc8be00640 1 --2- 172.21.15.60:0/2981591081 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7efc6806b9c0 0x7efc6806de80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7efc7c009560 tx=0x7efc7c03b040 comp rx=0 tx=0).stop 2024-04-14T23:48:48.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.690+0000 7efc8be00640 1 -- 172.21.15.60:0/2981591081 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc8c0b1e00 msgr2=0x7efc8c0c2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:48.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.690+0000 7efc8be00640 1 --2- 172.21.15.60:0/2981591081 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc8c0b1e00 0x7efc8c0c2e80 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7efc7400e890 tx=0x7efc7400ed50 comp rx=0 tx=0).stop 2024-04-14T23:48:48.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.691+0000 7efc8be00640 1 -- 172.21.15.60:0/2981591081 shutdown_connections 2024-04-14T23:48:48.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.691+0000 7efc8be00640 1 --2- 172.21.15.60:0/2981591081 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7efc6806b9c0 0x7efc6806de80 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:48.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.691+0000 7efc8be00640 1 --2- 172.21.15.60:0/2981591081 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efc8c0c84e0 0x7efc8c06b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:48.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.691+0000 7efc8be00640 1 --2- 172.21.15.60:0/2981591081 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc8c0b1e00 0x7efc8c0c2e80 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:48.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.691+0000 7efc8be00640 1 --2- 172.21.15.60:0/2981591081 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efc8c073320 0x7efc8c14fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:48.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.691+0000 7efc8be00640 1 -- 172.21.15.60:0/2981591081 >> 172.21.15.60:0/2981591081 conn(0x7efc8c0c8b50 msgr2=0x7efc8c14e650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:48.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.691+0000 7efc8be00640 1 -- 172.21.15.60:0/2981591081 shutdown_connections 2024-04-14T23:48:48.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.691+0000 7efc8be00640 1 -- 172.21.15.60:0/2981591081 wait complete. 2024-04-14T23:48:48.716 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-04-14T23:48:48.167455+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":152,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":217972,"kb_used_data":1752,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756748,"statfs":{"total":773094113280,"available":772870909952,"internally_reserved":0,"allocated":1794048,"data_stored":1747776,"data_compressed":65128,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":14,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001478"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.227573+0000","last_change":"2024-04-14T23:48:43.222542+0000","last_active":"2024-04-14T23:48:43.227573+0000","last_peered":"2024-04-14T23:48:43.227573+0000","last_clean":"2024-04-14T23:48:43.227573+0000","last_became_active":"2024-04-14T23:48:43.222278+0000","last_became_peered":"2024-04-14T23:48:43.222278+0000","last_unstale":"2024-04-14T23:48:43.227573+0000","last_undegraded":"2024-04-14T23:48:43.227573+0000","last_fullsized":"2024-04-14T23:48:43.227573+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:36:13.119301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.236713+0000","last_change":"2024-04-14T23:48:43.226781+0000","last_active":"2024-04-14T23:48:43.236713+0000","last_peered":"2024-04-14T23:48:43.236713+0000","last_clean":"2024-04-14T23:48:43.236713+0000","last_became_active":"2024-04-14T23:48:43.226426+0000","last_became_peered":"2024-04-14T23:48:43.226426+0000","last_unstale":"2024-04-14T23:48:43.236713+0000","last_undegraded":"2024-04-14T23:48:43.236713+0000","last_fullsized":"2024-04-14T23:48:43.236713+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T04:32:52.985782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.241614+0000","last_change":"2024-04-14T23:48:43.236939+0000","last_active":"2024-04-14T23:48:43.241614+0000","last_peered":"2024-04-14T23:48:43.241614+0000","last_clean":"2024-04-14T23:48:43.241614+0000","last_became_active":"2024-04-14T23:48:43.236616+0000","last_became_peered":"2024-04-14T23:48:43.236616+0000","last_unstale":"2024-04-14T23:48:43.241614+0000","last_undegraded":"2024-04-14T23:48:43.241614+0000","last_fullsized":"2024-04-14T23:48:43.241614+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:40:24.166032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230420+0000","last_change":"2024-04-14T23:48:43.242349+0000","last_active":"2024-04-14T23:48:44.230420+0000","last_peered":"2024-04-14T23:48:44.230420+0000","last_clean":"2024-04-14T23:48:44.230420+0000","last_became_active":"2024-04-14T23:48:43.242000+0000","last_became_peered":"2024-04-14T23:48:43.242000+0000","last_unstale":"2024-04-14T23:48:44.230420+0000","last_undegraded":"2024-04-14T23:48:44.230420+0000","last_fullsized":"2024-04-14T23:48:44.230420+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:33:52.458016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.238844+0000","last_change":"2024-04-14T23:48:43.233937+0000","last_active":"2024-04-14T23:48:43.238844+0000","last_peered":"2024-04-14T23:48:43.238844+0000","last_clean":"2024-04-14T23:48:43.238844+0000","last_became_active":"2024-04-14T23:48:43.233674+0000","last_became_peered":"2024-04-14T23:48:43.233674+0000","last_unstale":"2024-04-14T23:48:43.238844+0000","last_undegraded":"2024-04-14T23:48:43.238844+0000","last_fullsized":"2024-04-14T23:48:43.238844+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:45:47.377195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230260+0000","last_change":"2024-04-14T23:48:43.230925+0000","last_active":"2024-04-14T23:48:44.230260+0000","last_peered":"2024-04-14T23:48:44.230260+0000","last_clean":"2024-04-14T23:48:44.230260+0000","last_became_active":"2024-04-14T23:48:43.230405+0000","last_became_peered":"2024-04-14T23:48:43.230405+0000","last_unstale":"2024-04-14T23:48:44.230260+0000","last_undegraded":"2024-04-14T23:48:44.230260+0000","last_fullsized":"2024-04-14T23:48:44.230260+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:15:12.452964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.238591+0000","last_change":"2024-04-14T23:48:43.231088+0000","last_active":"2024-04-14T23:48:43.238591+0000","last_peered":"2024-04-14T23:48:43.238591+0000","last_clean":"2024-04-14T23:48:43.238591+0000","last_became_active":"2024-04-14T23:48:43.230784+0000","last_became_peered":"2024-04-14T23:48:43.230784+0000","last_unstale":"2024-04-14T23:48:43.238591+0000","last_undegraded":"2024-04-14T23:48:43.238591+0000","last_fullsized":"2024-04-14T23:48:43.238591+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:50:33.963376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.231230+0000","last_change":"2024-04-14T23:48:43.236019+0000","last_active":"2024-04-14T23:48:44.231230+0000","last_peered":"2024-04-14T23:48:44.231230+0000","last_clean":"2024-04-14T23:48:44.231230+0000","last_became_active":"2024-04-14T23:48:43.235719+0000","last_became_peered":"2024-04-14T23:48:43.235719+0000","last_unstale":"2024-04-14T23:48:44.231230+0000","last_undegraded":"2024-04-14T23:48:44.231230+0000","last_fullsized":"2024-04-14T23:48:44.231230+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T04:41:16.130730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.238782+0000","last_change":"2024-04-14T23:48:43.232166+0000","last_active":"2024-04-14T23:48:43.238782+0000","last_peered":"2024-04-14T23:48:43.238782+0000","last_clean":"2024-04-14T23:48:43.238782+0000","last_became_active":"2024-04-14T23:48:43.231751+0000","last_became_peered":"2024-04-14T23:48:43.231751+0000","last_unstale":"2024-04-14T23:48:43.238782+0000","last_undegraded":"2024-04-14T23:48:43.238782+0000","last_fullsized":"2024-04-14T23:48:43.238782+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T09:58:34.853704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.228924+0000","last_change":"2024-04-14T23:48:43.224938+0000","last_active":"2024-04-14T23:48:43.228924+0000","last_peered":"2024-04-14T23:48:43.228924+0000","last_clean":"2024-04-14T23:48:43.228924+0000","last_became_active":"2024-04-14T23:48:43.224592+0000","last_became_peered":"2024-04-14T23:48:43.224592+0000","last_unstale":"2024-04-14T23:48:43.228924+0000","last_undegraded":"2024-04-14T23:48:43.228924+0000","last_fullsized":"2024-04-14T23:48:43.228924+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T04:44:06.756499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.229987+0000","last_change":"2024-04-14T23:48:43.225290+0000","last_active":"2024-04-14T23:48:43.229987+0000","last_peered":"2024-04-14T23:48:43.229987+0000","last_clean":"2024-04-14T23:48:43.229987+0000","last_became_active":"2024-04-14T23:48:43.224989+0000","last_became_peered":"2024-04-14T23:48:43.224989+0000","last_unstale":"2024-04-14T23:48:43.229987+0000","last_undegraded":"2024-04-14T23:48:43.229987+0000","last_fullsized":"2024-04-14T23:48:43.229987+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T09:52:55.826011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.229660+0000","last_change":"2024-04-14T23:48:43.226194+0000","last_active":"2024-04-14T23:48:43.229660+0000","last_peered":"2024-04-14T23:48:43.229660+0000","last_clean":"2024-04-14T23:48:43.229660+0000","last_became_active":"2024-04-14T23:48:43.225892+0000","last_became_peered":"2024-04-14T23:48:43.225892+0000","last_unstale":"2024-04-14T23:48:43.229660+0000","last_undegraded":"2024-04-14T23:48:43.229660+0000","last_fullsized":"2024-04-14T23:48:43.229660+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:30:27.034185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.233039+0000","last_change":"2024-04-14T23:48:43.220571+0000","last_active":"2024-04-14T23:48:44.233039+0000","last_peered":"2024-04-14T23:48:44.233039+0000","last_clean":"2024-04-14T23:48:44.233039+0000","last_became_active":"2024-04-14T23:48:43.219614+0000","last_became_peered":"2024-04-14T23:48:43.219614+0000","last_unstale":"2024-04-14T23:48:44.233039+0000","last_undegraded":"2024-04-14T23:48:44.233039+0000","last_fullsized":"2024-04-14T23:48:44.233039+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:14:41.854957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.241634+0000","last_change":"2024-04-14T23:48:43.239403+0000","last_active":"2024-04-14T23:48:43.241634+0000","last_peered":"2024-04-14T23:48:43.241634+0000","last_clean":"2024-04-14T23:48:43.241634+0000","last_became_active":"2024-04-14T23:48:43.239136+0000","last_became_peered":"2024-04-14T23:48:43.239136+0000","last_unstale":"2024-04-14T23:48:43.241634+0000","last_undegraded":"2024-04-14T23:48:43.241634+0000","last_fullsized":"2024-04-14T23:48:43.241634+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:58:46.205808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.237076+0000","last_change":"2024-04-14T23:48:43.228157+0000","last_active":"2024-04-14T23:48:43.237076+0000","last_peered":"2024-04-14T23:48:43.237076+0000","last_clean":"2024-04-14T23:48:43.237076+0000","last_became_active":"2024-04-14T23:48:43.227898+0000","last_became_peered":"2024-04-14T23:48:43.227898+0000","last_unstale":"2024-04-14T23:48:43.237076+0000","last_undegraded":"2024-04-14T23:48:43.237076+0000","last_fullsized":"2024-04-14T23:48:43.237076+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:43:38.967154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230387+0000","last_change":"2024-04-14T23:48:43.238986+0000","last_active":"2024-04-14T23:48:44.230387+0000","last_peered":"2024-04-14T23:48:44.230387+0000","last_clean":"2024-04-14T23:48:44.230387+0000","last_became_active":"2024-04-14T23:48:43.238714+0000","last_became_peered":"2024-04-14T23:48:43.238714+0000","last_unstale":"2024-04-14T23:48:44.230387+0000","last_undegraded":"2024-04-14T23:48:44.230387+0000","last_fullsized":"2024-04-14T23:48:44.230387+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:05:30.655048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.231004+0000","last_change":"2024-04-14T23:48:43.229325+0000","last_active":"2024-04-14T23:48:44.231004+0000","last_peered":"2024-04-14T23:48:44.231004+0000","last_clean":"2024-04-14T23:48:44.231004+0000","last_became_active":"2024-04-14T23:48:43.228618+0000","last_became_peered":"2024-04-14T23:48:43.228618+0000","last_unstale":"2024-04-14T23:48:44.231004+0000","last_undegraded":"2024-04-14T23:48:44.231004+0000","last_fullsized":"2024-04-14T23:48:44.231004+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:06:43.745698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.238158+0000","last_change":"2024-04-14T23:48:43.233288+0000","last_active":"2024-04-14T23:48:43.238158+0000","last_peered":"2024-04-14T23:48:43.238158+0000","last_clean":"2024-04-14T23:48:43.238158+0000","last_became_active":"2024-04-14T23:48:43.232971+0000","last_became_peered":"2024-04-14T23:48:43.232971+0000","last_unstale":"2024-04-14T23:48:43.238158+0000","last_undegraded":"2024-04-14T23:48:43.238158+0000","last_fullsized":"2024-04-14T23:48:43.238158+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:35:56.958355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.238872+0000","last_change":"2024-04-14T23:48:43.233169+0000","last_active":"2024-04-14T23:48:43.238872+0000","last_peered":"2024-04-14T23:48:43.238872+0000","last_clean":"2024-04-14T23:48:43.238872+0000","last_became_active":"2024-04-14T23:48:43.232135+0000","last_became_peered":"2024-04-14T23:48:43.232135+0000","last_unstale":"2024-04-14T23:48:43.238872+0000","last_undegraded":"2024-04-14T23:48:43.238872+0000","last_fullsized":"2024-04-14T23:48:43.238872+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:17:08.188120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.233097+0000","last_change":"2024-04-14T23:48:43.228728+0000","last_active":"2024-04-14T23:48:43.233097+0000","last_peered":"2024-04-14T23:48:43.233097+0000","last_clean":"2024-04-14T23:48:43.233097+0000","last_became_active":"2024-04-14T23:48:43.226225+0000","last_became_peered":"2024-04-14T23:48:43.226225+0000","last_unstale":"2024-04-14T23:48:43.233097+0000","last_undegraded":"2024-04-14T23:48:43.233097+0000","last_fullsized":"2024-04-14T23:48:43.233097+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:09:05.048539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.241682+0000","last_change":"2024-04-14T23:48:43.237234+0000","last_active":"2024-04-14T23:48:43.241682+0000","last_peered":"2024-04-14T23:48:43.241682+0000","last_clean":"2024-04-14T23:48:43.241682+0000","last_became_active":"2024-04-14T23:48:43.230056+0000","last_became_peered":"2024-04-14T23:48:43.230056+0000","last_unstale":"2024-04-14T23:48:43.241682+0000","last_undegraded":"2024-04-14T23:48:43.241682+0000","last_fullsized":"2024-04-14T23:48:43.241682+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:30:02.685365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230697+0000","last_change":"2024-04-14T23:48:43.229387+0000","last_active":"2024-04-14T23:48:44.230697+0000","last_peered":"2024-04-14T23:48:44.230697+0000","last_clean":"2024-04-14T23:48:44.230697+0000","last_became_active":"2024-04-14T23:48:43.229110+0000","last_became_peered":"2024-04-14T23:48:43.229110+0000","last_unstale":"2024-04-14T23:48:44.230697+0000","last_undegraded":"2024-04-14T23:48:44.230697+0000","last_fullsized":"2024-04-14T23:48:44.230697+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T09:18:46.939940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.232750+0000","last_change":"2024-04-14T23:48:43.234154+0000","last_active":"2024-04-14T23:48:44.232750+0000","last_peered":"2024-04-14T23:48:44.232750+0000","last_clean":"2024-04-14T23:48:44.232750+0000","last_became_active":"2024-04-14T23:48:43.233853+0000","last_became_peered":"2024-04-14T23:48:43.233853+0000","last_unstale":"2024-04-14T23:48:44.232750+0000","last_undegraded":"2024-04-14T23:48:44.232750+0000","last_fullsized":"2024-04-14T23:48:44.232750+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T23:54:02.200200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230330+0000","last_change":"2024-04-14T23:48:43.242154+0000","last_active":"2024-04-14T23:48:44.230330+0000","last_peered":"2024-04-14T23:48:44.230330+0000","last_clean":"2024-04-14T23:48:44.230330+0000","last_became_active":"2024-04-14T23:48:43.241808+0000","last_became_peered":"2024-04-14T23:48:43.241808+0000","last_unstale":"2024-04-14T23:48:44.230330+0000","last_undegraded":"2024-04-14T23:48:44.230330+0000","last_fullsized":"2024-04-14T23:48:44.230330+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:48:42.680808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.232996+0000","last_change":"2024-04-14T23:48:43.233530+0000","last_active":"2024-04-14T23:48:44.232996+0000","last_peered":"2024-04-14T23:48:44.232996+0000","last_clean":"2024-04-14T23:48:44.232996+0000","last_became_active":"2024-04-14T23:48:43.233238+0000","last_became_peered":"2024-04-14T23:48:43.233238+0000","last_unstale":"2024-04-14T23:48:44.232996+0000","last_undegraded":"2024-04-14T23:48:44.232996+0000","last_fullsized":"2024-04-14T23:48:44.232996+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:10:58.525655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.228879+0000","last_change":"2024-04-14T23:48:43.222945+0000","last_active":"2024-04-14T23:48:43.228879+0000","last_peered":"2024-04-14T23:48:43.228879+0000","last_clean":"2024-04-14T23:48:43.228879+0000","last_became_active":"2024-04-14T23:48:43.222624+0000","last_became_peered":"2024-04-14T23:48:43.222624+0000","last_unstale":"2024-04-14T23:48:43.228879+0000","last_undegraded":"2024-04-14T23:48:43.228879+0000","last_fullsized":"2024-04-14T23:48:43.228879+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:03:28.128723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.240068+0000","last_change":"2024-04-14T23:48:43.232567+0000","last_active":"2024-04-14T23:48:43.240068+0000","last_peered":"2024-04-14T23:48:43.240068+0000","last_clean":"2024-04-14T23:48:43.240068+0000","last_became_active":"2024-04-14T23:48:43.232277+0000","last_became_peered":"2024-04-14T23:48:43.232277+0000","last_unstale":"2024-04-14T23:48:43.240068+0000","last_undegraded":"2024-04-14T23:48:43.240068+0000","last_fullsized":"2024-04-14T23:48:43.240068+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:32:30.528043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.232426+0000","last_change":"2024-04-14T23:48:43.229443+0000","last_active":"2024-04-14T23:48:44.232426+0000","last_peered":"2024-04-14T23:48:44.232426+0000","last_clean":"2024-04-14T23:48:44.232426+0000","last_became_active":"2024-04-14T23:48:43.229172+0000","last_became_peered":"2024-04-14T23:48:43.229172+0000","last_unstale":"2024-04-14T23:48:44.232426+0000","last_undegraded":"2024-04-14T23:48:44.232426+0000","last_fullsized":"2024-04-14T23:48:44.232426+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:41:31.405287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.244064+0000","last_change":"2024-04-14T23:48:43.242202+0000","last_active":"2024-04-14T23:48:43.244064+0000","last_peered":"2024-04-14T23:48:43.244064+0000","last_clean":"2024-04-14T23:48:43.244064+0000","last_became_active":"2024-04-14T23:48:43.241879+0000","last_became_peered":"2024-04-14T23:48:43.241879+0000","last_unstale":"2024-04-14T23:48:43.244064+0000","last_undegraded":"2024-04-14T23:48:43.244064+0000","last_fullsized":"2024-04-14T23:48:43.244064+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:47:46.306480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230871+0000","last_change":"2024-04-14T23:48:43.242068+0000","last_active":"2024-04-14T23:48:44.230871+0000","last_peered":"2024-04-14T23:48:44.230871+0000","last_clean":"2024-04-14T23:48:44.230871+0000","last_became_active":"2024-04-14T23:48:43.241734+0000","last_became_peered":"2024-04-14T23:48:43.241734+0000","last_unstale":"2024-04-14T23:48:44.230871+0000","last_undegraded":"2024-04-14T23:48:44.230871+0000","last_fullsized":"2024-04-14T23:48:44.230871+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:32:04.579906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.239805+0000","last_change":"2024-04-14T23:48:43.236034+0000","last_active":"2024-04-14T23:48:43.239805+0000","last_peered":"2024-04-14T23:48:43.239805+0000","last_clean":"2024-04-14T23:48:43.239805+0000","last_became_active":"2024-04-14T23:48:43.233567+0000","last_became_peered":"2024-04-14T23:48:43.233567+0000","last_unstale":"2024-04-14T23:48:43.239805+0000","last_undegraded":"2024-04-14T23:48:43.239805+0000","last_fullsized":"2024-04-14T23:48:43.239805+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T23:49:51.630628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.234495+0000","last_change":"2024-04-14T23:48:43.231045+0000","last_active":"2024-04-14T23:48:43.234495+0000","last_peered":"2024-04-14T23:48:43.234495+0000","last_clean":"2024-04-14T23:48:43.234495+0000","last_became_active":"2024-04-14T23:48:43.230626+0000","last_became_peered":"2024-04-14T23:48:43.230626+0000","last_unstale":"2024-04-14T23:48:43.234495+0000","last_undegraded":"2024-04-14T23:48:43.234495+0000","last_fullsized":"2024-04-14T23:48:43.234495+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:45:19.807215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.229591+0000","last_change":"2024-04-14T23:48:43.227543+0000","last_active":"2024-04-14T23:48:43.229591+0000","last_peered":"2024-04-14T23:48:43.229591+0000","last_clean":"2024-04-14T23:48:43.229591+0000","last_became_active":"2024-04-14T23:48:43.226773+0000","last_became_peered":"2024-04-14T23:48:43.226773+0000","last_unstale":"2024-04-14T23:48:43.229591+0000","last_undegraded":"2024-04-14T23:48:43.229591+0000","last_fullsized":"2024-04-14T23:48:43.229591+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:12:10.175172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.233181+0000","last_change":"2024-04-14T23:48:42.239254+0000","last_active":"2024-04-14T23:48:44.233181+0000","last_peered":"2024-04-14T23:48:44.233181+0000","last_clean":"2024-04-14T23:48:44.233181+0000","last_became_active":"2024-04-14T23:48:42.238847+0000","last_became_peered":"2024-04-14T23:48:42.238847+0000","last_unstale":"2024-04-14T23:48:44.233181+0000","last_undegraded":"2024-04-14T23:48:44.233181+0000","last_fullsized":"2024-04-14T23:48:44.233181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:46:16.301900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.241700+0000","last_change":"2024-04-14T23:48:43.237720+0000","last_active":"2024-04-14T23:48:43.241700+0000","last_peered":"2024-04-14T23:48:43.241700+0000","last_clean":"2024-04-14T23:48:43.241700+0000","last_became_active":"2024-04-14T23:48:43.237391+0000","last_became_peered":"2024-04-14T23:48:43.237391+0000","last_unstale":"2024-04-14T23:48:43.241700+0000","last_undegraded":"2024-04-14T23:48:43.241700+0000","last_fullsized":"2024-04-14T23:48:43.241700+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:15:38.587904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.216530+0000","last_change":"2024-04-14T23:48:42.219277+0000","last_active":"2024-04-14T23:48:43.216530+0000","last_peered":"2024-04-14T23:48:43.216530+0000","last_clean":"2024-04-14T23:48:43.216530+0000","last_became_active":"2024-04-14T23:48:42.218995+0000","last_became_peered":"2024-04-14T23:48:42.218995+0000","last_unstale":"2024-04-14T23:48:43.216530+0000","last_undegraded":"2024-04-14T23:48:43.216530+0000","last_fullsized":"2024-04-14T23:48:43.216530+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T23:51:33.370857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.232590+0000","last_change":"2024-04-14T23:48:42.236803+0000","last_active":"2024-04-14T23:48:44.232590+0000","last_peered":"2024-04-14T23:48:44.232590+0000","last_clean":"2024-04-14T23:48:44.232590+0000","last_became_active":"2024-04-14T23:48:42.225937+0000","last_became_peered":"2024-04-14T23:48:42.225937+0000","last_unstale":"2024-04-14T23:48:44.232590+0000","last_undegraded":"2024-04-14T23:48:44.232590+0000","last_fullsized":"2024-04-14T23:48:44.232590+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:52:32.609126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.231732+0000","last_change":"2024-04-14T23:48:43.229506+0000","last_active":"2024-04-14T23:48:44.231732+0000","last_peered":"2024-04-14T23:48:44.231732+0000","last_clean":"2024-04-14T23:48:44.231732+0000","last_became_active":"2024-04-14T23:48:43.227356+0000","last_became_peered":"2024-04-14T23:48:43.227356+0000","last_unstale":"2024-04-14T23:48:44.231732+0000","last_undegraded":"2024-04-14T23:48:44.231732+0000","last_fullsized":"2024-04-14T23:48:44.231732+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:13:04.840655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230660+0000","last_change":"2024-04-14T23:48:42.210883+0000","last_active":"2024-04-14T23:48:44.230660+0000","last_peered":"2024-04-14T23:48:44.230660+0000","last_clean":"2024-04-14T23:48:44.230660+0000","last_became_active":"2024-04-14T23:48:42.210607+0000","last_became_peered":"2024-04-14T23:48:42.210607+0000","last_unstale":"2024-04-14T23:48:44.230660+0000","last_undegraded":"2024-04-14T23:48:44.230660+0000","last_fullsized":"2024-04-14T23:48:44.230660+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:33:32.526223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.225460+0000","last_change":"2024-04-14T23:48:43.218259+0000","last_active":"2024-04-14T23:48:43.225460+0000","last_peered":"2024-04-14T23:48:43.225460+0000","last_clean":"2024-04-14T23:48:43.225460+0000","last_became_active":"2024-04-14T23:48:43.217801+0000","last_became_peered":"2024-04-14T23:48:43.217801+0000","last_unstale":"2024-04-14T23:48:43.225460+0000","last_undegraded":"2024-04-14T23:48:43.225460+0000","last_fullsized":"2024-04-14T23:48:43.225460+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:03:21.087919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.217021+0000","last_change":"2024-04-14T23:48:42.218652+0000","last_active":"2024-04-14T23:48:43.217021+0000","last_peered":"2024-04-14T23:48:43.217021+0000","last_clean":"2024-04-14T23:48:43.217021+0000","last_became_active":"2024-04-14T23:48:42.218321+0000","last_became_peered":"2024-04-14T23:48:42.218321+0000","last_unstale":"2024-04-14T23:48:43.217021+0000","last_undegraded":"2024-04-14T23:48:43.217021+0000","last_fullsized":"2024-04-14T23:48:43.217021+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T04:22:06.957747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230879+0000","last_change":"2024-04-14T23:48:43.229117+0000","last_active":"2024-04-14T23:48:44.230879+0000","last_peered":"2024-04-14T23:48:44.230879+0000","last_clean":"2024-04-14T23:48:44.230879+0000","last_became_active":"2024-04-14T23:48:43.228805+0000","last_became_peered":"2024-04-14T23:48:43.228805+0000","last_unstale":"2024-04-14T23:48:44.230879+0000","last_undegraded":"2024-04-14T23:48:44.230879+0000","last_fullsized":"2024-04-14T23:48:44.230879+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:10:39.043818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230930+0000","last_change":"2024-04-14T23:48:42.221104+0000","last_active":"2024-04-14T23:48:44.230930+0000","last_peered":"2024-04-14T23:48:44.230930+0000","last_clean":"2024-04-14T23:48:44.230930+0000","last_became_active":"2024-04-14T23:48:42.220767+0000","last_became_peered":"2024-04-14T23:48:42.220767+0000","last_unstale":"2024-04-14T23:48:44.230930+0000","last_undegraded":"2024-04-14T23:48:44.230930+0000","last_fullsized":"2024-04-14T23:48:44.230930+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:23:57.213764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.241543+0000","last_change":"2024-04-14T23:48:43.236916+0000","last_active":"2024-04-14T23:48:43.241543+0000","last_peered":"2024-04-14T23:48:43.241543+0000","last_clean":"2024-04-14T23:48:43.241543+0000","last_became_active":"2024-04-14T23:48:43.236168+0000","last_became_peered":"2024-04-14T23:48:43.236168+0000","last_unstale":"2024-04-14T23:48:43.241543+0000","last_undegraded":"2024-04-14T23:48:43.241543+0000","last_fullsized":"2024-04-14T23:48:43.241543+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:35:33.530129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.224228+0000","last_change":"2024-04-14T23:48:42.231329+0000","last_active":"2024-04-14T23:48:43.224228+0000","last_peered":"2024-04-14T23:48:43.224228+0000","last_clean":"2024-04-14T23:48:43.224228+0000","last_became_active":"2024-04-14T23:48:42.230969+0000","last_became_peered":"2024-04-14T23:48:42.230969+0000","last_unstale":"2024-04-14T23:48:43.224228+0000","last_undegraded":"2024-04-14T23:48:43.224228+0000","last_fullsized":"2024-04-14T23:48:43.224228+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:06:16.557589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.235654+0000","last_change":"2024-04-14T23:48:43.230531+0000","last_active":"2024-04-14T23:48:43.235654+0000","last_peered":"2024-04-14T23:48:43.235654+0000","last_clean":"2024-04-14T23:48:43.235654+0000","last_became_active":"2024-04-14T23:48:43.230253+0000","last_became_peered":"2024-04-14T23:48:43.230253+0000","last_unstale":"2024-04-14T23:48:43.235654+0000","last_undegraded":"2024-04-14T23:48:43.235654+0000","last_fullsized":"2024-04-14T23:48:43.235654+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:11:05.198292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.216588+0000","last_change":"2024-04-14T23:48:42.237069+0000","last_active":"2024-04-14T23:48:43.216588+0000","last_peered":"2024-04-14T23:48:43.216588+0000","last_clean":"2024-04-14T23:48:43.216588+0000","last_became_active":"2024-04-14T23:48:42.236804+0000","last_became_peered":"2024-04-14T23:48:42.236804+0000","last_unstale":"2024-04-14T23:48:43.216588+0000","last_undegraded":"2024-04-14T23:48:43.216588+0000","last_fullsized":"2024-04-14T23:48:43.216588+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:25:37.318379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.226657+0000","last_change":"2024-04-14T23:48:43.221771+0000","last_active":"2024-04-14T23:48:43.226657+0000","last_peered":"2024-04-14T23:48:43.226657+0000","last_clean":"2024-04-14T23:48:43.226657+0000","last_became_active":"2024-04-14T23:48:43.221462+0000","last_became_peered":"2024-04-14T23:48:43.221462+0000","last_unstale":"2024-04-14T23:48:43.226657+0000","last_undegraded":"2024-04-14T23:48:43.226657+0000","last_fullsized":"2024-04-14T23:48:43.226657+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:01:16.040458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.222096+0000","last_change":"2024-04-14T23:48:42.230518+0000","last_active":"2024-04-14T23:48:43.222096+0000","last_peered":"2024-04-14T23:48:43.222096+0000","last_clean":"2024-04-14T23:48:43.222096+0000","last_became_active":"2024-04-14T23:48:42.229758+0000","last_became_peered":"2024-04-14T23:48:42.229758+0000","last_unstale":"2024-04-14T23:48:43.222096+0000","last_undegraded":"2024-04-14T23:48:43.222096+0000","last_fullsized":"2024-04-14T23:48:43.222096+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:01:41.332553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.238539+0000","last_change":"2024-04-14T23:48:43.229873+0000","last_active":"2024-04-14T23:48:43.238539+0000","last_peered":"2024-04-14T23:48:43.238539+0000","last_clean":"2024-04-14T23:48:43.238539+0000","last_became_active":"2024-04-14T23:48:43.229476+0000","last_became_peered":"2024-04-14T23:48:43.229476+0000","last_unstale":"2024-04-14T23:48:43.238539+0000","last_undegraded":"2024-04-14T23:48:43.238539+0000","last_fullsized":"2024-04-14T23:48:43.238539+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:38:11.414948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230858+0000","last_change":"2024-04-14T23:48:42.225404+0000","last_active":"2024-04-14T23:48:44.230858+0000","last_peered":"2024-04-14T23:48:44.230858+0000","last_clean":"2024-04-14T23:48:44.230858+0000","last_became_active":"2024-04-14T23:48:42.225211+0000","last_became_peered":"2024-04-14T23:48:42.225211+0000","last_unstale":"2024-04-14T23:48:44.230858+0000","last_undegraded":"2024-04-14T23:48:44.230858+0000","last_fullsized":"2024-04-14T23:48:44.230858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:49:45.282259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.238724+0000","last_change":"2024-04-14T23:48:43.229543+0000","last_active":"2024-04-14T23:48:43.238724+0000","last_peered":"2024-04-14T23:48:43.238724+0000","last_clean":"2024-04-14T23:48:43.238724+0000","last_became_active":"2024-04-14T23:48:43.229297+0000","last_became_peered":"2024-04-14T23:48:43.229297+0000","last_unstale":"2024-04-14T23:48:43.238724+0000","last_undegraded":"2024-04-14T23:48:43.238724+0000","last_fullsized":"2024-04-14T23:48:43.238724+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:52:40.162331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.214624+0000","last_change":"2024-04-14T23:48:42.220492+0000","last_active":"2024-04-14T23:48:43.214624+0000","last_peered":"2024-04-14T23:48:43.214624+0000","last_clean":"2024-04-14T23:48:43.214624+0000","last_became_active":"2024-04-14T23:48:42.220145+0000","last_became_peered":"2024-04-14T23:48:42.220145+0000","last_unstale":"2024-04-14T23:48:43.214624+0000","last_undegraded":"2024-04-14T23:48:43.214624+0000","last_fullsized":"2024-04-14T23:48:43.214624+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:17:45.278993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.238431+0000","last_change":"2024-04-14T23:48:43.235676+0000","last_active":"2024-04-14T23:48:43.238431+0000","last_peered":"2024-04-14T23:48:43.238431+0000","last_clean":"2024-04-14T23:48:43.238431+0000","last_became_active":"2024-04-14T23:48:43.235319+0000","last_became_peered":"2024-04-14T23:48:43.235319+0000","last_unstale":"2024-04-14T23:48:43.238431+0000","last_undegraded":"2024-04-14T23:48:43.238431+0000","last_fullsized":"2024-04-14T23:48:43.238431+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:03:35.784237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.223088+0000","last_change":"2024-04-14T23:48:42.231178+0000","last_active":"2024-04-14T23:48:43.223088+0000","last_peered":"2024-04-14T23:48:43.223088+0000","last_clean":"2024-04-14T23:48:43.223088+0000","last_became_active":"2024-04-14T23:48:42.230792+0000","last_became_peered":"2024-04-14T23:48:42.230792+0000","last_unstale":"2024-04-14T23:48:43.223088+0000","last_undegraded":"2024-04-14T23:48:43.223088+0000","last_fullsized":"2024-04-14T23:48:43.223088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:22:40.744314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.235827+0000","last_change":"2024-04-14T23:48:43.228290+0000","last_active":"2024-04-14T23:48:43.235827+0000","last_peered":"2024-04-14T23:48:43.235827+0000","last_clean":"2024-04-14T23:48:43.235827+0000","last_became_active":"2024-04-14T23:48:43.228004+0000","last_became_peered":"2024-04-14T23:48:43.228004+0000","last_unstale":"2024-04-14T23:48:43.235827+0000","last_undegraded":"2024-04-14T23:48:43.235827+0000","last_fullsized":"2024-04-14T23:48:43.235827+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:44:50.560451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.216307+0000","last_change":"2024-04-14T23:48:42.231936+0000","last_active":"2024-04-14T23:48:43.216307+0000","last_peered":"2024-04-14T23:48:43.216307+0000","last_clean":"2024-04-14T23:48:43.216307+0000","last_became_active":"2024-04-14T23:48:42.231612+0000","last_became_peered":"2024-04-14T23:48:42.231612+0000","last_unstale":"2024-04-14T23:48:43.216307+0000","last_undegraded":"2024-04-14T23:48:43.216307+0000","last_fullsized":"2024-04-14T23:48:43.216307+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:45:03.098024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.235908+0000","last_change":"2024-04-14T23:48:43.228933+0000","last_active":"2024-04-14T23:48:43.235908+0000","last_peered":"2024-04-14T23:48:43.235908+0000","last_clean":"2024-04-14T23:48:43.235908+0000","last_became_active":"2024-04-14T23:48:43.228665+0000","last_became_peered":"2024-04-14T23:48:43.228665+0000","last_unstale":"2024-04-14T23:48:43.235908+0000","last_undegraded":"2024-04-14T23:48:43.235908+0000","last_fullsized":"2024-04-14T23:48:43.235908+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:00:11.267732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.223809+0000","last_change":"2024-04-14T23:48:42.239794+0000","last_active":"2024-04-14T23:48:43.223809+0000","last_peered":"2024-04-14T23:48:43.223809+0000","last_clean":"2024-04-14T23:48:43.223809+0000","last_became_active":"2024-04-14T23:48:42.239505+0000","last_became_peered":"2024-04-14T23:48:42.239505+0000","last_unstale":"2024-04-14T23:48:43.223809+0000","last_undegraded":"2024-04-14T23:48:43.223809+0000","last_fullsized":"2024-04-14T23:48:43.223809+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T09:43:16.582708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.238761+0000","last_change":"2024-04-14T23:48:43.230251+0000","last_active":"2024-04-14T23:48:43.238761+0000","last_peered":"2024-04-14T23:48:43.238761+0000","last_clean":"2024-04-14T23:48:43.238761+0000","last_became_active":"2024-04-14T23:48:43.229949+0000","last_became_peered":"2024-04-14T23:48:43.229949+0000","last_unstale":"2024-04-14T23:48:43.238761+0000","last_undegraded":"2024-04-14T23:48:43.238761+0000","last_fullsized":"2024-04-14T23:48:43.238761+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:13:45.621085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.222416+0000","last_change":"2024-04-14T23:48:42.227708+0000","last_active":"2024-04-14T23:48:43.222416+0000","last_peered":"2024-04-14T23:48:43.222416+0000","last_clean":"2024-04-14T23:48:43.222416+0000","last_became_active":"2024-04-14T23:48:42.226845+0000","last_became_peered":"2024-04-14T23:48:42.226845+0000","last_unstale":"2024-04-14T23:48:43.222416+0000","last_undegraded":"2024-04-14T23:48:43.222416+0000","last_fullsized":"2024-04-14T23:48:43.222416+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:02:54.468796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230624+0000","last_change":"2024-04-14T23:48:43.227439+0000","last_active":"2024-04-14T23:48:44.230624+0000","last_peered":"2024-04-14T23:48:44.230624+0000","last_clean":"2024-04-14T23:48:44.230624+0000","last_became_active":"2024-04-14T23:48:43.226717+0000","last_became_peered":"2024-04-14T23:48:43.226717+0000","last_unstale":"2024-04-14T23:48:44.230624+0000","last_undegraded":"2024-04-14T23:48:44.230624+0000","last_fullsized":"2024-04-14T23:48:44.230624+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:01:11.249999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.223049+0000","last_change":"2024-04-14T23:48:42.232164+0000","last_active":"2024-04-14T23:48:43.223049+0000","last_peered":"2024-04-14T23:48:43.223049+0000","last_clean":"2024-04-14T23:48:43.223049+0000","last_became_active":"2024-04-14T23:48:42.229785+0000","last_became_peered":"2024-04-14T23:48:42.229785+0000","last_unstale":"2024-04-14T23:48:43.223049+0000","last_undegraded":"2024-04-14T23:48:43.223049+0000","last_fullsized":"2024-04-14T23:48:43.223049+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T04:21:37.538735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.238850+0000","last_change":"2024-04-14T23:48:43.232949+0000","last_active":"2024-04-14T23:48:43.238850+0000","last_peered":"2024-04-14T23:48:43.238850+0000","last_clean":"2024-04-14T23:48:43.238850+0000","last_became_active":"2024-04-14T23:48:43.232612+0000","last_became_peered":"2024-04-14T23:48:43.232612+0000","last_unstale":"2024-04-14T23:48:43.238850+0000","last_undegraded":"2024-04-14T23:48:43.238850+0000","last_fullsized":"2024-04-14T23:48:43.238850+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:40:03.923776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":168,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.216039+0000","last_change":"2024-04-14T23:48:38.150200+0000","last_active":"2024-04-14T23:48:43.216039+0000","last_peered":"2024-04-14T23:48:43.216039+0000","last_clean":"2024-04-14T23:48:43.216039+0000","last_became_active":"2024-04-14T23:48:38.149301+0000","last_became_peered":"2024-04-14T23:48:38.149301+0000","last_unstale":"2024-04-14T23:48:43.216039+0000","last_undegraded":"2024-04-14T23:48:43.216039+0000","last_fullsized":"2024-04-14T23:48:43.216039+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:37.129663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:37.129663+0000","last_clean_scrub_stamp":"2024-04-14T23:48:37.129663+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T23:58:01.303851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230439+0000","last_change":"2024-04-14T23:48:42.229238+0000","last_active":"2024-04-14T23:48:44.230439+0000","last_peered":"2024-04-14T23:48:44.230439+0000","last_clean":"2024-04-14T23:48:44.230439+0000","last_became_active":"2024-04-14T23:48:42.228994+0000","last_became_peered":"2024-04-14T23:48:42.228994+0000","last_unstale":"2024-04-14T23:48:44.230439+0000","last_undegraded":"2024-04-14T23:48:44.230439+0000","last_fullsized":"2024-04-14T23:48:44.230439+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:47:15.406228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.232256+0000","last_change":"2024-04-14T23:48:43.223973+0000","last_active":"2024-04-14T23:48:44.232256+0000","last_peered":"2024-04-14T23:48:44.232256+0000","last_clean":"2024-04-14T23:48:44.232256+0000","last_became_active":"2024-04-14T23:48:43.223665+0000","last_became_peered":"2024-04-14T23:48:43.223665+0000","last_unstale":"2024-04-14T23:48:44.232256+0000","last_undegraded":"2024-04-14T23:48:44.232256+0000","last_fullsized":"2024-04-14T23:48:44.232256+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:23:28.495821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.215985+0000","last_change":"2024-04-14T23:48:42.230404+0000","last_active":"2024-04-14T23:48:43.215985+0000","last_peered":"2024-04-14T23:48:43.215985+0000","last_clean":"2024-04-14T23:48:43.215985+0000","last_became_active":"2024-04-14T23:48:42.230173+0000","last_became_peered":"2024-04-14T23:48:42.230173+0000","last_unstale":"2024-04-14T23:48:43.215985+0000","last_undegraded":"2024-04-14T23:48:43.215985+0000","last_fullsized":"2024-04-14T23:48:43.215985+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:23:45.666680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.240607+0000","last_change":"2024-04-14T23:48:43.231958+0000","last_active":"2024-04-14T23:48:43.240607+0000","last_peered":"2024-04-14T23:48:43.240607+0000","last_clean":"2024-04-14T23:48:43.240607+0000","last_became_active":"2024-04-14T23:48:43.231025+0000","last_became_peered":"2024-04-14T23:48:43.231025+0000","last_unstale":"2024-04-14T23:48:43.240607+0000","last_undegraded":"2024-04-14T23:48:43.240607+0000","last_fullsized":"2024-04-14T23:48:43.240607+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:49:56.020321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.223371+0000","last_change":"2024-04-14T23:48:42.240651+0000","last_active":"2024-04-14T23:48:43.223371+0000","last_peered":"2024-04-14T23:48:43.223371+0000","last_clean":"2024-04-14T23:48:43.223371+0000","last_became_active":"2024-04-14T23:48:42.240351+0000","last_became_peered":"2024-04-14T23:48:42.240351+0000","last_unstale":"2024-04-14T23:48:43.223371+0000","last_undegraded":"2024-04-14T23:48:43.223371+0000","last_fullsized":"2024-04-14T23:48:43.223371+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:03:02.804444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.240029+0000","last_change":"2024-04-14T23:48:43.232415+0000","last_active":"2024-04-14T23:48:43.240029+0000","last_peered":"2024-04-14T23:48:43.240029+0000","last_clean":"2024-04-14T23:48:43.240029+0000","last_became_active":"2024-04-14T23:48:43.232120+0000","last_became_peered":"2024-04-14T23:48:43.232120+0000","last_unstale":"2024-04-14T23:48:43.240029+0000","last_undegraded":"2024-04-14T23:48:43.240029+0000","last_fullsized":"2024-04-14T23:48:43.240029+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:30:01.619411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230988+0000","last_change":"2024-04-14T23:48:42.233743+0000","last_active":"2024-04-14T23:48:44.230988+0000","last_peered":"2024-04-14T23:48:44.230988+0000","last_clean":"2024-04-14T23:48:44.230988+0000","last_became_active":"2024-04-14T23:48:42.233354+0000","last_became_peered":"2024-04-14T23:48:42.233354+0000","last_unstale":"2024-04-14T23:48:44.230988+0000","last_undegraded":"2024-04-14T23:48:44.230988+0000","last_fullsized":"2024-04-14T23:48:44.230988+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:49:23.045968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.238816+0000","last_change":"2024-04-14T23:48:43.232468+0000","last_active":"2024-04-14T23:48:43.238816+0000","last_peered":"2024-04-14T23:48:43.238816+0000","last_clean":"2024-04-14T23:48:43.238816+0000","last_became_active":"2024-04-14T23:48:43.232119+0000","last_became_peered":"2024-04-14T23:48:43.232119+0000","last_unstale":"2024-04-14T23:48:43.238816+0000","last_undegraded":"2024-04-14T23:48:43.238816+0000","last_fullsized":"2024-04-14T23:48:43.238816+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:49:04.298324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.214850+0000","last_change":"2024-04-14T23:48:42.242249+0000","last_active":"2024-04-14T23:48:43.214850+0000","last_peered":"2024-04-14T23:48:43.214850+0000","last_clean":"2024-04-14T23:48:43.214850+0000","last_became_active":"2024-04-14T23:48:42.242039+0000","last_became_peered":"2024-04-14T23:48:42.242039+0000","last_unstale":"2024-04-14T23:48:43.214850+0000","last_undegraded":"2024-04-14T23:48:43.214850+0000","last_fullsized":"2024-04-14T23:48:43.214850+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:46:16.301900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230960+0000","last_change":"2024-04-14T23:48:43.242228+0000","last_active":"2024-04-14T23:48:44.230960+0000","last_peered":"2024-04-14T23:48:44.230960+0000","last_clean":"2024-04-14T23:48:44.230960+0000","last_became_active":"2024-04-14T23:48:43.241776+0000","last_became_peered":"2024-04-14T23:48:43.241776+0000","last_unstale":"2024-04-14T23:48:44.230960+0000","last_undegraded":"2024-04-14T23:48:44.230960+0000","last_fullsized":"2024-04-14T23:48:44.230960+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:48:23.653866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.213932+0000","last_change":"2024-04-14T23:48:42.232537+0000","last_active":"2024-04-14T23:48:43.213932+0000","last_peered":"2024-04-14T23:48:43.213932+0000","last_clean":"2024-04-14T23:48:43.213932+0000","last_became_active":"2024-04-14T23:48:42.232325+0000","last_became_peered":"2024-04-14T23:48:42.232325+0000","last_unstale":"2024-04-14T23:48:43.213932+0000","last_undegraded":"2024-04-14T23:48:43.213932+0000","last_fullsized":"2024-04-14T23:48:43.213932+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:34:37.448301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.239744+0000","last_change":"2024-04-14T23:48:43.234445+0000","last_active":"2024-04-14T23:48:43.239744+0000","last_peered":"2024-04-14T23:48:43.239744+0000","last_clean":"2024-04-14T23:48:43.239744+0000","last_became_active":"2024-04-14T23:48:43.234150+0000","last_became_peered":"2024-04-14T23:48:43.234150+0000","last_unstale":"2024-04-14T23:48:43.239744+0000","last_undegraded":"2024-04-14T23:48:43.239744+0000","last_fullsized":"2024-04-14T23:48:43.239744+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:39:36.833579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.216481+0000","last_change":"2024-04-14T23:48:42.230014+0000","last_active":"2024-04-14T23:48:43.216481+0000","last_peered":"2024-04-14T23:48:43.216481+0000","last_clean":"2024-04-14T23:48:43.216481+0000","last_became_active":"2024-04-14T23:48:42.229712+0000","last_became_peered":"2024-04-14T23:48:42.229712+0000","last_unstale":"2024-04-14T23:48:43.216481+0000","last_undegraded":"2024-04-14T23:48:43.216481+0000","last_fullsized":"2024-04-14T23:48:43.216481+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:54:20.595391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.233392+0000","last_change":"2024-04-14T23:48:43.229128+0000","last_active":"2024-04-14T23:48:44.233392+0000","last_peered":"2024-04-14T23:48:44.233392+0000","last_clean":"2024-04-14T23:48:44.233392+0000","last_became_active":"2024-04-14T23:48:43.228739+0000","last_became_peered":"2024-04-14T23:48:43.228739+0000","last_unstale":"2024-04-14T23:48:44.233392+0000","last_undegraded":"2024-04-14T23:48:44.233392+0000","last_fullsized":"2024-04-14T23:48:44.233392+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:08:08.148345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.217199+0000","last_change":"2024-04-14T23:48:42.220035+0000","last_active":"2024-04-14T23:48:43.217199+0000","last_peered":"2024-04-14T23:48:43.217199+0000","last_clean":"2024-04-14T23:48:43.217199+0000","last_became_active":"2024-04-14T23:48:42.219750+0000","last_became_peered":"2024-04-14T23:48:42.219750+0000","last_unstale":"2024-04-14T23:48:43.217199+0000","last_undegraded":"2024-04-14T23:48:43.217199+0000","last_fullsized":"2024-04-14T23:48:43.217199+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:00:58.928166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.240459+0000","last_change":"2024-04-14T23:48:43.232997+0000","last_active":"2024-04-14T23:48:43.240459+0000","last_peered":"2024-04-14T23:48:43.240459+0000","last_clean":"2024-04-14T23:48:43.240459+0000","last_became_active":"2024-04-14T23:48:43.232730+0000","last_became_peered":"2024-04-14T23:48:43.232730+0000","last_unstale":"2024-04-14T23:48:43.240459+0000","last_undegraded":"2024-04-14T23:48:43.240459+0000","last_fullsized":"2024-04-14T23:48:43.240459+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:49:33.985010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.215837+0000","last_change":"2024-04-14T23:48:42.232087+0000","last_active":"2024-04-14T23:48:43.215837+0000","last_peered":"2024-04-14T23:48:43.215837+0000","last_clean":"2024-04-14T23:48:43.215837+0000","last_became_active":"2024-04-14T23:48:42.231796+0000","last_became_peered":"2024-04-14T23:48:42.231796+0000","last_unstale":"2024-04-14T23:48:43.215837+0000","last_undegraded":"2024-04-14T23:48:43.215837+0000","last_fullsized":"2024-04-14T23:48:43.215837+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:14:35.050093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.238771+0000","last_change":"2024-04-14T23:48:43.232871+0000","last_active":"2024-04-14T23:48:43.238771+0000","last_peered":"2024-04-14T23:48:43.238771+0000","last_clean":"2024-04-14T23:48:43.238771+0000","last_became_active":"2024-04-14T23:48:43.232594+0000","last_became_peered":"2024-04-14T23:48:43.232594+0000","last_unstale":"2024-04-14T23:48:43.238771+0000","last_undegraded":"2024-04-14T23:48:43.238771+0000","last_fullsized":"2024-04-14T23:48:43.238771+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:36:16.823063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.214857+0000","last_change":"2024-04-14T23:48:42.231237+0000","last_active":"2024-04-14T23:48:43.214857+0000","last_peered":"2024-04-14T23:48:43.214857+0000","last_clean":"2024-04-14T23:48:43.214857+0000","last_became_active":"2024-04-14T23:48:42.230902+0000","last_became_peered":"2024-04-14T23:48:42.230902+0000","last_unstale":"2024-04-14T23:48:43.214857+0000","last_undegraded":"2024-04-14T23:48:43.214857+0000","last_fullsized":"2024-04-14T23:48:43.214857+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:28:37.228039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.232080+0000","last_change":"2024-04-14T23:48:43.233035+0000","last_active":"2024-04-14T23:48:44.232080+0000","last_peered":"2024-04-14T23:48:44.232080+0000","last_clean":"2024-04-14T23:48:44.232080+0000","last_became_active":"2024-04-14T23:48:43.232747+0000","last_became_peered":"2024-04-14T23:48:43.232747+0000","last_unstale":"2024-04-14T23:48:44.232080+0000","last_undegraded":"2024-04-14T23:48:44.232080+0000","last_fullsized":"2024-04-14T23:48:44.232080+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:38:57.324408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.214550+0000","last_change":"2024-04-14T23:48:42.240496+0000","last_active":"2024-04-14T23:48:43.214550+0000","last_peered":"2024-04-14T23:48:43.214550+0000","last_clean":"2024-04-14T23:48:43.214550+0000","last_became_active":"2024-04-14T23:48:42.238524+0000","last_became_peered":"2024-04-14T23:48:42.238524+0000","last_unstale":"2024-04-14T23:48:43.214550+0000","last_undegraded":"2024-04-14T23:48:43.214550+0000","last_fullsized":"2024-04-14T23:48:43.214550+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:29:35.218759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.233225+0000","last_change":"2024-04-14T23:48:43.234025+0000","last_active":"2024-04-14T23:48:44.233225+0000","last_peered":"2024-04-14T23:48:44.233225+0000","last_clean":"2024-04-14T23:48:44.233225+0000","last_became_active":"2024-04-14T23:48:43.233651+0000","last_became_peered":"2024-04-14T23:48:43.233651+0000","last_unstale":"2024-04-14T23:48:44.233225+0000","last_undegraded":"2024-04-14T23:48:44.233225+0000","last_fullsized":"2024-04-14T23:48:44.233225+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:26:58.094545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.213165+0000","last_change":"2024-04-14T23:48:42.218897+0000","last_active":"2024-04-14T23:48:43.213165+0000","last_peered":"2024-04-14T23:48:43.213165+0000","last_clean":"2024-04-14T23:48:43.213165+0000","last_became_active":"2024-04-14T23:48:42.218580+0000","last_became_peered":"2024-04-14T23:48:42.218580+0000","last_unstale":"2024-04-14T23:48:43.213165+0000","last_undegraded":"2024-04-14T23:48:43.213165+0000","last_fullsized":"2024-04-14T23:48:43.213165+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:26:44.477638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.224966+0000","last_change":"2024-04-14T23:48:43.219871+0000","last_active":"2024-04-14T23:48:43.224966+0000","last_peered":"2024-04-14T23:48:43.224966+0000","last_clean":"2024-04-14T23:48:43.224966+0000","last_became_active":"2024-04-14T23:48:43.219601+0000","last_became_peered":"2024-04-14T23:48:43.219601+0000","last_unstale":"2024-04-14T23:48:43.224966+0000","last_undegraded":"2024-04-14T23:48:43.224966+0000","last_fullsized":"2024-04-14T23:48:43.224966+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:46:56.543156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230971+0000","last_change":"2024-04-14T23:48:43.222259+0000","last_active":"2024-04-14T23:48:44.230971+0000","last_peered":"2024-04-14T23:48:44.230971+0000","last_clean":"2024-04-14T23:48:44.230971+0000","last_became_active":"2024-04-14T23:48:43.221842+0000","last_became_peered":"2024-04-14T23:48:43.221842+0000","last_unstale":"2024-04-14T23:48:44.230971+0000","last_undegraded":"2024-04-14T23:48:44.230971+0000","last_fullsized":"2024-04-14T23:48:44.230971+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:12:20.038987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.212374+0000","last_change":"2024-04-14T23:48:42.221314+0000","last_active":"2024-04-14T23:48:43.212374+0000","last_peered":"2024-04-14T23:48:43.212374+0000","last_clean":"2024-04-14T23:48:43.212374+0000","last_became_active":"2024-04-14T23:48:42.221028+0000","last_became_peered":"2024-04-14T23:48:42.221028+0000","last_unstale":"2024-04-14T23:48:43.212374+0000","last_undegraded":"2024-04-14T23:48:43.212374+0000","last_fullsized":"2024-04-14T23:48:43.212374+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T09:58:18.195400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230606+0000","last_change":"2024-04-14T23:48:43.224577+0000","last_active":"2024-04-14T23:48:44.230606+0000","last_peered":"2024-04-14T23:48:44.230606+0000","last_clean":"2024-04-14T23:48:44.230606+0000","last_became_active":"2024-04-14T23:48:43.224312+0000","last_became_peered":"2024-04-14T23:48:43.224312+0000","last_unstale":"2024-04-14T23:48:44.230606+0000","last_undegraded":"2024-04-14T23:48:44.230606+0000","last_fullsized":"2024-04-14T23:48:44.230606+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:52:44.955773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.213420+0000","last_change":"2024-04-14T23:48:42.221000+0000","last_active":"2024-04-14T23:48:43.213420+0000","last_peered":"2024-04-14T23:48:43.213420+0000","last_clean":"2024-04-14T23:48:43.213420+0000","last_became_active":"2024-04-14T23:48:42.219813+0000","last_became_peered":"2024-04-14T23:48:42.219813+0000","last_unstale":"2024-04-14T23:48:43.213420+0000","last_undegraded":"2024-04-14T23:48:43.213420+0000","last_fullsized":"2024-04-14T23:48:43.213420+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:00:38.107953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.241568+0000","last_change":"2024-04-14T23:48:43.236946+0000","last_active":"2024-04-14T23:48:43.241568+0000","last_peered":"2024-04-14T23:48:43.241568+0000","last_clean":"2024-04-14T23:48:43.241568+0000","last_became_active":"2024-04-14T23:48:43.236595+0000","last_became_peered":"2024-04-14T23:48:43.236595+0000","last_unstale":"2024-04-14T23:48:43.241568+0000","last_undegraded":"2024-04-14T23:48:43.241568+0000","last_fullsized":"2024-04-14T23:48:43.241568+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:57:34.820083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.234116+0000","last_change":"2024-04-14T23:48:42.239255+0000","last_active":"2024-04-14T23:48:44.234116+0000","last_peered":"2024-04-14T23:48:44.234116+0000","last_clean":"2024-04-14T23:48:44.234116+0000","last_became_active":"2024-04-14T23:48:42.238892+0000","last_became_peered":"2024-04-14T23:48:42.238892+0000","last_unstale":"2024-04-14T23:48:44.234116+0000","last_undegraded":"2024-04-14T23:48:44.234116+0000","last_fullsized":"2024-04-14T23:48:44.234116+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:44:39.632155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.233893+0000","last_change":"2024-04-14T23:48:43.227240+0000","last_active":"2024-04-14T23:48:44.233893+0000","last_peered":"2024-04-14T23:48:44.233893+0000","last_clean":"2024-04-14T23:48:44.233893+0000","last_became_active":"2024-04-14T23:48:43.222707+0000","last_became_peered":"2024-04-14T23:48:43.222707+0000","last_unstale":"2024-04-14T23:48:44.233893+0000","last_undegraded":"2024-04-14T23:48:44.233893+0000","last_fullsized":"2024-04-14T23:48:44.233893+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:02:17.231763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.223590+0000","last_change":"2024-04-14T23:48:42.226563+0000","last_active":"2024-04-14T23:48:43.223590+0000","last_peered":"2024-04-14T23:48:43.223590+0000","last_clean":"2024-04-14T23:48:43.223590+0000","last_became_active":"2024-04-14T23:48:42.226238+0000","last_became_peered":"2024-04-14T23:48:42.226238+0000","last_unstale":"2024-04-14T23:48:43.223590+0000","last_undegraded":"2024-04-14T23:48:43.223590+0000","last_fullsized":"2024-04-14T23:48:43.223590+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:01:36.516898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65128,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":270336,"data_stored":659673,"data_compressed":32564,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":69305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":69305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574851,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574852,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":69305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574852,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":69305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574851,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":270336,"data_stored":659673,"data_compressed":32564,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32564,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32564,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-14T23:48:48.720 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-14T23:48:48.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.979+0000 7fa1e6a00640 1 Processor -- start 2024-04-14T23:48:48.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.979+0000 7fa1e6a00640 1 -- start start 2024-04-14T23:48:48.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.981+0000 7fa1e6a00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa1e00b1e00 0x7fa1e00c6950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:48.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.981+0000 7fa1e6a00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa1e00c3d90 0x7fa1e00c41e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:48.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.981+0000 7fa1e6a00640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1e0152fb0 con 0x7fa1e0073320 2024-04-14T23:48:48.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.981+0000 7fa1e6a00640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fa1e00c6540 con 0x7fa1e00c3d90 2024-04-14T23:48:48.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.981+0000 7fa1e6a00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fa1e00b21e0 con 0x7fa1e00b1e00 2024-04-14T23:48:48.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.981+0000 7fa1e4c00640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fa1e0073320 0x7fa1e00c7fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.60:56792/0 (socket says 172.21.15.60:56792) 2024-04-14T23:48:48.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.981+0000 7fa1e4c00640 1 -- 172.21.15.60:0/566318731 learned_addr learned my addr 172.21.15.60:0/566318731 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:48:48.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.981+0000 7fa1e5600640 1 --2- 172.21.15.60:0/566318731 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa1e00b1e00 0x7fa1e00c6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:48.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.981+0000 7fa1e6000640 1 --2- 172.21.15.60:0/566318731 >> v2:172.21.15.60:3300/0 conn(0x7fa1e00c3d90 0x7fa1e00c41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:48.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.982+0000 7fa1e5600640 1 -- 172.21.15.60:0/566318731 >> v2:172.21.15.60:3300/0 conn(0x7fa1e00c3d90 msgr2=0x7fa1e00c41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:48.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.982+0000 7fa1e5600640 1 --2- 172.21.15.60:0/566318731 >> v2:172.21.15.60:3300/0 conn(0x7fa1e00c3d90 0x7fa1e00c41e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:48.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.982+0000 7fa1e5600640 1 -- 172.21.15.60:0/566318731 >> v1:172.21.15.186:6789/0 conn(0x7fa1e0073320 legacy=0x7fa1e00c7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:48.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.982+0000 7fa1e5600640 1 -- 172.21.15.60:0/566318731 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1e0159310 con 0x7fa1e00b1e00 2024-04-14T23:48:48.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.982+0000 7fa1dfe00640 1 -- 172.21.15.60:0/566318731 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4038068736 0 0) 0x7fa1e0152fb0 con 0x7fa1e0073320 2024-04-14T23:48:48.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.983+0000 7fa1e5600640 1 --2- 172.21.15.60:0/566318731 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa1e00b1e00 0x7fa1e00c6950 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa1d40097a0 tx=0x7fa1d402b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d796e06f86795c5c server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:48.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.983+0000 7fa1dfe00640 1 -- 172.21.15.60:0/566318731 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1d4034070 con 0x7fa1e00b1e00 2024-04-14T23:48:48.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.983+0000 7fa1dfe00640 1 -- 172.21.15.60:0/566318731 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1d4030610 con 0x7fa1e00b1e00 2024-04-14T23:48:48.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.983+0000 7fa1dfe00640 1 -- 172.21.15.60:0/566318731 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1d40309e0 con 0x7fa1e00b1e00 2024-04-14T23:48:48.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.984+0000 7fa1e6a00640 1 -- 172.21.15.60:0/566318731 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa1e00b1e00 msgr2=0x7fa1e00c6950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:48.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.984+0000 7fa1e6a00640 1 --2- 172.21.15.60:0/566318731 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa1e00b1e00 0x7fa1e00c6950 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa1d40097a0 tx=0x7fa1d402b960 comp rx=0 tx=0).stop 2024-04-14T23:48:48.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.984+0000 7fa1e6a00640 1 -- 172.21.15.60:0/566318731 shutdown_connections 2024-04-14T23:48:48.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.984+0000 7fa1e6a00640 1 --2- 172.21.15.60:0/566318731 >> v2:172.21.15.60:3300/0 conn(0x7fa1e00c3d90 0x7fa1e00c41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:48.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.984+0000 7fa1e6a00640 1 --2- 172.21.15.60:0/566318731 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa1e00b1e00 0x7fa1e00c6950 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:48.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.984+0000 7fa1e6a00640 1 -- 172.21.15.60:0/566318731 >> 172.21.15.60:0/566318731 conn(0x7fa1e00c8b50 msgr2=0x7fa1e00c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:48.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.984+0000 7fa1e6a00640 1 -- 172.21.15.60:0/566318731 shutdown_connections 2024-04-14T23:48:48.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.984+0000 7fa1e6a00640 1 -- 172.21.15.60:0/566318731 wait complete. 2024-04-14T23:48:48.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.986+0000 7fa1e6a00640 1 Processor -- start 2024-04-14T23:48:48.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.986+0000 7fa1e6a00640 1 -- start start 2024-04-14T23:48:48.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.987+0000 7fa1e6a00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1e0073320 0x7fa1e0153130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:48.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.987+0000 7fa1e6a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1e00b1e00 0x7fa1e00ab040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:48.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.987+0000 7fa1e6a00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa1e00c3d90 0x7fa1e01522b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:48.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.987+0000 7fa1e6a00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fa1e00c38b0 con 0x7fa1e0073320 2024-04-14T23:48:48.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.987+0000 7fa1e6a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa1e0159890 con 0x7fa1e00b1e00 2024-04-14T23:48:48.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.987+0000 7fa1e6a00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fa1e01595b0 con 0x7fa1e00c3d90 2024-04-14T23:48:48.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.987+0000 7fa1e5600640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1e0073320 0x7fa1e0153130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:48.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.987+0000 7fa1e5600640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1e0073320 0x7fa1e0153130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53938/0 (socket says 172.21.15.60:53938) 2024-04-14T23:48:48.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.987+0000 7fa1e5600640 1 -- 172.21.15.60:0/911200382 learned_addr learned my addr 172.21.15.60:0/911200382 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:48.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.987+0000 7fa1e4c00640 1 --2- 172.21.15.60:0/911200382 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1e00b1e00 0x7fa1e00ab040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:48.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.987+0000 7fa1e6000640 1 --2- 172.21.15.60:0/911200382 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa1e00c3d90 0x7fa1e01522b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:48.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.987+0000 7fa1e5600640 1 -- 172.21.15.60:0/911200382 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa1e00c3d90 msgr2=0x7fa1e01522b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:48.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.987+0000 7fa1e5600640 1 --2- 172.21.15.60:0/911200382 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa1e00c3d90 0x7fa1e01522b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:48.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.988+0000 7fa1e5600640 1 -- 172.21.15.60:0/911200382 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1e00b1e00 msgr2=0x7fa1e00ab040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:48.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.988+0000 7fa1e5600640 1 --2- 172.21.15.60:0/911200382 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1e00b1e00 0x7fa1e00ab040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:48.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.988+0000 7fa1e5600640 1 -- 172.21.15.60:0/911200382 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1e0073100 con 0x7fa1e0073320 2024-04-14T23:48:48.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.988+0000 7fa1e5600640 1 --2- 172.21.15.60:0/911200382 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1e0073320 0x7fa1e0153130 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa1d402bf70 tx=0x7fa1d40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:48.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.988+0000 7fa1dd600640 1 -- 172.21.15.60:0/911200382 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1d4034040 con 0x7fa1e0073320 2024-04-14T23:48:48.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.988+0000 7fa1dd600640 1 -- 172.21.15.60:0/911200382 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1d4030410 con 0x7fa1e0073320 2024-04-14T23:48:48.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.988+0000 7fa1dd600640 1 -- 172.21.15.60:0/911200382 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1d4039c30 con 0x7fa1e0073320 2024-04-14T23:48:48.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.988+0000 7fa1e6a00640 1 -- 172.21.15.60:0/911200382 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1e0071890 con 0x7fa1e0073320 2024-04-14T23:48:48.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.988+0000 7fa1e6a00640 1 -- 172.21.15.60:0/911200382 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1e0072e20 con 0x7fa1e0073320 2024-04-14T23:48:48.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.989+0000 7fa1e6a00640 1 -- 172.21.15.60:0/911200382 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1e004f540 con 0x7fa1e0073320 2024-04-14T23:48:48.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.990+0000 7fa1dd600640 1 -- 172.21.15.60:0/911200382 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7fa1d4037070 con 0x7fa1e0073320 2024-04-14T23:48:48.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.991+0000 7fa1dd600640 1 --2- 172.21.15.60:0/911200382 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fa1c006b990 0x7fa1c006de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:48.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.991+0000 7fa1dd600640 1 -- 172.21.15.60:0/911200382 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fa1d40ac780 con 0x7fa1e0073320 2024-04-14T23:48:48.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.992+0000 7fa1e4c00640 1 --2- 172.21.15.60:0/911200382 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fa1c006b990 0x7fa1c006de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:48.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.992+0000 7fa1e4c00640 1 --2- 172.21.15.60:0/911200382 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fa1c006b990 0x7fa1c006de50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa1e0152cd0 tx=0x7fa1cc007e40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:49.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:48.996+0000 7fa1dd600640 1 -- 172.21.15.60:0/911200382 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa1d40305e0 con 0x7fa1e0073320 2024-04-14T23:48:49.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.195+0000 7fa1e6a00640 1 -- 172.21.15.60:0/911200382 --> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa1e00ab580 con 0x7fa1c006b990 2024-04-14T23:48:49.209 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.203+0000 7fa1dd600640 1 -- 172.21.15.60:0/911200382 <== mgr.4100 v2:172.21.15.186:6832/714104835 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251890 (secure 0 0 0) 0x7fa1e00ab580 con 0x7fa1c006b990 2024-04-14T23:48:49.212 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:48:49.213 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-04-14T23:48:49.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.208+0000 7fa1e6a00640 1 -- 172.21.15.60:0/911200382 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fa1c006b990 msgr2=0x7fa1c006de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:49.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.208+0000 7fa1e6a00640 1 --2- 172.21.15.60:0/911200382 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fa1c006b990 0x7fa1c006de50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa1e0152cd0 tx=0x7fa1cc007e40 comp rx=0 tx=0).stop 2024-04-14T23:48:49.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.208+0000 7fa1e6a00640 1 -- 172.21.15.60:0/911200382 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1e0073320 msgr2=0x7fa1e0153130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:49.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.208+0000 7fa1e6a00640 1 --2- 172.21.15.60:0/911200382 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1e0073320 0x7fa1e0153130 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa1d402bf70 tx=0x7fa1d40046c0 comp rx=0 tx=0).stop 2024-04-14T23:48:49.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.209+0000 7fa1e6a00640 1 -- 172.21.15.60:0/911200382 shutdown_connections 2024-04-14T23:48:49.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.209+0000 7fa1e6a00640 1 --2- 172.21.15.60:0/911200382 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fa1c006b990 0x7fa1c006de50 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:49.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.209+0000 7fa1e6a00640 1 --2- 172.21.15.60:0/911200382 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa1e00c3d90 0x7fa1e01522b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:49.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.209+0000 7fa1e6a00640 1 --2- 172.21.15.60:0/911200382 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1e00b1e00 0x7fa1e00ab040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:49.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.209+0000 7fa1e6a00640 1 --2- 172.21.15.60:0/911200382 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1e0073320 0x7fa1e0153130 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:49.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.209+0000 7fa1e6a00640 1 -- 172.21.15.60:0/911200382 >> 172.21.15.60:0/911200382 conn(0x7fa1e00c8b50 msgr2=0x7fa1e00ca730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:49.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.209+0000 7fa1e6a00640 1 -- 172.21.15.60:0/911200382 shutdown_connections 2024-04-14T23:48:49.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.209+0000 7fa1e6a00640 1 -- 172.21.15.60:0/911200382 wait complete. 2024-04-14T23:48:49.235 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-04-14T23:48:48.167455+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":152,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":217972,"kb_used_data":1752,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756748,"statfs":{"total":773094113280,"available":772870909952,"internally_reserved":0,"allocated":1794048,"data_stored":1747776,"data_compressed":65128,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":14,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001478"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.227573+0000","last_change":"2024-04-14T23:48:43.222542+0000","last_active":"2024-04-14T23:48:43.227573+0000","last_peered":"2024-04-14T23:48:43.227573+0000","last_clean":"2024-04-14T23:48:43.227573+0000","last_became_active":"2024-04-14T23:48:43.222278+0000","last_became_peered":"2024-04-14T23:48:43.222278+0000","last_unstale":"2024-04-14T23:48:43.227573+0000","last_undegraded":"2024-04-14T23:48:43.227573+0000","last_fullsized":"2024-04-14T23:48:43.227573+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:36:13.119301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.236713+0000","last_change":"2024-04-14T23:48:43.226781+0000","last_active":"2024-04-14T23:48:43.236713+0000","last_peered":"2024-04-14T23:48:43.236713+0000","last_clean":"2024-04-14T23:48:43.236713+0000","last_became_active":"2024-04-14T23:48:43.226426+0000","last_became_peered":"2024-04-14T23:48:43.226426+0000","last_unstale":"2024-04-14T23:48:43.236713+0000","last_undegraded":"2024-04-14T23:48:43.236713+0000","last_fullsized":"2024-04-14T23:48:43.236713+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T04:32:52.985782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.241614+0000","last_change":"2024-04-14T23:48:43.236939+0000","last_active":"2024-04-14T23:48:43.241614+0000","last_peered":"2024-04-14T23:48:43.241614+0000","last_clean":"2024-04-14T23:48:43.241614+0000","last_became_active":"2024-04-14T23:48:43.236616+0000","last_became_peered":"2024-04-14T23:48:43.236616+0000","last_unstale":"2024-04-14T23:48:43.241614+0000","last_undegraded":"2024-04-14T23:48:43.241614+0000","last_fullsized":"2024-04-14T23:48:43.241614+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:40:24.166032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230420+0000","last_change":"2024-04-14T23:48:43.242349+0000","last_active":"2024-04-14T23:48:44.230420+0000","last_peered":"2024-04-14T23:48:44.230420+0000","last_clean":"2024-04-14T23:48:44.230420+0000","last_became_active":"2024-04-14T23:48:43.242000+0000","last_became_peered":"2024-04-14T23:48:43.242000+0000","last_unstale":"2024-04-14T23:48:44.230420+0000","last_undegraded":"2024-04-14T23:48:44.230420+0000","last_fullsized":"2024-04-14T23:48:44.230420+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:33:52.458016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.238844+0000","last_change":"2024-04-14T23:48:43.233937+0000","last_active":"2024-04-14T23:48:43.238844+0000","last_peered":"2024-04-14T23:48:43.238844+0000","last_clean":"2024-04-14T23:48:43.238844+0000","last_became_active":"2024-04-14T23:48:43.233674+0000","last_became_peered":"2024-04-14T23:48:43.233674+0000","last_unstale":"2024-04-14T23:48:43.238844+0000","last_undegraded":"2024-04-14T23:48:43.238844+0000","last_fullsized":"2024-04-14T23:48:43.238844+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:45:47.377195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230260+0000","last_change":"2024-04-14T23:48:43.230925+0000","last_active":"2024-04-14T23:48:44.230260+0000","last_peered":"2024-04-14T23:48:44.230260+0000","last_clean":"2024-04-14T23:48:44.230260+0000","last_became_active":"2024-04-14T23:48:43.230405+0000","last_became_peered":"2024-04-14T23:48:43.230405+0000","last_unstale":"2024-04-14T23:48:44.230260+0000","last_undegraded":"2024-04-14T23:48:44.230260+0000","last_fullsized":"2024-04-14T23:48:44.230260+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:15:12.452964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.238591+0000","last_change":"2024-04-14T23:48:43.231088+0000","last_active":"2024-04-14T23:48:43.238591+0000","last_peered":"2024-04-14T23:48:43.238591+0000","last_clean":"2024-04-14T23:48:43.238591+0000","last_became_active":"2024-04-14T23:48:43.230784+0000","last_became_peered":"2024-04-14T23:48:43.230784+0000","last_unstale":"2024-04-14T23:48:43.238591+0000","last_undegraded":"2024-04-14T23:48:43.238591+0000","last_fullsized":"2024-04-14T23:48:43.238591+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:50:33.963376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.231230+0000","last_change":"2024-04-14T23:48:43.236019+0000","last_active":"2024-04-14T23:48:44.231230+0000","last_peered":"2024-04-14T23:48:44.231230+0000","last_clean":"2024-04-14T23:48:44.231230+0000","last_became_active":"2024-04-14T23:48:43.235719+0000","last_became_peered":"2024-04-14T23:48:43.235719+0000","last_unstale":"2024-04-14T23:48:44.231230+0000","last_undegraded":"2024-04-14T23:48:44.231230+0000","last_fullsized":"2024-04-14T23:48:44.231230+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T04:41:16.130730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.238782+0000","last_change":"2024-04-14T23:48:43.232166+0000","last_active":"2024-04-14T23:48:43.238782+0000","last_peered":"2024-04-14T23:48:43.238782+0000","last_clean":"2024-04-14T23:48:43.238782+0000","last_became_active":"2024-04-14T23:48:43.231751+0000","last_became_peered":"2024-04-14T23:48:43.231751+0000","last_unstale":"2024-04-14T23:48:43.238782+0000","last_undegraded":"2024-04-14T23:48:43.238782+0000","last_fullsized":"2024-04-14T23:48:43.238782+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T09:58:34.853704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.228924+0000","last_change":"2024-04-14T23:48:43.224938+0000","last_active":"2024-04-14T23:48:43.228924+0000","last_peered":"2024-04-14T23:48:43.228924+0000","last_clean":"2024-04-14T23:48:43.228924+0000","last_became_active":"2024-04-14T23:48:43.224592+0000","last_became_peered":"2024-04-14T23:48:43.224592+0000","last_unstale":"2024-04-14T23:48:43.228924+0000","last_undegraded":"2024-04-14T23:48:43.228924+0000","last_fullsized":"2024-04-14T23:48:43.228924+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T04:44:06.756499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.229987+0000","last_change":"2024-04-14T23:48:43.225290+0000","last_active":"2024-04-14T23:48:43.229987+0000","last_peered":"2024-04-14T23:48:43.229987+0000","last_clean":"2024-04-14T23:48:43.229987+0000","last_became_active":"2024-04-14T23:48:43.224989+0000","last_became_peered":"2024-04-14T23:48:43.224989+0000","last_unstale":"2024-04-14T23:48:43.229987+0000","last_undegraded":"2024-04-14T23:48:43.229987+0000","last_fullsized":"2024-04-14T23:48:43.229987+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T09:52:55.826011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.229660+0000","last_change":"2024-04-14T23:48:43.226194+0000","last_active":"2024-04-14T23:48:43.229660+0000","last_peered":"2024-04-14T23:48:43.229660+0000","last_clean":"2024-04-14T23:48:43.229660+0000","last_became_active":"2024-04-14T23:48:43.225892+0000","last_became_peered":"2024-04-14T23:48:43.225892+0000","last_unstale":"2024-04-14T23:48:43.229660+0000","last_undegraded":"2024-04-14T23:48:43.229660+0000","last_fullsized":"2024-04-14T23:48:43.229660+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:30:27.034185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.233039+0000","last_change":"2024-04-14T23:48:43.220571+0000","last_active":"2024-04-14T23:48:44.233039+0000","last_peered":"2024-04-14T23:48:44.233039+0000","last_clean":"2024-04-14T23:48:44.233039+0000","last_became_active":"2024-04-14T23:48:43.219614+0000","last_became_peered":"2024-04-14T23:48:43.219614+0000","last_unstale":"2024-04-14T23:48:44.233039+0000","last_undegraded":"2024-04-14T23:48:44.233039+0000","last_fullsized":"2024-04-14T23:48:44.233039+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:14:41.854957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.241634+0000","last_change":"2024-04-14T23:48:43.239403+0000","last_active":"2024-04-14T23:48:43.241634+0000","last_peered":"2024-04-14T23:48:43.241634+0000","last_clean":"2024-04-14T23:48:43.241634+0000","last_became_active":"2024-04-14T23:48:43.239136+0000","last_became_peered":"2024-04-14T23:48:43.239136+0000","last_unstale":"2024-04-14T23:48:43.241634+0000","last_undegraded":"2024-04-14T23:48:43.241634+0000","last_fullsized":"2024-04-14T23:48:43.241634+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:58:46.205808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.237076+0000","last_change":"2024-04-14T23:48:43.228157+0000","last_active":"2024-04-14T23:48:43.237076+0000","last_peered":"2024-04-14T23:48:43.237076+0000","last_clean":"2024-04-14T23:48:43.237076+0000","last_became_active":"2024-04-14T23:48:43.227898+0000","last_became_peered":"2024-04-14T23:48:43.227898+0000","last_unstale":"2024-04-14T23:48:43.237076+0000","last_undegraded":"2024-04-14T23:48:43.237076+0000","last_fullsized":"2024-04-14T23:48:43.237076+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:43:38.967154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230387+0000","last_change":"2024-04-14T23:48:43.238986+0000","last_active":"2024-04-14T23:48:44.230387+0000","last_peered":"2024-04-14T23:48:44.230387+0000","last_clean":"2024-04-14T23:48:44.230387+0000","last_became_active":"2024-04-14T23:48:43.238714+0000","last_became_peered":"2024-04-14T23:48:43.238714+0000","last_unstale":"2024-04-14T23:48:44.230387+0000","last_undegraded":"2024-04-14T23:48:44.230387+0000","last_fullsized":"2024-04-14T23:48:44.230387+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:05:30.655048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.231004+0000","last_change":"2024-04-14T23:48:43.229325+0000","last_active":"2024-04-14T23:48:44.231004+0000","last_peered":"2024-04-14T23:48:44.231004+0000","last_clean":"2024-04-14T23:48:44.231004+0000","last_became_active":"2024-04-14T23:48:43.228618+0000","last_became_peered":"2024-04-14T23:48:43.228618+0000","last_unstale":"2024-04-14T23:48:44.231004+0000","last_undegraded":"2024-04-14T23:48:44.231004+0000","last_fullsized":"2024-04-14T23:48:44.231004+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:06:43.745698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.238158+0000","last_change":"2024-04-14T23:48:43.233288+0000","last_active":"2024-04-14T23:48:43.238158+0000","last_peered":"2024-04-14T23:48:43.238158+0000","last_clean":"2024-04-14T23:48:43.238158+0000","last_became_active":"2024-04-14T23:48:43.232971+0000","last_became_peered":"2024-04-14T23:48:43.232971+0000","last_unstale":"2024-04-14T23:48:43.238158+0000","last_undegraded":"2024-04-14T23:48:43.238158+0000","last_fullsized":"2024-04-14T23:48:43.238158+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:35:56.958355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.238872+0000","last_change":"2024-04-14T23:48:43.233169+0000","last_active":"2024-04-14T23:48:43.238872+0000","last_peered":"2024-04-14T23:48:43.238872+0000","last_clean":"2024-04-14T23:48:43.238872+0000","last_became_active":"2024-04-14T23:48:43.232135+0000","last_became_peered":"2024-04-14T23:48:43.232135+0000","last_unstale":"2024-04-14T23:48:43.238872+0000","last_undegraded":"2024-04-14T23:48:43.238872+0000","last_fullsized":"2024-04-14T23:48:43.238872+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:17:08.188120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.233097+0000","last_change":"2024-04-14T23:48:43.228728+0000","last_active":"2024-04-14T23:48:43.233097+0000","last_peered":"2024-04-14T23:48:43.233097+0000","last_clean":"2024-04-14T23:48:43.233097+0000","last_became_active":"2024-04-14T23:48:43.226225+0000","last_became_peered":"2024-04-14T23:48:43.226225+0000","last_unstale":"2024-04-14T23:48:43.233097+0000","last_undegraded":"2024-04-14T23:48:43.233097+0000","last_fullsized":"2024-04-14T23:48:43.233097+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:09:05.048539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.241682+0000","last_change":"2024-04-14T23:48:43.237234+0000","last_active":"2024-04-14T23:48:43.241682+0000","last_peered":"2024-04-14T23:48:43.241682+0000","last_clean":"2024-04-14T23:48:43.241682+0000","last_became_active":"2024-04-14T23:48:43.230056+0000","last_became_peered":"2024-04-14T23:48:43.230056+0000","last_unstale":"2024-04-14T23:48:43.241682+0000","last_undegraded":"2024-04-14T23:48:43.241682+0000","last_fullsized":"2024-04-14T23:48:43.241682+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:30:02.685365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230697+0000","last_change":"2024-04-14T23:48:43.229387+0000","last_active":"2024-04-14T23:48:44.230697+0000","last_peered":"2024-04-14T23:48:44.230697+0000","last_clean":"2024-04-14T23:48:44.230697+0000","last_became_active":"2024-04-14T23:48:43.229110+0000","last_became_peered":"2024-04-14T23:48:43.229110+0000","last_unstale":"2024-04-14T23:48:44.230697+0000","last_undegraded":"2024-04-14T23:48:44.230697+0000","last_fullsized":"2024-04-14T23:48:44.230697+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T09:18:46.939940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.232750+0000","last_change":"2024-04-14T23:48:43.234154+0000","last_active":"2024-04-14T23:48:44.232750+0000","last_peered":"2024-04-14T23:48:44.232750+0000","last_clean":"2024-04-14T23:48:44.232750+0000","last_became_active":"2024-04-14T23:48:43.233853+0000","last_became_peered":"2024-04-14T23:48:43.233853+0000","last_unstale":"2024-04-14T23:48:44.232750+0000","last_undegraded":"2024-04-14T23:48:44.232750+0000","last_fullsized":"2024-04-14T23:48:44.232750+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T23:54:02.200200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230330+0000","last_change":"2024-04-14T23:48:43.242154+0000","last_active":"2024-04-14T23:48:44.230330+0000","last_peered":"2024-04-14T23:48:44.230330+0000","last_clean":"2024-04-14T23:48:44.230330+0000","last_became_active":"2024-04-14T23:48:43.241808+0000","last_became_peered":"2024-04-14T23:48:43.241808+0000","last_unstale":"2024-04-14T23:48:44.230330+0000","last_undegraded":"2024-04-14T23:48:44.230330+0000","last_fullsized":"2024-04-14T23:48:44.230330+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:48:42.680808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.232996+0000","last_change":"2024-04-14T23:48:43.233530+0000","last_active":"2024-04-14T23:48:44.232996+0000","last_peered":"2024-04-14T23:48:44.232996+0000","last_clean":"2024-04-14T23:48:44.232996+0000","last_became_active":"2024-04-14T23:48:43.233238+0000","last_became_peered":"2024-04-14T23:48:43.233238+0000","last_unstale":"2024-04-14T23:48:44.232996+0000","last_undegraded":"2024-04-14T23:48:44.232996+0000","last_fullsized":"2024-04-14T23:48:44.232996+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:10:58.525655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.228879+0000","last_change":"2024-04-14T23:48:43.222945+0000","last_active":"2024-04-14T23:48:43.228879+0000","last_peered":"2024-04-14T23:48:43.228879+0000","last_clean":"2024-04-14T23:48:43.228879+0000","last_became_active":"2024-04-14T23:48:43.222624+0000","last_became_peered":"2024-04-14T23:48:43.222624+0000","last_unstale":"2024-04-14T23:48:43.228879+0000","last_undegraded":"2024-04-14T23:48:43.228879+0000","last_fullsized":"2024-04-14T23:48:43.228879+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:03:28.128723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.240068+0000","last_change":"2024-04-14T23:48:43.232567+0000","last_active":"2024-04-14T23:48:43.240068+0000","last_peered":"2024-04-14T23:48:43.240068+0000","last_clean":"2024-04-14T23:48:43.240068+0000","last_became_active":"2024-04-14T23:48:43.232277+0000","last_became_peered":"2024-04-14T23:48:43.232277+0000","last_unstale":"2024-04-14T23:48:43.240068+0000","last_undegraded":"2024-04-14T23:48:43.240068+0000","last_fullsized":"2024-04-14T23:48:43.240068+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:32:30.528043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.232426+0000","last_change":"2024-04-14T23:48:43.229443+0000","last_active":"2024-04-14T23:48:44.232426+0000","last_peered":"2024-04-14T23:48:44.232426+0000","last_clean":"2024-04-14T23:48:44.232426+0000","last_became_active":"2024-04-14T23:48:43.229172+0000","last_became_peered":"2024-04-14T23:48:43.229172+0000","last_unstale":"2024-04-14T23:48:44.232426+0000","last_undegraded":"2024-04-14T23:48:44.232426+0000","last_fullsized":"2024-04-14T23:48:44.232426+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:41:31.405287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.244064+0000","last_change":"2024-04-14T23:48:43.242202+0000","last_active":"2024-04-14T23:48:43.244064+0000","last_peered":"2024-04-14T23:48:43.244064+0000","last_clean":"2024-04-14T23:48:43.244064+0000","last_became_active":"2024-04-14T23:48:43.241879+0000","last_became_peered":"2024-04-14T23:48:43.241879+0000","last_unstale":"2024-04-14T23:48:43.244064+0000","last_undegraded":"2024-04-14T23:48:43.244064+0000","last_fullsized":"2024-04-14T23:48:43.244064+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:47:46.306480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230871+0000","last_change":"2024-04-14T23:48:43.242068+0000","last_active":"2024-04-14T23:48:44.230871+0000","last_peered":"2024-04-14T23:48:44.230871+0000","last_clean":"2024-04-14T23:48:44.230871+0000","last_became_active":"2024-04-14T23:48:43.241734+0000","last_became_peered":"2024-04-14T23:48:43.241734+0000","last_unstale":"2024-04-14T23:48:44.230871+0000","last_undegraded":"2024-04-14T23:48:44.230871+0000","last_fullsized":"2024-04-14T23:48:44.230871+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:32:04.579906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.239805+0000","last_change":"2024-04-14T23:48:43.236034+0000","last_active":"2024-04-14T23:48:43.239805+0000","last_peered":"2024-04-14T23:48:43.239805+0000","last_clean":"2024-04-14T23:48:43.239805+0000","last_became_active":"2024-04-14T23:48:43.233567+0000","last_became_peered":"2024-04-14T23:48:43.233567+0000","last_unstale":"2024-04-14T23:48:43.239805+0000","last_undegraded":"2024-04-14T23:48:43.239805+0000","last_fullsized":"2024-04-14T23:48:43.239805+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T23:49:51.630628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.234495+0000","last_change":"2024-04-14T23:48:43.231045+0000","last_active":"2024-04-14T23:48:43.234495+0000","last_peered":"2024-04-14T23:48:43.234495+0000","last_clean":"2024-04-14T23:48:43.234495+0000","last_became_active":"2024-04-14T23:48:43.230626+0000","last_became_peered":"2024-04-14T23:48:43.230626+0000","last_unstale":"2024-04-14T23:48:43.234495+0000","last_undegraded":"2024-04-14T23:48:43.234495+0000","last_fullsized":"2024-04-14T23:48:43.234495+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:45:19.807215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.229591+0000","last_change":"2024-04-14T23:48:43.227543+0000","last_active":"2024-04-14T23:48:43.229591+0000","last_peered":"2024-04-14T23:48:43.229591+0000","last_clean":"2024-04-14T23:48:43.229591+0000","last_became_active":"2024-04-14T23:48:43.226773+0000","last_became_peered":"2024-04-14T23:48:43.226773+0000","last_unstale":"2024-04-14T23:48:43.229591+0000","last_undegraded":"2024-04-14T23:48:43.229591+0000","last_fullsized":"2024-04-14T23:48:43.229591+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:12:10.175172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.233181+0000","last_change":"2024-04-14T23:48:42.239254+0000","last_active":"2024-04-14T23:48:44.233181+0000","last_peered":"2024-04-14T23:48:44.233181+0000","last_clean":"2024-04-14T23:48:44.233181+0000","last_became_active":"2024-04-14T23:48:42.238847+0000","last_became_peered":"2024-04-14T23:48:42.238847+0000","last_unstale":"2024-04-14T23:48:44.233181+0000","last_undegraded":"2024-04-14T23:48:44.233181+0000","last_fullsized":"2024-04-14T23:48:44.233181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:46:16.301900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.241700+0000","last_change":"2024-04-14T23:48:43.237720+0000","last_active":"2024-04-14T23:48:43.241700+0000","last_peered":"2024-04-14T23:48:43.241700+0000","last_clean":"2024-04-14T23:48:43.241700+0000","last_became_active":"2024-04-14T23:48:43.237391+0000","last_became_peered":"2024-04-14T23:48:43.237391+0000","last_unstale":"2024-04-14T23:48:43.241700+0000","last_undegraded":"2024-04-14T23:48:43.241700+0000","last_fullsized":"2024-04-14T23:48:43.241700+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:15:38.587904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.216530+0000","last_change":"2024-04-14T23:48:42.219277+0000","last_active":"2024-04-14T23:48:43.216530+0000","last_peered":"2024-04-14T23:48:43.216530+0000","last_clean":"2024-04-14T23:48:43.216530+0000","last_became_active":"2024-04-14T23:48:42.218995+0000","last_became_peered":"2024-04-14T23:48:42.218995+0000","last_unstale":"2024-04-14T23:48:43.216530+0000","last_undegraded":"2024-04-14T23:48:43.216530+0000","last_fullsized":"2024-04-14T23:48:43.216530+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T23:51:33.370857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.232590+0000","last_change":"2024-04-14T23:48:42.236803+0000","last_active":"2024-04-14T23:48:44.232590+0000","last_peered":"2024-04-14T23:48:44.232590+0000","last_clean":"2024-04-14T23:48:44.232590+0000","last_became_active":"2024-04-14T23:48:42.225937+0000","last_became_peered":"2024-04-14T23:48:42.225937+0000","last_unstale":"2024-04-14T23:48:44.232590+0000","last_undegraded":"2024-04-14T23:48:44.232590+0000","last_fullsized":"2024-04-14T23:48:44.232590+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:52:32.609126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.231732+0000","last_change":"2024-04-14T23:48:43.229506+0000","last_active":"2024-04-14T23:48:44.231732+0000","last_peered":"2024-04-14T23:48:44.231732+0000","last_clean":"2024-04-14T23:48:44.231732+0000","last_became_active":"2024-04-14T23:48:43.227356+0000","last_became_peered":"2024-04-14T23:48:43.227356+0000","last_unstale":"2024-04-14T23:48:44.231732+0000","last_undegraded":"2024-04-14T23:48:44.231732+0000","last_fullsized":"2024-04-14T23:48:44.231732+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:13:04.840655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230660+0000","last_change":"2024-04-14T23:48:42.210883+0000","last_active":"2024-04-14T23:48:44.230660+0000","last_peered":"2024-04-14T23:48:44.230660+0000","last_clean":"2024-04-14T23:48:44.230660+0000","last_became_active":"2024-04-14T23:48:42.210607+0000","last_became_peered":"2024-04-14T23:48:42.210607+0000","last_unstale":"2024-04-14T23:48:44.230660+0000","last_undegraded":"2024-04-14T23:48:44.230660+0000","last_fullsized":"2024-04-14T23:48:44.230660+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:33:32.526223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.225460+0000","last_change":"2024-04-14T23:48:43.218259+0000","last_active":"2024-04-14T23:48:43.225460+0000","last_peered":"2024-04-14T23:48:43.225460+0000","last_clean":"2024-04-14T23:48:43.225460+0000","last_became_active":"2024-04-14T23:48:43.217801+0000","last_became_peered":"2024-04-14T23:48:43.217801+0000","last_unstale":"2024-04-14T23:48:43.225460+0000","last_undegraded":"2024-04-14T23:48:43.225460+0000","last_fullsized":"2024-04-14T23:48:43.225460+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:03:21.087919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.217021+0000","last_change":"2024-04-14T23:48:42.218652+0000","last_active":"2024-04-14T23:48:43.217021+0000","last_peered":"2024-04-14T23:48:43.217021+0000","last_clean":"2024-04-14T23:48:43.217021+0000","last_became_active":"2024-04-14T23:48:42.218321+0000","last_became_peered":"2024-04-14T23:48:42.218321+0000","last_unstale":"2024-04-14T23:48:43.217021+0000","last_undegraded":"2024-04-14T23:48:43.217021+0000","last_fullsized":"2024-04-14T23:48:43.217021+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T04:22:06.957747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230879+0000","last_change":"2024-04-14T23:48:43.229117+0000","last_active":"2024-04-14T23:48:44.230879+0000","last_peered":"2024-04-14T23:48:44.230879+0000","last_clean":"2024-04-14T23:48:44.230879+0000","last_became_active":"2024-04-14T23:48:43.228805+0000","last_became_peered":"2024-04-14T23:48:43.228805+0000","last_unstale":"2024-04-14T23:48:44.230879+0000","last_undegraded":"2024-04-14T23:48:44.230879+0000","last_fullsized":"2024-04-14T23:48:44.230879+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:10:39.043818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230930+0000","last_change":"2024-04-14T23:48:42.221104+0000","last_active":"2024-04-14T23:48:44.230930+0000","last_peered":"2024-04-14T23:48:44.230930+0000","last_clean":"2024-04-14T23:48:44.230930+0000","last_became_active":"2024-04-14T23:48:42.220767+0000","last_became_peered":"2024-04-14T23:48:42.220767+0000","last_unstale":"2024-04-14T23:48:44.230930+0000","last_undegraded":"2024-04-14T23:48:44.230930+0000","last_fullsized":"2024-04-14T23:48:44.230930+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:23:57.213764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.241543+0000","last_change":"2024-04-14T23:48:43.236916+0000","last_active":"2024-04-14T23:48:43.241543+0000","last_peered":"2024-04-14T23:48:43.241543+0000","last_clean":"2024-04-14T23:48:43.241543+0000","last_became_active":"2024-04-14T23:48:43.236168+0000","last_became_peered":"2024-04-14T23:48:43.236168+0000","last_unstale":"2024-04-14T23:48:43.241543+0000","last_undegraded":"2024-04-14T23:48:43.241543+0000","last_fullsized":"2024-04-14T23:48:43.241543+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:35:33.530129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.224228+0000","last_change":"2024-04-14T23:48:42.231329+0000","last_active":"2024-04-14T23:48:43.224228+0000","last_peered":"2024-04-14T23:48:43.224228+0000","last_clean":"2024-04-14T23:48:43.224228+0000","last_became_active":"2024-04-14T23:48:42.230969+0000","last_became_peered":"2024-04-14T23:48:42.230969+0000","last_unstale":"2024-04-14T23:48:43.224228+0000","last_undegraded":"2024-04-14T23:48:43.224228+0000","last_fullsized":"2024-04-14T23:48:43.224228+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:06:16.557589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.235654+0000","last_change":"2024-04-14T23:48:43.230531+0000","last_active":"2024-04-14T23:48:43.235654+0000","last_peered":"2024-04-14T23:48:43.235654+0000","last_clean":"2024-04-14T23:48:43.235654+0000","last_became_active":"2024-04-14T23:48:43.230253+0000","last_became_peered":"2024-04-14T23:48:43.230253+0000","last_unstale":"2024-04-14T23:48:43.235654+0000","last_undegraded":"2024-04-14T23:48:43.235654+0000","last_fullsized":"2024-04-14T23:48:43.235654+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:11:05.198292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.216588+0000","last_change":"2024-04-14T23:48:42.237069+0000","last_active":"2024-04-14T23:48:43.216588+0000","last_peered":"2024-04-14T23:48:43.216588+0000","last_clean":"2024-04-14T23:48:43.216588+0000","last_became_active":"2024-04-14T23:48:42.236804+0000","last_became_peered":"2024-04-14T23:48:42.236804+0000","last_unstale":"2024-04-14T23:48:43.216588+0000","last_undegraded":"2024-04-14T23:48:43.216588+0000","last_fullsized":"2024-04-14T23:48:43.216588+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:25:37.318379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.226657+0000","last_change":"2024-04-14T23:48:43.221771+0000","last_active":"2024-04-14T23:48:43.226657+0000","last_peered":"2024-04-14T23:48:43.226657+0000","last_clean":"2024-04-14T23:48:43.226657+0000","last_became_active":"2024-04-14T23:48:43.221462+0000","last_became_peered":"2024-04-14T23:48:43.221462+0000","last_unstale":"2024-04-14T23:48:43.226657+0000","last_undegraded":"2024-04-14T23:48:43.226657+0000","last_fullsized":"2024-04-14T23:48:43.226657+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:01:16.040458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.222096+0000","last_change":"2024-04-14T23:48:42.230518+0000","last_active":"2024-04-14T23:48:43.222096+0000","last_peered":"2024-04-14T23:48:43.222096+0000","last_clean":"2024-04-14T23:48:43.222096+0000","last_became_active":"2024-04-14T23:48:42.229758+0000","last_became_peered":"2024-04-14T23:48:42.229758+0000","last_unstale":"2024-04-14T23:48:43.222096+0000","last_undegraded":"2024-04-14T23:48:43.222096+0000","last_fullsized":"2024-04-14T23:48:43.222096+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:01:41.332553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.238539+0000","last_change":"2024-04-14T23:48:43.229873+0000","last_active":"2024-04-14T23:48:43.238539+0000","last_peered":"2024-04-14T23:48:43.238539+0000","last_clean":"2024-04-14T23:48:43.238539+0000","last_became_active":"2024-04-14T23:48:43.229476+0000","last_became_peered":"2024-04-14T23:48:43.229476+0000","last_unstale":"2024-04-14T23:48:43.238539+0000","last_undegraded":"2024-04-14T23:48:43.238539+0000","last_fullsized":"2024-04-14T23:48:43.238539+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:38:11.414948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230858+0000","last_change":"2024-04-14T23:48:42.225404+0000","last_active":"2024-04-14T23:48:44.230858+0000","last_peered":"2024-04-14T23:48:44.230858+0000","last_clean":"2024-04-14T23:48:44.230858+0000","last_became_active":"2024-04-14T23:48:42.225211+0000","last_became_peered":"2024-04-14T23:48:42.225211+0000","last_unstale":"2024-04-14T23:48:44.230858+0000","last_undegraded":"2024-04-14T23:48:44.230858+0000","last_fullsized":"2024-04-14T23:48:44.230858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:49:45.282259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.238724+0000","last_change":"2024-04-14T23:48:43.229543+0000","last_active":"2024-04-14T23:48:43.238724+0000","last_peered":"2024-04-14T23:48:43.238724+0000","last_clean":"2024-04-14T23:48:43.238724+0000","last_became_active":"2024-04-14T23:48:43.229297+0000","last_became_peered":"2024-04-14T23:48:43.229297+0000","last_unstale":"2024-04-14T23:48:43.238724+0000","last_undegraded":"2024-04-14T23:48:43.238724+0000","last_fullsized":"2024-04-14T23:48:43.238724+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:52:40.162331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.214624+0000","last_change":"2024-04-14T23:48:42.220492+0000","last_active":"2024-04-14T23:48:43.214624+0000","last_peered":"2024-04-14T23:48:43.214624+0000","last_clean":"2024-04-14T23:48:43.214624+0000","last_became_active":"2024-04-14T23:48:42.220145+0000","last_became_peered":"2024-04-14T23:48:42.220145+0000","last_unstale":"2024-04-14T23:48:43.214624+0000","last_undegraded":"2024-04-14T23:48:43.214624+0000","last_fullsized":"2024-04-14T23:48:43.214624+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:17:45.278993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.238431+0000","last_change":"2024-04-14T23:48:43.235676+0000","last_active":"2024-04-14T23:48:43.238431+0000","last_peered":"2024-04-14T23:48:43.238431+0000","last_clean":"2024-04-14T23:48:43.238431+0000","last_became_active":"2024-04-14T23:48:43.235319+0000","last_became_peered":"2024-04-14T23:48:43.235319+0000","last_unstale":"2024-04-14T23:48:43.238431+0000","last_undegraded":"2024-04-14T23:48:43.238431+0000","last_fullsized":"2024-04-14T23:48:43.238431+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:03:35.784237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.223088+0000","last_change":"2024-04-14T23:48:42.231178+0000","last_active":"2024-04-14T23:48:43.223088+0000","last_peered":"2024-04-14T23:48:43.223088+0000","last_clean":"2024-04-14T23:48:43.223088+0000","last_became_active":"2024-04-14T23:48:42.230792+0000","last_became_peered":"2024-04-14T23:48:42.230792+0000","last_unstale":"2024-04-14T23:48:43.223088+0000","last_undegraded":"2024-04-14T23:48:43.223088+0000","last_fullsized":"2024-04-14T23:48:43.223088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:22:40.744314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.235827+0000","last_change":"2024-04-14T23:48:43.228290+0000","last_active":"2024-04-14T23:48:43.235827+0000","last_peered":"2024-04-14T23:48:43.235827+0000","last_clean":"2024-04-14T23:48:43.235827+0000","last_became_active":"2024-04-14T23:48:43.228004+0000","last_became_peered":"2024-04-14T23:48:43.228004+0000","last_unstale":"2024-04-14T23:48:43.235827+0000","last_undegraded":"2024-04-14T23:48:43.235827+0000","last_fullsized":"2024-04-14T23:48:43.235827+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:44:50.560451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.216307+0000","last_change":"2024-04-14T23:48:42.231936+0000","last_active":"2024-04-14T23:48:43.216307+0000","last_peered":"2024-04-14T23:48:43.216307+0000","last_clean":"2024-04-14T23:48:43.216307+0000","last_became_active":"2024-04-14T23:48:42.231612+0000","last_became_peered":"2024-04-14T23:48:42.231612+0000","last_unstale":"2024-04-14T23:48:43.216307+0000","last_undegraded":"2024-04-14T23:48:43.216307+0000","last_fullsized":"2024-04-14T23:48:43.216307+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:45:03.098024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.235908+0000","last_change":"2024-04-14T23:48:43.228933+0000","last_active":"2024-04-14T23:48:43.235908+0000","last_peered":"2024-04-14T23:48:43.235908+0000","last_clean":"2024-04-14T23:48:43.235908+0000","last_became_active":"2024-04-14T23:48:43.228665+0000","last_became_peered":"2024-04-14T23:48:43.228665+0000","last_unstale":"2024-04-14T23:48:43.235908+0000","last_undegraded":"2024-04-14T23:48:43.235908+0000","last_fullsized":"2024-04-14T23:48:43.235908+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:00:11.267732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.223809+0000","last_change":"2024-04-14T23:48:42.239794+0000","last_active":"2024-04-14T23:48:43.223809+0000","last_peered":"2024-04-14T23:48:43.223809+0000","last_clean":"2024-04-14T23:48:43.223809+0000","last_became_active":"2024-04-14T23:48:42.239505+0000","last_became_peered":"2024-04-14T23:48:42.239505+0000","last_unstale":"2024-04-14T23:48:43.223809+0000","last_undegraded":"2024-04-14T23:48:43.223809+0000","last_fullsized":"2024-04-14T23:48:43.223809+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T09:43:16.582708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.238761+0000","last_change":"2024-04-14T23:48:43.230251+0000","last_active":"2024-04-14T23:48:43.238761+0000","last_peered":"2024-04-14T23:48:43.238761+0000","last_clean":"2024-04-14T23:48:43.238761+0000","last_became_active":"2024-04-14T23:48:43.229949+0000","last_became_peered":"2024-04-14T23:48:43.229949+0000","last_unstale":"2024-04-14T23:48:43.238761+0000","last_undegraded":"2024-04-14T23:48:43.238761+0000","last_fullsized":"2024-04-14T23:48:43.238761+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:13:45.621085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.222416+0000","last_change":"2024-04-14T23:48:42.227708+0000","last_active":"2024-04-14T23:48:43.222416+0000","last_peered":"2024-04-14T23:48:43.222416+0000","last_clean":"2024-04-14T23:48:43.222416+0000","last_became_active":"2024-04-14T23:48:42.226845+0000","last_became_peered":"2024-04-14T23:48:42.226845+0000","last_unstale":"2024-04-14T23:48:43.222416+0000","last_undegraded":"2024-04-14T23:48:43.222416+0000","last_fullsized":"2024-04-14T23:48:43.222416+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:02:54.468796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230624+0000","last_change":"2024-04-14T23:48:43.227439+0000","last_active":"2024-04-14T23:48:44.230624+0000","last_peered":"2024-04-14T23:48:44.230624+0000","last_clean":"2024-04-14T23:48:44.230624+0000","last_became_active":"2024-04-14T23:48:43.226717+0000","last_became_peered":"2024-04-14T23:48:43.226717+0000","last_unstale":"2024-04-14T23:48:44.230624+0000","last_undegraded":"2024-04-14T23:48:44.230624+0000","last_fullsized":"2024-04-14T23:48:44.230624+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:01:11.249999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.223049+0000","last_change":"2024-04-14T23:48:42.232164+0000","last_active":"2024-04-14T23:48:43.223049+0000","last_peered":"2024-04-14T23:48:43.223049+0000","last_clean":"2024-04-14T23:48:43.223049+0000","last_became_active":"2024-04-14T23:48:42.229785+0000","last_became_peered":"2024-04-14T23:48:42.229785+0000","last_unstale":"2024-04-14T23:48:43.223049+0000","last_undegraded":"2024-04-14T23:48:43.223049+0000","last_fullsized":"2024-04-14T23:48:43.223049+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T04:21:37.538735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.238850+0000","last_change":"2024-04-14T23:48:43.232949+0000","last_active":"2024-04-14T23:48:43.238850+0000","last_peered":"2024-04-14T23:48:43.238850+0000","last_clean":"2024-04-14T23:48:43.238850+0000","last_became_active":"2024-04-14T23:48:43.232612+0000","last_became_peered":"2024-04-14T23:48:43.232612+0000","last_unstale":"2024-04-14T23:48:43.238850+0000","last_undegraded":"2024-04-14T23:48:43.238850+0000","last_fullsized":"2024-04-14T23:48:43.238850+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:40:03.923776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":168,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.216039+0000","last_change":"2024-04-14T23:48:38.150200+0000","last_active":"2024-04-14T23:48:43.216039+0000","last_peered":"2024-04-14T23:48:43.216039+0000","last_clean":"2024-04-14T23:48:43.216039+0000","last_became_active":"2024-04-14T23:48:38.149301+0000","last_became_peered":"2024-04-14T23:48:38.149301+0000","last_unstale":"2024-04-14T23:48:43.216039+0000","last_undegraded":"2024-04-14T23:48:43.216039+0000","last_fullsized":"2024-04-14T23:48:43.216039+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:37.129663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:37.129663+0000","last_clean_scrub_stamp":"2024-04-14T23:48:37.129663+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T23:58:01.303851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230439+0000","last_change":"2024-04-14T23:48:42.229238+0000","last_active":"2024-04-14T23:48:44.230439+0000","last_peered":"2024-04-14T23:48:44.230439+0000","last_clean":"2024-04-14T23:48:44.230439+0000","last_became_active":"2024-04-14T23:48:42.228994+0000","last_became_peered":"2024-04-14T23:48:42.228994+0000","last_unstale":"2024-04-14T23:48:44.230439+0000","last_undegraded":"2024-04-14T23:48:44.230439+0000","last_fullsized":"2024-04-14T23:48:44.230439+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:47:15.406228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.232256+0000","last_change":"2024-04-14T23:48:43.223973+0000","last_active":"2024-04-14T23:48:44.232256+0000","last_peered":"2024-04-14T23:48:44.232256+0000","last_clean":"2024-04-14T23:48:44.232256+0000","last_became_active":"2024-04-14T23:48:43.223665+0000","last_became_peered":"2024-04-14T23:48:43.223665+0000","last_unstale":"2024-04-14T23:48:44.232256+0000","last_undegraded":"2024-04-14T23:48:44.232256+0000","last_fullsized":"2024-04-14T23:48:44.232256+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:23:28.495821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.215985+0000","last_change":"2024-04-14T23:48:42.230404+0000","last_active":"2024-04-14T23:48:43.215985+0000","last_peered":"2024-04-14T23:48:43.215985+0000","last_clean":"2024-04-14T23:48:43.215985+0000","last_became_active":"2024-04-14T23:48:42.230173+0000","last_became_peered":"2024-04-14T23:48:42.230173+0000","last_unstale":"2024-04-14T23:48:43.215985+0000","last_undegraded":"2024-04-14T23:48:43.215985+0000","last_fullsized":"2024-04-14T23:48:43.215985+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:23:45.666680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.240607+0000","last_change":"2024-04-14T23:48:43.231958+0000","last_active":"2024-04-14T23:48:43.240607+0000","last_peered":"2024-04-14T23:48:43.240607+0000","last_clean":"2024-04-14T23:48:43.240607+0000","last_became_active":"2024-04-14T23:48:43.231025+0000","last_became_peered":"2024-04-14T23:48:43.231025+0000","last_unstale":"2024-04-14T23:48:43.240607+0000","last_undegraded":"2024-04-14T23:48:43.240607+0000","last_fullsized":"2024-04-14T23:48:43.240607+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:49:56.020321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.223371+0000","last_change":"2024-04-14T23:48:42.240651+0000","last_active":"2024-04-14T23:48:43.223371+0000","last_peered":"2024-04-14T23:48:43.223371+0000","last_clean":"2024-04-14T23:48:43.223371+0000","last_became_active":"2024-04-14T23:48:42.240351+0000","last_became_peered":"2024-04-14T23:48:42.240351+0000","last_unstale":"2024-04-14T23:48:43.223371+0000","last_undegraded":"2024-04-14T23:48:43.223371+0000","last_fullsized":"2024-04-14T23:48:43.223371+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:03:02.804444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.240029+0000","last_change":"2024-04-14T23:48:43.232415+0000","last_active":"2024-04-14T23:48:43.240029+0000","last_peered":"2024-04-14T23:48:43.240029+0000","last_clean":"2024-04-14T23:48:43.240029+0000","last_became_active":"2024-04-14T23:48:43.232120+0000","last_became_peered":"2024-04-14T23:48:43.232120+0000","last_unstale":"2024-04-14T23:48:43.240029+0000","last_undegraded":"2024-04-14T23:48:43.240029+0000","last_fullsized":"2024-04-14T23:48:43.240029+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:30:01.619411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230988+0000","last_change":"2024-04-14T23:48:42.233743+0000","last_active":"2024-04-14T23:48:44.230988+0000","last_peered":"2024-04-14T23:48:44.230988+0000","last_clean":"2024-04-14T23:48:44.230988+0000","last_became_active":"2024-04-14T23:48:42.233354+0000","last_became_peered":"2024-04-14T23:48:42.233354+0000","last_unstale":"2024-04-14T23:48:44.230988+0000","last_undegraded":"2024-04-14T23:48:44.230988+0000","last_fullsized":"2024-04-14T23:48:44.230988+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:49:23.045968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.238816+0000","last_change":"2024-04-14T23:48:43.232468+0000","last_active":"2024-04-14T23:48:43.238816+0000","last_peered":"2024-04-14T23:48:43.238816+0000","last_clean":"2024-04-14T23:48:43.238816+0000","last_became_active":"2024-04-14T23:48:43.232119+0000","last_became_peered":"2024-04-14T23:48:43.232119+0000","last_unstale":"2024-04-14T23:48:43.238816+0000","last_undegraded":"2024-04-14T23:48:43.238816+0000","last_fullsized":"2024-04-14T23:48:43.238816+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:49:04.298324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.214850+0000","last_change":"2024-04-14T23:48:42.242249+0000","last_active":"2024-04-14T23:48:43.214850+0000","last_peered":"2024-04-14T23:48:43.214850+0000","last_clean":"2024-04-14T23:48:43.214850+0000","last_became_active":"2024-04-14T23:48:42.242039+0000","last_became_peered":"2024-04-14T23:48:42.242039+0000","last_unstale":"2024-04-14T23:48:43.214850+0000","last_undegraded":"2024-04-14T23:48:43.214850+0000","last_fullsized":"2024-04-14T23:48:43.214850+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:46:16.301900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230960+0000","last_change":"2024-04-14T23:48:43.242228+0000","last_active":"2024-04-14T23:48:44.230960+0000","last_peered":"2024-04-14T23:48:44.230960+0000","last_clean":"2024-04-14T23:48:44.230960+0000","last_became_active":"2024-04-14T23:48:43.241776+0000","last_became_peered":"2024-04-14T23:48:43.241776+0000","last_unstale":"2024-04-14T23:48:44.230960+0000","last_undegraded":"2024-04-14T23:48:44.230960+0000","last_fullsized":"2024-04-14T23:48:44.230960+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:48:23.653866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.213932+0000","last_change":"2024-04-14T23:48:42.232537+0000","last_active":"2024-04-14T23:48:43.213932+0000","last_peered":"2024-04-14T23:48:43.213932+0000","last_clean":"2024-04-14T23:48:43.213932+0000","last_became_active":"2024-04-14T23:48:42.232325+0000","last_became_peered":"2024-04-14T23:48:42.232325+0000","last_unstale":"2024-04-14T23:48:43.213932+0000","last_undegraded":"2024-04-14T23:48:43.213932+0000","last_fullsized":"2024-04-14T23:48:43.213932+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:34:37.448301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.239744+0000","last_change":"2024-04-14T23:48:43.234445+0000","last_active":"2024-04-14T23:48:43.239744+0000","last_peered":"2024-04-14T23:48:43.239744+0000","last_clean":"2024-04-14T23:48:43.239744+0000","last_became_active":"2024-04-14T23:48:43.234150+0000","last_became_peered":"2024-04-14T23:48:43.234150+0000","last_unstale":"2024-04-14T23:48:43.239744+0000","last_undegraded":"2024-04-14T23:48:43.239744+0000","last_fullsized":"2024-04-14T23:48:43.239744+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:39:36.833579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.216481+0000","last_change":"2024-04-14T23:48:42.230014+0000","last_active":"2024-04-14T23:48:43.216481+0000","last_peered":"2024-04-14T23:48:43.216481+0000","last_clean":"2024-04-14T23:48:43.216481+0000","last_became_active":"2024-04-14T23:48:42.229712+0000","last_became_peered":"2024-04-14T23:48:42.229712+0000","last_unstale":"2024-04-14T23:48:43.216481+0000","last_undegraded":"2024-04-14T23:48:43.216481+0000","last_fullsized":"2024-04-14T23:48:43.216481+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:54:20.595391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.233392+0000","last_change":"2024-04-14T23:48:43.229128+0000","last_active":"2024-04-14T23:48:44.233392+0000","last_peered":"2024-04-14T23:48:44.233392+0000","last_clean":"2024-04-14T23:48:44.233392+0000","last_became_active":"2024-04-14T23:48:43.228739+0000","last_became_peered":"2024-04-14T23:48:43.228739+0000","last_unstale":"2024-04-14T23:48:44.233392+0000","last_undegraded":"2024-04-14T23:48:44.233392+0000","last_fullsized":"2024-04-14T23:48:44.233392+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:08:08.148345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.217199+0000","last_change":"2024-04-14T23:48:42.220035+0000","last_active":"2024-04-14T23:48:43.217199+0000","last_peered":"2024-04-14T23:48:43.217199+0000","last_clean":"2024-04-14T23:48:43.217199+0000","last_became_active":"2024-04-14T23:48:42.219750+0000","last_became_peered":"2024-04-14T23:48:42.219750+0000","last_unstale":"2024-04-14T23:48:43.217199+0000","last_undegraded":"2024-04-14T23:48:43.217199+0000","last_fullsized":"2024-04-14T23:48:43.217199+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:00:58.928166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.240459+0000","last_change":"2024-04-14T23:48:43.232997+0000","last_active":"2024-04-14T23:48:43.240459+0000","last_peered":"2024-04-14T23:48:43.240459+0000","last_clean":"2024-04-14T23:48:43.240459+0000","last_became_active":"2024-04-14T23:48:43.232730+0000","last_became_peered":"2024-04-14T23:48:43.232730+0000","last_unstale":"2024-04-14T23:48:43.240459+0000","last_undegraded":"2024-04-14T23:48:43.240459+0000","last_fullsized":"2024-04-14T23:48:43.240459+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:49:33.985010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.215837+0000","last_change":"2024-04-14T23:48:42.232087+0000","last_active":"2024-04-14T23:48:43.215837+0000","last_peered":"2024-04-14T23:48:43.215837+0000","last_clean":"2024-04-14T23:48:43.215837+0000","last_became_active":"2024-04-14T23:48:42.231796+0000","last_became_peered":"2024-04-14T23:48:42.231796+0000","last_unstale":"2024-04-14T23:48:43.215837+0000","last_undegraded":"2024-04-14T23:48:43.215837+0000","last_fullsized":"2024-04-14T23:48:43.215837+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:14:35.050093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.238771+0000","last_change":"2024-04-14T23:48:43.232871+0000","last_active":"2024-04-14T23:48:43.238771+0000","last_peered":"2024-04-14T23:48:43.238771+0000","last_clean":"2024-04-14T23:48:43.238771+0000","last_became_active":"2024-04-14T23:48:43.232594+0000","last_became_peered":"2024-04-14T23:48:43.232594+0000","last_unstale":"2024-04-14T23:48:43.238771+0000","last_undegraded":"2024-04-14T23:48:43.238771+0000","last_fullsized":"2024-04-14T23:48:43.238771+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:36:16.823063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.214857+0000","last_change":"2024-04-14T23:48:42.231237+0000","last_active":"2024-04-14T23:48:43.214857+0000","last_peered":"2024-04-14T23:48:43.214857+0000","last_clean":"2024-04-14T23:48:43.214857+0000","last_became_active":"2024-04-14T23:48:42.230902+0000","last_became_peered":"2024-04-14T23:48:42.230902+0000","last_unstale":"2024-04-14T23:48:43.214857+0000","last_undegraded":"2024-04-14T23:48:43.214857+0000","last_fullsized":"2024-04-14T23:48:43.214857+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:28:37.228039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.232080+0000","last_change":"2024-04-14T23:48:43.233035+0000","last_active":"2024-04-14T23:48:44.232080+0000","last_peered":"2024-04-14T23:48:44.232080+0000","last_clean":"2024-04-14T23:48:44.232080+0000","last_became_active":"2024-04-14T23:48:43.232747+0000","last_became_peered":"2024-04-14T23:48:43.232747+0000","last_unstale":"2024-04-14T23:48:44.232080+0000","last_undegraded":"2024-04-14T23:48:44.232080+0000","last_fullsized":"2024-04-14T23:48:44.232080+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:38:57.324408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.214550+0000","last_change":"2024-04-14T23:48:42.240496+0000","last_active":"2024-04-14T23:48:43.214550+0000","last_peered":"2024-04-14T23:48:43.214550+0000","last_clean":"2024-04-14T23:48:43.214550+0000","last_became_active":"2024-04-14T23:48:42.238524+0000","last_became_peered":"2024-04-14T23:48:42.238524+0000","last_unstale":"2024-04-14T23:48:43.214550+0000","last_undegraded":"2024-04-14T23:48:43.214550+0000","last_fullsized":"2024-04-14T23:48:43.214550+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:29:35.218759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.233225+0000","last_change":"2024-04-14T23:48:43.234025+0000","last_active":"2024-04-14T23:48:44.233225+0000","last_peered":"2024-04-14T23:48:44.233225+0000","last_clean":"2024-04-14T23:48:44.233225+0000","last_became_active":"2024-04-14T23:48:43.233651+0000","last_became_peered":"2024-04-14T23:48:43.233651+0000","last_unstale":"2024-04-14T23:48:44.233225+0000","last_undegraded":"2024-04-14T23:48:44.233225+0000","last_fullsized":"2024-04-14T23:48:44.233225+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:26:58.094545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.213165+0000","last_change":"2024-04-14T23:48:42.218897+0000","last_active":"2024-04-14T23:48:43.213165+0000","last_peered":"2024-04-14T23:48:43.213165+0000","last_clean":"2024-04-14T23:48:43.213165+0000","last_became_active":"2024-04-14T23:48:42.218580+0000","last_became_peered":"2024-04-14T23:48:42.218580+0000","last_unstale":"2024-04-14T23:48:43.213165+0000","last_undegraded":"2024-04-14T23:48:43.213165+0000","last_fullsized":"2024-04-14T23:48:43.213165+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:26:44.477638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.224966+0000","last_change":"2024-04-14T23:48:43.219871+0000","last_active":"2024-04-14T23:48:43.224966+0000","last_peered":"2024-04-14T23:48:43.224966+0000","last_clean":"2024-04-14T23:48:43.224966+0000","last_became_active":"2024-04-14T23:48:43.219601+0000","last_became_peered":"2024-04-14T23:48:43.219601+0000","last_unstale":"2024-04-14T23:48:43.224966+0000","last_undegraded":"2024-04-14T23:48:43.224966+0000","last_fullsized":"2024-04-14T23:48:43.224966+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:46:56.543156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230971+0000","last_change":"2024-04-14T23:48:43.222259+0000","last_active":"2024-04-14T23:48:44.230971+0000","last_peered":"2024-04-14T23:48:44.230971+0000","last_clean":"2024-04-14T23:48:44.230971+0000","last_became_active":"2024-04-14T23:48:43.221842+0000","last_became_peered":"2024-04-14T23:48:43.221842+0000","last_unstale":"2024-04-14T23:48:44.230971+0000","last_undegraded":"2024-04-14T23:48:44.230971+0000","last_fullsized":"2024-04-14T23:48:44.230971+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:12:20.038987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.212374+0000","last_change":"2024-04-14T23:48:42.221314+0000","last_active":"2024-04-14T23:48:43.212374+0000","last_peered":"2024-04-14T23:48:43.212374+0000","last_clean":"2024-04-14T23:48:43.212374+0000","last_became_active":"2024-04-14T23:48:42.221028+0000","last_became_peered":"2024-04-14T23:48:42.221028+0000","last_unstale":"2024-04-14T23:48:43.212374+0000","last_undegraded":"2024-04-14T23:48:43.212374+0000","last_fullsized":"2024-04-14T23:48:43.212374+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T09:58:18.195400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230606+0000","last_change":"2024-04-14T23:48:43.224577+0000","last_active":"2024-04-14T23:48:44.230606+0000","last_peered":"2024-04-14T23:48:44.230606+0000","last_clean":"2024-04-14T23:48:44.230606+0000","last_became_active":"2024-04-14T23:48:43.224312+0000","last_became_peered":"2024-04-14T23:48:43.224312+0000","last_unstale":"2024-04-14T23:48:44.230606+0000","last_undegraded":"2024-04-14T23:48:44.230606+0000","last_fullsized":"2024-04-14T23:48:44.230606+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:52:44.955773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.213420+0000","last_change":"2024-04-14T23:48:42.221000+0000","last_active":"2024-04-14T23:48:43.213420+0000","last_peered":"2024-04-14T23:48:43.213420+0000","last_clean":"2024-04-14T23:48:43.213420+0000","last_became_active":"2024-04-14T23:48:42.219813+0000","last_became_peered":"2024-04-14T23:48:42.219813+0000","last_unstale":"2024-04-14T23:48:43.213420+0000","last_undegraded":"2024-04-14T23:48:43.213420+0000","last_fullsized":"2024-04-14T23:48:43.213420+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:00:38.107953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.241568+0000","last_change":"2024-04-14T23:48:43.236946+0000","last_active":"2024-04-14T23:48:43.241568+0000","last_peered":"2024-04-14T23:48:43.241568+0000","last_clean":"2024-04-14T23:48:43.241568+0000","last_became_active":"2024-04-14T23:48:43.236595+0000","last_became_peered":"2024-04-14T23:48:43.236595+0000","last_unstale":"2024-04-14T23:48:43.241568+0000","last_undegraded":"2024-04-14T23:48:43.241568+0000","last_fullsized":"2024-04-14T23:48:43.241568+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:57:34.820083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.234116+0000","last_change":"2024-04-14T23:48:42.239255+0000","last_active":"2024-04-14T23:48:44.234116+0000","last_peered":"2024-04-14T23:48:44.234116+0000","last_clean":"2024-04-14T23:48:44.234116+0000","last_became_active":"2024-04-14T23:48:42.238892+0000","last_became_peered":"2024-04-14T23:48:42.238892+0000","last_unstale":"2024-04-14T23:48:44.234116+0000","last_undegraded":"2024-04-14T23:48:44.234116+0000","last_fullsized":"2024-04-14T23:48:44.234116+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:44:39.632155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.233893+0000","last_change":"2024-04-14T23:48:43.227240+0000","last_active":"2024-04-14T23:48:44.233893+0000","last_peered":"2024-04-14T23:48:44.233893+0000","last_clean":"2024-04-14T23:48:44.233893+0000","last_became_active":"2024-04-14T23:48:43.222707+0000","last_became_peered":"2024-04-14T23:48:43.222707+0000","last_unstale":"2024-04-14T23:48:44.233893+0000","last_undegraded":"2024-04-14T23:48:44.233893+0000","last_fullsized":"2024-04-14T23:48:44.233893+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:02:17.231763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-14T23:48:43.223590+0000","last_change":"2024-04-14T23:48:42.226563+0000","last_active":"2024-04-14T23:48:43.223590+0000","last_peered":"2024-04-14T23:48:43.223590+0000","last_clean":"2024-04-14T23:48:43.223590+0000","last_became_active":"2024-04-14T23:48:42.226238+0000","last_became_peered":"2024-04-14T23:48:42.226238+0000","last_unstale":"2024-04-14T23:48:43.223590+0000","last_undegraded":"2024-04-14T23:48:43.223590+0000","last_fullsized":"2024-04-14T23:48:43.223590+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:01:36.516898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65128,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":270336,"data_stored":659673,"data_compressed":32564,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":69305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":69305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574851,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574852,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":69305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574852,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":69305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574851,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":270336,"data_stored":659673,"data_compressed":32564,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32564,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32564,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-14T23:48:49.239 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-14T23:48:49.240 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-14T23:48:49.240 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-14T23:48:49.240 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-14T23:48:49.240 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-14T23:48:49.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.502+0000 7f337b000640 1 Processor -- start 2024-04-14T23:48:49.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.502+0000 7f337b000640 1 -- start start 2024-04-14T23:48:49.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.504+0000 7f337b000640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f33740c3910 0x7f33740c52b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:49.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.504+0000 7f337b000640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33740c3d70 con 0x7f337415ae00 2024-04-14T23:48:49.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.504+0000 7f337b000640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33740ccf80 con 0x7f3374151ae0 2024-04-14T23:48:49.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.504+0000 7f337b000640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f337406e920 con 0x7f33740c3910 2024-04-14T23:48:49.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.504+0000 7f336b400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f33740c3910 0x7f33740c52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:49.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.504+0000 7f3373e00640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f3374151ae0 0x7f337406e210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.60:56800/0 (socket says 172.21.15.60:56800) 2024-04-14T23:48:49.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.504+0000 7f3373e00640 1 -- 172.21.15.60:0/2069708039 learned_addr learned my addr 172.21.15.60:0/2069708039 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:48:49.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.505+0000 7f336b400640 1 -- 172.21.15.60:0/2069708039 >> v1:172.21.15.186:6789/0 conn(0x7f3374151ae0 legacy=0x7f337406e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:49.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.505+0000 7f336b400640 1 -- 172.21.15.60:0/2069708039 >> v1:172.21.15.60:6789/0 conn(0x7f337415ae00 legacy=0x7f337415b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:49.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.505+0000 7f3373400640 1 -- 172.21.15.60:0/2069708039 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3523544045 0 0) 0x7f33740ccf80 con 0x7f3374151ae0 2024-04-14T23:48:49.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.505+0000 7f336b400640 1 -- 172.21.15.60:0/2069708039 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3374072dc0 con 0x7f33740c3910 2024-04-14T23:48:49.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.506+0000 7f336b400640 1 --2- 172.21.15.60:0/2069708039 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f33740c3910 0x7f33740c52b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f335c0097d0 tx=0x7f335c02b9d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1161917bf3a6d553 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:49.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.506+0000 7f3373400640 1 -- 172.21.15.60:0/2069708039 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f335c034070 con 0x7f33740c3910 2024-04-14T23:48:49.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.506+0000 7f3373400640 1 -- 172.21.15.60:0/2069708039 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f335c030890 con 0x7f33740c3910 2024-04-14T23:48:49.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.506+0000 7f3373400640 1 -- 172.21.15.60:0/2069708039 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f335c030c60 con 0x7f33740c3910 2024-04-14T23:48:49.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.507+0000 7f337b000640 1 -- 172.21.15.60:0/2069708039 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f33740c3910 msgr2=0x7f33740c52b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:49.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.507+0000 7f337b000640 1 --2- 172.21.15.60:0/2069708039 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f33740c3910 0x7f33740c52b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f335c0097d0 tx=0x7f335c02b9d0 comp rx=0 tx=0).stop 2024-04-14T23:48:49.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.507+0000 7f337b000640 1 -- 172.21.15.60:0/2069708039 shutdown_connections 2024-04-14T23:48:49.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.507+0000 7f337b000640 1 --2- 172.21.15.60:0/2069708039 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f33740c3910 0x7f33740c52b0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:49.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.507+0000 7f337b000640 1 -- 172.21.15.60:0/2069708039 >> 172.21.15.60:0/2069708039 conn(0x7f33740cc170 msgr2=0x7f33740cc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:49.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.507+0000 7f337b000640 1 -- 172.21.15.60:0/2069708039 shutdown_connections 2024-04-14T23:48:49.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.507+0000 7f337b000640 1 -- 172.21.15.60:0/2069708039 wait complete. 2024-04-14T23:48:49.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.509+0000 7f337b000640 1 Processor -- start 2024-04-14T23:48:49.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.509+0000 7f337b000640 1 -- start start 2024-04-14T23:48:49.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.510+0000 7f337b000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f33740c3910 0x7f337415a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:49.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.510+0000 7f337b000640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3374151ae0 0x7f3374071a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:49.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.510+0000 7f337b000640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f337415ae00 0x7f33741f0a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:49.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.510+0000 7f337b000640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f33740726c0 con 0x7f337415ae00 2024-04-14T23:48:49.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.510+0000 7f337b000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f33740723e0 con 0x7f33740c3910 2024-04-14T23:48:49.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.510+0000 7f337b000640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f337415b960 con 0x7f3374151ae0 2024-04-14T23:48:49.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.510+0000 7f3379000640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f337415ae00 0x7f33741f0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:49.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.510+0000 7f3379000640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f337415ae00 0x7f33741f0a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53944/0 (socket says 172.21.15.60:53944) 2024-04-14T23:48:49.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.510+0000 7f3379000640 1 -- 172.21.15.60:0/2366939680 learned_addr learned my addr 172.21.15.60:0/2366939680 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:49.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.510+0000 7f3373e00640 1 --2- 172.21.15.60:0/2366939680 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f33740c3910 0x7f337415a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:49.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.510+0000 7f336b400640 1 --2- 172.21.15.60:0/2366939680 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3374151ae0 0x7f3374071a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:49.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.511+0000 7f3379000640 1 -- 172.21.15.60:0/2366939680 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3374151ae0 msgr2=0x7f3374071a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:49.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.511+0000 7f3379000640 1 --2- 172.21.15.60:0/2366939680 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3374151ae0 0x7f3374071a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:49.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.511+0000 7f3379000640 1 -- 172.21.15.60:0/2366939680 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f33740c3910 msgr2=0x7f337415a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:49.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.511+0000 7f3379000640 1 --2- 172.21.15.60:0/2366939680 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f33740c3910 0x7f337415a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:49.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.511+0000 7f3379000640 1 -- 172.21.15.60:0/2366939680 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33741f4e70 con 0x7f337415ae00 2024-04-14T23:48:49.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.511+0000 7f3379000640 1 --2- 172.21.15.60:0/2366939680 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f337415ae00 0x7f33741f0a10 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f336c00c7a0 tx=0x7f336c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:49.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.511+0000 7f3370c00640 1 -- 172.21.15.60:0/2366939680 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f336c003b70 con 0x7f337415ae00 2024-04-14T23:48:49.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.511+0000 7f3370c00640 1 -- 172.21.15.60:0/2366939680 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f336c003d30 con 0x7f337415ae00 2024-04-14T23:48:49.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.511+0000 7f3370c00640 1 -- 172.21.15.60:0/2366939680 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f336c00aa90 con 0x7f337415ae00 2024-04-14T23:48:49.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.511+0000 7f337b000640 1 -- 172.21.15.60:0/2366939680 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3374158d80 con 0x7f337415ae00 2024-04-14T23:48:49.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.511+0000 7f337b000640 1 -- 172.21.15.60:0/2366939680 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33741f4390 con 0x7f337415ae00 2024-04-14T23:48:49.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.512+0000 7f337b000640 1 -- 172.21.15.60:0/2366939680 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3374076970 con 0x7f337415ae00 2024-04-14T23:48:49.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.513+0000 7f3370c00640 1 -- 172.21.15.60:0/2366939680 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f336c00ac30 con 0x7f337415ae00 2024-04-14T23:48:49.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.514+0000 7f3370c00640 1 --2- 172.21.15.60:0/2366939680 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f335006b990 0x7f335006de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:49.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.514+0000 7f3370c00640 1 -- 172.21.15.60:0/2366939680 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f336c08a710 con 0x7f337415ae00 2024-04-14T23:48:49.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.515+0000 7f3373e00640 1 --2- 172.21.15.60:0/2366939680 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f335006b990 0x7f335006de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:49.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.515+0000 7f3373e00640 1 --2- 172.21.15.60:0/2366939680 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f335006b990 0x7f335006de50 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3364007f70 tx=0x7f3364007f00 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:49.524 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.519+0000 7f3370c00640 1 -- 172.21.15.60:0/2366939680 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f33740c3d70 con 0x7f337415ae00 2024-04-14T23:48:49.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.775+0000 7f337b000640 1 -- 172.21.15.60:0/2366939680 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f33740715a0 con 0x7f337415ae00 2024-04-14T23:48:49.790 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.784+0000 7f3370c00640 1 -- 172.21.15.60:0/2366939680 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+172380 (secure 0 0 0) 0x7f336c058c30 con 0x7f337415ae00 2024-04-14T23:48:49.793 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:48:49.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.789+0000 7f337b000640 1 -- 172.21.15.60:0/2366939680 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f335006b990 msgr2=0x7f335006de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:49.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.789+0000 7f337b000640 1 --2- 172.21.15.60:0/2366939680 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f335006b990 0x7f335006de50 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3364007f70 tx=0x7f3364007f00 comp rx=0 tx=0).stop 2024-04-14T23:48:49.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.789+0000 7f337b000640 1 -- 172.21.15.60:0/2366939680 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f337415ae00 msgr2=0x7f33741f0a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:49.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.789+0000 7f337b000640 1 --2- 172.21.15.60:0/2366939680 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f337415ae00 0x7f33741f0a10 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f336c00c7a0 tx=0x7f336c00cc60 comp rx=0 tx=0).stop 2024-04-14T23:48:49.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.789+0000 7f337b000640 1 -- 172.21.15.60:0/2366939680 shutdown_connections 2024-04-14T23:48:49.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.789+0000 7f337b000640 1 --2- 172.21.15.60:0/2366939680 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f335006b990 0x7f335006de50 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:49.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.789+0000 7f337b000640 1 --2- 172.21.15.60:0/2366939680 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f337415ae00 0x7f33741f0a10 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:49.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.789+0000 7f337b000640 1 --2- 172.21.15.60:0/2366939680 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3374151ae0 0x7f3374071a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:49.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.789+0000 7f337b000640 1 --2- 172.21.15.60:0/2366939680 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f33740c3910 0x7f337415a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:49.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.789+0000 7f337b000640 1 -- 172.21.15.60:0/2366939680 >> 172.21.15.60:0/2366939680 conn(0x7f33740cc170 msgr2=0x7f33740cac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:49.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.789+0000 7f337b000640 1 -- 172.21.15.60:0/2366939680 shutdown_connections 2024-04-14T23:48:49.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:49.789+0000 7f337b000640 1 -- 172.21.15.60:0/2366939680 wait complete. 2024-04-14T23:48:49.814 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":5,"flags":0,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":714104835},{"type":"v1","addr":"172.21.15.186:6833","nonce":714104835}]},"active_addr":"172.21.15.186:6833/714104835","active_change":"2024-04-14T23:48:28.155348+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4103,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.186:0","nonce":2412030303}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.186:0","nonce":2724820008}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.186:0","nonce":2365826864}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.186:0","nonce":148496467}]}]} 2024-04-14T23:48:49.817 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-14T23:48:49.817 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-14T23:48:49.817 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-14T23:48:50.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.035+0000 7f2948400640 1 Processor -- start 2024-04-14T23:48:50.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.035+0000 7f2948400640 1 -- start start 2024-04-14T23:48:50.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.037+0000 7f2948400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2940152cf0 0x7f294015c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:50.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.037+0000 7f2948400640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f29400c63c0 0x7f294015c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:50.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.037+0000 7f2948400640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29401530d0 con 0x7f2940159450 2024-04-14T23:48:50.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.037+0000 7f2948400640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f29400751f0 con 0x7f29400c63c0 2024-04-14T23:48:50.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.037+0000 7f2948400640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f29400c8370 con 0x7f2940152cf0 2024-04-14T23:48:50.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.037+0000 7f2945000640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f29400c63c0 0x7f294015c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:50.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.037+0000 7f2945a00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2940152cf0 0x7f294015c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:50.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.037+0000 7f2946400640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f2940159450 0x7f29400c3f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.60:56802/0 (socket says 172.21.15.60:56802) 2024-04-14T23:48:50.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.037+0000 7f2946400640 1 -- 172.21.15.60:0/3334006481 learned_addr learned my addr 172.21.15.60:0/3334006481 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:48:50.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.038+0000 7f293fe00640 1 -- 172.21.15.60:0/3334006481 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1087021159 0 0) 0x7f29401530d0 con 0x7f2940159450 2024-04-14T23:48:50.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.038+0000 7f293fe00640 1 -- 172.21.15.60:0/3334006481 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2924003520 con 0x7f2940159450 2024-04-14T23:48:50.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.038+0000 7f2945000640 1 -- 172.21.15.60:0/3334006481 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2940152cf0 msgr2=0x7f294015c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:50.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.038+0000 7f2945000640 1 --2- 172.21.15.60:0/3334006481 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2940152cf0 0x7f294015c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:50.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.038+0000 7f2945000640 1 -- 172.21.15.60:0/3334006481 >> v1:172.21.15.186:6789/0 conn(0x7f2940159450 legacy=0x7f29400c3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:50.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.038+0000 7f2945000640 1 -- 172.21.15.60:0/3334006481 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29400c3810 con 0x7f29400c63c0 2024-04-14T23:48:50.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.038+0000 7f2945a00640 1 --2- 172.21.15.60:0/3334006481 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2940152cf0 0x7f294015c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:48:50.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.039+0000 7f2945000640 1 --2- 172.21.15.60:0/3334006481 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f29400c63c0 0x7f294015c7e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f292c0097a0 tx=0x7f292c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e76460e3409bea1a server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:50.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.040+0000 7f293fe00640 1 -- 172.21.15.60:0/3334006481 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f292c034070 con 0x7f29400c63c0 2024-04-14T23:48:50.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.040+0000 7f293fe00640 1 -- 172.21.15.60:0/3334006481 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f292c030610 con 0x7f29400c63c0 2024-04-14T23:48:50.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.040+0000 7f293fe00640 1 -- 172.21.15.60:0/3334006481 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f292c0309c0 con 0x7f29400c63c0 2024-04-14T23:48:50.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.040+0000 7f2948400640 1 -- 172.21.15.60:0/3334006481 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f29400c63c0 msgr2=0x7f294015c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:50.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.040+0000 7f2948400640 1 --2- 172.21.15.60:0/3334006481 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f29400c63c0 0x7f294015c7e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f292c0097a0 tx=0x7f292c02b960 comp rx=0 tx=0).stop 2024-04-14T23:48:50.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.040+0000 7f2948400640 1 -- 172.21.15.60:0/3334006481 shutdown_connections 2024-04-14T23:48:50.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.040+0000 7f2948400640 1 --2- 172.21.15.60:0/3334006481 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f29400c63c0 0x7f294015c7e0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:50.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.040+0000 7f2948400640 1 --2- 172.21.15.60:0/3334006481 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2940152cf0 0x7f294015c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:50.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.040+0000 7f2948400640 1 -- 172.21.15.60:0/3334006481 >> 172.21.15.60:0/3334006481 conn(0x7f29400cc170 msgr2=0x7f29400cc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:50.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.040+0000 7f2948400640 1 -- 172.21.15.60:0/3334006481 shutdown_connections 2024-04-14T23:48:50.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.040+0000 7f2948400640 1 -- 172.21.15.60:0/3334006481 wait complete. 2024-04-14T23:48:50.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.042+0000 7f2948400640 1 Processor -- start 2024-04-14T23:48:50.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.042+0000 7f2948400640 1 -- start start 2024-04-14T23:48:50.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.043+0000 7f2948400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f29400c63c0 0x7f294015b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:50.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.043+0000 7f2948400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2940152cf0 0x7f29401f1050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:50.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.043+0000 7f2948400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2940159450 0x7f29400cb500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:50.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.043+0000 7f2948400640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f29400c88f0 con 0x7f2940152cf0 2024-04-14T23:48:50.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.043+0000 7f2948400640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f29400c8610 con 0x7f29400c63c0 2024-04-14T23:48:50.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.043+0000 7f2948400640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f29400c4690 con 0x7f2940159450 2024-04-14T23:48:50.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.044+0000 7f2946400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2940159450 0x7f29400cb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:50.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.044+0000 7f2945a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f29400c63c0 0x7f294015b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:50.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.044+0000 7f2946400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2940159450 0x7f29400cb500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.60:48488/0 (socket says 172.21.15.60:48488) 2024-04-14T23:48:50.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.044+0000 7f2945000640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2940152cf0 0x7f29401f1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:50.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.044+0000 7f2946400640 1 -- 172.21.15.60:0/4170958906 learned_addr learned my addr 172.21.15.60:0/4170958906 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:50.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.044+0000 7f2946400640 1 -- 172.21.15.60:0/4170958906 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f29400c63c0 msgr2=0x7f294015b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:50.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.044+0000 7f2946400640 1 --2- 172.21.15.60:0/4170958906 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f29400c63c0 0x7f294015b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:50.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.044+0000 7f2946400640 1 -- 172.21.15.60:0/4170958906 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2940152cf0 msgr2=0x7f29401f1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:50.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.044+0000 7f2946400640 1 --2- 172.21.15.60:0/4170958906 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2940152cf0 0x7f29401f1050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:50.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.044+0000 7f2946400640 1 -- 172.21.15.60:0/4170958906 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29401f4600 con 0x7f2940159450 2024-04-14T23:48:50.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.044+0000 7f2945a00640 1 --2- 172.21.15.60:0/4170958906 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f29400c63c0 0x7f294015b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:48:50.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.045+0000 7f2946400640 1 --2- 172.21.15.60:0/4170958906 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2940159450 0x7f29400cb500 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f293800c390 tx=0x7f293800c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:50.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.045+0000 7f293d600640 1 -- 172.21.15.60:0/4170958906 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2938017070 con 0x7f2940159450 2024-04-14T23:48:50.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.045+0000 7f293d600640 1 -- 172.21.15.60:0/4170958906 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2938005d00 con 0x7f2940159450 2024-04-14T23:48:50.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.045+0000 7f293d600640 1 -- 172.21.15.60:0/4170958906 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2938007310 con 0x7f2940159450 2024-04-14T23:48:50.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.045+0000 7f2948400640 1 -- 172.21.15.60:0/4170958906 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2940150a40 con 0x7f2940159450 2024-04-14T23:48:50.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.045+0000 7f2948400640 1 -- 172.21.15.60:0/4170958906 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f29401f0850 con 0x7f2940159450 2024-04-14T23:48:50.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.046+0000 7f2948400640 1 -- 172.21.15.60:0/4170958906 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2940076970 con 0x7f2940159450 2024-04-14T23:48:50.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.047+0000 7f293d600640 1 -- 172.21.15.60:0/4170958906 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f293801a070 con 0x7f2940159450 2024-04-14T23:48:50.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.048+0000 7f293d600640 1 --2- 172.21.15.60:0/4170958906 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f292006b9c0 0x7f292006de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:50.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.048+0000 7f293d600640 1 -- 172.21.15.60:0/4170958906 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f2938089ac0 con 0x7f2940159450 2024-04-14T23:48:50.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.048+0000 7f2945a00640 1 --2- 172.21.15.60:0/4170958906 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f292006b9c0 0x7f292006de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:50.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.049+0000 7f2945a00640 1 --2- 172.21.15.60:0/4170958906 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f292006b9c0 0x7f292006de80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2930009fd0 tx=0x7f293000b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:50.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.053+0000 7f293d600640 1 -- 172.21.15.60:0/4170958906 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f29401f4600 con 0x7f2940159450 2024-04-14T23:48:50.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.257+0000 7f2948400640 1 -- 172.21.15.60:0/4170958906 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f29400c7f40 con 0x7f2940159450 2024-04-14T23:48:50.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.259+0000 7f293d600640 1 -- 172.21.15.60:0/4170958906 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16605 (secure 0 0 0) 0x7f2938089e20 con 0x7f2940159450 2024-04-14T23:48:50.265 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:48:50.265 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":21,"fsid":"5847db56-a76d-4486-b563-d83f1186af92","created":"2024-04-14T23:48:23.900003+0000","modified":"2024-04-14T23:48:44.219352+0000","last_up_change":"2024-04-14T23:48:34.203573+0000","last_in_change":"2024-04-14T23:48:26.345499+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-14T23:48:36.247372+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-14T23:48:40.274195+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-14T23:48:41.722183+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b2b2b755-7f42-4f9b-af9a-a178f7cc5a52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6817","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6818","nonce":2536528895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6822","nonce":2536528895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6827","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6828","nonce":2536528895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6825","nonce":2536528895}]},"public_addr":"172.21.15.60:6818/2536528895","cluster_addr":"172.21.15.60:6822/2536528895","heartbeat_back_addr":"172.21.15.60:6828/2536528895","heartbeat_front_addr":"172.21.15.60:6825/2536528895","state":["exists","up"]},{"osd":1,"uuid":"321418cd-5ab7-41a5-b9c8-b3fc12f68ebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6801","nonce":3115695676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6803","nonce":3115695676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6807","nonce":3115695676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6805","nonce":3115695676}]},"public_addr":"172.21.15.60:6801/3115695676","cluster_addr":"172.21.15.60:6803/3115695676","heartbeat_back_addr":"172.21.15.60:6807/3115695676","heartbeat_front_addr":"172.21.15.60:6805/3115695676","state":["exists","up"]},{"osd":2,"uuid":"07874e9f-3a38-41dc-be9f-2b75e84e53d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6819","nonce":3475168698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6824","nonce":3475168698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6831","nonce":3475168698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6829","nonce":3475168698}]},"public_addr":"172.21.15.60:6819/3475168698","cluster_addr":"172.21.15.60:6824/3475168698","heartbeat_back_addr":"172.21.15.60:6831/3475168698","heartbeat_front_addr":"172.21.15.60:6829/3475168698","state":["exists","up"]},{"osd":3,"uuid":"7e4ef6c9-b17a-47c6-a320-f5d16a6fe05e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6809","nonce":1730813674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6811","nonce":1730813674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6815","nonce":1730813674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6813","nonce":1730813674}]},"public_addr":"172.21.15.60:6809/1730813674","cluster_addr":"172.21.15.60:6811/1730813674","heartbeat_back_addr":"172.21.15.60:6815/1730813674","heartbeat_front_addr":"172.21.15.60:6813/1730813674","state":["exists","up"]},{"osd":4,"uuid":"759a3efc-dade-44c5-b57a-1623499aba50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6805","nonce":1464587038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6810","nonce":1464587038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6819","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6822","nonce":1464587038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6813","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6816","nonce":1464587038}]},"public_addr":"172.21.15.186:6805/1464587038","cluster_addr":"172.21.15.186:6810/1464587038","heartbeat_back_addr":"172.21.15.186:6822/1464587038","heartbeat_front_addr":"172.21.15.186:6816/1464587038","state":["exists","up"]},{"osd":5,"uuid":"24cd58c1-0cf1-4077-ab33-84c39c6907ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6801","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6803","nonce":1697098067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6809","nonce":1697098067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6820","nonce":1697098067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6815","nonce":1697098067}]},"public_addr":"172.21.15.186:6803/1697098067","cluster_addr":"172.21.15.186:6809/1697098067","heartbeat_back_addr":"172.21.15.186:6820/1697098067","heartbeat_front_addr":"172.21.15.186:6815/1697098067","state":["exists","up"]},{"osd":6,"uuid":"6add38f1-65e9-4b0d-927a-4ab77f5e2323","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6804","nonce":1837108559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6807","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6811","nonce":1837108559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6823","nonce":1837108559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6817","nonce":1837108559}]},"public_addr":"172.21.15.186:6804/1837108559","cluster_addr":"172.21.15.186:6811/1837108559","heartbeat_back_addr":"172.21.15.186:6823/1837108559","heartbeat_front_addr":"172.21.15.186:6817/1837108559","state":["exists","up"]},{"osd":7,"uuid":"a2326b6f-7601-4dda-a456-67c443ea5929","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6825","nonce":842537875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6827","nonce":842537875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6831","nonce":842537875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6829","nonce":842537875}]},"public_addr":"172.21.15.186:6825/842537875","cluster_addr":"172.21.15.186:6827/842537875","heartbeat_back_addr":"172.21.15.186:6831/842537875","heartbeat_front_addr":"172.21.15.186:6829/842537875","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.975583+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.976806+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.722725+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.763705+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.502140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.528972+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.517961+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.513350+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-14T23:48:50.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.262+0000 7f2948400640 1 -- 172.21.15.60:0/4170958906 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f292006b9c0 msgr2=0x7f292006de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:50.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.262+0000 7f2948400640 1 --2- 172.21.15.60:0/4170958906 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f292006b9c0 0x7f292006de80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2930009fd0 tx=0x7f293000b040 comp rx=0 tx=0).stop 2024-04-14T23:48:50.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.262+0000 7f2948400640 1 -- 172.21.15.60:0/4170958906 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2940159450 msgr2=0x7f29400cb500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:50.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.262+0000 7f2948400640 1 --2- 172.21.15.60:0/4170958906 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2940159450 0x7f29400cb500 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f293800c390 tx=0x7f293800c850 comp rx=0 tx=0).stop 2024-04-14T23:48:50.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.262+0000 7f2948400640 1 -- 172.21.15.60:0/4170958906 shutdown_connections 2024-04-14T23:48:50.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.262+0000 7f2948400640 1 --2- 172.21.15.60:0/4170958906 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f292006b9c0 0x7f292006de80 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:50.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.262+0000 7f2948400640 1 --2- 172.21.15.60:0/4170958906 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2940159450 0x7f29400cb500 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:50.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.262+0000 7f2948400640 1 --2- 172.21.15.60:0/4170958906 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2940152cf0 0x7f29401f1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:50.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.262+0000 7f2948400640 1 --2- 172.21.15.60:0/4170958906 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f29400c63c0 0x7f294015b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:50.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.262+0000 7f2948400640 1 -- 172.21.15.60:0/4170958906 >> 172.21.15.60:0/4170958906 conn(0x7f29400cc170 msgr2=0x7f294015c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:50.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.262+0000 7f2948400640 1 -- 172.21.15.60:0/4170958906 shutdown_connections 2024-04-14T23:48:50.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.262+0000 7f2948400640 1 -- 172.21.15.60:0/4170958906 wait complete. 2024-04-14T23:48:50.287 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-14T23:48:50.287 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-14T23:48:50.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.499+0000 7ff637200640 1 Processor -- start 2024-04-14T23:48:50.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.499+0000 7ff637200640 1 -- start start 2024-04-14T23:48:50.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.501+0000 7ff637200640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff638073320 0x7ff6380c6950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:50.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.501+0000 7ff637200640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff6380c3d90 0x7ff6380c41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:50.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.501+0000 7ff637200640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff638152fb0 con 0x7ff6380b1e00 2024-04-14T23:48:50.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.501+0000 7ff637200640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7ff6380c6540 con 0x7ff6380c3d90 2024-04-14T23:48:50.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.501+0000 7ff637200640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7ff6380b21e0 con 0x7ff638073320 2024-04-14T23:48:50.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.501+0000 7ff636800640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff6380c3d90 0x7ff6380c41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:50.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.501+0000 7ff635e00640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7ff6380b1e00 0x7ff6380c7fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.60:56806/0 (socket says 172.21.15.60:56806) 2024-04-14T23:48:50.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.501+0000 7ff635400640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff638073320 0x7ff6380c6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:50.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.501+0000 7ff635e00640 1 -- 172.21.15.60:0/751289447 learned_addr learned my addr 172.21.15.60:0/751289447 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:48:50.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.502+0000 7ff635400640 1 -- 172.21.15.60:0/751289447 >> v2:172.21.15.60:3300/0 conn(0x7ff6380c3d90 msgr2=0x7ff6380c41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:50.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.502+0000 7ff635400640 1 --2- 172.21.15.60:0/751289447 >> v2:172.21.15.60:3300/0 conn(0x7ff6380c3d90 0x7ff6380c41c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:50.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.502+0000 7ff635400640 1 -- 172.21.15.60:0/751289447 >> v1:172.21.15.186:6789/0 conn(0x7ff6380b1e00 legacy=0x7ff6380c7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:50.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.502+0000 7ff635400640 1 -- 172.21.15.60:0/751289447 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff638159310 con 0x7ff638073320 2024-04-14T23:48:50.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.502+0000 7ff636800640 1 --2- 172.21.15.60:0/751289447 >> v2:172.21.15.60:3300/0 conn(0x7ff6380c3d90 0x7ff6380c41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:48:50.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.502+0000 7ff634a00640 1 -- 172.21.15.60:0/751289447 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 983851 0 0) 0x7ff638152fb0 con 0x7ff6380b1e00 2024-04-14T23:48:50.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.503+0000 7ff635400640 1 --2- 172.21.15.60:0/751289447 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff638073320 0x7ff6380c6950 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff63806e450 tx=0x7ff62002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f4fb89aa082b9691 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:50.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.503+0000 7ff634a00640 1 -- 172.21.15.60:0/751289447 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff620034070 con 0x7ff638073320 2024-04-14T23:48:50.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.503+0000 7ff634a00640 1 -- 172.21.15.60:0/751289447 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff620030610 con 0x7ff638073320 2024-04-14T23:48:50.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.503+0000 7ff634a00640 1 -- 172.21.15.60:0/751289447 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6200309e0 con 0x7ff638073320 2024-04-14T23:48:50.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.504+0000 7ff637200640 1 -- 172.21.15.60:0/751289447 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff638073320 msgr2=0x7ff6380c6950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:50.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.504+0000 7ff637200640 1 --2- 172.21.15.60:0/751289447 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff638073320 0x7ff6380c6950 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff63806e450 tx=0x7ff62002b960 comp rx=0 tx=0).stop 2024-04-14T23:48:50.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.504+0000 7ff637200640 1 -- 172.21.15.60:0/751289447 shutdown_connections 2024-04-14T23:48:50.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.504+0000 7ff637200640 1 --2- 172.21.15.60:0/751289447 >> v2:172.21.15.60:3300/0 conn(0x7ff6380c3d90 0x7ff6380c41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:50.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.504+0000 7ff637200640 1 --2- 172.21.15.60:0/751289447 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff638073320 0x7ff6380c6950 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:50.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.504+0000 7ff637200640 1 -- 172.21.15.60:0/751289447 >> 172.21.15.60:0/751289447 conn(0x7ff6380c8b50 msgr2=0x7ff6380c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:50.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.504+0000 7ff637200640 1 -- 172.21.15.60:0/751289447 shutdown_connections 2024-04-14T23:48:50.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.504+0000 7ff637200640 1 -- 172.21.15.60:0/751289447 wait complete. 2024-04-14T23:48:50.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.506+0000 7ff637200640 1 Processor -- start 2024-04-14T23:48:50.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.506+0000 7ff637200640 1 -- start start 2024-04-14T23:48:50.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.507+0000 7ff637200640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff638073320 0x7ff638149120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:50.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.507+0000 7ff637200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6380b1e00 0x7ff6381ef1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:50.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.507+0000 7ff637200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6380c3d90 0x7ff6381ef6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:50.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.507+0000 7ff637200640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff6380c38b0 con 0x7ff638073320 2024-04-14T23:48:50.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.507+0000 7ff637200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff638159890 con 0x7ff6380c3d90 2024-04-14T23:48:50.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.507+0000 7ff637200640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7ff6381595b0 con 0x7ff6380b1e00 2024-04-14T23:48:50.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.507+0000 7ff635e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff638073320 0x7ff638149120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:50.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.507+0000 7ff635e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff638073320 0x7ff638149120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53972/0 (socket says 172.21.15.60:53972) 2024-04-14T23:48:50.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.507+0000 7ff635e00640 1 -- 172.21.15.60:0/1538382135 learned_addr learned my addr 172.21.15.60:0/1538382135 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:50.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.507+0000 7ff635400640 1 --2- 172.21.15.60:0/1538382135 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6380b1e00 0x7ff6381ef1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:50.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.508+0000 7ff636800640 1 --2- 172.21.15.60:0/1538382135 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6380c3d90 0x7ff6381ef6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:50.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.508+0000 7ff635e00640 1 -- 172.21.15.60:0/1538382135 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6380b1e00 msgr2=0x7ff6381ef1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:50.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.508+0000 7ff635e00640 1 --2- 172.21.15.60:0/1538382135 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6380b1e00 0x7ff6381ef1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:50.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.508+0000 7ff635e00640 1 -- 172.21.15.60:0/1538382135 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6380c3d90 msgr2=0x7ff6381ef6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:50.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.508+0000 7ff635e00640 1 --2- 172.21.15.60:0/1538382135 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6380c3d90 0x7ff6381ef6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:50.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.508+0000 7ff635e00640 1 -- 172.21.15.60:0/1538382135 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6381553a0 con 0x7ff638073320 2024-04-14T23:48:50.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.508+0000 7ff635e00640 1 --2- 172.21.15.60:0/1538382135 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff638073320 0x7ff638149120 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff62800c4a0 tx=0x7ff62800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:50.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.508+0000 7ff62e000640 1 -- 172.21.15.60:0/1538382135 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6280074e0 con 0x7ff638073320 2024-04-14T23:48:50.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.508+0000 7ff62e000640 1 -- 172.21.15.60:0/1538382135 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6280047e0 con 0x7ff638073320 2024-04-14T23:48:50.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.509+0000 7ff62e000640 1 -- 172.21.15.60:0/1538382135 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff628004c50 con 0x7ff638073320 2024-04-14T23:48:50.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.509+0000 7ff637200640 1 -- 172.21.15.60:0/1538382135 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff638155680 con 0x7ff638073320 2024-04-14T23:48:50.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.509+0000 7ff637200640 1 -- 172.21.15.60:0/1538382135 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff63814b010 con 0x7ff638073320 2024-04-14T23:48:50.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.510+0000 7ff637200640 1 -- 172.21.15.60:0/1538382135 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff638068bc0 con 0x7ff638073320 2024-04-14T23:48:50.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.511+0000 7ff62e000640 1 -- 172.21.15.60:0/1538382135 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7ff628011070 con 0x7ff638073320 2024-04-14T23:48:50.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.512+0000 7ff62e000640 1 --2- 172.21.15.60:0/1538382135 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff61406b990 0x7ff61406de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:50.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.512+0000 7ff62e000640 1 -- 172.21.15.60:0/1538382135 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ff62808a3d0 con 0x7ff638073320 2024-04-14T23:48:50.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.512+0000 7ff635400640 1 --2- 172.21.15.60:0/1538382135 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff61406b990 0x7ff61406de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:50.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.513+0000 7ff635400640 1 --2- 172.21.15.60:0/1538382135 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff61406b990 0x7ff61406de50 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff620036000 tx=0x7ff62003b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:50.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.516+0000 7ff62e000640 1 -- 172.21.15.60:0/1538382135 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff638068bc0 con 0x7ff638073320 2024-04-14T23:48:50.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.719+0000 7ff637200640 1 -- 172.21.15.60:0/1538382135 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff6381eeaf0 con 0x7ff638073320 2024-04-14T23:48:50.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.722+0000 7ff62e000640 1 -- 172.21.15.60:0/1538382135 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16605 (secure 0 0 0) 0x7ff628058fd0 con 0x7ff638073320 2024-04-14T23:48:50.728 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:48:50.728 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":21,"fsid":"5847db56-a76d-4486-b563-d83f1186af92","created":"2024-04-14T23:48:23.900003+0000","modified":"2024-04-14T23:48:44.219352+0000","last_up_change":"2024-04-14T23:48:34.203573+0000","last_in_change":"2024-04-14T23:48:26.345499+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-14T23:48:36.247372+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-14T23:48:40.274195+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-14T23:48:41.722183+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b2b2b755-7f42-4f9b-af9a-a178f7cc5a52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6817","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6818","nonce":2536528895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6822","nonce":2536528895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6827","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6828","nonce":2536528895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6825","nonce":2536528895}]},"public_addr":"172.21.15.60:6818/2536528895","cluster_addr":"172.21.15.60:6822/2536528895","heartbeat_back_addr":"172.21.15.60:6828/2536528895","heartbeat_front_addr":"172.21.15.60:6825/2536528895","state":["exists","up"]},{"osd":1,"uuid":"321418cd-5ab7-41a5-b9c8-b3fc12f68ebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6801","nonce":3115695676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6803","nonce":3115695676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6807","nonce":3115695676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6805","nonce":3115695676}]},"public_addr":"172.21.15.60:6801/3115695676","cluster_addr":"172.21.15.60:6803/3115695676","heartbeat_back_addr":"172.21.15.60:6807/3115695676","heartbeat_front_addr":"172.21.15.60:6805/3115695676","state":["exists","up"]},{"osd":2,"uuid":"07874e9f-3a38-41dc-be9f-2b75e84e53d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6819","nonce":3475168698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6824","nonce":3475168698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6831","nonce":3475168698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6829","nonce":3475168698}]},"public_addr":"172.21.15.60:6819/3475168698","cluster_addr":"172.21.15.60:6824/3475168698","heartbeat_back_addr":"172.21.15.60:6831/3475168698","heartbeat_front_addr":"172.21.15.60:6829/3475168698","state":["exists","up"]},{"osd":3,"uuid":"7e4ef6c9-b17a-47c6-a320-f5d16a6fe05e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6809","nonce":1730813674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6811","nonce":1730813674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6815","nonce":1730813674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6813","nonce":1730813674}]},"public_addr":"172.21.15.60:6809/1730813674","cluster_addr":"172.21.15.60:6811/1730813674","heartbeat_back_addr":"172.21.15.60:6815/1730813674","heartbeat_front_addr":"172.21.15.60:6813/1730813674","state":["exists","up"]},{"osd":4,"uuid":"759a3efc-dade-44c5-b57a-1623499aba50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6805","nonce":1464587038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6810","nonce":1464587038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6819","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6822","nonce":1464587038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6813","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6816","nonce":1464587038}]},"public_addr":"172.21.15.186:6805/1464587038","cluster_addr":"172.21.15.186:6810/1464587038","heartbeat_back_addr":"172.21.15.186:6822/1464587038","heartbeat_front_addr":"172.21.15.186:6816/1464587038","state":["exists","up"]},{"osd":5,"uuid":"24cd58c1-0cf1-4077-ab33-84c39c6907ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6801","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6803","nonce":1697098067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6809","nonce":1697098067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6820","nonce":1697098067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6815","nonce":1697098067}]},"public_addr":"172.21.15.186:6803/1697098067","cluster_addr":"172.21.15.186:6809/1697098067","heartbeat_back_addr":"172.21.15.186:6820/1697098067","heartbeat_front_addr":"172.21.15.186:6815/1697098067","state":["exists","up"]},{"osd":6,"uuid":"6add38f1-65e9-4b0d-927a-4ab77f5e2323","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6804","nonce":1837108559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6807","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6811","nonce":1837108559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6823","nonce":1837108559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6817","nonce":1837108559}]},"public_addr":"172.21.15.186:6804/1837108559","cluster_addr":"172.21.15.186:6811/1837108559","heartbeat_back_addr":"172.21.15.186:6823/1837108559","heartbeat_front_addr":"172.21.15.186:6817/1837108559","state":["exists","up"]},{"osd":7,"uuid":"a2326b6f-7601-4dda-a456-67c443ea5929","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6825","nonce":842537875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6827","nonce":842537875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6831","nonce":842537875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6829","nonce":842537875}]},"public_addr":"172.21.15.186:6825/842537875","cluster_addr":"172.21.15.186:6827/842537875","heartbeat_back_addr":"172.21.15.186:6831/842537875","heartbeat_front_addr":"172.21.15.186:6829/842537875","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.975583+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.976806+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.722725+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.763705+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.502140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.528972+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.517961+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.513350+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-14T23:48:50.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.725+0000 7ff637200640 1 -- 172.21.15.60:0/1538382135 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff61406b990 msgr2=0x7ff61406de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:50.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.725+0000 7ff637200640 1 --2- 172.21.15.60:0/1538382135 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff61406b990 0x7ff61406de50 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff620036000 tx=0x7ff62003b040 comp rx=0 tx=0).stop 2024-04-14T23:48:50.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.725+0000 7ff637200640 1 -- 172.21.15.60:0/1538382135 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff638073320 msgr2=0x7ff638149120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:50.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.725+0000 7ff637200640 1 --2- 172.21.15.60:0/1538382135 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff638073320 0x7ff638149120 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff62800c4a0 tx=0x7ff62800c8a0 comp rx=0 tx=0).stop 2024-04-14T23:48:50.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.725+0000 7ff637200640 1 -- 172.21.15.60:0/1538382135 shutdown_connections 2024-04-14T23:48:50.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.725+0000 7ff637200640 1 --2- 172.21.15.60:0/1538382135 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff61406b990 0x7ff61406de50 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:50.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.725+0000 7ff637200640 1 --2- 172.21.15.60:0/1538382135 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6380c3d90 0x7ff6381ef6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:50.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.725+0000 7ff637200640 1 --2- 172.21.15.60:0/1538382135 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6380b1e00 0x7ff6381ef1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:50.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.725+0000 7ff637200640 1 --2- 172.21.15.60:0/1538382135 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff638073320 0x7ff638149120 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:50.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.725+0000 7ff637200640 1 -- 172.21.15.60:0/1538382135 >> 172.21.15.60:0/1538382135 conn(0x7ff6380c8b50 msgr2=0x7ff6381516f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:50.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.725+0000 7ff637200640 1 -- 172.21.15.60:0/1538382135 shutdown_connections 2024-04-14T23:48:50.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:50.725+0000 7ff637200640 1 -- 172.21.15.60:0/1538382135 wait complete. 2024-04-14T23:48:50.751 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-14T23:48:50.751 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-14T23:48:50.752 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-14T23:48:50.752 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-14T23:48:50.753 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-04-14T23:48:50.753 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-04-14T23:48:50.753 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-04-14T23:48:50.754 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-04-14T23:48:51.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.048+0000 7f8918c00640 1 Processor -- start 2024-04-14T23:48:51.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.048+0000 7f8918c00640 1 -- start start 2024-04-14T23:48:51.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.050+0000 7f8918c00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8914152bf0 0x7f891406edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.050+0000 7f8918c00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f891406f330 0x7f8914158fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.050+0000 7f8918c00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8914152fd0 con 0x7f89140cb450 2024-04-14T23:48:51.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.050+0000 7f8918c00640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f89140cb990 con 0x7f8914152bf0 2024-04-14T23:48:51.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.050+0000 7f8918c00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f891406e630 con 0x7f891406f330 2024-04-14T23:48:51.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.053+0000 7f8911e00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8914152bf0 0x7f891406edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.053+0000 7f8911400640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f89140cb450 0x7f89141588a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:43670/0 (socket says 172.21.15.60:43670) 2024-04-14T23:48:51.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.053+0000 7f8911400640 1 -- 172.21.15.60:0/1365508610 learned_addr learned my addr 172.21.15.60:0/1365508610 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:48:51.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.053+0000 7f8912800640 1 --2- 172.21.15.60:0/1365508610 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f891406f330 0x7f8914158fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.053+0000 7f8911e00640 1 -- 172.21.15.60:0/1365508610 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f891406f330 msgr2=0x7f8914158fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:51.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.053+0000 7f8911e00640 1 --2- 172.21.15.60:0/1365508610 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f891406f330 0x7f8914158fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.053+0000 7f8911e00640 1 -- 172.21.15.60:0/1365508610 >> v1:172.21.15.60:6789/0 conn(0x7f89140cb450 legacy=0x7f89141588a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.053+0000 7f8911e00640 1 -- 172.21.15.60:0/1365508610 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89140c3570 con 0x7f8914152bf0 2024-04-14T23:48:51.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.056+0000 7f870ac00640 1 Processor -- start 2024-04-14T23:48:51.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.056+0000 7f8911e00640 1 --2- 172.21.15.60:0/1365508610 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8914152bf0 0x7f891406edf0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8904009800 tx=0x7f890403c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f19b2de03f59f50 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:51.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.057+0000 7f8910a00640 1 -- 172.21.15.60:0/1365508610 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8904046020 con 0x7f8914152bf0 2024-04-14T23:48:51.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.057+0000 7f8910a00640 1 -- 172.21.15.60:0/1365508610 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8904042640 con 0x7f8914152bf0 2024-04-14T23:48:51.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.057+0000 7f8910a00640 1 -- 172.21.15.60:0/1365508610 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8904046020 con 0x7f8914152bf0 2024-04-14T23:48:51.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.058+0000 7f8918c00640 1 -- 172.21.15.60:0/1365508610 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8914152bf0 msgr2=0x7f891406edf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.059+0000 7f8918c00640 1 --2- 172.21.15.60:0/1365508610 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8914152bf0 0x7f891406edf0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8904009800 tx=0x7f890403c8e0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.056+0000 7f870ac00640 1 -- start start 2024-04-14T23:48:51.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.059+0000 7f870ac00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f870406b450 0x7f87040cb1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.059+0000 7f870ac00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87040c7f00 con 0x7f8704155030 2024-04-14T23:48:51.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.059+0000 7f870ac00640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f870415bb30 con 0x7f870406b890 2024-04-14T23:48:51.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.059+0000 7f870ac00640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f8704155410 con 0x7f870406b450 2024-04-14T23:48:51.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.059+0000 7f8918c00640 1 -- 172.21.15.60:0/1365508610 shutdown_connections 2024-04-14T23:48:51.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.059+0000 7f8918c00640 1 --2- 172.21.15.60:0/1365508610 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f891406f330 0x7f8914158fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.059+0000 7f8918c00640 1 --2- 172.21.15.60:0/1365508610 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8914152bf0 0x7f891406edf0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.059+0000 7f8918c00640 1 -- 172.21.15.60:0/1365508610 >> 172.21.15.60:0/1365508610 conn(0x7f8914070d20 msgr2=0x7f8914071100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:51.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.060+0000 7f8918c00640 1 -- 172.21.15.60:0/1365508610 shutdown_connections 2024-04-14T23:48:51.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.060+0000 7f8918c00640 1 -- 172.21.15.60:0/1365508610 wait complete. 2024-04-14T23:48:51.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.060+0000 7f8708c00640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f870406b890 0x7f8704165500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.60:56818/0 (socket says 172.21.15.60:56818) 2024-04-14T23:48:51.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.060+0000 7f8708c00640 1 -- 172.21.15.60:0/2287086576 learned_addr learned my addr 172.21.15.60:0/2287086576 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:48:51.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.060+0000 7f09efe00640 1 Processor -- start 2024-04-14T23:48:51.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.060+0000 7f09efe00640 1 -- start start 2024-04-14T23:48:51.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.061+0000 7f8702a00640 1 -- 172.21.15.60:0/2287086576 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1960921878 0 0) 0x7f870415bb30 con 0x7f870406b890 2024-04-14T23:48:51.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.061+0000 7f8702a00640 1 -- 172.21.15.60:0/2287086576 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86e8003520 con 0x7f870406b890 2024-04-14T23:48:51.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.061+0000 7f8702a00640 1 -- 172.21.15.60:0/2287086576 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 41574010 0 0) 0x7f86e8003520 con 0x7f870406b890 2024-04-14T23:48:51.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.061+0000 7f8702a00640 1 -- 172.21.15.60:0/2287086576 >> v2:172.21.15.186:3300/0 conn(0x7f870406b450 msgr2=0x7f87040cb1e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-14T23:48:51.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.061+0000 7f8702a00640 1 --2- 172.21.15.60:0/2287086576 >> v2:172.21.15.186:3300/0 conn(0x7f870406b450 0x7f87040cb1e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.061+0000 7f8702a00640 1 -- 172.21.15.60:0/2287086576 >> v1:172.21.15.60:6789/0 conn(0x7f8704155030 legacy=0x7f87040c77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:51.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.062+0000 7f8702a00640 1 -- 172.21.15.60:0/2287086576 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87040cc3f0 con 0x7f870406b890 2024-04-14T23:48:51.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.062+0000 7f8702a00640 1 -- 172.21.15.60:0/2287086576 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2814038680 0 0) 0x7f86fc0019f0 con 0x7f870406b890 2024-04-14T23:48:51.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.062+0000 7f8702a00640 1 -- 172.21.15.60:0/2287086576 >> v1:172.21.15.186:6789/0 conn(0x7f870406b890 legacy=0x7f8704165500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.062+0000 7f8702a00640 1 --2- 172.21.15.60:0/2287086576 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86e80039b0 0x7f86e8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.062+0000 7f8702a00640 1 --2- 172.21.15.60:0/2287086576 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86e8004800 0x7f86e8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.062+0000 7f8702a00640 1 --2- 172.21.15.60:0/2287086576 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86e80052e0 0x7f86e800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.062+0000 7f8702a00640 1 -- 172.21.15.60:0/2287086576 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f8704155410 con 0x7f86e80052e0 2024-04-14T23:48:51.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.062+0000 7f8702a00640 1 -- 172.21.15.60:0/2287086576 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f86e800c1d0 con 0x7f86e80039b0 2024-04-14T23:48:51.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.062+0000 7f8702a00640 1 -- 172.21.15.60:0/2287086576 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f86e800c3d0 con 0x7f86e8004800 2024-04-14T23:48:51.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.063+0000 7f8703e00640 1 --2- 172.21.15.60:0/2287086576 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86e80039b0 0x7f86e8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.062+0000 7f8918c00640 1 Processor -- start 2024-04-14T23:48:51.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.063+0000 7f8918c00640 1 -- start start 2024-04-14T23:48:51.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.064+0000 7f09efe00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f09e804b500 0x7f09e80dac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.063+0000 7f8703e00640 1 -- 172.21.15.60:0/2287086576 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86e8004800 msgr2=0x7f86e8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:51.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.063+0000 7f8703e00640 1 --2- 172.21.15.60:0/2287086576 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86e8004800 0x7f86e8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.063+0000 7f8703e00640 1 -- 172.21.15.60:0/2287086576 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86e80052e0 msgr2=0x7f86e800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:51.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.063+0000 7f8703e00640 1 --2- 172.21.15.60:0/2287086576 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86e80052e0 0x7f86e800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.064+0000 7f09efe00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09e8007710 con 0x7f09e8034a40 2024-04-14T23:48:51.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.064+0000 7f09efe00640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09e80068b0 con 0x7f09e804b8e0 2024-04-14T23:48:51.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.063+0000 7f8703e00640 1 -- 172.21.15.60:0/2287086576 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87040cc3f0 con 0x7f86e80039b0 2024-04-14T23:48:51.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.064+0000 7f8708c00640 1 -- 172.21.15.60:0/2287086576 reap_dead start 2024-04-14T23:48:51.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.064+0000 7f09efe00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f09e8006b90 con 0x7f09e804b500 2024-04-14T23:48:51.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.064+0000 7f09eea00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f09e804b500 0x7f09e80dac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.064+0000 7f09eea00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f09e804b500 0x7f09e80dac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.60:48518/0 (socket says 172.21.15.60:48518) 2024-04-14T23:48:51.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.064+0000 7f09eea00640 1 -- 172.21.15.60:0/2801013076 learned_addr learned my addr 172.21.15.60:0/2801013076 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:51.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.065+0000 7f09ed600640 1 -- 172.21.15.60:0/2801013076 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3713340369 0 0) 0x7f09e80068b0 con 0x7f09e804b8e0 2024-04-14T23:48:51.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.065+0000 7f09ed600640 1 -- 172.21.15.60:0/2801013076 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f09d4003520 con 0x7f09e804b8e0 2024-04-14T23:48:51.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.065+0000 7f09eea00640 1 -- 172.21.15.60:0/2801013076 >> v1:172.21.15.186:6789/0 conn(0x7f09e804b8e0 legacy=0x7f09e8005e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.065+0000 7f09eea00640 1 -- 172.21.15.60:0/2801013076 >> v1:172.21.15.60:6789/0 conn(0x7f09e8034a40 legacy=0x7f09e8034ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.065+0000 7f8703e00640 1 --2- 172.21.15.60:0/2287086576 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86e80039b0 0x7f86e8003dd0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f86f0001750 tx=0x7f86e8005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db7ee36a6110adec server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:51.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.066+0000 7f8918c00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f891406f330 0x7f89141f08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.066+0000 7f8918c00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89140cb450 0x7f89141ec7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.066+0000 7f8702a00640 1 -- 172.21.15.60:0/2287086576 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86f0001370 con 0x7f86e80039b0 2024-04-14T23:48:51.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.066+0000 7f8702a00640 1 -- 172.21.15.60:0/2287086576 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86f00052d0 con 0x7f86e80039b0 2024-04-14T23:48:51.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.066+0000 7f8702a00640 1 -- 172.21.15.60:0/2287086576 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86f0003830 con 0x7f86e80039b0 2024-04-14T23:48:51.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.066+0000 7f09eea00640 1 -- 172.21.15.60:0/2801013076 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09e8006e70 con 0x7f09e804b500 2024-04-14T23:48:51.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.066+0000 7f5366000640 1 Processor -- start 2024-04-14T23:48:51.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.067+0000 7f8911e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f891406f330 0x7f89141f08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.067+0000 7f8911e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f891406f330 0x7f89141f08a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.60:48528/0 (socket says 172.21.15.60:48528) 2024-04-14T23:48:51.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.067+0000 7f8918c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8914152bf0 0x7f891414e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.067+0000 7f8911e00640 1 -- 172.21.15.60:0/2906597355 learned_addr learned my addr 172.21.15.60:0/2906597355 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:51.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.067+0000 7f8918c00640 1 -- 172.21.15.60:0/2906597355 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f891406e8d0 con 0x7f89140cb450 2024-04-14T23:48:51.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.067+0000 7f8918c00640 1 -- 172.21.15.60:0/2906597355 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f89140c9ab0 con 0x7f8914152bf0 2024-04-14T23:48:51.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.067+0000 7f8918c00640 1 -- 172.21.15.60:0/2906597355 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f89140c9d90 con 0x7f891406f330 2024-04-14T23:48:51.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.067+0000 7f8911400640 1 --2- 172.21.15.60:0/2906597355 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89140cb450 0x7f89141ec7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.067+0000 7f09eea00640 1 --2- 172.21.15.60:0/2801013076 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f09e804b500 0x7f09e80dac60 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f09e00098a0 tx=0x7f09e002b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a99b7bd59752c761 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:51.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.068+0000 7f8911400640 1 -- 172.21.15.60:0/2906597355 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f891406f330 msgr2=0x7f89141f08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.068+0000 7f09ed600640 1 -- 172.21.15.60:0/2801013076 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09e0031790 con 0x7f09e804b500 2024-04-14T23:48:51.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.068+0000 7f8911400640 1 --2- 172.21.15.60:0/2906597355 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f891406f330 0x7f89141f08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.068+0000 7f8912800640 1 --2- 172.21.15.60:0/2906597355 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8914152bf0 0x7f891414e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.068+0000 7f8911400640 1 -- 172.21.15.60:0/2906597355 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8914152bf0 msgr2=0x7f891414e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.068+0000 7f870ac00640 1 -- 172.21.15.60:0/2287086576 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86e80039b0 msgr2=0x7f86e8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.068+0000 7f870ac00640 1 --2- 172.21.15.60:0/2287086576 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86e80039b0 0x7f86e8003dd0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f86f0001750 tx=0x7f86e8005830 comp rx=0 tx=0).stop 2024-04-14T23:48:51.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.068+0000 7f09ed600640 1 -- 172.21.15.60:0/2801013076 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09e0037070 con 0x7f09e804b500 2024-04-14T23:48:51.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.068+0000 7f09ed600640 1 -- 172.21.15.60:0/2801013076 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09e0031b50 con 0x7f09e804b500 2024-04-14T23:48:51.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.068+0000 7f870ac00640 1 -- 172.21.15.60:0/2287086576 shutdown_connections 2024-04-14T23:48:51.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.068+0000 7f870ac00640 1 --2- 172.21.15.60:0/2287086576 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86e80039b0 0x7f86e8003dd0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.068+0000 7f870ac00640 1 -- 172.21.15.60:0/2287086576 >> 172.21.15.60:0/2287086576 conn(0x7f8704066f10 msgr2=0x7f8704067320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:51.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.068+0000 7f5366000640 1 -- start start 2024-04-14T23:48:51.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.068+0000 7f870ac00640 1 -- 172.21.15.60:0/2287086576 shutdown_connections 2024-04-14T23:48:51.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.068+0000 7f8911400640 1 --2- 172.21.15.60:0/2906597355 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8914152bf0 0x7f891414e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.068+0000 7f8911400640 1 -- 172.21.15.60:0/2906597355 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8914153b10 con 0x7f89140cb450 2024-04-14T23:48:51.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.069+0000 7f8912800640 1 --2- 172.21.15.60:0/2906597355 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8914152bf0 0x7f891414e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-14T23:48:51.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.069+0000 7f8911400640 1 --2- 172.21.15.60:0/2906597355 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89140cb450 0x7f89141ec7b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f890000c470 tx=0x7f890000c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:51.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.070+0000 7f09efe00640 1 -- 172.21.15.60:0/2801013076 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f09e804b500 msgr2=0x7f09e80dac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.070+0000 7f09efe00640 1 --2- 172.21.15.60:0/2801013076 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f09e804b500 0x7f09e80dac60 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f09e00098a0 tx=0x7f09e002b8b0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.070+0000 7f890a000640 1 -- 172.21.15.60:0/2906597355 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8900017070 con 0x7f89140cb450 2024-04-14T23:48:51.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.069+0000 7f870ac00640 1 -- 172.21.15.60:0/2287086576 wait complete. 2024-04-14T23:48:51.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.070+0000 7f09efe00640 1 -- 172.21.15.60:0/2801013076 shutdown_connections 2024-04-14T23:48:51.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.070+0000 7f09efe00640 1 --2- 172.21.15.60:0/2801013076 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f09e804b500 0x7f09e80dac60 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.070+0000 7f09efe00640 1 -- 172.21.15.60:0/2801013076 >> 172.21.15.60:0/2801013076 conn(0x7f09e8050180 msgr2=0x7f09e8050590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:51.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.070+0000 7f09efe00640 1 -- 172.21.15.60:0/2801013076 shutdown_connections 2024-04-14T23:48:51.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.070+0000 7f09efe00640 1 -- 172.21.15.60:0/2801013076 wait complete. 2024-04-14T23:48:51.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.070+0000 7f5366000640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f53600c8360 0x7f536015ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.070+0000 7f5366000640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f53600c8740 0x7f536006f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.070+0000 7f5366000640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f53600c75c0 0x7f5360158e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.070+0000 7f890a000640 1 -- 172.21.15.60:0/2906597355 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8900005ce0 con 0x7f89140cb450 2024-04-14T23:48:51.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.070+0000 7f890a000640 1 -- 172.21.15.60:0/2906597355 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89000072d0 con 0x7f89140cb450 2024-04-14T23:48:51.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.070+0000 7f5366000640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f53600c6b70 con 0x7f53600c8740 2024-04-14T23:48:51.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.070+0000 7f5366000640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f536015f7c0 con 0x7f53600c8360 2024-04-14T23:48:51.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.070+0000 7f5366000640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f536015faa0 con 0x7f53600c75c0 2024-04-14T23:48:51.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.071+0000 7ff604e00640 1 Processor -- start 2024-04-14T23:48:51.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.071+0000 7ff604e00640 1 -- start start 2024-04-14T23:48:51.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.071+0000 7f535ea00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f53600c8740 0x7f536006f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.071+0000 7f535ea00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f53600c8740 0x7f536006f310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54010/0 (socket says 172.21.15.60:54010) 2024-04-14T23:48:51.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.072+0000 7f535fe00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f53600c75c0 0x7f5360158e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.072+0000 7f535fe00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f53600c75c0 0x7f5360158e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.60:48538/0 (socket says 172.21.15.60:48538) 2024-04-14T23:48:51.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.072+0000 7f535fe00640 1 -- 172.21.15.60:0/2058953296 learned_addr learned my addr 172.21.15.60:0/2058953296 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:51.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.072+0000 7f870ac00640 1 Processor -- start 2024-04-14T23:48:51.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.072+0000 7f535f400640 1 --2- 172.21.15.60:0/2058953296 >> v2:172.21.15.186:3300/0 conn(0x7f53600c8360 0x7f536015ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.073+0000 7ff604e00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff60006e740 0x7ff6000b1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.073+0000 7f870ac00640 1 -- start start 2024-04-14T23:48:51.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.073+0000 7f535fe00640 1 -- 172.21.15.60:0/2058953296 >> v2:172.21.15.186:3300/0 conn(0x7f53600c8360 msgr2=0x7f536015ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:51.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.073+0000 7f535fe00640 1 --2- 172.21.15.60:0/2058953296 >> v2:172.21.15.186:3300/0 conn(0x7f53600c8360 0x7f536015ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.071+0000 7f8918c00640 1 -- 172.21.15.60:0/2906597355 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8914153df0 con 0x7f89140cb450 2024-04-14T23:48:51.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.073+0000 7f8918c00640 1 -- 172.21.15.60:0/2906597355 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89141f00a0 con 0x7f89140cb450 2024-04-14T23:48:51.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.073+0000 7f535fe00640 1 -- 172.21.15.60:0/2058953296 >> v2:172.21.15.60:3300/0 conn(0x7f53600c8740 msgr2=0x7f536006f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:51.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.073+0000 7f09efe00640 1 Processor -- start 2024-04-14T23:48:51.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.073+0000 7ff604e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6000c78f0 0x7ff600161460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.073+0000 7ff604e00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6000c6420 con 0x7ff600070d10 2024-04-14T23:48:51.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.073+0000 7ff604e00640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7ff60006b250 con 0x7ff60006e740 2024-04-14T23:48:51.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.073+0000 7ff604e00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7ff6000b2030 con 0x7ff6000c78f0 2024-04-14T23:48:51.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.073+0000 7ff5ff400640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff60006e740 0x7ff6000b1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.073+0000 7ff5ff400640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff60006e740 0x7ff6000b1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54024/0 (socket says 172.21.15.60:54024) 2024-04-14T23:48:51.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.073+0000 7ff5ff400640 1 -- 172.21.15.60:0/3321968657 learned_addr learned my addr 172.21.15.60:0/3321968657 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:51.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.074+0000 7ff5ffe00640 1 --2- 172.21.15.60:0/3321968657 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6000c78f0 0x7ff600161460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.073+0000 7f535fe00640 1 --2- 172.21.15.60:0/2058953296 >> v2:172.21.15.60:3300/0 conn(0x7f53600c8740 0x7f536006f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.073+0000 7f535fe00640 1 -- 172.21.15.60:0/2058953296 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f536015fd80 con 0x7f53600c75c0 2024-04-14T23:48:51.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.074+0000 7f535f400640 1 --2- 172.21.15.60:0/2058953296 >> v2:172.21.15.186:3300/0 conn(0x7f53600c8360 0x7f536015ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:48:51.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.074+0000 7f535ea00640 1 --2- 172.21.15.60:0/2058953296 >> v2:172.21.15.60:3300/0 conn(0x7f53600c8740 0x7f536006f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-14T23:48:51.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.074+0000 7ff5ff400640 1 -- 172.21.15.60:0/3321968657 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6000c78f0 msgr2=0x7ff600161460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:51.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.074+0000 7ff5ff400640 1 --2- 172.21.15.60:0/3321968657 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6000c78f0 0x7ff600161460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.074+0000 7ff5ff400640 1 -- 172.21.15.60:0/3321968657 >> v1:172.21.15.60:6789/0 conn(0x7ff600070d10 legacy=0x7ff6000c71e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:51.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.074+0000 7ff5ff400640 1 -- 172.21.15.60:0/3321968657 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6000c6140 con 0x7ff60006e740 2024-04-14T23:48:51.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.074+0000 7f8918c00640 1 -- 172.21.15.60:0/2906597355 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8914076970 con 0x7f89140cb450 2024-04-14T23:48:51.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.075+0000 7f535fe00640 1 --2- 172.21.15.60:0/2058953296 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f53600c75c0 0x7f5360158e30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5360073940 tx=0x7f535802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8650289557f24f62 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:51.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.075+0000 7f870ac00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86e80039b0 0x7f87041f11f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.075+0000 7f870ac00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f870415bdc0 0x7f87040d0880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.075+0000 7f870ac00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87040d0dc0 0x7f87041e5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.074+0000 7f09efe00640 1 -- start start 2024-04-14T23:48:51.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.076+0000 7f09efe00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f09e8034a40 0x7f09e80d3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.076+0000 7f09efe00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f09e804b500 0x7f09e80d0000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.076+0000 7f09efe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09e804b8e0 0x7f09e817cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.076+0000 7f09efe00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f09e80065a0 con 0x7f09e8034a40 2024-04-14T23:48:51.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.076+0000 7f09efe00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f09e8007430 con 0x7f09e804b8e0 2024-04-14T23:48:51.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.076+0000 7f09efe00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f09e804bd00 con 0x7f09e804b500 2024-04-14T23:48:51.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.076+0000 7f8708c00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87040d0dc0 0x7f87041e5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.076+0000 7f8708c00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87040d0dc0 0x7f87041e5530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54026/0 (socket says 172.21.15.60:54026) 2024-04-14T23:48:51.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.076+0000 7f09ef400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09e804b8e0 0x7f09e817cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.076+0000 7f09ef400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09e804b8e0 0x7f09e817cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:43482/0 (socket says 172.21.15.60:43482) 2024-04-14T23:48:51.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.076+0000 7f09ef400640 1 -- 172.21.15.60:0/3117765181 learned_addr learned my addr 172.21.15.60:0/3117765181 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:51.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.077+0000 7f09eea00640 1 --2- 172.21.15.60:0/3117765181 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f09e8034a40 0x7f09e80d3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.077+0000 7f09ee000640 1 --2- 172.21.15.60:0/3117765181 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f09e804b500 0x7f09e80d0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.076+0000 7f535e000640 1 -- 172.21.15.60:0/2058953296 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f535802cb00 con 0x7f53600c75c0 2024-04-14T23:48:51.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.077+0000 7f535e000640 1 -- 172.21.15.60:0/2058953296 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f535802cca0 con 0x7f53600c75c0 2024-04-14T23:48:51.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.077+0000 7f535e000640 1 -- 172.21.15.60:0/2058953296 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5358034610 con 0x7f53600c75c0 2024-04-14T23:48:51.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.076+0000 7f8703400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f870415bdc0 0x7f87040d0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.077+0000 7f870ac00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f870415b050 con 0x7f87040d0dc0 2024-04-14T23:48:51.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.077+0000 7ff5ff400640 1 --2- 172.21.15.60:0/3321968657 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff60006e740 0x7ff6000b1af0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff5f8014a00 tx=0x7ff5f8034a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8b8f59e8b2cf9d server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:51.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.077+0000 7f870ac00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f870415ad70 con 0x7f86e80039b0 2024-04-14T23:48:51.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.077+0000 7f870ac00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f870415b330 con 0x7f870415bdc0 2024-04-14T23:48:51.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.078+0000 7f5366000640 1 -- 172.21.15.60:0/2058953296 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f53600c75c0 msgr2=0x7f5360158e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.078+0000 7f5366000640 1 --2- 172.21.15.60:0/2058953296 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f53600c75c0 0x7f5360158e30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5360073940 tx=0x7f535802bf60 comp rx=0 tx=0).stop 2024-04-14T23:48:51.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.077+0000 7f8708c00640 1 -- 172.21.15.60:0/1888925289 learned_addr learned my addr 172.21.15.60:0/1888925289 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:51.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.077+0000 7f8703e00640 1 --2- 172.21.15.60:0/1888925289 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86e80039b0 0x7f87041f11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.078+0000 7f5366000640 1 -- 172.21.15.60:0/2058953296 shutdown_connections 2024-04-14T23:48:51.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.078+0000 7f5366000640 1 --2- 172.21.15.60:0/2058953296 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f53600c75c0 0x7f5360158e30 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.078+0000 7f5366000640 1 --2- 172.21.15.60:0/2058953296 >> v2:172.21.15.60:3300/0 conn(0x7f53600c8740 0x7f536006f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.078+0000 7f5366000640 1 --2- 172.21.15.60:0/2058953296 >> v2:172.21.15.186:3300/0 conn(0x7f53600c8360 0x7f536015ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.078+0000 7f5366000640 1 -- 172.21.15.60:0/2058953296 >> 172.21.15.60:0/2058953296 conn(0x7f53600cc170 msgr2=0x7f53600cc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:51.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.078+0000 7ff5fe000640 1 -- 172.21.15.60:0/3321968657 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5f803c070 con 0x7ff60006e740 2024-04-14T23:48:51.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.078+0000 7ff5fe000640 1 -- 172.21.15.60:0/3321968657 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5f80114e0 con 0x7ff60006e740 2024-04-14T23:48:51.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.078+0000 7ff5fe000640 1 -- 172.21.15.60:0/3321968657 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5f80118b0 con 0x7ff60006e740 2024-04-14T23:48:51.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.079+0000 7f09ef400640 1 -- 172.21.15.60:0/3117765181 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f09e804b500 msgr2=0x7f09e80d0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.078+0000 7f8703e00640 1 -- 172.21.15.60:0/1888925289 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f870415bdc0 msgr2=0x7f87040d0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.078+0000 7f8703e00640 1 --2- 172.21.15.60:0/1888925289 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f870415bdc0 0x7f87040d0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.078+0000 7f8703e00640 1 -- 172.21.15.60:0/1888925289 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87040d0dc0 msgr2=0x7f87041e5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.079+0000 7f09ef400640 1 --2- 172.21.15.60:0/3117765181 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f09e804b500 0x7f09e80d0000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.078+0000 7f8703e00640 1 --2- 172.21.15.60:0/1888925289 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87040d0dc0 0x7f87041e5530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.078+0000 7f8703e00640 1 -- 172.21.15.60:0/1888925289 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87040c3b70 con 0x7f86e80039b0 2024-04-14T23:48:51.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.078+0000 7f8708c00640 1 --2- 172.21.15.60:0/1888925289 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87040d0dc0 0x7f87041e5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:48:51.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.078+0000 7f8703e00640 1 --2- 172.21.15.60:0/1888925289 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86e80039b0 0x7f87041f11f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f86f0004000 tx=0x7f86f0001120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:51.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.079+0000 7f86fbe00640 1 -- 172.21.15.60:0/1888925289 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86f0041020 con 0x7f86e80039b0 2024-04-14T23:48:51.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.079+0000 7f86fbe00640 1 -- 172.21.15.60:0/1888925289 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86f0005480 con 0x7f86e80039b0 2024-04-14T23:48:51.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.079+0000 7f86fbe00640 1 -- 172.21.15.60:0/1888925289 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86f0001370 con 0x7f86e80039b0 2024-04-14T23:48:51.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.079+0000 7f870ac00640 1 -- 172.21.15.60:0/1888925289 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87040ce540 con 0x7f86e80039b0 2024-04-14T23:48:51.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.079+0000 7f09ef400640 1 -- 172.21.15.60:0/3117765181 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f09e8034a40 msgr2=0x7f09e80d3fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.079+0000 7f09ef400640 1 --2- 172.21.15.60:0/3117765181 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f09e8034a40 0x7f09e80d3fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.079+0000 7f09ef400640 1 -- 172.21.15.60:0/3117765181 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09e80d2330 con 0x7f09e804b8e0 2024-04-14T23:48:51.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.077+0000 7f890a000640 1 -- 172.21.15.60:0/2906597355 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f890001a070 con 0x7f89140cb450 2024-04-14T23:48:51.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.078+0000 7f890a000640 1 --2- 172.21.15.60:0/2906597355 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f88f406b9e0 0x7f88f406dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.079+0000 7f890a000640 1 -- 172.21.15.60:0/2906597355 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f8900089970 con 0x7f89140cb450 2024-04-14T23:48:51.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.079+0000 7f890a000640 1 --2- 172.21.15.60:0/2906597355 >> [v2:172.21.15.60:6817/2536528895,v1:172.21.15.60:6818/2536528895] conn(0x7f88f4073b80 0x7f88f4075fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.079+0000 7f890a000640 1 -- 172.21.15.60:0/2906597355 --> [v2:172.21.15.60:6817/2536528895,v1:172.21.15.60:6818/2536528895] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f88f4076650 con 0x7f88f4073b80 2024-04-14T23:48:51.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.079+0000 7f890a000640 1 -- 172.21.15.60:0/2906597355 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f89000540e0 con 0x7f89140cb450 2024-04-14T23:48:51.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.079+0000 7f870ac00640 1 -- 172.21.15.60:0/1888925289 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87040ceb00 con 0x7f86e80039b0 2024-04-14T23:48:51.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.079+0000 7f633a200640 1 Processor -- start 2024-04-14T23:48:51.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.079+0000 7f09ef400640 1 --2- 172.21.15.60:0/3117765181 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09e804b8e0 0x7f09e817cf80 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f09f0064cd0 tx=0x7f09f0068880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:51.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.079+0000 7f09ee000640 1 --2- 172.21.15.60:0/3117765181 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f09e804b500 0x7f09e80d0000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:48:51.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.080+0000 7f5366000640 1 -- 172.21.15.60:0/2058953296 shutdown_connections 2024-04-14T23:48:51.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.080+0000 7ff604e00640 1 -- 172.21.15.60:0/3321968657 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff60006e740 msgr2=0x7ff6000b1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.080+0000 7ff604e00640 1 --2- 172.21.15.60:0/3321968657 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff60006e740 0x7ff6000b1af0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff5f8014a00 tx=0x7ff5f8034a80 comp rx=0 tx=0).stop 2024-04-14T23:48:51.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.080+0000 7f09eea00640 1 --2- 172.21.15.60:0/3117765181 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f09e8034a40 0x7f09e80d3fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:48:51.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.080+0000 7f09e7400640 1 -- 172.21.15.60:0/3117765181 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09f0073450 con 0x7f09e804b8e0 2024-04-14T23:48:51.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.080+0000 7f09e7400640 1 -- 172.21.15.60:0/3117765181 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09f0090030 con 0x7f09e804b8e0 2024-04-14T23:48:51.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.080+0000 7f09e7400640 1 -- 172.21.15.60:0/3117765181 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09f006f460 con 0x7f09e804b8e0 2024-04-14T23:48:51.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.080+0000 7f09efe00640 1 -- 172.21.15.60:0/3117765181 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09e80d2610 con 0x7f09e804b8e0 2024-04-14T23:48:51.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.080+0000 7f8911e00640 1 --2- 172.21.15.60:0/2906597355 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f88f406b9e0 0x7f88f406dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.080+0000 7f8912800640 1 --2- 172.21.15.60:0/2906597355 >> [v2:172.21.15.60:6817/2536528895,v1:172.21.15.60:6818/2536528895] conn(0x7f88f4073b80 0x7f88f4075fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.081+0000 7f8911e00640 1 --2- 172.21.15.60:0/2906597355 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f88f406b9e0 0x7f88f406dea0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8904009560 tx=0x7f8904008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:51.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.081+0000 7f8912800640 1 --2- 172.21.15.60:0/2906597355 >> [v2:172.21.15.60:6817/2536528895,v1:172.21.15.60:6818/2536528895] conn(0x7f88f4073b80 0x7f88f4075fa0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:51.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.080+0000 7f09efe00640 1 -- 172.21.15.60:0/3117765181 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09e80d2bd0 con 0x7f09e804b8e0 2024-04-14T23:48:51.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.080+0000 7f5366000640 1 -- 172.21.15.60:0/2058953296 wait complete. 2024-04-14T23:48:51.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.080+0000 7f633a200640 1 -- start start 2024-04-14T23:48:51.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.082+0000 7f09efe00640 1 -- 172.21.15.60:0/3117765181 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f09c8001030 con 0x7f09e804b8e0 2024-04-14T23:48:51.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.082+0000 7f870ac00640 1 -- 172.21.15.60:0/1888925289 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f86dc001030 con 0x7f86e80039b0 2024-04-14T23:48:51.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.082+0000 7ff604e00640 1 -- 172.21.15.60:0/3321968657 shutdown_connections 2024-04-14T23:48:51.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.082+0000 7ff604e00640 1 --2- 172.21.15.60:0/3321968657 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6000c78f0 0x7ff600161460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.082+0000 7ff604e00640 1 --2- 172.21.15.60:0/3321968657 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff60006e740 0x7ff6000b1af0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.082+0000 7ff604e00640 1 -- 172.21.15.60:0/3321968657 >> 172.21.15.60:0/3321968657 conn(0x7ff6000cc2f0 msgr2=0x7ff6000cc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:51.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.082+0000 7ff604e00640 1 -- 172.21.15.60:0/3321968657 shutdown_connections 2024-04-14T23:48:51.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.082+0000 7f633a200640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f63340cd160 0x7f633415ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.083+0000 7f633a200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63340cb750 0x7f6334154b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.083+0000 7f633a200640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63340b3f50 con 0x7f63340cd540 2024-04-14T23:48:51.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.083+0000 7f633a200640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f633415b370 con 0x7f63340cd160 2024-04-14T23:48:51.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.083+0000 7f633a200640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f63340b39b0 con 0x7f63340cb750 2024-04-14T23:48:51.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.083+0000 7f890a000640 1 -- 172.21.15.60:0/2906597355 <== osd.0 v2:172.21.15.60:6817/2536528895 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f88f4076650 con 0x7f88f4073b80 2024-04-14T23:48:51.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.083+0000 7ff604e00640 1 -- 172.21.15.60:0/3321968657 wait complete. 2024-04-14T23:48:51.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.083+0000 7f5366000640 1 Processor -- start 2024-04-14T23:48:51.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.083+0000 7f09e7400640 1 -- 172.21.15.60:0/3117765181 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f09f006f600 con 0x7f09e804b8e0 2024-04-14T23:48:51.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.084+0000 7f86fbe00640 1 -- 172.21.15.60:0/1888925289 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f86f0044020 con 0x7f86e80039b0 2024-04-14T23:48:51.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.084+0000 7f6333400640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f63340cd160 0x7f633415ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.084+0000 7f6332a00640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f63340cd540 0x7f63340b3190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:43710/0 (socket says 172.21.15.60:43710) 2024-04-14T23:48:51.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.084+0000 7f6332a00640 1 -- 172.21.15.60:0/3088564095 learned_addr learned my addr 172.21.15.60:0/3088564095 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:48:51.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.085+0000 7f86fbe00640 1 --2- 172.21.15.60:0/1888925289 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f86e406ba10 0x7f86e406ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.083+0000 7f5366000640 1 -- start start 2024-04-14T23:48:51.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.085+0000 7f86fbe00640 1 -- 172.21.15.60:0/1888925289 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f86f00ac520 con 0x7f86e80039b0 2024-04-14T23:48:51.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.085+0000 7fd9f6400640 1 Processor -- start 2024-04-14T23:48:51.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.085+0000 7fd9f6400640 1 -- start start 2024-04-14T23:48:51.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.085+0000 7ff604e00640 1 Processor -- start 2024-04-14T23:48:51.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.085+0000 7f8703400640 1 --2- 172.21.15.60:0/1888925289 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f86e406ba10 0x7f86e406ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.085+0000 7f86fbe00640 1 --2- 172.21.15.60:0/1888925289 >> [v2:172.21.15.60:6800/3115695676,v1:172.21.15.60:6801/3115695676] conn(0x7f86e4073c90 0x7f86e40760b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.085+0000 7f86fbe00640 1 -- 172.21.15.60:0/1888925289 --> [v2:172.21.15.60:6800/3115695676,v1:172.21.15.60:6801/3115695676] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f86f00052d0 con 0x7f86e4073c90 2024-04-14T23:48:51.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.085+0000 7f86fbe00640 1 -- 172.21.15.60:0/1888925289 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f86f00ac8d0 con 0x7f86e80039b0 2024-04-14T23:48:51.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.085+0000 7f5366000640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f53600c75c0 0x7f536014de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.085+0000 7f5366000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f53600c8360 0x7f53600cb360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.085+0000 7f5366000640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f53600c8740 0x7f53601f51a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.085+0000 7f5366000640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f536015f4c0 con 0x7f53600c75c0 2024-04-14T23:48:51.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.085+0000 7f5366000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f53600c70a0 con 0x7f53600c8360 2024-04-14T23:48:51.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.085+0000 7f5366000640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f5360071030 con 0x7f53600c8740 2024-04-14T23:48:51.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.085+0000 7f6333400640 1 -- 172.21.15.60:0/3088564095 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63340cb750 msgr2=0x7f6334154b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:51.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.086+0000 7f6333400640 1 --2- 172.21.15.60:0/3088564095 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63340cb750 0x7f6334154b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.086+0000 7f6333400640 1 -- 172.21.15.60:0/3088564095 >> v1:172.21.15.60:6789/0 conn(0x7f63340cd540 legacy=0x7f63340b3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.085+0000 7f535ea00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f53600c8360 0x7f53600cb360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.085+0000 7f535ea00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f53600c8360 0x7f53600cb360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:43512/0 (socket says 172.21.15.60:43512) 2024-04-14T23:48:51.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.085+0000 7f535ea00640 1 -- 172.21.15.60:0/589539644 learned_addr learned my addr 172.21.15.60:0/589539644 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:51.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.086+0000 7ff604e00640 1 -- start start 2024-04-14T23:48:51.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.086+0000 7f6333400640 1 -- 172.21.15.60:0/3088564095 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63340b3c70 con 0x7f63340cd160 2024-04-14T23:48:51.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.086+0000 7f535ea00640 1 -- 172.21.15.60:0/589539644 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f53600c8740 msgr2=0x7f53601f51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.086+0000 7f535fe00640 1 --2- 172.21.15.60:0/589539644 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f53600c8740 0x7f53601f51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.086+0000 7f535ea00640 1 --2- 172.21.15.60:0/589539644 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f53600c8740 0x7f53601f51a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.086+0000 7f535ea00640 1 -- 172.21.15.60:0/589539644 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f53600c75c0 msgr2=0x7f536014de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.086+0000 7f535ea00640 1 --2- 172.21.15.60:0/589539644 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f53600c75c0 0x7f536014de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.086+0000 7f535ea00640 1 -- 172.21.15.60:0/589539644 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53601f8440 con 0x7f53600c8360 2024-04-14T23:48:51.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.086+0000 7f8708c00640 1 --2- 172.21.15.60:0/1888925289 >> [v2:172.21.15.60:6800/3115695676,v1:172.21.15.60:6801/3115695676] conn(0x7f86e4073c90 0x7f86e40760b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.125 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.086+0000 7f535fe00640 1 --2- 172.21.15.60:0/589539644 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f53600c8740 0x7f53601f51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-14T23:48:51.125 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.087+0000 7f535ea00640 1 --2- 172.21.15.60:0/589539644 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f53600c8360 0x7f53600cb360 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f534c00d800 tx=0x7f534c00dcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:51.125 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.087+0000 7f5357400640 1 -- 172.21.15.60:0/589539644 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f534c004030 con 0x7f53600c8360 2024-04-14T23:48:51.125 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.087+0000 7f34fae00640 1 Processor -- start 2024-04-14T23:48:51.125 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.087+0000 7f8703400640 1 --2- 172.21.15.60:0/1888925289 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f86e406ba10 0x7f86e406ded0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f87040b5e60 tx=0x7f86f40090e0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:51.125 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.087+0000 7f8708c00640 1 --2- 172.21.15.60:0/1888925289 >> [v2:172.21.15.60:6800/3115695676,v1:172.21.15.60:6801/3115695676] conn(0x7f86e4073c90 0x7f86e40760b0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:51.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.087+0000 7ff604e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff60006e740 0x7ff6001ec7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.087+0000 7ff604e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff600070d10 0x7ff60006c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.087+0000 7ff604e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff6000c78f0 0x7ff6000ca680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.087+0000 7ff604e00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff600072cc0 con 0x7ff6000c78f0 2024-04-14T23:48:51.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.087+0000 7ff604e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff6000729e0 con 0x7ff60006e740 2024-04-14T23:48:51.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.087+0000 7ff604e00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7ff6000c7d40 con 0x7ff600070d10 2024-04-14T23:48:51.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.087+0000 7f5357400640 1 -- 172.21.15.60:0/589539644 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f534c0041d0 con 0x7f53600c8360 2024-04-14T23:48:51.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.087+0000 7f5357400640 1 -- 172.21.15.60:0/589539644 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f534c0096a0 con 0x7f53600c8360 2024-04-14T23:48:51.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.087+0000 7f5366000640 1 -- 172.21.15.60:0/589539644 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53601f4ca0 con 0x7f53600c8360 2024-04-14T23:48:51.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.087+0000 7f5366000640 1 -- 172.21.15.60:0/589539644 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53601f89e0 con 0x7f53600c8360 2024-04-14T23:48:51.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.088+0000 7ff5ffe00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff6000c78f0 0x7ff6000ca680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.088+0000 7fd9f6400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd9f00c8360 0x7fd9f015ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.088+0000 7ff5ffe00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff6000c78f0 0x7ff6000ca680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54056/0 (socket says 172.21.15.60:54056) 2024-04-14T23:48:51.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.088+0000 7ff5ffe00640 1 -- 172.21.15.60:0/2276777177 learned_addr learned my addr 172.21.15.60:0/2276777177 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:51.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.088+0000 7f6333400640 1 --2- 172.21.15.60:0/3088564095 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63340cd160 0x7f633415ae30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f632c009d90 tx=0x7f632c02bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a98d01702a9a13c server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:51.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.088+0000 7fd9f6400640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd9f00c8740 0x7fd9f006f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.129 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.088+0000 7ff5fea00640 1 --2- 172.21.15.60:0/2276777177 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff600070d10 0x7ff60006c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.129 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.088+0000 7ff5ff400640 1 --2- 172.21.15.60:0/2276777177 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff60006e740 0x7ff6001ec7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.129 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.089+0000 7f86fbe00640 1 -- 172.21.15.60:0/1888925289 <== osd.1 v2:172.21.15.60:6800/3115695676 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f86f00052d0 con 0x7f86e4073c90 2024-04-14T23:48:51.129 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.089+0000 7f6332000640 1 -- 172.21.15.60:0/3088564095 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f632c039070 con 0x7f63340cd160 2024-04-14T23:48:51.129 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.089+0000 7f6332000640 1 -- 172.21.15.60:0/3088564095 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f632c02cce0 con 0x7f63340cd160 2024-04-14T23:48:51.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.089+0000 7f6332000640 1 -- 172.21.15.60:0/3088564095 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f632c034660 con 0x7f63340cd160 2024-04-14T23:48:51.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.088+0000 7fd9f6400640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9f015f7c0 con 0x7fd9f00c75c0 2024-04-14T23:48:51.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.088+0000 7fd9f6400640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fd9f00c6b70 con 0x7fd9f00c8740 2024-04-14T23:48:51.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.088+0000 7fd9f6400640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fd9f015faa0 con 0x7fd9f00c8360 2024-04-14T23:48:51.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.089+0000 7fd9eea00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd9f00c8740 0x7fd9f006f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.089+0000 7fd9eea00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd9f00c8740 0x7fd9f006f310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:43534/0 (socket says 172.21.15.60:43534) 2024-04-14T23:48:51.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.089+0000 7fd9eea00640 1 -- 172.21.15.60:0/645744492 learned_addr learned my addr 172.21.15.60:0/645744492 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:51.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.090+0000 7f633a200640 1 -- 172.21.15.60:0/3088564095 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63340cd160 msgr2=0x7f633415ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.090+0000 7f633a200640 1 --2- 172.21.15.60:0/3088564095 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63340cd160 0x7f633415ae30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f632c009d90 tx=0x7f632c02bf30 comp rx=0 tx=0).stop 2024-04-14T23:48:51.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.090+0000 7f633a200640 1 -- 172.21.15.60:0/3088564095 shutdown_connections 2024-04-14T23:48:51.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.090+0000 7f633a200640 1 --2- 172.21.15.60:0/3088564095 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63340cb750 0x7f6334154b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.090+0000 7f633a200640 1 --2- 172.21.15.60:0/3088564095 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63340cd160 0x7f633415ae30 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.090+0000 7f633a200640 1 -- 172.21.15.60:0/3088564095 >> 172.21.15.60:0/3088564095 conn(0x7f6334066f10 msgr2=0x7f63340672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:51.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.089+0000 7ff5ff400640 1 -- 172.21.15.60:0/2276777177 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff600070d10 msgr2=0x7ff60006c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.089+0000 7ff5ff400640 1 --2- 172.21.15.60:0/2276777177 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff600070d10 0x7ff60006c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.089+0000 7ff5ff400640 1 -- 172.21.15.60:0/2276777177 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff6000c78f0 msgr2=0x7ff6000ca680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.089+0000 7ff5ff400640 1 --2- 172.21.15.60:0/2276777177 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff6000c78f0 0x7ff6000ca680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.089+0000 7ff5ff400640 1 -- 172.21.15.60:0/2276777177 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff60006c7a0 con 0x7ff60006e740 2024-04-14T23:48:51.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.090+0000 7ff5ffe00640 1 --2- 172.21.15.60:0/2276777177 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff6000c78f0 0x7ff6000ca680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:48:51.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.090+0000 7ff5fea00640 1 --2- 172.21.15.60:0/2276777177 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff600070d10 0x7ff60006c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-14T23:48:51.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.090+0000 7fd9ef400640 1 --2- 172.21.15.60:0/645744492 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd9f00c8360 0x7fd9f015ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.090+0000 7fd9eea00640 1 -- 172.21.15.60:0/645744492 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd9f00c8360 msgr2=0x7fd9f015ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:51.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.090+0000 7fd9eea00640 1 --2- 172.21.15.60:0/645744492 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd9f00c8360 0x7fd9f015ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.090+0000 7fd9eea00640 1 -- 172.21.15.60:0/645744492 >> v1:172.21.15.60:6789/0 conn(0x7fd9f00c75c0 legacy=0x7fd9f0158e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.090+0000 7fd9eea00640 1 -- 172.21.15.60:0/645744492 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9f015fd80 con 0x7fd9f00c8740 2024-04-14T23:48:51.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.090+0000 7f633a200640 1 -- 172.21.15.60:0/3088564095 shutdown_connections 2024-04-14T23:48:51.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.089+0000 7f5366000640 1 -- 172.21.15.60:0/589539644 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5360072d00 con 0x7f53600c8360 2024-04-14T23:48:51.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.091+0000 7ff5ff400640 1 --2- 172.21.15.60:0/2276777177 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff60006e740 0x7ff6001ec7b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff60006f450 tx=0x7ff5f8004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:51.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.090+0000 7f09e7400640 1 --2- 172.21.15.60:0/3117765181 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f09cc06b9e0 0x7f09cc06dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.090+0000 7f09e7400640 1 -- 172.21.15.60:0/3117765181 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f09f0106a10 con 0x7f09e804b8e0 2024-04-14T23:48:51.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.090+0000 7f09eea00640 1 --2- 172.21.15.60:0/3117765181 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f09cc06b9e0 0x7f09cc06dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.090+0000 7f09e7400640 1 --2- 172.21.15.60:0/3117765181 >> [v2:172.21.15.60:6816/3475168698,v1:172.21.15.60:6819/3475168698] conn(0x7f09cc073c60 0x7f09cc076080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.091+0000 7f09e7400640 1 -- 172.21.15.60:0/3117765181 --> [v2:172.21.15.60:6816/3475168698,v1:172.21.15.60:6819/3475168698] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f09f009aec0 con 0x7f09cc073c60 2024-04-14T23:48:51.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.091+0000 7f09e7400640 1 -- 172.21.15.60:0/3117765181 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f09f010b2a0 con 0x7f09e804b8e0 2024-04-14T23:48:51.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.091+0000 7ff5f7400640 1 -- 172.21.15.60:0/2276777177 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5f8051070 con 0x7ff60006e740 2024-04-14T23:48:51.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.087+0000 7f34fae00640 1 -- start start 2024-04-14T23:48:51.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.090+0000 7f34fae00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f34f4154bc0 0x7f34f415b550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.090+0000 7f34fae00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34f415ba90 con 0x7f34f40cd110 2024-04-14T23:48:51.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.090+0000 7f34fae00640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34f4154fa0 con 0x7f34f40cc7d0 2024-04-14T23:48:51.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.090+0000 7f34fae00640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f34f406c270 con 0x7f34f4154bc0 2024-04-14T23:48:51.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.090+0000 7f34f9a00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f34f4154bc0 0x7f34f415b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.091+0000 7f34f9a00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f34f4154bc0 0x7f34f415b550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54066/0 (socket says 172.21.15.60:54066) 2024-04-14T23:48:51.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.091+0000 7f34f9a00640 1 -- 172.21.15.60:0/1251559829 learned_addr learned my addr 172.21.15.60:0/1251559829 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:51.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.091+0000 7f633a200640 1 -- 172.21.15.60:0/3088564095 wait complete. 2024-04-14T23:48:51.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.091+0000 7ff604e00640 1 -- 172.21.15.60:0/2276777177 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6000aa640 con 0x7ff60006e740 2024-04-14T23:48:51.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.091+0000 7ff604e00640 1 -- 172.21.15.60:0/2276777177 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6000aac00 con 0x7ff60006e740 2024-04-14T23:48:51.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.092+0000 7f34f9a00640 1 -- 172.21.15.60:0/1251559829 >> v1:172.21.15.186:6789/0 conn(0x7f34f40cc7d0 legacy=0x7f34f40b45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:51.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.092+0000 7f34f9a00640 1 -- 172.21.15.60:0/1251559829 >> v1:172.21.15.60:6789/0 conn(0x7f34f40cd110 legacy=0x7f34f4165450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:51.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.092+0000 7f34f9a00640 1 -- 172.21.15.60:0/1251559829 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34f40b4cb0 con 0x7f34f4154bc0 2024-04-14T23:48:51.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.092+0000 7f09ee000640 1 --2- 172.21.15.60:0/3117765181 >> [v2:172.21.15.60:6816/3475168698,v1:172.21.15.60:6819/3475168698] conn(0x7f09cc073c60 0x7f09cc076080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.092+0000 7ff5f7400640 1 -- 172.21.15.60:0/2276777177 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5f80112e0 con 0x7ff60006e740 2024-04-14T23:48:51.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.092+0000 7ff5f7400640 1 -- 172.21.15.60:0/2276777177 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5f8051070 con 0x7ff60006e740 2024-04-14T23:48:51.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.092+0000 7ff604e00640 1 -- 172.21.15.60:0/2276777177 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff5d8001030 con 0x7ff60006e740 2024-04-14T23:48:51.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.092+0000 7fd9eea00640 1 --2- 172.21.15.60:0/645744492 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9f00c8740 0x7fd9f006f310 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd9e0009870 tx=0x7fd9e002ba10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c96fcfbdf1a3a609 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:51.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.093+0000 7fd9ee000640 1 -- 172.21.15.60:0/645744492 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9e002f830 con 0x7fd9f00c8740 2024-04-14T23:48:51.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.093+0000 7fd9ee000640 1 -- 172.21.15.60:0/645744492 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9e002f9d0 con 0x7fd9f00c8740 2024-04-14T23:48:51.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.093+0000 7f09eea00640 1 --2- 172.21.15.60:0/3117765181 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f09cc06b9e0 0x7f09cc06dea0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f09e0036000 tx=0x7f09e003d040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:51.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.093+0000 7fd9ee000640 1 -- 172.21.15.60:0/645744492 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9e002fda0 con 0x7fd9f00c8740 2024-04-14T23:48:51.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.093+0000 7f5357400640 1 -- 172.21.15.60:0/589539644 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f534c005990 con 0x7f53600c8360 2024-04-14T23:48:51.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.094+0000 7fd9f6400640 1 -- 172.21.15.60:0/645744492 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9f00c8740 msgr2=0x7fd9f006f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.093+0000 7f633a200640 1 Processor -- start 2024-04-14T23:48:51.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.094+0000 7f633a200640 1 -- start start 2024-04-14T23:48:51.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.094+0000 7f34f9a00640 1 --2- 172.21.15.60:0/1251559829 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f34f4154bc0 0x7f34f415b550 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f34e8009870 tx=0x7f34e802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2115339de98b2942 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:51.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.094+0000 7fd9f6400640 1 --2- 172.21.15.60:0/645744492 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9f00c8740 0x7fd9f006f310 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd9e0009870 tx=0x7fd9e002ba10 comp rx=0 tx=0).stop 2024-04-14T23:48:51.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.093+0000 7f09ee000640 1 --2- 172.21.15.60:0/3117765181 >> [v2:172.21.15.60:6816/3475168698,v1:172.21.15.60:6819/3475168698] conn(0x7f09cc073c60 0x7f09cc076080 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:51.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.094+0000 7fd9f6400640 1 -- 172.21.15.60:0/645744492 shutdown_connections 2024-04-14T23:48:51.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.094+0000 7fd9f6400640 1 --2- 172.21.15.60:0/645744492 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9f00c8740 0x7fd9f006f310 secure :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd9e0009870 tx=0x7fd9e002ba10 comp rx=0 tx=0).stop 2024-04-14T23:48:51.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.094+0000 7fd9f6400640 1 --2- 172.21.15.60:0/645744492 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd9f00c8360 0x7fd9f015ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.094+0000 7fd9f6400640 1 -- 172.21.15.60:0/645744492 >> 172.21.15.60:0/645744492 conn(0x7fd9f00cc170 msgr2=0x7fd9f00cc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:51.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.095+0000 7ff5f7400640 1 -- 172.21.15.60:0/2276777177 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7ff5f8054070 con 0x7ff60006e740 2024-04-14T23:48:51.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.095+0000 7f09e7400640 1 -- 172.21.15.60:0/3117765181 <== osd.2 v2:172.21.15.60:6816/3475168698 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f09f009aec0 con 0x7f09cc073c60 2024-04-14T23:48:51.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.095+0000 7fd9f6400640 1 -- 172.21.15.60:0/645744492 shutdown_connections 2024-04-14T23:48:51.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.095+0000 7fd9f6400640 1 -- 172.21.15.60:0/645744492 wait complete. 2024-04-14T23:48:51.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.095+0000 7f633a200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63340cb750 0x7f633415aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.095+0000 7f633a200640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f63340cd160 0x7f63340c8740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.095+0000 7f633a200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63340cd540 0x7f63340c8c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.095+0000 7f633a200640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f633415ea70 con 0x7f63340cd160 2024-04-14T23:48:51.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.095+0000 7f633a200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f633415e790 con 0x7f63340cb750 2024-04-14T23:48:51.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.095+0000 7f633a200640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f63340ccb20 con 0x7f63340cd540 2024-04-14T23:48:51.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.096+0000 7f5357400640 1 --2- 172.21.15.60:0/589539644 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f534006ba60 0x7f534006df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.096+0000 7f5357400640 1 -- 172.21.15.60:0/589539644 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f534c08b120 con 0x7f53600c8360 2024-04-14T23:48:51.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.096+0000 7f535f400640 1 --2- 172.21.15.60:0/589539644 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f534006ba60 0x7f534006df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.096+0000 7f5357400640 1 --2- 172.21.15.60:0/589539644 >> [v2:172.21.15.60:6808/1730813674,v1:172.21.15.60:6809/1730813674] conn(0x7f5340073ce0 0x7f5340076100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.096+0000 7f5357400640 1 -- 172.21.15.60:0/589539644 --> [v2:172.21.15.60:6808/1730813674,v1:172.21.15.60:6809/1730813674] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f534c00fe50 con 0x7f5340073ce0 2024-04-14T23:48:51.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.096+0000 7f5357400640 1 -- 172.21.15.60:0/589539644 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f534c08b510 con 0x7f53600c8360 2024-04-14T23:48:51.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.096+0000 7f6333400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63340cb750 0x7f633415aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.096+0000 7f6333400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63340cb750 0x7f633415aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:43538/0 (socket says 172.21.15.60:43538) 2024-04-14T23:48:51.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.096+0000 7f6333400640 1 -- 172.21.15.60:0/1837537875 learned_addr learned my addr 172.21.15.60:0/1837537875 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:51.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.096+0000 7f6333e00640 1 --2- 172.21.15.60:0/1837537875 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63340cd540 0x7f63340c8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.096+0000 7f6332a00640 1 --2- 172.21.15.60:0/1837537875 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f63340cd160 0x7f63340c8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.096+0000 7f6333400640 1 -- 172.21.15.60:0/1837537875 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63340cd540 msgr2=0x7f63340c8c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.096+0000 7f6333400640 1 --2- 172.21.15.60:0/1837537875 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63340cd540 0x7f63340c8c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.096+0000 7f6333400640 1 -- 172.21.15.60:0/1837537875 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f63340cd160 msgr2=0x7f63340c8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.096+0000 7f6333400640 1 --2- 172.21.15.60:0/1837537875 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f63340cd160 0x7f63340c8740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.096+0000 7f6333400640 1 -- 172.21.15.60:0/1837537875 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63340d0790 con 0x7f63340cb750 2024-04-14T23:48:51.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.096+0000 7f6332a00640 1 --2- 172.21.15.60:0/1837537875 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f63340cd160 0x7f63340c8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-14T23:48:51.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.097+0000 7f6333e00640 1 --2- 172.21.15.60:0/1837537875 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63340cd540 0x7f63340c8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:48:51.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.097+0000 7f6333400640 1 --2- 172.21.15.60:0/1837537875 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63340cb750 0x7f633415aae0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f632c009eb0 tx=0x7f632c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:51.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.097+0000 7f632b400640 1 -- 172.21.15.60:0/1837537875 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f632c044020 con 0x7f63340cb750 2024-04-14T23:48:51.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.097+0000 7f632b400640 1 -- 172.21.15.60:0/1837537875 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f632c03a040 con 0x7f63340cb750 2024-04-14T23:48:51.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.097+0000 7f632b400640 1 -- 172.21.15.60:0/1837537875 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f632c039070 con 0x7f63340cb750 2024-04-14T23:48:51.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.097+0000 7f633a200640 1 -- 172.21.15.60:0/1837537875 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63340d0a30 con 0x7f63340cb750 2024-04-14T23:48:51.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.098+0000 7f633a200640 1 -- 172.21.15.60:0/1837537875 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63340c7ed0 con 0x7f63340cb750 2024-04-14T23:48:51.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.096+0000 7ff5f7400640 1 --2- 172.21.15.60:0/2276777177 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff5e006bb90 0x7ff5e006e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.096+0000 7ff5f7400640 1 -- 172.21.15.60:0/2276777177 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ff5f80b5df0 con 0x7ff60006e740 2024-04-14T23:48:51.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.097+0000 7ff5f7400640 1 --2- 172.21.15.60:0/2276777177 >> [v2:172.21.15.186:6801/1697098067,v1:172.21.15.186:6803/1697098067] conn(0x7ff5e0073d90 0x7ff5e00761b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.097+0000 7ff5f7400640 1 -- 172.21.15.60:0/2276777177 --> [v2:172.21.15.186:6801/1697098067,v1:172.21.15.186:6803/1697098067] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff5e0076860 con 0x7ff5e0073d90 2024-04-14T23:48:51.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.097+0000 7ff5f7400640 1 -- 172.21.15.60:0/2276777177 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7ff5f80b61a0 con 0x7ff60006e740 2024-04-14T23:48:51.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.097+0000 7ff5fea00640 1 --2- 172.21.15.60:0/2276777177 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff5e006bb90 0x7ff5e006e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.095+0000 7f34f3e00640 1 -- 172.21.15.60:0/1251559829 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34e8034070 con 0x7f34f4154bc0 2024-04-14T23:48:51.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.095+0000 7f34f3e00640 1 -- 172.21.15.60:0/1251559829 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34e8030860 con 0x7f34f4154bc0 2024-04-14T23:48:51.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.095+0000 7f34f3e00640 1 -- 172.21.15.60:0/1251559829 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34e8030c30 con 0x7f34f4154bc0 2024-04-14T23:48:51.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.097+0000 7f34fae00640 1 -- 172.21.15.60:0/1251559829 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f34f4154bc0 msgr2=0x7f34f415b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.097+0000 7f34fae00640 1 --2- 172.21.15.60:0/1251559829 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f34f4154bc0 0x7f34f415b550 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f34e8009870 tx=0x7f34e802b9a0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.098+0000 7ff5fea00640 1 --2- 172.21.15.60:0/2276777177 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff5e006bb90 0x7ff5e006e050 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff60006cbf0 tx=0x7ff5ec009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:51.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.097+0000 7f535fe00640 1 --2- 172.21.15.60:0/589539644 >> [v2:172.21.15.60:6808/1730813674,v1:172.21.15.60:6809/1730813674] conn(0x7f5340073ce0 0x7f5340076100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.098+0000 7f535f400640 1 --2- 172.21.15.60:0/589539644 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f534006ba60 0x7f534006df20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5350009a60 tx=0x7f5350009460 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:51.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.098+0000 7fd9f6400640 1 Processor -- start 2024-04-14T23:48:51.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.099+0000 7fd9f6400640 1 -- start start 2024-04-14T23:48:51.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.099+0000 7f34fae00640 1 -- 172.21.15.60:0/1251559829 shutdown_connections 2024-04-14T23:48:51.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.099+0000 7f34fae00640 1 --2- 172.21.15.60:0/1251559829 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f34f4154bc0 0x7f34f415b550 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.099+0000 7f34fae00640 1 -- 172.21.15.60:0/1251559829 >> 172.21.15.60:0/1251559829 conn(0x7f34f40670f0 msgr2=0x7f34f4067500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:51.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.099+0000 7ff5ffe00640 1 --2- 172.21.15.60:0/2276777177 >> [v2:172.21.15.186:6801/1697098067,v1:172.21.15.186:6803/1697098067] conn(0x7ff5e0073d90 0x7ff5e00761b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.099+0000 7f34fae00640 1 -- 172.21.15.60:0/1251559829 shutdown_connections 2024-04-14T23:48:51.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.100+0000 7f535fe00640 1 --2- 172.21.15.60:0/589539644 >> [v2:172.21.15.60:6808/1730813674,v1:172.21.15.60:6809/1730813674] conn(0x7f5340073ce0 0x7f5340076100 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:51.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.099+0000 7f6328c00640 1 -- 172.21.15.60:0/1837537875 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6334153e70 con 0x7f63340cb750 2024-04-14T23:48:51.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.100+0000 7f632b400640 1 -- 172.21.15.60:0/1837537875 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f632c047020 con 0x7f63340cb750 2024-04-14T23:48:51.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.101+0000 7fd9f6400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd9f00c75c0 0x7fd9f014e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.101+0000 7fd9f6400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9f00c8360 0x7fd9f01ea8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.101+0000 7fd9f6400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9f0071440 0x7fd9f01e9af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.099+0000 7ff5ffe00640 1 --2- 172.21.15.60:0/2276777177 >> [v2:172.21.15.186:6801/1697098067,v1:172.21.15.186:6803/1697098067] conn(0x7ff5e0073d90 0x7ff5e00761b0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:51.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.101+0000 7ff5f7400640 1 -- 172.21.15.60:0/2276777177 <== osd.5 v2:172.21.15.186:6801/1697098067 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff5e0076860 con 0x7ff5e0073d90 2024-04-14T23:48:51.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.101+0000 7f632b400640 1 --2- 172.21.15.60:0/1837537875 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f631006ba60 0x7f631006df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.101+0000 7fd9f6400640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fd9f015f4c0 con 0x7fd9f00c8360 2024-04-14T23:48:51.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.099+0000 7f34fae00640 1 -- 172.21.15.60:0/1251559829 wait complete. 2024-04-14T23:48:51.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.101+0000 7f632b400640 1 -- 172.21.15.60:0/1837537875 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f632c0ad750 con 0x7f63340cb750 2024-04-14T23:48:51.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.101+0000 7fd9ef400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd9f00c75c0 0x7fd9f014e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.101+0000 7fd9f6400640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd9f00c70a0 con 0x7fd9f0071440 2024-04-14T23:48:51.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.102+0000 7fd9f6400640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fd9f0071030 con 0x7fd9f00c75c0 2024-04-14T23:48:51.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.102+0000 7f5357400640 1 -- 172.21.15.60:0/589539644 <== osd.3 v2:172.21.15.60:6808/1730813674 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f534c00fe50 con 0x7f5340073ce0 2024-04-14T23:48:51.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.102+0000 7fd9efe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9f0071440 0x7fd9f01e9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.102+0000 7f6332a00640 1 --2- 172.21.15.60:0/1837537875 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f631006ba60 0x7f631006df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.102+0000 7fd9ef400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd9f00c75c0 0x7fd9f014e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.60:48614/0 (socket says 172.21.15.60:48614) 2024-04-14T23:48:51.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.102+0000 7fd9ef400640 1 -- 172.21.15.60:0/810191551 learned_addr learned my addr 172.21.15.60:0/810191551 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:51.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.102+0000 7f632b400640 1 --2- 172.21.15.60:0/1837537875 >> [v2:172.21.15.186:6800/1464587038,v1:172.21.15.186:6805/1464587038] conn(0x7f6310073c80 0x7f63100760a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.102+0000 7f632b400640 1 -- 172.21.15.60:0/1837537875 --> [v2:172.21.15.186:6800/1464587038,v1:172.21.15.186:6805/1464587038] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f632c04ae50 con 0x7f6310073c80 2024-04-14T23:48:51.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.102+0000 7f632b400640 1 -- 172.21.15.60:0/1837537875 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f632c0adb40 con 0x7f63340cb750 2024-04-14T23:48:51.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.102+0000 7fd9ef400640 1 -- 172.21.15.60:0/810191551 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9f0071440 msgr2=0x7fd9f01e9af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.102+0000 7f6333e00640 1 --2- 172.21.15.60:0/1837537875 >> [v2:172.21.15.186:6800/1464587038,v1:172.21.15.186:6805/1464587038] conn(0x7f6310073c80 0x7f63100760a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.102+0000 7f6332a00640 1 --2- 172.21.15.60:0/1837537875 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f631006ba60 0x7f631006df20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f63341e51a0 tx=0x7f631c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:51.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.103+0000 7f34fae00640 1 Processor -- start 2024-04-14T23:48:51.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.103+0000 7fd9ef400640 1 --2- 172.21.15.60:0/810191551 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9f0071440 0x7fd9f01e9af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.103+0000 7fd9ef400640 1 -- 172.21.15.60:0/810191551 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9f00c8360 msgr2=0x7fd9f01ea8a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-14T23:48:51.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.103+0000 7f34fae00640 1 -- start start 2024-04-14T23:48:51.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.103+0000 7f6333e00640 1 --2- 172.21.15.60:0/1837537875 >> [v2:172.21.15.186:6800/1464587038,v1:172.21.15.186:6805/1464587038] conn(0x7f6310073c80 0x7f63100760a0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:51.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.103+0000 7fd9ef400640 1 --2- 172.21.15.60:0/810191551 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9f00c8360 0x7fd9f01ea8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.103+0000 7fd9ef400640 1 -- 172.21.15.60:0/810191551 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9f014eba0 con 0x7fd9f00c75c0 2024-04-14T23:48:51.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.104+0000 7f632b400640 1 -- 172.21.15.60:0/1837537875 <== osd.4 v2:172.21.15.186:6800/1464587038 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f632c04ae50 con 0x7f6310073c80 2024-04-14T23:48:51.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.105+0000 7f34fae00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f34f40cc7d0 0x7f34f4155970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.105+0000 7f34fae00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f34f40cd110 0x7f34f41562b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.105+0000 7f34f9a00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f34f40cc7d0 0x7f34f4155970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.105+0000 7f34f9a00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f34f40cc7d0 0x7f34f4155970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54100/0 (socket says 172.21.15.60:54100) 2024-04-14T23:48:51.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.105+0000 7fd9ef400640 1 --2- 172.21.15.60:0/810191551 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd9f00c75c0 0x7fd9f014e660 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc00d800 tx=0x7fd9dc00dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:51.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.106+0000 7f34f9a00640 1 -- 172.21.15.60:0/3492673479 learned_addr learned my addr 172.21.15.60:0/3492673479 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:51.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.106+0000 7f34fae00640 1 --2- 172.21.15.60:0/3492673479 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f34f4154bc0 0x7f34f41567f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.106+0000 7fd9e7400640 1 -- 172.21.15.60:0/810191551 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9dc00b660 con 0x7fd9f00c75c0 2024-04-14T23:48:51.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.106+0000 7fd9e7400640 1 -- 172.21.15.60:0/810191551 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9dc019070 con 0x7fd9f00c75c0 2024-04-14T23:48:51.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.106+0000 7fd9e7400640 1 -- 172.21.15.60:0/810191551 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9dc00ba20 con 0x7fd9f00c75c0 2024-04-14T23:48:51.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.106+0000 7f34f9000640 1 --2- 172.21.15.60:0/3492673479 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f34f40cd110 0x7f34f41562b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.106+0000 7f34fa400640 1 --2- 172.21.15.60:0/3492673479 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f34f4154bc0 0x7f34f41567f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.106+0000 7f34fae00640 1 -- 172.21.15.60:0/3492673479 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f34f406bc20 con 0x7f34f40cc7d0 2024-04-14T23:48:51.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.106+0000 7f34fae00640 1 -- 172.21.15.60:0/3492673479 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f34f406b940 con 0x7f34f40cd110 2024-04-14T23:48:51.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.106+0000 7fd9f6400640 1 -- 172.21.15.60:0/810191551 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9f014fd10 con 0x7fd9f00c75c0 2024-04-14T23:48:51.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.106+0000 7fd9f6400640 1 -- 172.21.15.60:0/810191551 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9f014ffb0 con 0x7fd9f00c75c0 2024-04-14T23:48:51.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.106+0000 7f34fae00640 1 -- 172.21.15.60:0/3492673479 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f34f406bf00 con 0x7f34f4154bc0 2024-04-14T23:48:51.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.107+0000 7f34f9000640 1 -- 172.21.15.60:0/3492673479 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f34f4154bc0 msgr2=0x7f34f41567f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.107+0000 7f34f9000640 1 --2- 172.21.15.60:0/3492673479 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f34f4154bc0 0x7f34f41567f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.107+0000 7f34f9000640 1 -- 172.21.15.60:0/3492673479 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f34f40cc7d0 msgr2=0x7f34f4155970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.107+0000 7f34f9000640 1 --2- 172.21.15.60:0/3492673479 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f34f40cc7d0 0x7f34f4155970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.107+0000 7f34f9000640 1 -- 172.21.15.60:0/3492673479 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34f41f1690 con 0x7f34f40cd110 2024-04-14T23:48:51.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.109+0000 7fd9f6400640 1 -- 172.21.15.60:0/810191551 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd9f0076970 con 0x7fd9f00c75c0 2024-04-14T23:48:51.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.110+0000 7fd9e7400640 1 -- 172.21.15.60:0/810191551 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7fd9dc00bbc0 con 0x7fd9f00c75c0 2024-04-14T23:48:51.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.110+0000 7f34f9000640 1 --2- 172.21.15.60:0/3492673479 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f34f40cd110 0x7f34f41562b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f34e000c410 tx=0x7f34e000c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:51.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.110+0000 7f34f1600640 1 -- 172.21.15.60:0/3492673479 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34e0017070 con 0x7f34f40cd110 2024-04-14T23:48:51.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.110+0000 7f34f1600640 1 -- 172.21.15.60:0/3492673479 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34e0005230 con 0x7f34f40cd110 2024-04-14T23:48:51.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.110+0000 7f34f1600640 1 -- 172.21.15.60:0/3492673479 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34e00045e0 con 0x7f34f40cd110 2024-04-14T23:48:51.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.111+0000 7f34fae00640 1 -- 172.21.15.60:0/3492673479 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34f41e6fe0 con 0x7f34f40cd110 2024-04-14T23:48:51.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.111+0000 7f34fae00640 1 -- 172.21.15.60:0/3492673479 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34f41538c0 con 0x7f34f40cd110 2024-04-14T23:48:51.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.111+0000 7fd9e7400640 1 --2- 172.21.15.60:0/810191551 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fd9d006ba60 0x7fd9d006df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.112+0000 7fd9e7400640 1 -- 172.21.15.60:0/810191551 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fd9dc08abf0 con 0x7fd9f00c75c0 2024-04-14T23:48:51.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.112+0000 7fd9e7400640 1 --2- 172.21.15.60:0/810191551 >> [v2:172.21.15.186:6824/842537875,v1:172.21.15.186:6825/842537875] conn(0x7fd9d0073c80 0x7fd9d00760a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.112+0000 7f34fae00640 1 -- 172.21.15.60:0/3492673479 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f34f404f540 con 0x7f34f40cd110 2024-04-14T23:48:51.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.112+0000 7fd9e7400640 1 -- 172.21.15.60:0/810191551 --> [v2:172.21.15.186:6824/842537875,v1:172.21.15.186:6825/842537875] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd9dc001d30 con 0x7fd9d0073c80 2024-04-14T23:48:51.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.112+0000 7fd9e7400640 1 -- 172.21.15.60:0/810191551 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fd9dc08f2a0 con 0x7fd9f00c75c0 2024-04-14T23:48:51.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.113+0000 7fd9efe00640 1 --2- 172.21.15.60:0/810191551 >> [v2:172.21.15.186:6824/842537875,v1:172.21.15.186:6825/842537875] conn(0x7fd9d0073c80 0x7fd9d00760a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.114+0000 7fd9eea00640 1 --2- 172.21.15.60:0/810191551 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fd9d006ba60 0x7fd9d006df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.114+0000 7fd9efe00640 1 --2- 172.21.15.60:0/810191551 >> [v2:172.21.15.186:6824/842537875,v1:172.21.15.186:6825/842537875] conn(0x7fd9d0073c80 0x7fd9d00760a0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:51.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.115+0000 7fd9eea00640 1 --2- 172.21.15.60:0/810191551 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fd9d006ba60 0x7fd9d006df20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd9e0036000 tx=0x7fd9e0030000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:51.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.115+0000 7f34f1600640 1 -- 172.21.15.60:0/3492673479 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f34e001a070 con 0x7f34f40cd110 2024-04-14T23:48:51.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.115+0000 7fd9e7400640 1 -- 172.21.15.60:0/810191551 <== osd.7 v2:172.21.15.186:6824/842537875 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd9dc001d30 con 0x7fd9d0073c80 2024-04-14T23:48:51.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.116+0000 7f34f1600640 1 --2- 172.21.15.60:0/3492673479 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f34d406b990 0x7f34d406de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.116+0000 7f34f1600640 1 -- 172.21.15.60:0/3492673479 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f34e0089b30 con 0x7f34f40cd110 2024-04-14T23:48:51.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.117+0000 7f34f9a00640 1 --2- 172.21.15.60:0/3492673479 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f34d406b990 0x7f34d406de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.117+0000 7f34f1600640 1 --2- 172.21.15.60:0/3492673479 >> [v2:172.21.15.186:6802/1837108559,v1:172.21.15.186:6804/1837108559] conn(0x7f34d4073bb0 0x7f34d4075fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.117+0000 7f34f1600640 1 -- 172.21.15.60:0/3492673479 --> [v2:172.21.15.186:6802/1837108559,v1:172.21.15.186:6804/1837108559] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f34d4076680 con 0x7f34d4073bb0 2024-04-14T23:48:51.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.117+0000 7f34f1600640 1 -- 172.21.15.60:0/3492673479 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f34e008e2a0 con 0x7f34f40cd110 2024-04-14T23:48:51.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.117+0000 7f34fa400640 1 --2- 172.21.15.60:0/3492673479 >> [v2:172.21.15.186:6802/1837108559,v1:172.21.15.186:6804/1837108559] conn(0x7f34d4073bb0 0x7f34d4075fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.118+0000 7f34fa400640 1 --2- 172.21.15.60:0/3492673479 >> [v2:172.21.15.186:6802/1837108559,v1:172.21.15.186:6804/1837108559] conn(0x7f34d4073bb0 0x7f34d4075fd0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:51.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.118+0000 7f34f9a00640 1 --2- 172.21.15.60:0/3492673479 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f34d406b990 0x7f34d406de50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f34e80090b0 tx=0x7f34e803b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:51.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.119+0000 7f34f1600640 1 -- 172.21.15.60:0/3492673479 <== osd.6 v2:172.21.15.186:6802/1837108559 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f34d4076680 con 0x7f34d4073bb0 2024-04-14T23:48:51.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.120+0000 7f8918c00640 1 -- 172.21.15.60:0/2906597355 --> [v2:172.21.15.60:6817/2536528895,v1:172.21.15.60:6818/2536528895] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f89140b3c10 con 0x7f88f4073b80 2024-04-14T23:48:51.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.122+0000 7f09efe00640 1 -- 172.21.15.60:0/3117765181 --> [v2:172.21.15.60:6816/3475168698,v1:172.21.15.60:6819/3475168698] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f09c8002dd0 con 0x7f09cc073c60 2024-04-14T23:48:51.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.124+0000 7f09e7400640 1 -- 172.21.15.60:0/3117765181 <== osd.2 v2:172.21.15.60:6816/3475168698 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f09c8002dd0 con 0x7f09cc073c60 2024-04-14T23:48:51.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.124+0000 7f890a000640 1 -- 172.21.15.60:0/2906597355 <== osd.0 v2:172.21.15.60:6817/2536528895 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f89140b3c10 con 0x7f88f4073b80 2024-04-14T23:48:51.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.125+0000 7f8918c00640 1 -- 172.21.15.60:0/2906597355 >> [v2:172.21.15.60:6817/2536528895,v1:172.21.15.60:6818/2536528895] conn(0x7f88f4073b80 msgr2=0x7f88f4075fa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.125+0000 7f8918c00640 1 --2- 172.21.15.60:0/2906597355 >> [v2:172.21.15.60:6817/2536528895,v1:172.21.15.60:6818/2536528895] conn(0x7f88f4073b80 0x7f88f4075fa0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.125+0000 7f8918c00640 1 -- 172.21.15.60:0/2906597355 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f88f406b9e0 msgr2=0x7f88f406dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.125+0000 7f8918c00640 1 --2- 172.21.15.60:0/2906597355 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f88f406b9e0 0x7f88f406dea0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8904009560 tx=0x7f8904008030 comp rx=0 tx=0).stop 2024-04-14T23:48:51.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.125+0000 7f8918c00640 1 -- 172.21.15.60:0/2906597355 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89140cb450 msgr2=0x7f89141ec7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.125+0000 7f8918c00640 1 --2- 172.21.15.60:0/2906597355 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89140cb450 0x7f89141ec7b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f890000c470 tx=0x7f890000c930 comp rx=0 tx=0).stop 2024-04-14T23:48:51.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.125+0000 7f8912800640 1 -- 172.21.15.60:0/2906597355 reap_dead start 2024-04-14T23:48:51.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.125+0000 7f09efe00640 1 -- 172.21.15.60:0/3117765181 >> [v2:172.21.15.60:6816/3475168698,v1:172.21.15.60:6819/3475168698] conn(0x7f09cc073c60 msgr2=0x7f09cc076080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.125+0000 7f09efe00640 1 --2- 172.21.15.60:0/3117765181 >> [v2:172.21.15.60:6816/3475168698,v1:172.21.15.60:6819/3475168698] conn(0x7f09cc073c60 0x7f09cc076080 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.126+0000 7f09efe00640 1 -- 172.21.15.60:0/3117765181 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f09cc06b9e0 msgr2=0x7f09cc06dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.126+0000 7f09efe00640 1 --2- 172.21.15.60:0/3117765181 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f09cc06b9e0 0x7f09cc06dea0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f09e0036000 tx=0x7f09e003d040 comp rx=0 tx=0).stop 2024-04-14T23:48:51.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.126+0000 7f09efe00640 1 -- 172.21.15.60:0/3117765181 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09e804b8e0 msgr2=0x7f09e817cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.126+0000 7f09efe00640 1 --2- 172.21.15.60:0/3117765181 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09e804b8e0 0x7f09e817cf80 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f09f0064cd0 tx=0x7f09f0068880 comp rx=0 tx=0).stop 2024-04-14T23:48:51.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.127+0000 7f09ef400640 1 -- 172.21.15.60:0/3117765181 reap_dead start 2024-04-14T23:48:51.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.128+0000 7f09efe00640 1 -- 172.21.15.60:0/3117765181 shutdown_connections 2024-04-14T23:48:51.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.128+0000 7f09efe00640 1 -- 172.21.15.60:0/3117765181 >> 172.21.15.60:0/3117765181 conn(0x7f09e8050180 msgr2=0x7f09e80055a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:51.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.128+0000 7f09efe00640 1 -- 172.21.15.60:0/3117765181 shutdown_connections 2024-04-14T23:48:51.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.128+0000 7f09efe00640 1 -- 172.21.15.60:0/3117765181 wait complete. 2024-04-14T23:48:51.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.130+0000 7f8918c00640 1 -- 172.21.15.60:0/2906597355 shutdown_connections 2024-04-14T23:48:51.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.130+0000 7f8918c00640 1 -- 172.21.15.60:0/2906597355 >> 172.21.15.60:0/2906597355 conn(0x7f8914070d20 msgr2=0x7f891414f020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:51.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.130+0000 7f8918c00640 1 -- 172.21.15.60:0/2906597355 shutdown_connections 2024-04-14T23:48:51.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.130+0000 7f8918c00640 1 -- 172.21.15.60:0/2906597355 wait complete. 2024-04-14T23:48:51.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.135+0000 7f870ac00640 1 -- 172.21.15.60:0/1888925289 --> [v2:172.21.15.60:6800/3115695676,v1:172.21.15.60:6801/3115695676] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f86dc002dd0 con 0x7f86e4073c90 2024-04-14T23:48:51.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.135+0000 7f5366000640 1 -- 172.21.15.60:0/589539644 --> [v2:172.21.15.60:6808/1730813674,v1:172.21.15.60:6809/1730813674] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5360076970 con 0x7f5340073ce0 2024-04-14T23:48:51.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.136+0000 7f86fbe00640 1 -- 172.21.15.60:0/1888925289 <== osd.1 v2:172.21.15.60:6800/3115695676 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f86dc002dd0 con 0x7f86e4073c90 2024-04-14T23:48:51.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.136+0000 7f5357400640 1 -- 172.21.15.60:0/589539644 <== osd.3 v2:172.21.15.60:6808/1730813674 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f5360076970 con 0x7f5340073ce0 2024-04-14T23:48:51.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.136+0000 7f870ac00640 1 -- 172.21.15.60:0/1888925289 >> [v2:172.21.15.60:6800/3115695676,v1:172.21.15.60:6801/3115695676] conn(0x7f86e4073c90 msgr2=0x7f86e40760b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.136+0000 7f870ac00640 1 --2- 172.21.15.60:0/1888925289 >> [v2:172.21.15.60:6800/3115695676,v1:172.21.15.60:6801/3115695676] conn(0x7f86e4073c90 0x7f86e40760b0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.137+0000 7f870ac00640 1 -- 172.21.15.60:0/1888925289 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f86e406ba10 msgr2=0x7f86e406ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.137+0000 7f870ac00640 1 --2- 172.21.15.60:0/1888925289 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f86e406ba10 0x7f86e406ded0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f87040b5e60 tx=0x7f86f40090e0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.137+0000 7f870ac00640 1 -- 172.21.15.60:0/1888925289 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86e80039b0 msgr2=0x7f87041f11f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.137+0000 7f870ac00640 1 --2- 172.21.15.60:0/1888925289 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86e80039b0 0x7f87041f11f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f86f0004000 tx=0x7f86f0001120 comp rx=0 tx=0).stop 2024-04-14T23:48:51.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.137+0000 7f8708c00640 1 -- 172.21.15.60:0/1888925289 reap_dead start 2024-04-14T23:48:51.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.137+0000 7f870ac00640 1 -- 172.21.15.60:0/1888925289 shutdown_connections 2024-04-14T23:48:51.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.137+0000 7f870ac00640 1 -- 172.21.15.60:0/1888925289 >> 172.21.15.60:0/1888925289 conn(0x7f8704066f10 msgr2=0x7f8704153360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:51.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.137+0000 7f870ac00640 1 -- 172.21.15.60:0/1888925289 shutdown_connections 2024-04-14T23:48:51.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.138+0000 7f870ac00640 1 -- 172.21.15.60:0/1888925289 wait complete. 2024-04-14T23:48:51.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.138+0000 7f6328c00640 1 -- 172.21.15.60:0/1837537875 --> [v2:172.21.15.186:6800/1464587038,v1:172.21.15.186:6805/1464587038] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6334068bc0 con 0x7f6310073c80 2024-04-14T23:48:51.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.138+0000 7f5366000640 1 -- 172.21.15.60:0/589539644 >> [v2:172.21.15.60:6808/1730813674,v1:172.21.15.60:6809/1730813674] conn(0x7f5340073ce0 msgr2=0x7f5340076100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.138+0000 7f5366000640 1 --2- 172.21.15.60:0/589539644 >> [v2:172.21.15.60:6808/1730813674,v1:172.21.15.60:6809/1730813674] conn(0x7f5340073ce0 0x7f5340076100 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.138+0000 7f5366000640 1 -- 172.21.15.60:0/589539644 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f534006ba60 msgr2=0x7f534006df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.139+0000 7f632b400640 1 -- 172.21.15.60:0/1837537875 <== osd.4 v2:172.21.15.186:6800/1464587038 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f6334068bc0 con 0x7f6310073c80 2024-04-14T23:48:51.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.138+0000 7f5366000640 1 --2- 172.21.15.60:0/589539644 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f534006ba60 0x7f534006df20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5350009a60 tx=0x7f5350009460 comp rx=0 tx=0).stop 2024-04-14T23:48:51.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.138+0000 7f5366000640 1 -- 172.21.15.60:0/589539644 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f53600c8360 msgr2=0x7f53600cb360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.138+0000 7f5366000640 1 --2- 172.21.15.60:0/589539644 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f53600c8360 0x7f53600cb360 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f534c00d800 tx=0x7f534c00dcc0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.139+0000 7f535fe00640 1 -- 172.21.15.60:0/589539644 reap_dead start 2024-04-14T23:48:51.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.139+0000 7f5366000640 1 -- 172.21.15.60:0/589539644 shutdown_connections 2024-04-14T23:48:51.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.139+0000 7ff604e00640 1 -- 172.21.15.60:0/2276777177 --> [v2:172.21.15.186:6801/1697098067,v1:172.21.15.186:6803/1697098067] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff5d8002dd0 con 0x7ff5e0073d90 2024-04-14T23:48:51.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.139+0000 7f5366000640 1 -- 172.21.15.60:0/589539644 >> 172.21.15.60:0/589539644 conn(0x7f53600cc170 msgr2=0x7f5360071c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:51.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.139+0000 7f5366000640 1 -- 172.21.15.60:0/589539644 shutdown_connections 2024-04-14T23:48:51.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.140+0000 7ff5f7400640 1 -- 172.21.15.60:0/2276777177 <== osd.5 v2:172.21.15.186:6801/1697098067 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7ff5d8002dd0 con 0x7ff5e0073d90 2024-04-14T23:48:51.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.140+0000 7f5366000640 1 -- 172.21.15.60:0/589539644 wait complete. 2024-04-14T23:48:51.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.140+0000 7ff604e00640 1 -- 172.21.15.60:0/2276777177 >> [v2:172.21.15.186:6801/1697098067,v1:172.21.15.186:6803/1697098067] conn(0x7ff5e0073d90 msgr2=0x7ff5e00761b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.140+0000 7ff604e00640 1 --2- 172.21.15.60:0/2276777177 >> [v2:172.21.15.186:6801/1697098067,v1:172.21.15.186:6803/1697098067] conn(0x7ff5e0073d90 0x7ff5e00761b0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.140+0000 7ff604e00640 1 -- 172.21.15.60:0/2276777177 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff5e006bb90 msgr2=0x7ff5e006e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.140+0000 7ff604e00640 1 --2- 172.21.15.60:0/2276777177 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff5e006bb90 0x7ff5e006e050 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff60006cbf0 tx=0x7ff5ec009040 comp rx=0 tx=0).stop 2024-04-14T23:48:51.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.141+0000 7ff604e00640 1 -- 172.21.15.60:0/2276777177 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff60006e740 msgr2=0x7ff6001ec7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.141+0000 7ff604e00640 1 --2- 172.21.15.60:0/2276777177 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff60006e740 0x7ff6001ec7b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff60006f450 tx=0x7ff5f8004a40 comp rx=0 tx=0).stop 2024-04-14T23:48:51.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.141+0000 7ff5ffe00640 1 -- 172.21.15.60:0/2276777177 reap_dead start 2024-04-14T23:48:51.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.141+0000 7ff604e00640 1 -- 172.21.15.60:0/2276777177 shutdown_connections 2024-04-14T23:48:51.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.141+0000 7ff604e00640 1 -- 172.21.15.60:0/2276777177 >> 172.21.15.60:0/2276777177 conn(0x7ff6000cc2f0 msgr2=0x7ff6000cb670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:51.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.142+0000 7ff604e00640 1 -- 172.21.15.60:0/2276777177 shutdown_connections 2024-04-14T23:48:51.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.142+0000 7ff604e00640 1 -- 172.21.15.60:0/2276777177 wait complete. 2024-04-14T23:48:51.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.144+0000 7f6328c00640 1 -- 172.21.15.60:0/1837537875 >> [v2:172.21.15.186:6800/1464587038,v1:172.21.15.186:6805/1464587038] conn(0x7f6310073c80 msgr2=0x7f63100760a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.144+0000 7f6328c00640 1 --2- 172.21.15.60:0/1837537875 >> [v2:172.21.15.186:6800/1464587038,v1:172.21.15.186:6805/1464587038] conn(0x7f6310073c80 0x7f63100760a0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.145+0000 7f6328c00640 1 -- 172.21.15.60:0/1837537875 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f631006ba60 msgr2=0x7f631006df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.145+0000 7f6328c00640 1 --2- 172.21.15.60:0/1837537875 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f631006ba60 0x7f631006df20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f63341e51a0 tx=0x7f631c009040 comp rx=0 tx=0).stop 2024-04-14T23:48:51.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.145+0000 7f6328c00640 1 -- 172.21.15.60:0/1837537875 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63340cb750 msgr2=0x7f633415aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.145+0000 7f6328c00640 1 --2- 172.21.15.60:0/1837537875 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63340cb750 0x7f633415aae0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f632c009eb0 tx=0x7f632c004a40 comp rx=0 tx=0).stop 2024-04-14T23:48:51.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.145+0000 7f6333e00640 1 -- 172.21.15.60:0/1837537875 reap_dead start 2024-04-14T23:48:51.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.145+0000 7f6328c00640 1 -- 172.21.15.60:0/1837537875 shutdown_connections 2024-04-14T23:48:51.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.145+0000 7f6328c00640 1 -- 172.21.15.60:0/1837537875 >> 172.21.15.60:0/1837537875 conn(0x7f6334066f10 msgr2=0x7f633415d520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:51.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.145+0000 7f6328c00640 1 -- 172.21.15.60:0/1837537875 shutdown_connections 2024-04-14T23:48:51.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.145+0000 7f6328c00640 1 -- 172.21.15.60:0/1837537875 wait complete. 2024-04-14T23:48:51.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.146+0000 7fd9f6400640 1 -- 172.21.15.60:0/810191551 --> [v2:172.21.15.186:6824/842537875,v1:172.21.15.186:6825/842537875] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd9f01ea680 con 0x7fd9d0073c80 2024-04-14T23:48:51.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.147+0000 7fd9e7400640 1 -- 172.21.15.60:0/810191551 <== osd.7 v2:172.21.15.186:6824/842537875 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fd9f01ea680 con 0x7fd9d0073c80 2024-04-14T23:48:51.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.148+0000 7fd9f6400640 1 -- 172.21.15.60:0/810191551 >> [v2:172.21.15.186:6824/842537875,v1:172.21.15.186:6825/842537875] conn(0x7fd9d0073c80 msgr2=0x7fd9d00760a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.148+0000 7fd9f6400640 1 --2- 172.21.15.60:0/810191551 >> [v2:172.21.15.186:6824/842537875,v1:172.21.15.186:6825/842537875] conn(0x7fd9d0073c80 0x7fd9d00760a0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.151+0000 7fd9f6400640 1 -- 172.21.15.60:0/810191551 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fd9d006ba60 msgr2=0x7fd9d006df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.151+0000 7fd9f6400640 1 --2- 172.21.15.60:0/810191551 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fd9d006ba60 0x7fd9d006df20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd9e0036000 tx=0x7fd9e0030000 comp rx=0 tx=0).stop 2024-04-14T23:48:51.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.151+0000 7fd9f6400640 1 -- 172.21.15.60:0/810191551 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd9f00c75c0 msgr2=0x7fd9f014e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.151+0000 7fd9f6400640 1 --2- 172.21.15.60:0/810191551 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd9f00c75c0 0x7fd9f014e660 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc00d800 tx=0x7fd9dc00dcc0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.151+0000 7fd9efe00640 1 -- 172.21.15.60:0/810191551 reap_dead start 2024-04-14T23:48:51.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.151+0000 7fd9f6400640 1 -- 172.21.15.60:0/810191551 shutdown_connections 2024-04-14T23:48:51.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.151+0000 7fd9f6400640 1 -- 172.21.15.60:0/810191551 >> 172.21.15.60:0/810191551 conn(0x7fd9f00cc170 msgr2=0x7fd9f0071cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:51.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.153+0000 7fd9f6400640 1 -- 172.21.15.60:0/810191551 shutdown_connections 2024-04-14T23:48:51.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.153+0000 7fd9f6400640 1 -- 172.21.15.60:0/810191551 wait complete. 2024-04-14T23:48:51.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.153+0000 7f34fae00640 1 -- 172.21.15.60:0/3492673479 --> [v2:172.21.15.186:6802/1837108559,v1:172.21.15.186:6804/1837108559] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f34f41521c0 con 0x7f34d4073bb0 2024-04-14T23:48:51.191 INFO:teuthology.orchestra.run.smithi060.stdout:55834574854 2024-04-14T23:48:51.192 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-14T23:48:51.193 INFO:teuthology.orchestra.run.smithi060.stdout:55834574853 2024-04-14T23:48:51.193 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-14T23:48:51.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.154+0000 7f34f1600640 1 -- 172.21.15.60:0/3492673479 <== osd.6 v2:172.21.15.186:6802/1837108559 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f34f41521c0 con 0x7f34d4073bb0 2024-04-14T23:48:51.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.155+0000 7f34fae00640 1 -- 172.21.15.60:0/3492673479 >> [v2:172.21.15.186:6802/1837108559,v1:172.21.15.186:6804/1837108559] conn(0x7f34d4073bb0 msgr2=0x7f34d4075fd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.155+0000 7f34fae00640 1 --2- 172.21.15.60:0/3492673479 >> [v2:172.21.15.186:6802/1837108559,v1:172.21.15.186:6804/1837108559] conn(0x7f34d4073bb0 0x7f34d4075fd0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.155+0000 7f34fae00640 1 -- 172.21.15.60:0/3492673479 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f34d406b990 msgr2=0x7f34d406de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.155+0000 7f34fae00640 1 --2- 172.21.15.60:0/3492673479 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f34d406b990 0x7f34d406de50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f34e80090b0 tx=0x7f34e803b040 comp rx=0 tx=0).stop 2024-04-14T23:48:51.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.156+0000 7f34fae00640 1 -- 172.21.15.60:0/3492673479 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f34f40cd110 msgr2=0x7f34f41562b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.156+0000 7f34fae00640 1 --2- 172.21.15.60:0/3492673479 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f34f40cd110 0x7f34f41562b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f34e000c410 tx=0x7f34e000c8d0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.157+0000 7f34fa400640 1 -- 172.21.15.60:0/3492673479 reap_dead start 2024-04-14T23:48:51.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.158+0000 7f34fae00640 1 -- 172.21.15.60:0/3492673479 shutdown_connections 2024-04-14T23:48:51.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.158+0000 7f34fae00640 1 -- 172.21.15.60:0/3492673479 >> 172.21.15.60:0/3492673479 conn(0x7f34f40670f0 msgr2=0x7f34f40cf860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:51.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.158+0000 7f34fae00640 1 -- 172.21.15.60:0/3492673479 shutdown_connections 2024-04-14T23:48:51.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.158+0000 7f34fae00640 1 -- 172.21.15.60:0/3492673479 wait complete. 2024-04-14T23:48:51.196 INFO:teuthology.orchestra.run.smithi060.stdout:55834574854 2024-04-14T23:48:51.196 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-14T23:48:51.197 INFO:teuthology.orchestra.run.smithi060.stdout:55834574853 2024-04-14T23:48:51.197 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-14T23:48:51.198 INFO:teuthology.orchestra.run.smithi060.stdout:55834574854 2024-04-14T23:48:51.198 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-14T23:48:51.199 INFO:teuthology.orchestra.run.smithi060.stdout:55834574855 2024-04-14T23:48:51.200 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-14T23:48:51.200 INFO:teuthology.orchestra.run.smithi060.stdout:55834574854 2024-04-14T23:48:51.200 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-14T23:48:51.202 INFO:teuthology.orchestra.run.smithi060.stdout:55834574855 2024-04-14T23:48:51.202 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-14T23:48:51.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.528+0000 7febd7200640 1 Processor -- start 2024-04-14T23:48:51.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.529+0000 7febd7200640 1 -- start start 2024-04-14T23:48:51.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.529+0000 7fbf8a400640 1 Processor -- start 2024-04-14T23:48:51.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.531+0000 7febd7200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7febd00c8360 0x7febd015ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.531+0000 7febd7200640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7febd00c8740 0x7febd006f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.531+0000 7febd7200640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7febd00c75c0 0x7febd0158e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.531+0000 7febd7200640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7febd00c6b70 con 0x7febd00c8740 2024-04-14T23:48:51.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.531+0000 7febd7200640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7febd015f7c0 con 0x7febd00c75c0 2024-04-14T23:48:51.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.531+0000 7febd7200640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7febd015faa0 con 0x7febd00c8360 2024-04-14T23:48:51.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.529+0000 7fbf8a400640 1 -- start start 2024-04-14T23:48:51.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.531+0000 7fbf8a400640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbf840cd160 0x7fbf8415ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.532+0000 7fbf8a400640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fbf840cb750 0x7fbf84154b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.532+0000 7fbf8a400640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf840b3f50 con 0x7fbf840cd540 2024-04-14T23:48:51.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.532+0000 7fbf8a400640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fbf8415b370 con 0x7fbf840cd160 2024-04-14T23:48:51.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.532+0000 7fbf8a400640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fbf840b39b0 con 0x7fbf840cb750 2024-04-14T23:48:51.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.531+0000 7febcf400640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7febd00c8740 0x7febd006f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.531+0000 7febd5200640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7febd00c75c0 0x7febd0158e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.532+0000 7febd5200640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7febd00c75c0 0x7febd0158e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:43566/0 (socket says 172.21.15.60:43566) 2024-04-14T23:48:51.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.532+0000 7febd5200640 1 -- 172.21.15.60:0/2680748631 learned_addr learned my addr 172.21.15.60:0/2680748631 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:51.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.532+0000 7febcfe00640 1 --2- 172.21.15.60:0/2680748631 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7febd00c8360 0x7febd015ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.532+0000 7fbf83400640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbf840cd160 0x7fbf8415ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.532+0000 7fbf83e00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fbf840cb750 0x7fbf84154b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.532+0000 7fbf83400640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbf840cd160 0x7fbf8415ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54110/0 (socket says 172.21.15.60:54110) 2024-04-14T23:48:51.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.532+0000 7fbf7aa00640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fbf840cd540 0x7fbf840b3190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:43732/0 (socket says 172.21.15.60:43732) 2024-04-14T23:48:51.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.532+0000 7fbf83400640 1 -- 172.21.15.60:0/2058343036 learned_addr learned my addr 172.21.15.60:0/2058343036 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:51.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.532+0000 7febd5200640 1 -- 172.21.15.60:0/2680748631 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7febd00c8360 msgr2=0x7febd015ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:51.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.532+0000 7febd5200640 1 --2- 172.21.15.60:0/2680748631 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7febd00c8360 0x7febd015ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.532+0000 7febd5200640 1 -- 172.21.15.60:0/2680748631 >> v2:172.21.15.60:3300/0 conn(0x7febd00c8740 msgr2=0x7febd006f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:51.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.533+0000 7fbf82a00640 1 -- 172.21.15.60:0/2058343036 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2309389995 0 0) 0x7fbf840b3f50 con 0x7fbf840cd540 2024-04-14T23:48:51.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.533+0000 7fbf82a00640 1 -- 172.21.15.60:0/2058343036 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf68003520 con 0x7fbf840cd540 2024-04-14T23:48:51.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.533+0000 7fbf83400640 1 -- 172.21.15.60:0/2058343036 >> v2:172.21.15.186:3300/0 conn(0x7fbf840cb750 msgr2=0x7fbf84154b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:51.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.533+0000 7fbf83400640 1 --2- 172.21.15.60:0/2058343036 >> v2:172.21.15.186:3300/0 conn(0x7fbf840cb750 0x7fbf84154b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.533+0000 7fbf83400640 1 -- 172.21.15.60:0/2058343036 >> v1:172.21.15.60:6789/0 conn(0x7fbf840cd540 legacy=0x7fbf840b3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.533+0000 7fbf83400640 1 -- 172.21.15.60:0/2058343036 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf840b3c70 con 0x7fbf840cd160 2024-04-14T23:48:51.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.533+0000 7febd5200640 1 --2- 172.21.15.60:0/2680748631 >> v2:172.21.15.60:3300/0 conn(0x7febd00c8740 0x7febd006f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.533+0000 7febd5200640 1 -- 172.21.15.60:0/2680748631 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febd015fd80 con 0x7febd00c75c0 2024-04-14T23:48:51.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.534+0000 7febcf400640 1 --2- 172.21.15.60:0/2680748631 >> v2:172.21.15.60:3300/0 conn(0x7febd00c8740 0x7febd006f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-14T23:48:51.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.535+0000 7fbf83400640 1 --2- 172.21.15.60:0/2058343036 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf840cd160 0x7fbf8415ae30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbf8415e3a0 tx=0x7fbf6c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9b40a462f4f857f server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:51.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.536+0000 7febd5200640 1 --2- 172.21.15.60:0/2680748631 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7febd00c75c0 0x7febd0158e30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7febd00cd1f0 tx=0x7febc802bf80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57dffbc584963853 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:51.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.536+0000 7fbf82a00640 1 -- 172.21.15.60:0/2058343036 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf6c034070 con 0x7fbf840cd160 2024-04-14T23:48:51.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.536+0000 7fbf82a00640 1 -- 172.21.15.60:0/2058343036 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf6c030860 con 0x7fbf840cd160 2024-04-14T23:48:51.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.536+0000 7fbf82a00640 1 -- 172.21.15.60:0/2058343036 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf6c030c30 con 0x7fbf840cd160 2024-04-14T23:48:51.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.537+0000 7febcea00640 1 -- 172.21.15.60:0/2680748631 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febc802cb20 con 0x7febd00c75c0 2024-04-14T23:48:51.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.537+0000 7febcea00640 1 -- 172.21.15.60:0/2680748631 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febc802ccc0 con 0x7febd00c75c0 2024-04-14T23:48:51.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.537+0000 7febcea00640 1 -- 172.21.15.60:0/2680748631 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febc8034610 con 0x7febd00c75c0 2024-04-14T23:48:51.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.537+0000 7fbf8a400640 1 -- 172.21.15.60:0/2058343036 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf840cd160 msgr2=0x7fbf8415ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.537+0000 7fbf8a400640 1 --2- 172.21.15.60:0/2058343036 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf840cd160 0x7fbf8415ae30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbf8415e3a0 tx=0x7fbf6c02b9a0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.538+0000 7fbf8a400640 1 -- 172.21.15.60:0/2058343036 shutdown_connections 2024-04-14T23:48:51.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.538+0000 7fbf8a400640 1 --2- 172.21.15.60:0/2058343036 >> v2:172.21.15.186:3300/0 conn(0x7fbf840cb750 0x7fbf84154b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.538+0000 7fbf8a400640 1 --2- 172.21.15.60:0/2058343036 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf840cd160 0x7fbf8415ae30 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.538+0000 7febd7200640 1 -- 172.21.15.60:0/2680748631 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7febd00c75c0 msgr2=0x7febd0158e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.538+0000 7febd7200640 1 --2- 172.21.15.60:0/2680748631 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7febd00c75c0 0x7febd0158e30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7febd00cd1f0 tx=0x7febc802bf80 comp rx=0 tx=0).stop 2024-04-14T23:48:51.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.538+0000 7fbf8a400640 1 -- 172.21.15.60:0/2058343036 >> 172.21.15.60:0/2058343036 conn(0x7fbf84066f10 msgr2=0x7fbf840672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:51.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.539+0000 7febd7200640 1 -- 172.21.15.60:0/2680748631 shutdown_connections 2024-04-14T23:48:51.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.539+0000 7fbf8a400640 1 -- 172.21.15.60:0/2058343036 shutdown_connections 2024-04-14T23:48:51.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.539+0000 7febd7200640 1 --2- 172.21.15.60:0/2680748631 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7febd00c75c0 0x7febd0158e30 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.539+0000 7febd7200640 1 --2- 172.21.15.60:0/2680748631 >> v2:172.21.15.60:3300/0 conn(0x7febd00c8740 0x7febd006f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.539+0000 7febd7200640 1 --2- 172.21.15.60:0/2680748631 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7febd00c8360 0x7febd015ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.539+0000 7febd7200640 1 -- 172.21.15.60:0/2680748631 >> 172.21.15.60:0/2680748631 conn(0x7febd00cc170 msgr2=0x7febd00cc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:51.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.539+0000 7fbf8a400640 1 -- 172.21.15.60:0/2058343036 wait complete. 2024-04-14T23:48:51.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.540+0000 7febd7200640 1 -- 172.21.15.60:0/2680748631 shutdown_connections 2024-04-14T23:48:51.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.540+0000 7febd7200640 1 -- 172.21.15.60:0/2680748631 wait complete. 2024-04-14T23:48:51.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.541+0000 7fbf8a400640 1 Processor -- start 2024-04-14T23:48:51.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.542+0000 7fbf8a400640 1 -- start start 2024-04-14T23:48:51.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.543+0000 7febd7200640 1 Processor -- start 2024-04-14T23:48:51.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.544+0000 7febd7200640 1 -- start start 2024-04-14T23:48:51.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.542+0000 7ff2b5c00640 1 Processor -- start 2024-04-14T23:48:51.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.542+0000 7ff2b5c00640 1 -- start start 2024-04-14T23:48:51.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.544+0000 7ff2b5c00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff2b0150e00 0x7ff2b006ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.544+0000 7ff2b5c00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff2b0072ec0 0x7ff2b00c6b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.544+0000 7ff2b5c00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2b006fa10 con 0x7ff2b00c70b0 2024-04-14T23:48:51.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.544+0000 7ff2b5c00640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7ff2b00c3a00 con 0x7ff2b0072ec0 2024-04-14T23:48:51.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.544+0000 7ff2b5c00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7ff2b0156ce0 con 0x7ff2b0150e00 2024-04-14T23:48:51.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.545+0000 7ff2ae000640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff2b0072ec0 0x7ff2b00c6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.545+0000 7ff2ae000640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff2b0072ec0 0x7ff2b00c6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54122/0 (socket says 172.21.15.60:54122) 2024-04-14T23:48:51.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.545+0000 7ff2ae000640 1 -- 172.21.15.60:0/3801688461 learned_addr learned my addr 172.21.15.60:0/3801688461 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:51.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.545+0000 7ff2aea00640 1 --2- 172.21.15.60:0/3801688461 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff2b0150e00 0x7ff2b006ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.545+0000 7febd7200640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7febd00c75c0 0x7febd0071520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.545+0000 7febd7200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7febd00c8360 0x7febd01600e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.545+0000 7febd7200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7febd00c8740 0x7febd00c4840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.545+0000 7febd7200640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7febd015f4c0 con 0x7febd00c75c0 2024-04-14T23:48:51.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.545+0000 7febd7200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7febd00c70a0 con 0x7febd00c8360 2024-04-14T23:48:51.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.545+0000 7febd7200640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7febd0071030 con 0x7febd00c8740 2024-04-14T23:48:51.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.546+0000 7ff2ae000640 1 -- 172.21.15.60:0/3801688461 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff2b0150e00 msgr2=0x7ff2b006ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:51.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.546+0000 7ff2ae000640 1 --2- 172.21.15.60:0/3801688461 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff2b0150e00 0x7ff2b006ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.546+0000 7ff2ae000640 1 -- 172.21.15.60:0/3801688461 >> v1:172.21.15.60:6789/0 conn(0x7ff2b00c70b0 legacy=0x7ff2b015a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:51.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.546+0000 7ff2ae000640 1 -- 172.21.15.60:0/3801688461 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2b0156fc0 con 0x7ff2b0072ec0 2024-04-14T23:48:51.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.546+0000 7febcf400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7febd00c8360 0x7febd01600e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.546+0000 7febcfe00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7febd00c75c0 0x7febd0071520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.546+0000 7febcf400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7febd00c8360 0x7febd01600e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:43586/0 (socket says 172.21.15.60:43586) 2024-04-14T23:48:51.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.546+0000 7febcf400640 1 -- 172.21.15.60:0/2161769252 learned_addr learned my addr 172.21.15.60:0/2161769252 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:51.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.546+0000 7febd5200640 1 --2- 172.21.15.60:0/2161769252 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7febd00c8740 0x7febd00c4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.547+0000 7febcf400640 1 -- 172.21.15.60:0/2161769252 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7febd00c8740 msgr2=0x7febd00c4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.547+0000 7febcf400640 1 --2- 172.21.15.60:0/2161769252 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7febd00c8740 0x7febd00c4840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.547+0000 7febcf400640 1 -- 172.21.15.60:0/2161769252 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7febd00c75c0 msgr2=0x7febd0071520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.547+0000 7febcf400640 1 --2- 172.21.15.60:0/2161769252 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7febd00c75c0 0x7febd0071520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.547+0000 7febcf400640 1 -- 172.21.15.60:0/2161769252 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febd00c58c0 con 0x7febd00c8360 2024-04-14T23:48:51.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.547+0000 7febcfe00640 1 --2- 172.21.15.60:0/2161769252 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7febd00c75c0 0x7febd0071520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:48:51.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.547+0000 7fbf8a400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf840cb750 0x7fbf840c9f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.547+0000 7fbf8a400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbf840cd160 0x7fbf84070010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.547+0000 7fbf8a400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbf840cd540 0x7fbf840c4480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.547+0000 7febcf400640 1 --2- 172.21.15.60:0/2161769252 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7febd00c8360 0x7febd01600e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7febbc00e810 tx=0x7febbc00ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:51.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.548+0000 7febc7e00640 1 -- 172.21.15.60:0/2161769252 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febbc00f040 con 0x7febd00c8360 2024-04-14T23:48:51.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.548+0000 7febc7e00640 1 -- 172.21.15.60:0/2161769252 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febbc004030 con 0x7febd00c8360 2024-04-14T23:48:51.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.548+0000 7febc7e00640 1 -- 172.21.15.60:0/2161769252 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febbc004400 con 0x7febd00c8360 2024-04-14T23:48:51.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.548+0000 7febd7200640 1 -- 172.21.15.60:0/2161769252 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febd00c5ba0 con 0x7febd00c8360 2024-04-14T23:48:51.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.548+0000 7fbf83e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbf840cd540 0x7fbf840c4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.548+0000 7fbf83e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbf840cd540 0x7fbf840c4480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.60:48656/0 (socket says 172.21.15.60:48656) 2024-04-14T23:48:51.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.548+0000 7fbf83e00640 1 -- 172.21.15.60:0/3372844360 learned_addr learned my addr 172.21.15.60:0/3372844360 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:51.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.548+0000 7fbf7aa00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbf840cd160 0x7fbf84070010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.548+0000 7fbf8a400640 1 -- 172.21.15.60:0/3372844360 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fbf8415ea70 con 0x7fbf840cb750 2024-04-14T23:48:51.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.548+0000 7fbf8a400640 1 -- 172.21.15.60:0/3372844360 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbf8415e790 con 0x7fbf840cd160 2024-04-14T23:48:51.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.548+0000 7fbf8a400640 1 -- 172.21.15.60:0/3372844360 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fbf840ccb20 con 0x7fbf840cd540 2024-04-14T23:48:51.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.548+0000 7fbf83400640 1 --2- 172.21.15.60:0/3372844360 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf840cb750 0x7fbf840c9f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.548+0000 7ff2ae000640 1 --2- 172.21.15.60:0/3801688461 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2b0072ec0 0x7ff2b00c6b70 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff2a0009ad0 tx=0x7ff2a002bc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8917ddc9d76690e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:51.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.549+0000 7ff2ad600640 1 -- 172.21.15.60:0/3801688461 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2a0039070 con 0x7ff2b0072ec0 2024-04-14T23:48:51.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.549+0000 7ff2ad600640 1 -- 172.21.15.60:0/3801688461 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2a002c830 con 0x7ff2b0072ec0 2024-04-14T23:48:51.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.549+0000 7ff2ad600640 1 -- 172.21.15.60:0/3801688461 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2a0039070 con 0x7ff2b0072ec0 2024-04-14T23:48:51.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.548+0000 7febd7200640 1 -- 172.21.15.60:0/2161769252 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febd00c6160 con 0x7febd00c8360 2024-04-14T23:48:51.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.550+0000 7fbf83e00640 1 -- 172.21.15.60:0/3372844360 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbf840cd160 msgr2=0x7fbf84070010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.550+0000 7ff2b5c00640 1 -- 172.21.15.60:0/3801688461 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2b0072ec0 msgr2=0x7ff2b00c6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.550+0000 7ff2b5c00640 1 --2- 172.21.15.60:0/3801688461 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2b0072ec0 0x7ff2b00c6b70 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff2a0009ad0 tx=0x7ff2a002bc90 comp rx=0 tx=0).stop 2024-04-14T23:48:51.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.550+0000 7febd7200640 1 -- 172.21.15.60:0/2161769252 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febd0076970 con 0x7febd00c8360 2024-04-14T23:48:51.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.550+0000 7fbf83e00640 1 --2- 172.21.15.60:0/3372844360 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbf840cd160 0x7fbf84070010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.550+0000 7fbf83e00640 1 -- 172.21.15.60:0/3372844360 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf840cb750 msgr2=0x7fbf840c9f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.550+0000 7fbf83e00640 1 --2- 172.21.15.60:0/3372844360 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf840cb750 0x7fbf840c9f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.550+0000 7fbf83e00640 1 -- 172.21.15.60:0/3372844360 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf84070d10 con 0x7fbf840cd540 2024-04-14T23:48:51.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.551+0000 7fbf83e00640 1 --2- 172.21.15.60:0/3372844360 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbf840cd540 0x7fbf840c4480 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c00efc0 tx=0x7fbf7c00c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:51.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.551+0000 7ff2b5c00640 1 -- 172.21.15.60:0/3801688461 shutdown_connections 2024-04-14T23:48:51.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.551+0000 7ff2b5c00640 1 --2- 172.21.15.60:0/3801688461 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2b0072ec0 0x7ff2b00c6b70 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.551+0000 7ff2b5c00640 1 --2- 172.21.15.60:0/3801688461 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff2b0150e00 0x7ff2b006ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.551+0000 7ff2b5c00640 1 -- 172.21.15.60:0/3801688461 >> 172.21.15.60:0/3801688461 conn(0x7ff2b00cc170 msgr2=0x7ff2b00cc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:51.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.551+0000 7ff2b5c00640 1 -- 172.21.15.60:0/3801688461 shutdown_connections 2024-04-14T23:48:51.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.551+0000 7ff2b5c00640 1 -- 172.21.15.60:0/3801688461 wait complete. 2024-04-14T23:48:51.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.552+0000 7fbf7be00640 1 -- 172.21.15.60:0/3372844360 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf7c018020 con 0x7fbf840cd540 2024-04-14T23:48:51.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.552+0000 7fbf7be00640 1 -- 172.21.15.60:0/3372844360 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf7c014480 con 0x7fbf840cd540 2024-04-14T23:48:51.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.552+0000 7fbf7be00640 1 -- 172.21.15.60:0/3372844360 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf7c014870 con 0x7fbf840cd540 2024-04-14T23:48:51.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.552+0000 7fbf8a400640 1 -- 172.21.15.60:0/3372844360 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf84070ff0 con 0x7fbf840cd540 2024-04-14T23:48:51.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.553+0000 7fbf8a400640 1 -- 172.21.15.60:0/3372844360 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf840715b0 con 0x7fbf840cd540 2024-04-14T23:48:51.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.553+0000 7ff2b5c00640 1 Processor -- start 2024-04-14T23:48:51.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.553+0000 7f89ed400640 1 Processor -- start 2024-04-14T23:48:51.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.553+0000 7f89ed400640 1 -- start start 2024-04-14T23:48:51.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.553+0000 7fd6d8000640 1 Processor -- start 2024-04-14T23:48:51.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.554+0000 7ff2b5c00640 1 -- start start 2024-04-14T23:48:51.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.554+0000 7febc7e00640 1 -- 172.21.15.60:0/2161769252 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7febbc00c6d0 con 0x7febd00c8360 2024-04-14T23:48:51.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.554+0000 7fbf8a400640 1 -- 172.21.15.60:0/3372844360 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf84076970 con 0x7fbf840cd540 2024-04-14T23:48:51.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.555+0000 7fec4cc00640 1 Processor -- start 2024-04-14T23:48:51.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.555+0000 7fec4cc00640 1 -- start start 2024-04-14T23:48:51.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.555+0000 7fcec0a00640 1 Processor -- start 2024-04-14T23:48:51.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.555+0000 7febc7e00640 1 --2- 172.21.15.60:0/2161769252 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7febb006ba90 0x7febb006df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.555+0000 7febc7e00640 1 -- 172.21.15.60:0/2161769252 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7febbc08a520 con 0x7febd00c8360 2024-04-14T23:48:51.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.555+0000 7ff2b5c00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff2b0072ec0 0x7ff2b013ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.555+0000 7ff2b5c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff2b00c70b0 0x7ff2b01edb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.555+0000 7ff2b5c00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2b0150e00 0x7ff2b0156180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.556+0000 7ff2b5c00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff2b0157860 con 0x7ff2b0150e00 2024-04-14T23:48:51.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.556+0000 7ff2b5c00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff2b0157580 con 0x7ff2b00c70b0 2024-04-14T23:48:51.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.556+0000 7ff2b5c00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7ff2b01511e0 con 0x7ff2b0072ec0 2024-04-14T23:48:51.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.556+0000 7ff2ae000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff2b00c70b0 0x7ff2b01edb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.556+0000 7ff2ae000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff2b00c70b0 0x7ff2b01edb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:43598/0 (socket says 172.21.15.60:43598) 2024-04-14T23:48:51.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.556+0000 7ff2ae000640 1 -- 172.21.15.60:0/889492960 learned_addr learned my addr 172.21.15.60:0/889492960 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:51.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.556+0000 7ff2af400640 1 --2- 172.21.15.60:0/889492960 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2b0150e00 0x7ff2b0156180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.555+0000 7fd6d8000640 1 -- start start 2024-04-14T23:48:51.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.557+0000 7fd6d8000640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd6d00cd540 0x7fd6d015ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.557+0000 7fd6d8000640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6d00b3f50 con 0x7fd6d00cd160 2024-04-14T23:48:51.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.557+0000 7f89ed400640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f89e8071030 0x7f89e806ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.557+0000 7f89ed400640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89e81512a0 con 0x7f89e80c6b70 2024-04-14T23:48:51.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.557+0000 7f89ed400640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89e8156ce0 con 0x7f89e8072ec0 2024-04-14T23:48:51.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.557+0000 7fd6d8000640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6d00b39b0 con 0x7fd6d00cb750 2024-04-14T23:48:51.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.557+0000 7f89ed400640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f89e80c3a00 con 0x7f89e8071030 2024-04-14T23:48:51.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.557+0000 7f89e6800640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f89e8071030 0x7f89e806ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.557+0000 7f89e6800640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f89e8071030 0x7f89e806ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:43608/0 (socket says 172.21.15.60:43608) 2024-04-14T23:48:51.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.557+0000 7f89e6800640 1 -- 172.21.15.60:0/3873807571 learned_addr learned my addr 172.21.15.60:0/3873807571 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:51.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.555+0000 7fcec0a00640 1 -- start start 2024-04-14T23:48:51.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.557+0000 7fec4cc00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fec4815c9b0 0x7fec480c7280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.557+0000 7fec4cc00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec481530d0 con 0x7fec48152cf0 2024-04-14T23:48:51.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.557+0000 7fec4cc00640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec480751f0 con 0x7fec480c63c0 2024-04-14T23:48:51.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.557+0000 7fec4cc00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fec480c8370 con 0x7fec4815c9b0 2024-04-14T23:48:51.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.557+0000 7febcfe00640 1 --2- 172.21.15.60:0/2161769252 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7febb006ba90 0x7febb006df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.557+0000 7fd6d8000640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fd6d015b370 con 0x7fd6d00cd540 2024-04-14T23:48:51.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.557+0000 7f89e6800640 1 -- 172.21.15.60:0/3873807571 >> v1:172.21.15.186:6789/0 conn(0x7f89e8072ec0 legacy=0x7f89e8150b90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-14T23:48:51.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.557+0000 7f89e6800640 1 -- 172.21.15.60:0/3873807571 >> v1:172.21.15.60:6789/0 conn(0x7f89e80c6b70 legacy=0x7f89e815a490 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-14T23:48:51.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.557+0000 7f89e6800640 1 -- 172.21.15.60:0/3873807571 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89e8156fc0 con 0x7f89e8071030 2024-04-14T23:48:51.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.558+0000 7fcec0a00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fcebc0b1af0 0x7fcebc06b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.558+0000 7fbf7be00640 1 -- 172.21.15.60:0/3372844360 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7fbf7c005d10 con 0x7fbf840cd540 2024-04-14T23:48:51.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.557+0000 7ff2aea00640 1 --2- 172.21.15.60:0/889492960 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff2b0072ec0 0x7ff2b013ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.557+0000 7ff2af400640 1 -- 172.21.15.60:0/889492960 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff2b0072ec0 msgr2=0x7ff2b013ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.557+0000 7fec45400640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fec480c63c0 0x7fec480c6b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.60:56870/0 (socket says 172.21.15.60:56870) 2024-04-14T23:48:51.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.558+0000 7fec46800640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fec4815c9b0 0x7fec480c7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.558+0000 7fd6d5600640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fd6d00cd160 0x7fd6d00b3190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:43744/0 (socket says 172.21.15.60:43744) 2024-04-14T23:48:51.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.558+0000 7fd6d5600640 1 -- 172.21.15.60:0/2604648585 learned_addr learned my addr 172.21.15.60:0/2604648585 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:48:51.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.558+0000 7fec46800640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fec4815c9b0 0x7fec480c7280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.60:48684/0 (socket says 172.21.15.60:48684) 2024-04-14T23:48:51.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.558+0000 7fcec0a00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcebc0c6420 con 0x7fcebc070d10 2024-04-14T23:48:51.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.558+0000 7fcec0a00640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcebc0c6140 con 0x7fcebc06bc20 2024-04-14T23:48:51.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.559+0000 7fcec0a00640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fcebc06b250 con 0x7fcebc0b1af0 2024-04-14T23:48:51.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.559+0000 7fcebb200640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fcebc06bc20 0x7fcebc0c78f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.60:56882/0 (socket says 172.21.15.60:56882) 2024-04-14T23:48:51.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.559+0000 7fcebb200640 1 -- 172.21.15.60:0/2308254020 learned_addr learned my addr 172.21.15.60:0/2308254020 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:48:51.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.559+0000 7fceba800640 1 --2- 172.21.15.60:0/2308254020 >> v2:172.21.15.60:3300/0 conn(0x7fcebc0b1af0 0x7fcebc06b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.558+0000 7fd6d4c00640 1 --2- 172.21.15.60:0/2604648585 >> v2:172.21.15.60:3300/0 conn(0x7fd6d00cd540 0x7fd6d015ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.558+0000 7ff2af400640 1 --2- 172.21.15.60:0/889492960 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff2b0072ec0 0x7ff2b013ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.558+0000 7ff2af400640 1 -- 172.21.15.60:0/889492960 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff2b00c70b0 msgr2=0x7ff2b01edb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.558+0000 7ff2af400640 1 --2- 172.21.15.60:0/889492960 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff2b00c70b0 0x7ff2b01edb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.558+0000 7ff2af400640 1 -- 172.21.15.60:0/889492960 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2b00c6dd0 con 0x7ff2b0150e00 2024-04-14T23:48:51.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.558+0000 7ff2ae000640 1 --2- 172.21.15.60:0/889492960 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff2b00c70b0 0x7ff2b01edb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-14T23:48:51.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.558+0000 7ff2aea00640 1 --2- 172.21.15.60:0/889492960 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff2b0072ec0 0x7ff2b013ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-14T23:48:51.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.559+0000 7fbf7be00640 1 --2- 172.21.15.60:0/3372844360 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fbf6006ba60 0x7fbf6006df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.559+0000 7ff2af400640 1 --2- 172.21.15.60:0/889492960 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2b0150e00 0x7ff2b0156180 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff2a800c730 tx=0x7ff2a800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:51.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.558+0000 7fec46800640 1 -- 172.21.15.60:0/3958701622 learned_addr learned my addr 172.21.15.60:0/3958701622 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:51.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.558+0000 7fec46800640 1 -- 172.21.15.60:0/3958701622 >> v1:172.21.15.186:6789/0 conn(0x7fec480c63c0 legacy=0x7fec480c6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:51.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.558+0000 7fec46800640 1 -- 172.21.15.60:0/3958701622 >> v1:172.21.15.60:6789/0 conn(0x7fec48152cf0 legacy=0x7fec4815c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:51.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.559+0000 7fec46800640 1 -- 172.21.15.60:0/3958701622 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec480c3810 con 0x7fec4815c9b0 2024-04-14T23:48:51.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.559+0000 7fbf7be00640 1 -- 172.21.15.60:0/3372844360 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fbf7c08b160 con 0x7fbf840cd540 2024-04-14T23:48:51.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.560+0000 7fd6cfe00640 1 -- 172.21.15.60:0/2604648585 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2631379714 0 0) 0x7fd6d00b39b0 con 0x7fd6d00cb750 2024-04-14T23:48:51.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.560+0000 7fd6cfe00640 1 -- 172.21.15.60:0/2604648585 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd6b8003520 con 0x7fd6d00cb750 2024-04-14T23:48:51.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.560+0000 7fceb9400640 1 -- 172.21.15.60:0/2308254020 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 362912980 0 0) 0x7fcebc0c6140 con 0x7fcebc06bc20 2024-04-14T23:48:51.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.560+0000 7fceb9400640 1 -- 172.21.15.60:0/2308254020 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcea0003520 con 0x7fcebc06bc20 2024-04-14T23:48:51.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.560+0000 7febc7e00640 1 -- 172.21.15.60:0/2161769252 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7febbc017020 con 0x7febd00c8360 2024-04-14T23:48:51.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.560+0000 7fceba800640 1 -- 172.21.15.60:0/2308254020 >> v1:172.21.15.186:6789/0 conn(0x7fcebc06bc20 legacy=0x7fcebc0c78f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.560+0000 7fceba800640 1 -- 172.21.15.60:0/2308254020 >> v1:172.21.15.60:6789/0 conn(0x7fcebc070d10 legacy=0x7fcebc0c71e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.560+0000 7fceba800640 1 -- 172.21.15.60:0/2308254020 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcebc072700 con 0x7fcebc0b1af0 2024-04-14T23:48:51.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.560+0000 7fd6cfe00640 1 -- 172.21.15.60:0/2604648585 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 293471323 0 0) 0x7fd6d00b3f50 con 0x7fd6d00cd160 2024-04-14T23:48:51.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.560+0000 7fd6cfe00640 1 -- 172.21.15.60:0/2604648585 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd6d00b39b0 con 0x7fd6d00cd160 2024-04-14T23:48:51.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.560+0000 7fd6cfe00640 1 -- 172.21.15.60:0/2604648585 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1611616211 0 0) 0x7fd6b8003520 con 0x7fd6d00cb750 2024-04-14T23:48:51.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.560+0000 7fd6cfe00640 1 -- 172.21.15.60:0/2604648585 >> v2:172.21.15.60:3300/0 conn(0x7fd6d00cd540 msgr2=0x7fd6d015ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:51.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.560+0000 7fd6cfe00640 1 --2- 172.21.15.60:0/2604648585 >> v2:172.21.15.60:3300/0 conn(0x7fd6d00cd540 0x7fd6d015ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.560+0000 7fd6cfe00640 1 -- 172.21.15.60:0/2604648585 >> v1:172.21.15.60:6789/0 conn(0x7fd6d00cd160 legacy=0x7fd6d00b3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.560+0000 7fd6cfe00640 1 -- 172.21.15.60:0/2604648585 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6d00b3c70 con 0x7fd6d00cb750 2024-04-14T23:48:51.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.560+0000 7fd6cfe00640 1 -- 172.21.15.60:0/2604648585 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2814038680 0 0) 0x7fd6c4001970 con 0x7fd6d00cb750 2024-04-14T23:48:51.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.560+0000 7fd6cfe00640 1 -- 172.21.15.60:0/2604648585 >> v1:172.21.15.186:6789/0 conn(0x7fd6d00cb750 legacy=0x7fd6d0154b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.560+0000 7fd6cfe00640 1 --2- 172.21.15.60:0/2604648585 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6b8003b40 0x7fd6b8003f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.560+0000 7fd6cfe00640 1 --2- 172.21.15.60:0/2604648585 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd6b8004800 0x7fd6b8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.560+0000 7fd6cfe00640 1 --2- 172.21.15.60:0/2604648585 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd6b80051c0 0x7fd6b800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.560+0000 7fd6cfe00640 1 -- 172.21.15.60:0/2604648585 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fd6b800bf00 con 0x7fd6b80051c0 2024-04-14T23:48:51.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.560+0000 7fd6cfe00640 1 -- 172.21.15.60:0/2604648585 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd6b800c0b0 con 0x7fd6b8003b40 2024-04-14T23:48:51.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.560+0000 7fd6cfe00640 1 -- 172.21.15.60:0/2604648585 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fd6b800c2b0 con 0x7fd6b8004800 2024-04-14T23:48:51.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.561+0000 7f89e6800640 1 --2- 172.21.15.60:0/3873807571 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89e8071030 0x7f89e806ef80 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f89d8009870 tx=0x7f89d802b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=28a2ee521f0d86af server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:51.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.560+0000 7fbf83400640 1 --2- 172.21.15.60:0/3372844360 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fbf6006ba60 0x7fbf6006df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.561+0000 7ff2a6a00640 1 -- 172.21.15.60:0/889492960 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2a8016020 con 0x7ff2b0150e00 2024-04-14T23:48:51.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.561+0000 7ff2a6a00640 1 -- 172.21.15.60:0/889492960 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2a8006d50 con 0x7ff2b0150e00 2024-04-14T23:48:51.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.561+0000 7ff2a6a00640 1 -- 172.21.15.60:0/889492960 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2a8003bb0 con 0x7ff2b0150e00 2024-04-14T23:48:51.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.561+0000 7fd6d5600640 1 --2- 172.21.15.60:0/2604648585 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6b8003b40 0x7fd6b8003f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.561+0000 7fec46800640 1 --2- 172.21.15.60:0/3958701622 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fec4815c9b0 0x7fec480c7280 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fec380098a0 tx=0x7fec3802fd90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4a5df56193e5759 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:51.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.561+0000 7febcfe00640 1 --2- 172.21.15.60:0/2161769252 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7febb006ba90 0x7febb006df50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7febc000b880 tx=0x7febc0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:51.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.561+0000 7fd6d6000640 1 --2- 172.21.15.60:0/2604648585 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd6b80051c0 0x7fd6b800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.561+0000 7fd6d4c00640 1 --2- 172.21.15.60:0/2604648585 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd6b8004800 0x7fd6b8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.562+0000 7fd6d5600640 1 -- 172.21.15.60:0/2604648585 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd6b8004800 msgr2=0x7fd6b8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:51.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.562+0000 7fd6d5600640 1 --2- 172.21.15.60:0/2604648585 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd6b8004800 0x7fd6b8004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.562+0000 7fd6d5600640 1 -- 172.21.15.60:0/2604648585 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd6b80051c0 msgr2=0x7fd6b800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:51.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.562+0000 7fd6d5600640 1 --2- 172.21.15.60:0/2604648585 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd6b80051c0 0x7fd6b800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.562+0000 7fd6d5600640 1 -- 172.21.15.60:0/2604648585 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6d00b3c70 con 0x7fd6b8003b40 2024-04-14T23:48:51.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.562+0000 7fd6d6000640 1 -- 172.21.15.60:0/2604648585 reap_dead start 2024-04-14T23:48:51.581 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.562+0000 7fec44a00640 1 -- 172.21.15.60:0/3958701622 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec3803d070 con 0x7fec4815c9b0 2024-04-14T23:48:51.581 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.562+0000 7fec44a00640 1 -- 172.21.15.60:0/3958701622 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec38030b40 con 0x7fec4815c9b0 2024-04-14T23:48:51.581 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.562+0000 7fec44a00640 1 -- 172.21.15.60:0/3958701622 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec38038440 con 0x7fec4815c9b0 2024-04-14T23:48:51.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.561+0000 7ff2b5c00640 1 -- 172.21.15.60:0/889492960 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2b013da30 con 0x7ff2b0150e00 2024-04-14T23:48:51.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.561+0000 7ff2b5c00640 1 -- 172.21.15.60:0/889492960 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2b00c6af0 con 0x7ff2b0150e00 2024-04-14T23:48:51.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.562+0000 7f89e5400640 1 -- 172.21.15.60:0/3873807571 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d8035020 con 0x7f89e8071030 2024-04-14T23:48:51.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.562+0000 7f89e5400640 1 -- 172.21.15.60:0/3873807571 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89d8031870 con 0x7f89e8071030 2024-04-14T23:48:51.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.562+0000 7f89e5400640 1 -- 172.21.15.60:0/3873807571 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d8031c40 con 0x7f89e8071030 2024-04-14T23:48:51.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.562+0000 7fbf83400640 1 --2- 172.21.15.60:0/3372844360 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fbf6006ba60 0x7fbf6006df20 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c036000 tx=0x7fbf6c008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:51.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.562+0000 7fbf7be00640 1 -- 172.21.15.60:0/3372844360 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fbf7c022070 con 0x7fbf840cd540 2024-04-14T23:48:51.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.563+0000 7fceba800640 1 --2- 172.21.15.60:0/2308254020 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcebc0b1af0 0x7fcebc06b6e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fceac0097a0 tx=0x7fceac02b9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d28d50e52d66e71 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:51.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.563+0000 7fec4cc00640 1 -- 172.21.15.60:0/3958701622 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fec4815c9b0 msgr2=0x7fec480c7280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.563+0000 7fec4cc00640 1 --2- 172.21.15.60:0/3958701622 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fec4815c9b0 0x7fec480c7280 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fec380098a0 tx=0x7fec3802fd90 comp rx=0 tx=0).stop 2024-04-14T23:48:51.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.563+0000 7fceb9400640 1 -- 172.21.15.60:0/2308254020 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceac02f800 con 0x7fcebc0b1af0 2024-04-14T23:48:51.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.563+0000 7fceb9400640 1 -- 172.21.15.60:0/2308254020 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fceac02f9a0 con 0x7fcebc0b1af0 2024-04-14T23:48:51.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.563+0000 7fceb9400640 1 -- 172.21.15.60:0/2308254020 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceac02fd70 con 0x7fcebc0b1af0 2024-04-14T23:48:51.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.563+0000 7f89ed400640 1 -- 172.21.15.60:0/3873807571 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89e8071030 msgr2=0x7f89e806ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.563+0000 7f89ed400640 1 --2- 172.21.15.60:0/3873807571 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89e8071030 0x7f89e806ef80 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f89d8009870 tx=0x7f89d802b910 comp rx=0 tx=0).stop 2024-04-14T23:48:51.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.564+0000 7fcec0a00640 1 -- 172.21.15.60:0/2308254020 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcebc0b1af0 msgr2=0x7fcebc06b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.564+0000 7fcec0a00640 1 --2- 172.21.15.60:0/2308254020 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcebc0b1af0 0x7fcebc06b6e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fceac0097a0 tx=0x7fceac02b9e0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.566+0000 7f89ed400640 1 -- 172.21.15.60:0/3873807571 shutdown_connections 2024-04-14T23:48:51.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.566+0000 7f89ed400640 1 --2- 172.21.15.60:0/3873807571 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89e8071030 0x7f89e806ef80 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.566+0000 7f89ed400640 1 -- 172.21.15.60:0/3873807571 >> 172.21.15.60:0/3873807571 conn(0x7f89e80cc170 msgr2=0x7f89e80cc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:51.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.564+0000 7fd6d5600640 1 --2- 172.21.15.60:0/2604648585 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6b8003b40 0x7fd6b8003f20 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd6c8001a00 tx=0x7fd6b8005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f775c8714da1d547 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:51.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.565+0000 7fd6cfe00640 1 -- 172.21.15.60:0/2604648585 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c8037070 con 0x7fd6b8003b40 2024-04-14T23:48:51.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.565+0000 7fd6cfe00640 1 -- 172.21.15.60:0/2604648585 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6c8005ad0 con 0x7fd6b8003b40 2024-04-14T23:48:51.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.565+0000 7fd6cfe00640 1 -- 172.21.15.60:0/2604648585 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c80015d0 con 0x7fd6b8003b40 2024-04-14T23:48:51.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.566+0000 7fd6d8000640 1 -- 172.21.15.60:0/2604648585 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6b8003b40 msgr2=0x7fd6b8003f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.566+0000 7fd6d8000640 1 --2- 172.21.15.60:0/2604648585 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6b8003b40 0x7fd6b8003f20 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd6c8001a00 tx=0x7fd6b8005830 comp rx=0 tx=0).stop 2024-04-14T23:48:51.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.566+0000 7fd6d8000640 1 -- 172.21.15.60:0/2604648585 shutdown_connections 2024-04-14T23:48:51.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.564+0000 7fec4cc00640 1 -- 172.21.15.60:0/3958701622 shutdown_connections 2024-04-14T23:48:51.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.564+0000 7fec4cc00640 1 --2- 172.21.15.60:0/3958701622 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fec4815c9b0 0x7fec480c7280 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.564+0000 7fec4cc00640 1 -- 172.21.15.60:0/3958701622 >> 172.21.15.60:0/3958701622 conn(0x7fec480cc170 msgr2=0x7fec480cc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:51.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.564+0000 7fec4cc00640 1 -- 172.21.15.60:0/3958701622 shutdown_connections 2024-04-14T23:48:51.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.564+0000 7fec4cc00640 1 -- 172.21.15.60:0/3958701622 wait complete. 2024-04-14T23:48:51.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.563+0000 7f3b14000640 1 Processor -- start 2024-04-14T23:48:51.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.564+0000 7f3b14000640 1 -- start start 2024-04-14T23:48:51.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.566+0000 7f3b14000640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3b0c0c8360 0x7f3b0c15ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.566+0000 7fd6d8000640 1 --2- 172.21.15.60:0/2604648585 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6b8003b40 0x7fd6b8003f20 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.566+0000 7fd6d8000640 1 -- 172.21.15.60:0/2604648585 >> 172.21.15.60:0/2604648585 conn(0x7fd6d0066f10 msgr2=0x7fd6d00672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:51.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.566+0000 7f3b14000640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3b0c0c75c0 0x7f3b0c158e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.566+0000 7f3b14000640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b0c15f7c0 con 0x7f3b0c0c8740 2024-04-14T23:48:51.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.566+0000 7f3b14000640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f3b0c0c6b70 con 0x7f3b0c0c8360 2024-04-14T23:48:51.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.566+0000 7f3b14000640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f3b0c06fa20 con 0x7f3b0c0c75c0 2024-04-14T23:48:51.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.566+0000 7f3b11600640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3b0c0c8360 0x7f3b0c15ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.566+0000 7fec4cc00640 1 Processor -- start 2024-04-14T23:48:51.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.567+0000 7f3b11600640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3b0c0c8360 0x7f3b0c15ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54178/0 (socket says 172.21.15.60:54178) 2024-04-14T23:48:51.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.567+0000 7f3b11600640 1 -- 172.21.15.60:0/3061292502 learned_addr learned my addr 172.21.15.60:0/3061292502 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:51.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.567+0000 7fec4cc00640 1 -- start start 2024-04-14T23:48:51.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.567+0000 7fcec0a00640 1 -- 172.21.15.60:0/2308254020 shutdown_connections 2024-04-14T23:48:51.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.567+0000 7fcec0a00640 1 --2- 172.21.15.60:0/2308254020 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcebc0b1af0 0x7fcebc06b6e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.567+0000 7fcec0a00640 1 -- 172.21.15.60:0/2308254020 >> 172.21.15.60:0/2308254020 conn(0x7fcebc0cc2f0 msgr2=0x7fcebc0cc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:51.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.567+0000 7f89ed400640 1 -- 172.21.15.60:0/3873807571 shutdown_connections 2024-04-14T23:48:51.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.567+0000 7f3b11600640 1 -- 172.21.15.60:0/3061292502 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3b0c0c75c0 msgr2=0x7f3b0c158e30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-14T23:48:51.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.567+0000 7f3b11600640 1 --2- 172.21.15.60:0/3061292502 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3b0c0c75c0 0x7f3b0c158e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.567+0000 7f3b11600640 1 -- 172.21.15.60:0/3061292502 >> v1:172.21.15.60:6789/0 conn(0x7f3b0c0c8740 legacy=0x7f3b0c06f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.567+0000 7fcec0a00640 1 -- 172.21.15.60:0/2308254020 shutdown_connections 2024-04-14T23:48:51.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.567+0000 7fcec0a00640 1 -- 172.21.15.60:0/2308254020 wait complete. 2024-04-14T23:48:51.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.568+0000 7f3b11600640 1 -- 172.21.15.60:0/3061292502 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b0c15faa0 con 0x7f3b0c0c8360 2024-04-14T23:48:51.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.567+0000 7f89ed400640 1 -- 172.21.15.60:0/3873807571 wait complete. 2024-04-14T23:48:51.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.568+0000 7fec4cc00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fec480c63c0 0x7fec481f5e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.568+0000 7fec4cc00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec48152cf0 0x7fec480c9f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.568+0000 7fec4cc00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec4815c9b0 0x7fec48071db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.568+0000 7fec4cc00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fec480c88f0 con 0x7fec4815c9b0 2024-04-14T23:48:51.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.568+0000 7fec4cc00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fec480c8610 con 0x7fec48152cf0 2024-04-14T23:48:51.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.568+0000 7fec4cc00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fec480c77c0 con 0x7fec480c63c0 2024-04-14T23:48:51.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.569+0000 7fec45e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fec480c63c0 0x7fec481f5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.569+0000 7fec45400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec48152cf0 0x7fec480c9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.569+0000 7fec45400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec48152cf0 0x7fec480c9f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:43622/0 (socket says 172.21.15.60:43622) 2024-04-14T23:48:51.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.569+0000 7fec45e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fec480c63c0 0x7fec481f5e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.60:48714/0 (socket says 172.21.15.60:48714) 2024-04-14T23:48:51.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.569+0000 7fec45400640 1 -- 172.21.15.60:0/2937194093 learned_addr learned my addr 172.21.15.60:0/2937194093 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:51.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.569+0000 7f3b11600640 1 --2- 172.21.15.60:0/3061292502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3b0c0c8360 0x7f3b0c15ef80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3b00009870 tx=0x7f3b0002ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cddabb186f34b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:51.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.569+0000 7fec46800640 1 --2- 172.21.15.60:0/2937194093 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec4815c9b0 0x7fec48071db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.570+0000 7fec45400640 1 -- 172.21.15.60:0/2937194093 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fec480c63c0 msgr2=0x7fec481f5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.570+0000 7fec45400640 1 --2- 172.21.15.60:0/2937194093 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fec480c63c0 0x7fec481f5e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.570+0000 7fec45400640 1 -- 172.21.15.60:0/2937194093 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec4815c9b0 msgr2=0x7fec48071db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.570+0000 7fec45400640 1 --2- 172.21.15.60:0/2937194093 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec4815c9b0 0x7fec48071db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.570+0000 7fec45400640 1 -- 172.21.15.60:0/2937194093 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec480cbdc0 con 0x7fec48152cf0 2024-04-14T23:48:51.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.570+0000 7fec46800640 1 --2- 172.21.15.60:0/2937194093 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec4815c9b0 0x7fec48071db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-14T23:48:51.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.570+0000 7fec45400640 1 --2- 172.21.15.60:0/2937194093 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec48152cf0 0x7fec480c9f70 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fec3000b3e0 tx=0x7fec3000b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:51.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.570+0000 7fec3e000640 1 -- 172.21.15.60:0/2937194093 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec30015020 con 0x7fec48152cf0 2024-04-14T23:48:51.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.570+0000 7fec3e000640 1 -- 172.21.15.60:0/2937194093 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec30011830 con 0x7fec48152cf0 2024-04-14T23:48:51.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.570+0000 7fec3e000640 1 -- 172.21.15.60:0/2937194093 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec30011c20 con 0x7fec48152cf0 2024-04-14T23:48:51.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.570+0000 7f89ed400640 1 Processor -- start 2024-04-14T23:48:51.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.570+0000 7fec4cc00640 1 -- 172.21.15.60:0/2937194093 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec48150ba0 con 0x7fec48152cf0 2024-04-14T23:48:51.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.571+0000 7fec4cc00640 1 -- 172.21.15.60:0/2937194093 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec481f5660 con 0x7fec48152cf0 2024-04-14T23:48:51.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.571+0000 7f89ed400640 1 -- start start 2024-04-14T23:48:51.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.571+0000 7fd6d8000640 1 -- 172.21.15.60:0/2604648585 shutdown_connections 2024-04-14T23:48:51.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.571+0000 7fcec0a00640 1 Processor -- start 2024-04-14T23:48:51.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.571+0000 7fd6d8000640 1 -- 172.21.15.60:0/2604648585 wait complete. 2024-04-14T23:48:51.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.572+0000 7fcec0a00640 1 -- start start 2024-04-14T23:48:51.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.573+0000 7f89ed400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89e8071030 0x7f89e80c5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.573+0000 7f89ed400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89e8072ec0 0x7f89e81f0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.573+0000 7fd6d8000640 1 Processor -- start 2024-04-14T23:48:51.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.565+0000 7ff2b5c00640 1 -- 172.21.15.60:0/889492960 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2b0072a90 con 0x7ff2b0150e00 2024-04-14T23:48:51.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.573+0000 7ff2a6a00640 1 -- 172.21.15.60:0/889492960 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7ff2a80050d0 con 0x7ff2b0150e00 2024-04-14T23:48:51.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.573+0000 7f89e6800640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89e8071030 0x7f89e80c5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.573+0000 7fd6d8000640 1 -- start start 2024-04-14T23:48:51.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.573+0000 7f89ed400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f89e80c6b70 0x7f89e814fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.573+0000 7f89e6800640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89e8071030 0x7f89e80c5020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54196/0 (socket says 172.21.15.60:54196) 2024-04-14T23:48:51.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.573+0000 7fcec0a00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcebc06bc20 0x7fcebc1f0b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.573+0000 7fcec0a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcebc070d10 0x7fcebc156700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.573+0000 7fcec0a00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcebc0b1af0 0x7fcebc156c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.574+0000 7fcec0a00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fcebc06c030 con 0x7fcebc06bc20 2024-04-14T23:48:51.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.574+0000 7fcec0a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcebc072cc0 con 0x7fcebc070d10 2024-04-14T23:48:51.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.574+0000 7fcec0a00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fcebc0c85a0 con 0x7fcebc0b1af0 2024-04-14T23:48:51.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.574+0000 7fceba800640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcebc06bc20 0x7fcebc1f0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.574+0000 7f89e5e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89e8072ec0 0x7f89e81f0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.574+0000 7f89e5e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89e8072ec0 0x7f89e81f0c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:43636/0 (socket says 172.21.15.60:43636) 2024-04-14T23:48:51.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.574+0000 7f89e7200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f89e80c6b70 0x7f89e814fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.574+0000 7f89e7200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f89e80c6b70 0x7f89e814fd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.60:48746/0 (socket says 172.21.15.60:48746) 2024-04-14T23:48:51.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.574+0000 7f89e6800640 1 -- 172.21.15.60:0/538934297 learned_addr learned my addr 172.21.15.60:0/538934297 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:51.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.573+0000 7fec4cc00640 1 -- 172.21.15.60:0/2937194093 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec480c6030 con 0x7fec48152cf0 2024-04-14T23:48:51.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.574+0000 7fd6d8000640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd6b8003b40 0x7fd6d00cee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.575+0000 7fd6d8000640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd6d0158040 0x7fd6d01561b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.575+0000 7fd6d8000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6d0158480 0x7fd6d0158a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.575+0000 7f89ed400640 1 -- 172.21.15.60:0/538934297 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f89e8157860 con 0x7f89e8071030 2024-04-14T23:48:51.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.575+0000 7f89ed400640 1 -- 172.21.15.60:0/538934297 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f89e8157580 con 0x7f89e8072ec0 2024-04-14T23:48:51.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.575+0000 7fd6d8000640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d015ea70 con 0x7fd6b8003b40 2024-04-14T23:48:51.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.575+0000 7fd6d8000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d015e790 con 0x7fd6d0158480 2024-04-14T23:48:51.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.575+0000 7fd6d8000640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fd6d00ccb20 con 0x7fd6d0158040 2024-04-14T23:48:51.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.575+0000 7f89ed400640 1 -- 172.21.15.60:0/538934297 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f89e806f4c0 con 0x7f89e80c6b70 2024-04-14T23:48:51.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.574+0000 7fec3e000640 1 -- 172.21.15.60:0/2937194093 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7fec30018020 con 0x7fec48152cf0 2024-04-14T23:48:51.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.574+0000 7fcebb200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcebc0b1af0 0x7fcebc156c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.574+0000 7fceb9e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcebc070d10 0x7fcebc156700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.575+0000 7fcebb200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcebc0b1af0 0x7fcebc156c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.60:48730/0 (socket says 172.21.15.60:48730) 2024-04-14T23:48:51.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.575+0000 7fcebb200640 1 -- 172.21.15.60:0/763024427 learned_addr learned my addr 172.21.15.60:0/763024427 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:51.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.575+0000 7fcebb200640 1 -- 172.21.15.60:0/763024427 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcebc070d10 msgr2=0x7fcebc156700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.575+0000 7fcebb200640 1 --2- 172.21.15.60:0/763024427 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcebc070d10 0x7fcebc156700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.575+0000 7fcebb200640 1 -- 172.21.15.60:0/763024427 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcebc06bc20 msgr2=0x7fcebc1f0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.575+0000 7fcebb200640 1 --2- 172.21.15.60:0/763024427 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcebc06bc20 0x7fcebc1f0b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.575+0000 7fcebb200640 1 -- 172.21.15.60:0/763024427 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcebc0b0750 con 0x7fcebc0b1af0 2024-04-14T23:48:51.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.575+0000 7fd6d4c00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd6d0158040 0x7fd6d01561b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.576+0000 7fd6d4c00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd6d0158040 0x7fd6d01561b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.60:48762/0 (socket says 172.21.15.60:48762) 2024-04-14T23:48:51.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.576+0000 7fd6d4c00640 1 -- 172.21.15.60:0/1854056203 learned_addr learned my addr 172.21.15.60:0/1854056203 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:51.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.576+0000 7f89e6800640 1 -- 172.21.15.60:0/538934297 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f89e80c6b70 msgr2=0x7f89e814fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.576+0000 7f89e6800640 1 --2- 172.21.15.60:0/538934297 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f89e80c6b70 0x7f89e814fd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.576+0000 7f89e6800640 1 -- 172.21.15.60:0/538934297 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89e8072ec0 msgr2=0x7f89e81f0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.576+0000 7f89e6800640 1 --2- 172.21.15.60:0/538934297 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89e8072ec0 0x7f89e81f0c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.576+0000 7f89e6800640 1 -- 172.21.15.60:0/538934297 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89e80c5c30 con 0x7f89e8071030 2024-04-14T23:48:51.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.576+0000 7fd6d5600640 1 --2- 172.21.15.60:0/1854056203 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd6b8003b40 0x7fd6d00cee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.576+0000 7f89e7200640 1 --2- 172.21.15.60:0/538934297 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f89e80c6b70 0x7f89e814fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-04-14T23:48:51.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.576+0000 7f89e6800640 1 --2- 172.21.15.60:0/538934297 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89e8071030 0x7f89e80c5020 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f89d8009870 tx=0x7f89d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:51.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.576+0000 7f89e5e00640 1 --2- 172.21.15.60:0/538934297 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89e8072ec0 0x7f89e81f0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-04-14T23:48:51.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.576+0000 7fcebb200640 1 --2- 172.21.15.60:0/763024427 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcebc0b1af0 0x7fcebc156c40 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fceb400cbe0 tx=0x7fceb400a430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:51.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.576+0000 7fec3e000640 1 --2- 172.21.15.60:0/2937194093 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fec2406b9e0 0x7fec2406dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.576+0000 7fec3e000640 1 -- 172.21.15.60:0/2937194093 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fec3008a670 con 0x7fec48152cf0 2024-04-14T23:48:51.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.577+0000 7fd6d6000640 1 --2- 172.21.15.60:0/1854056203 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6d0158480 0x7fd6d0158a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.577+0000 7fceb2a00640 1 -- 172.21.15.60:0/763024427 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb4017070 con 0x7fcebc0b1af0 2024-04-14T23:48:51.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.577+0000 7fceb2a00640 1 -- 172.21.15.60:0/763024427 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fceb400ad40 con 0x7fcebc0b1af0 2024-04-14T23:48:51.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.577+0000 7fceb2a00640 1 -- 172.21.15.60:0/763024427 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb4012600 con 0x7fcebc0b1af0 2024-04-14T23:48:51.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.577+0000 7fcec0a00640 1 -- 172.21.15.60:0/763024427 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcebc1f1580 con 0x7fcebc0b1af0 2024-04-14T23:48:51.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.577+0000 7fec45e00640 1 --2- 172.21.15.60:0/2937194093 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fec2406b9e0 0x7fec2406dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.577+0000 7fcec0a00640 1 -- 172.21.15.60:0/763024427 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcebc0b0470 con 0x7fcebc0b1af0 2024-04-14T23:48:51.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.577+0000 7fd6d5600640 1 -- 172.21.15.60:0/1854056203 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd6d0158040 msgr2=0x7fd6d01561b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.577+0000 7fd6d5600640 1 --2- 172.21.15.60:0/1854056203 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd6d0158040 0x7fd6d01561b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.577+0000 7fd6d5600640 1 -- 172.21.15.60:0/1854056203 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6d0158480 msgr2=0x7fd6d0158a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.577+0000 7fd6d5600640 1 --2- 172.21.15.60:0/1854056203 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6d0158480 0x7fd6d0158a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.577+0000 7fd6d5600640 1 -- 172.21.15.60:0/1854056203 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6d00d0340 con 0x7fd6b8003b40 2024-04-14T23:48:51.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.577+0000 7fd6d4c00640 1 --2- 172.21.15.60:0/1854056203 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd6d0158040 0x7fd6d01561b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:48:51.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.577+0000 7fd6d6000640 1 --2- 172.21.15.60:0/1854056203 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6d0158480 0x7fd6d0158a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-14T23:48:51.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.578+0000 7fd6d5600640 1 --2- 172.21.15.60:0/1854056203 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd6b8003b40 0x7fd6d00cee60 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd6b800c280 tx=0x7fd6c80012b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:51.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.578+0000 7fec45e00640 1 --2- 172.21.15.60:0/2937194093 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fec2406b9e0 0x7fec2406dea0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fec40005660 tx=0x7fec40003d90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:51.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.578+0000 7ff2a6a00640 1 --2- 172.21.15.60:0/889492960 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff29006ba60 0x7ff29006df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.578+0000 7ff2a6a00640 1 -- 172.21.15.60:0/889492960 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ff2a808a460 con 0x7ff2b0150e00 2024-04-14T23:48:51.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.578+0000 7ff2aea00640 1 --2- 172.21.15.60:0/889492960 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff29006ba60 0x7ff29006df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.579+0000 7ff2aea00640 1 --2- 172.21.15.60:0/889492960 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff29006ba60 0x7ff29006df20 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff29c009370 tx=0x7ff29c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:51.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.579+0000 7fcec0a00640 1 -- 172.21.15.60:0/763024427 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcebc0c6d40 con 0x7fcebc0b1af0 2024-04-14T23:48:51.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.580+0000 7fec3e000640 1 -- 172.21.15.60:0/2937194093 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fec3001f020 con 0x7fec48152cf0 2024-04-14T23:48:51.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.580+0000 7f3b0be00640 1 -- 172.21.15.60:0/3061292502 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b0002f850 con 0x7f3b0c0c8360 2024-04-14T23:48:51.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.580+0000 7fceb2a00640 1 -- 172.21.15.60:0/763024427 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7fceb401a070 con 0x7fcebc0b1af0 2024-04-14T23:48:51.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.581+0000 7fd6cd600640 1 -- 172.21.15.60:0/1854056203 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c8042070 con 0x7fd6b8003b40 2024-04-14T23:48:51.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.581+0000 7fd6d8000640 1 -- 172.21.15.60:0/1854056203 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6d00cf580 con 0x7fd6b8003b40 2024-04-14T23:48:51.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.581+0000 7fd6d8000640 1 -- 172.21.15.60:0/1854056203 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6d00d08e0 con 0x7fd6b8003b40 2024-04-14T23:48:51.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.581+0000 7f3b0be00640 1 -- 172.21.15.60:0/3061292502 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b0002f9f0 con 0x7f3b0c0c8360 2024-04-14T23:48:51.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.582+0000 7f3b0be00640 1 -- 172.21.15.60:0/3061292502 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b0002fdc0 con 0x7f3b0c0c8360 2024-04-14T23:48:51.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.581+0000 7fceb2a00640 1 --2- 172.21.15.60:0/763024427 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fce9c06b9e0 0x7fce9c06dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.582+0000 7f89dea00640 1 -- 172.21.15.60:0/538934297 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d80443f0 con 0x7f89e8071030 2024-04-14T23:48:51.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.582+0000 7f89dea00640 1 -- 172.21.15.60:0/538934297 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89d8044590 con 0x7f89e8071030 2024-04-14T23:48:51.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.582+0000 7f89dea00640 1 -- 172.21.15.60:0/538934297 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d8044980 con 0x7f89e8071030 2024-04-14T23:48:51.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.582+0000 7fceb2a00640 1 -- 172.21.15.60:0/763024427 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fceb408a5b0 con 0x7fcebc0b1af0 2024-04-14T23:48:51.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.582+0000 7fd6cd600640 1 -- 172.21.15.60:0/1854056203 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6c8032470 con 0x7fd6b8003b40 2024-04-14T23:48:51.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.582+0000 7fd6cd600640 1 -- 172.21.15.60:0/1854056203 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c8037040 con 0x7fd6b8003b40 2024-04-14T23:48:51.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.582+0000 7f3b14000640 1 -- 172.21.15.60:0/3061292502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3b0c0c8360 msgr2=0x7f3b0c15ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.582+0000 7f3b14000640 1 --2- 172.21.15.60:0/3061292502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3b0c0c8360 0x7f3b0c15ef80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3b00009870 tx=0x7f3b0002ba30 comp rx=0 tx=0).stop 2024-04-14T23:48:51.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.583+0000 7f3b14000640 1 -- 172.21.15.60:0/3061292502 shutdown_connections 2024-04-14T23:48:51.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.583+0000 7f3b14000640 1 --2- 172.21.15.60:0/3061292502 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3b0c0c75c0 0x7f3b0c158e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.583+0000 7f3b14000640 1 --2- 172.21.15.60:0/3061292502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3b0c0c8360 0x7f3b0c15ef80 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.583+0000 7f3b14000640 1 -- 172.21.15.60:0/3061292502 >> 172.21.15.60:0/3061292502 conn(0x7f3b0c0cc170 msgr2=0x7f3b0c0cc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:51.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.583+0000 7fceba800640 1 --2- 172.21.15.60:0/763024427 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fce9c06b9e0 0x7fce9c06dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.583+0000 7f89ed400640 1 -- 172.21.15.60:0/538934297 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89e81560b0 con 0x7f89e8071030 2024-04-14T23:48:51.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.583+0000 7f89ed400640 1 -- 172.21.15.60:0/538934297 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89e81f4310 con 0x7f89e8071030 2024-04-14T23:48:51.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.583+0000 7ff2a6a00640 1 -- 172.21.15.60:0/889492960 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff2a8020020 con 0x7ff2b0150e00 2024-04-14T23:48:51.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.583+0000 7fd6d8000640 1 -- 172.21.15.60:0/1854056203 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6d00b5500 con 0x7fd6b8003b40 2024-04-14T23:48:51.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.584+0000 7f3b14000640 1 -- 172.21.15.60:0/3061292502 shutdown_connections 2024-04-14T23:48:51.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.584+0000 7fceba800640 1 --2- 172.21.15.60:0/763024427 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fce9c06b9e0 0x7fce9c06dea0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fceac036000 tx=0x7fceac039040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:51.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.584+0000 7f3b14000640 1 -- 172.21.15.60:0/3061292502 wait complete. 2024-04-14T23:48:51.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.586+0000 7fceb2a00640 1 -- 172.21.15.60:0/763024427 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fceb40592a0 con 0x7fcebc0b1af0 2024-04-14T23:48:51.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.587+0000 7f89ed400640 1 -- 172.21.15.60:0/538934297 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89c0005290 con 0x7f89e8071030 2024-04-14T23:48:51.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.587+0000 7f3b14000640 1 Processor -- start 2024-04-14T23:48:51.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.587+0000 7f3b14000640 1 -- start start 2024-04-14T23:48:51.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.588+0000 7f3b14000640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3b0c0c75c0 0x7f3b0c1f5630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.588+0000 7f3b14000640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3b0c0c8360 0x7f3b0c15cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.588+0000 7f3b14000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b0c0c8740 0x7f3b0c15a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.588+0000 7f3b14000640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f3b0c0c70a0 con 0x7f3b0c0c75c0 2024-04-14T23:48:51.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.588+0000 7f3b14000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3b0c0c6dc0 con 0x7f3b0c0c8740 2024-04-14T23:48:51.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.588+0000 7f3b14000640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f3b0c15f4c0 con 0x7f3b0c0c8360 2024-04-14T23:48:51.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.589+0000 7f3b11600640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3b0c0c75c0 0x7f3b0c1f5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.589+0000 7f3b11600640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3b0c0c75c0 0x7f3b0c1f5630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54230/0 (socket says 172.21.15.60:54230) 2024-04-14T23:48:51.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.589+0000 7f3b11600640 1 -- 172.21.15.60:0/562870125 learned_addr learned my addr 172.21.15.60:0/562870125 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:51.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.589+0000 7f3b10c00640 1 --2- 172.21.15.60:0/562870125 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3b0c0c8360 0x7f3b0c15cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.589+0000 7f3b12000640 1 --2- 172.21.15.60:0/562870125 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b0c0c8740 0x7f3b0c15a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.590+0000 7f3b10c00640 1 -- 172.21.15.60:0/562870125 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b0c0c8740 msgr2=0x7f3b0c15a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.590+0000 7f3b10c00640 1 --2- 172.21.15.60:0/562870125 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b0c0c8740 0x7f3b0c15a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.590+0000 7f3b10c00640 1 -- 172.21.15.60:0/562870125 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3b0c0c75c0 msgr2=0x7f3b0c1f5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:51.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.590+0000 7f3b10c00640 1 --2- 172.21.15.60:0/562870125 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3b0c0c75c0 0x7f3b0c1f5630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:51.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.590+0000 7f3b10c00640 1 -- 172.21.15.60:0/562870125 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b0c15c900 con 0x7f3b0c0c8360 2024-04-14T23:48:51.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.590+0000 7f3b10c00640 1 --2- 172.21.15.60:0/562870125 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3b0c0c8360 0x7f3b0c15cdf0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3afc00c390 tx=0x7f3afc00c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:51.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.588+0000 7fd6cd600640 1 -- 172.21.15.60:0/1854056203 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7fd6c8045070 con 0x7fd6b8003b40 2024-04-14T23:48:51.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.590+0000 7fd6cd600640 1 --2- 172.21.15.60:0/1854056203 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fd6b006b9c0 0x7fd6b006de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.590+0000 7fd6cd600640 1 -- 172.21.15.60:0/1854056203 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fd6c80ac9b0 con 0x7fd6b8003b40 2024-04-14T23:48:51.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.591+0000 7fd6d4c00640 1 --2- 172.21.15.60:0/1854056203 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fd6b006b9c0 0x7fd6b006de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.592+0000 7fd6d4c00640 1 --2- 172.21.15.60:0/1854056203 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fd6b006b9c0 0x7fd6b006de80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd6d00ceb80 tx=0x7fd6bc008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:51.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.590+0000 7f3b09600640 1 -- 172.21.15.60:0/562870125 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3afc005220 con 0x7f3b0c0c8360 2024-04-14T23:48:51.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.591+0000 7f3b09600640 1 -- 172.21.15.60:0/562870125 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3afc0053c0 con 0x7f3b0c0c8360 2024-04-14T23:48:51.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.591+0000 7f3b09600640 1 -- 172.21.15.60:0/562870125 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3afc007250 con 0x7f3b0c0c8360 2024-04-14T23:48:51.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.591+0000 7f3b14000640 1 -- 172.21.15.60:0/562870125 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b0c15eb50 con 0x7f3b0c0c8360 2024-04-14T23:48:51.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.591+0000 7f3b14000640 1 -- 172.21.15.60:0/562870125 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b0c159e80 con 0x7f3b0c0c8360 2024-04-14T23:48:51.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.593+0000 7f3b14000640 1 -- 172.21.15.60:0/562870125 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b0c076970 con 0x7f3b0c0c8360 2024-04-14T23:48:51.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.594+0000 7f3b09600640 1 -- 172.21.15.60:0/562870125 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f3afc019020 con 0x7f3b0c0c8360 2024-04-14T23:48:51.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.596+0000 7f3b09600640 1 --2- 172.21.15.60:0/562870125 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f3af006ba10 0x7f3af006ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.596+0000 7f3b09600640 1 -- 172.21.15.60:0/562870125 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f3afc0899d0 con 0x7f3b0c0c8360 2024-04-14T23:48:51.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.597+0000 7f3b11600640 1 --2- 172.21.15.60:0/562870125 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f3af006ba10 0x7f3af006ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.595+0000 7f89dea00640 1 -- 172.21.15.60:0/538934297 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f89d804b060 con 0x7f89e8071030 2024-04-14T23:48:51.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.597+0000 7f89dea00640 1 --2- 172.21.15.60:0/538934297 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f89c806ba90 0x7f89c806df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:51.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.597+0000 7f89dea00640 1 -- 172.21.15.60:0/538934297 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f89d80ac980 con 0x7f89e8071030 2024-04-14T23:48:51.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.597+0000 7f89e5e00640 1 --2- 172.21.15.60:0/538934297 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f89c806ba90 0x7f89c806df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:51.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.598+0000 7f3b11600640 1 --2- 172.21.15.60:0/562870125 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f3af006ba10 0x7f3af006ded0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3b000090b0 tx=0x7f3b00008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:51.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.598+0000 7f89e5e00640 1 --2- 172.21.15.60:0/538934297 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f89c806ba90 0x7f89c806df50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f89e81473b0 tx=0x7f89d4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:51.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.600+0000 7f3b09600640 1 -- 172.21.15.60:0/562870125 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3afc0585c0 con 0x7f3b0c0c8360 2024-04-14T23:48:51.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.604+0000 7fd6cd600640 1 -- 172.21.15.60:0/1854056203 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd6d00b5500 con 0x7fd6b8003b40 2024-04-14T23:48:51.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:51.611+0000 7f89dea00640 1 -- 172.21.15.60:0/538934297 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f89d803f070 con 0x7f89e8071030 2024-04-14T23:48:52.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.002+0000 7fbf8a400640 1 -- 172.21.15.60:0/3372844360 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fbf840717b0 con 0x7fbf840cd540 2024-04-14T23:48:52.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.003+0000 7febd7200640 1 -- 172.21.15.60:0/2161769252 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7febd00721b0 con 0x7febd00c8360 2024-04-14T23:48:52.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.003+0000 7fbf7be00640 1 -- 172.21.15.60:0/3372844360 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fbf7c010040 con 0x7fbf840cd540 2024-04-14T23:48:52.009 INFO:teuthology.orchestra.run.smithi060.stdout:55834574853 2024-04-14T23:48:52.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.004+0000 7febc7e00640 1 -- 172.21.15.60:0/2161769252 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7febbc058b50 con 0x7febd00c8360 2024-04-14T23:48:52.013 INFO:teuthology.orchestra.run.smithi060.stdout:55834574852 2024-04-14T23:48:52.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.006+0000 7fbf8a400640 1 -- 172.21.15.60:0/3372844360 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fbf6006ba60 msgr2=0x7fbf6006df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:52.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.006+0000 7fbf8a400640 1 --2- 172.21.15.60:0/3372844360 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fbf6006ba60 0x7fbf6006df20 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c036000 tx=0x7fbf6c008030 comp rx=0 tx=0).stop 2024-04-14T23:48:52.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.006+0000 7fbf8a400640 1 -- 172.21.15.60:0/3372844360 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbf840cd540 msgr2=0x7fbf840c4480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:52.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.006+0000 7fbf8a400640 1 --2- 172.21.15.60:0/3372844360 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbf840cd540 0x7fbf840c4480 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c00efc0 tx=0x7fbf7c00c490 comp rx=0 tx=0).stop 2024-04-14T23:48:52.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.007+0000 7fbf8a400640 1 -- 172.21.15.60:0/3372844360 shutdown_connections 2024-04-14T23:48:52.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.007+0000 7fbf8a400640 1 --2- 172.21.15.60:0/3372844360 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fbf6006ba60 0x7fbf6006df20 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:52.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.007+0000 7fbf8a400640 1 --2- 172.21.15.60:0/3372844360 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbf840cd540 0x7fbf840c4480 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:52.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.007+0000 7fbf8a400640 1 --2- 172.21.15.60:0/3372844360 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbf840cd160 0x7fbf84070010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:52.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.007+0000 7fbf8a400640 1 --2- 172.21.15.60:0/3372844360 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf840cb750 0x7fbf840c9f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:52.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.007+0000 7fbf8a400640 1 -- 172.21.15.60:0/3372844360 >> 172.21.15.60:0/3372844360 conn(0x7fbf84066f10 msgr2=0x7fbf84153eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:52.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.007+0000 7fbf8a400640 1 -- 172.21.15.60:0/3372844360 shutdown_connections 2024-04-14T23:48:52.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.007+0000 7fbf8a400640 1 -- 172.21.15.60:0/3372844360 wait complete. 2024-04-14T23:48:52.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.008+0000 7febd7200640 1 -- 172.21.15.60:0/2161769252 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7febb006ba90 msgr2=0x7febb006df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:52.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.008+0000 7febd7200640 1 --2- 172.21.15.60:0/2161769252 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7febb006ba90 0x7febb006df50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7febc000b880 tx=0x7febc0008040 comp rx=0 tx=0).stop 2024-04-14T23:48:52.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.008+0000 7febd7200640 1 -- 172.21.15.60:0/2161769252 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7febd00c8360 msgr2=0x7febd01600e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:52.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.008+0000 7febd7200640 1 --2- 172.21.15.60:0/2161769252 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7febd00c8360 0x7febd01600e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7febbc00e810 tx=0x7febbc00ecd0 comp rx=0 tx=0).stop 2024-04-14T23:48:52.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.008+0000 7febd7200640 1 -- 172.21.15.60:0/2161769252 shutdown_connections 2024-04-14T23:48:52.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.008+0000 7febd7200640 1 --2- 172.21.15.60:0/2161769252 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7febb006ba90 0x7febb006df50 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:52.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.008+0000 7febd7200640 1 --2- 172.21.15.60:0/2161769252 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7febd00c8740 0x7febd00c4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:52.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.008+0000 7febd7200640 1 --2- 172.21.15.60:0/2161769252 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7febd00c8360 0x7febd01600e0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:52.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.008+0000 7febd7200640 1 --2- 172.21.15.60:0/2161769252 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7febd00c75c0 0x7febd0071520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:52.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.008+0000 7febd7200640 1 -- 172.21.15.60:0/2161769252 >> 172.21.15.60:0/2161769252 conn(0x7febd00cc170 msgr2=0x7febd0156500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:52.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.009+0000 7febd7200640 1 -- 172.21.15.60:0/2161769252 shutdown_connections 2024-04-14T23:48:52.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.009+0000 7febd7200640 1 -- 172.21.15.60:0/2161769252 wait complete. 2024-04-14T23:48:52.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.019+0000 7fcec0a00640 1 -- 172.21.15.60:0/763024427 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fcebc04f540 con 0x7fcebc0b1af0 2024-04-14T23:48:52.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.020+0000 7fceb2a00640 1 -- 172.21.15.60:0/763024427 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fceb4058ba0 con 0x7fcebc0b1af0 2024-04-14T23:48:52.028 INFO:teuthology.orchestra.run.smithi060.stdout:55834574851 2024-04-14T23:48:52.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.023+0000 7fcec0a00640 1 -- 172.21.15.60:0/763024427 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fce9c06b9e0 msgr2=0x7fce9c06dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:52.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.024+0000 7fcec0a00640 1 --2- 172.21.15.60:0/763024427 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fce9c06b9e0 0x7fce9c06dea0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fceac036000 tx=0x7fceac039040 comp rx=0 tx=0).stop 2024-04-14T23:48:52.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.024+0000 7fcec0a00640 1 -- 172.21.15.60:0/763024427 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcebc0b1af0 msgr2=0x7fcebc156c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:52.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.024+0000 7fcec0a00640 1 --2- 172.21.15.60:0/763024427 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcebc0b1af0 0x7fcebc156c40 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fceb400cbe0 tx=0x7fceb400a430 comp rx=0 tx=0).stop 2024-04-14T23:48:52.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.024+0000 7fcec0a00640 1 -- 172.21.15.60:0/763024427 shutdown_connections 2024-04-14T23:48:52.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.024+0000 7fcec0a00640 1 --2- 172.21.15.60:0/763024427 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fce9c06b9e0 0x7fce9c06dea0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:52.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.024+0000 7fcec0a00640 1 --2- 172.21.15.60:0/763024427 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcebc0b1af0 0x7fcebc156c40 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:52.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.024+0000 7fcec0a00640 1 --2- 172.21.15.60:0/763024427 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcebc070d10 0x7fcebc156700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:52.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.024+0000 7fcec0a00640 1 --2- 172.21.15.60:0/763024427 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcebc06bc20 0x7fcebc1f0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:52.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.024+0000 7fcec0a00640 1 -- 172.21.15.60:0/763024427 >> 172.21.15.60:0/763024427 conn(0x7fcebc0cc2f0 msgr2=0x7fcebc0c2910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:52.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.025+0000 7fcec0a00640 1 -- 172.21.15.60:0/763024427 shutdown_connections 2024-04-14T23:48:52.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.025+0000 7fcec0a00640 1 -- 172.21.15.60:0/763024427 wait complete. 2024-04-14T23:48:52.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.027+0000 7fec4cc00640 1 -- 172.21.15.60:0/2937194093 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fec48076970 con 0x7fec48152cf0 2024-04-14T23:48:52.035 INFO:teuthology.orchestra.run.smithi060.stdout:55834574852 2024-04-14T23:48:52.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.028+0000 7fec3e000640 1 -- 172.21.15.60:0/2937194093 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fec300592f0 con 0x7fec48152cf0 2024-04-14T23:48:52.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.029+0000 7ff2b5c00640 1 -- 172.21.15.60:0/889492960 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7ff2b00c9380 con 0x7ff2b0150e00 2024-04-14T23:48:52.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.030+0000 7ff2a6a00640 1 -- 172.21.15.60:0/889492960 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ff2a80079c0 con 0x7ff2b0150e00 2024-04-14T23:48:52.038 INFO:teuthology.orchestra.run.smithi060.stdout:55834574852 2024-04-14T23:48:52.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.033+0000 7fec4cc00640 1 -- 172.21.15.60:0/2937194093 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fec2406b9e0 msgr2=0x7fec2406dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:52.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.033+0000 7fec4cc00640 1 --2- 172.21.15.60:0/2937194093 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fec2406b9e0 0x7fec2406dea0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fec40005660 tx=0x7fec40003d90 comp rx=0 tx=0).stop 2024-04-14T23:48:52.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.033+0000 7fec4cc00640 1 -- 172.21.15.60:0/2937194093 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec48152cf0 msgr2=0x7fec480c9f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:52.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.033+0000 7fec4cc00640 1 --2- 172.21.15.60:0/2937194093 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec48152cf0 0x7fec480c9f70 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fec3000b3e0 tx=0x7fec3000b8a0 comp rx=0 tx=0).stop 2024-04-14T23:48:52.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.033+0000 7fec4cc00640 1 -- 172.21.15.60:0/2937194093 shutdown_connections 2024-04-14T23:48:52.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.033+0000 7fec4cc00640 1 --2- 172.21.15.60:0/2937194093 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fec2406b9e0 0x7fec2406dea0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:52.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.033+0000 7fec4cc00640 1 --2- 172.21.15.60:0/2937194093 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec4815c9b0 0x7fec48071db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:52.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.033+0000 7fec4cc00640 1 --2- 172.21.15.60:0/2937194093 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec48152cf0 0x7fec480c9f70 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:52.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.033+0000 7fec4cc00640 1 --2- 172.21.15.60:0/2937194093 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fec480c63c0 0x7fec481f5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:52.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.033+0000 7fec4cc00640 1 -- 172.21.15.60:0/2937194093 >> 172.21.15.60:0/2937194093 conn(0x7fec480cc170 msgr2=0x7fec4806e9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:52.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.034+0000 7fec4cc00640 1 -- 172.21.15.60:0/2937194093 shutdown_connections 2024-04-14T23:48:52.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.034+0000 7fec4cc00640 1 -- 172.21.15.60:0/2937194093 wait complete. 2024-04-14T23:48:52.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.037+0000 7ff2b5c00640 1 -- 172.21.15.60:0/889492960 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff29006ba60 msgr2=0x7ff29006df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:52.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.037+0000 7ff2b5c00640 1 --2- 172.21.15.60:0/889492960 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff29006ba60 0x7ff29006df20 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff29c009370 tx=0x7ff29c008040 comp rx=0 tx=0).stop 2024-04-14T23:48:52.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.037+0000 7ff2b5c00640 1 -- 172.21.15.60:0/889492960 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2b0150e00 msgr2=0x7ff2b0156180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:52.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.037+0000 7ff2b5c00640 1 --2- 172.21.15.60:0/889492960 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2b0150e00 0x7ff2b0156180 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff2a800c730 tx=0x7ff2a800cbf0 comp rx=0 tx=0).stop 2024-04-14T23:48:52.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.038+0000 7f3b14000640 1 -- 172.21.15.60:0/562870125 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f3b0c071720 con 0x7f3b0c0c8360 2024-04-14T23:48:52.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.038+0000 7ff2b5c00640 1 -- 172.21.15.60:0/889492960 shutdown_connections 2024-04-14T23:48:52.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.038+0000 7ff2b5c00640 1 --2- 172.21.15.60:0/889492960 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff29006ba60 0x7ff29006df20 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:52.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.038+0000 7ff2b5c00640 1 --2- 172.21.15.60:0/889492960 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2b0150e00 0x7ff2b0156180 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:52.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.038+0000 7ff2b5c00640 1 --2- 172.21.15.60:0/889492960 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff2b00c70b0 0x7ff2b01edb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:52.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.038+0000 7ff2b5c00640 1 --2- 172.21.15.60:0/889492960 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff2b0072ec0 0x7ff2b013ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:52.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.038+0000 7ff2b5c00640 1 -- 172.21.15.60:0/889492960 >> 172.21.15.60:0/889492960 conn(0x7ff2b00cc170 msgr2=0x7ff2b006e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:52.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.038+0000 7ff2b5c00640 1 -- 172.21.15.60:0/889492960 shutdown_connections 2024-04-14T23:48:52.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.038+0000 7f3b09600640 1 -- 172.21.15.60:0/562870125 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f3afc057ec0 con 0x7f3b0c0c8360 2024-04-14T23:48:52.047 INFO:teuthology.orchestra.run.smithi060.stdout:55834574853 2024-04-14T23:48:52.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.038+0000 7ff2b5c00640 1 -- 172.21.15.60:0/889492960 wait complete. 2024-04-14T23:48:52.050 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.2 2024-04-14T23:48:52.051 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.0 2024-04-14T23:48:52.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.042+0000 7f3b14000640 1 -- 172.21.15.60:0/562870125 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f3af006ba10 msgr2=0x7f3af006ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:52.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.042+0000 7f3b14000640 1 --2- 172.21.15.60:0/562870125 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f3af006ba10 0x7f3af006ded0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3b000090b0 tx=0x7f3b00008030 comp rx=0 tx=0).stop 2024-04-14T23:48:52.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.042+0000 7f3b14000640 1 -- 172.21.15.60:0/562870125 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3b0c0c8360 msgr2=0x7f3b0c15cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:52.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.042+0000 7f3b14000640 1 --2- 172.21.15.60:0/562870125 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3b0c0c8360 0x7f3b0c15cdf0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3afc00c390 tx=0x7f3afc00c850 comp rx=0 tx=0).stop 2024-04-14T23:48:52.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.042+0000 7f3b14000640 1 -- 172.21.15.60:0/562870125 shutdown_connections 2024-04-14T23:48:52.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.042+0000 7f3b14000640 1 --2- 172.21.15.60:0/562870125 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f3af006ba10 0x7f3af006ded0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:52.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.042+0000 7f3b14000640 1 --2- 172.21.15.60:0/562870125 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b0c0c8740 0x7f3b0c15a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:52.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.042+0000 7f3b14000640 1 --2- 172.21.15.60:0/562870125 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3b0c0c8360 0x7f3b0c15cdf0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:52.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.042+0000 7f3b14000640 1 --2- 172.21.15.60:0/562870125 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3b0c0c75c0 0x7f3b0c1f5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:52.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.042+0000 7f3b14000640 1 -- 172.21.15.60:0/562870125 >> 172.21.15.60:0/562870125 conn(0x7f3b0c0cc170 msgr2=0x7f3b0c06e4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:52.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.042+0000 7f3b14000640 1 -- 172.21.15.60:0/562870125 shutdown_connections 2024-04-14T23:48:52.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.042+0000 7f3b14000640 1 -- 172.21.15.60:0/562870125 wait complete. 2024-04-14T23:48:52.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.042+0000 7f89ed400640 1 -- 172.21.15.60:0/538934297 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f89c0005580 con 0x7f89e8071030 2024-04-14T23:48:52.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.042+0000 7f89dea00640 1 -- 172.21.15.60:0/538934297 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f89d807b4e0 con 0x7f89e8071030 2024-04-14T23:48:52.055 INFO:teuthology.orchestra.run.smithi060.stdout:55834574853 2024-04-14T23:48:52.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.045+0000 7f89ed400640 1 -- 172.21.15.60:0/538934297 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f89c806ba90 msgr2=0x7f89c806df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:52.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.045+0000 7f89ed400640 1 --2- 172.21.15.60:0/538934297 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f89c806ba90 0x7f89c806df50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f89e81473b0 tx=0x7f89d4008040 comp rx=0 tx=0).stop 2024-04-14T23:48:52.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.045+0000 7f89ed400640 1 -- 172.21.15.60:0/538934297 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89e8071030 msgr2=0x7f89e80c5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:52.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.045+0000 7f89ed400640 1 --2- 172.21.15.60:0/538934297 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89e8071030 0x7f89e80c5020 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f89d8009870 tx=0x7f89d80046c0 comp rx=0 tx=0).stop 2024-04-14T23:48:52.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.046+0000 7f89ed400640 1 -- 172.21.15.60:0/538934297 shutdown_connections 2024-04-14T23:48:52.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.046+0000 7f89ed400640 1 --2- 172.21.15.60:0/538934297 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f89c806ba90 0x7f89c806df50 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:52.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.046+0000 7f89ed400640 1 --2- 172.21.15.60:0/538934297 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f89e80c6b70 0x7f89e814fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:52.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.046+0000 7f89ed400640 1 --2- 172.21.15.60:0/538934297 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89e8072ec0 0x7f89e81f0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:52.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.046+0000 7f89ed400640 1 --2- 172.21.15.60:0/538934297 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89e8071030 0x7f89e80c5020 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:52.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.046+0000 7f89ed400640 1 -- 172.21.15.60:0/538934297 >> 172.21.15.60:0/538934297 conn(0x7f89e80cc170 msgr2=0x7f89e80cc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:52.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.047+0000 7f89ed400640 1 -- 172.21.15.60:0/538934297 shutdown_connections 2024-04-14T23:48:52.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.047+0000 7f89ed400640 1 -- 172.21.15.60:0/538934297 wait complete. 2024-04-14T23:48:52.059 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574851 for osd.3 2024-04-14T23:48:52.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.050+0000 7fd6d8000640 1 -- 172.21.15.60:0/1854056203 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fd6d0076970 con 0x7fd6b8003b40 2024-04-14T23:48:52.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.050+0000 7fd6cd600640 1 -- 172.21.15.60:0/1854056203 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd6c804c030 con 0x7fd6b8003b40 2024-04-14T23:48:52.060 INFO:teuthology.orchestra.run.smithi060.stdout:55834574853 2024-04-14T23:48:52.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.053+0000 7fd6d8000640 1 -- 172.21.15.60:0/1854056203 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fd6b006b9c0 msgr2=0x7fd6b006de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:52.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.053+0000 7fd6d8000640 1 --2- 172.21.15.60:0/1854056203 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fd6b006b9c0 0x7fd6b006de80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd6d00ceb80 tx=0x7fd6bc008040 comp rx=0 tx=0).stop 2024-04-14T23:48:52.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.053+0000 7fd6d8000640 1 -- 172.21.15.60:0/1854056203 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd6b8003b40 msgr2=0x7fd6d00cee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:52.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.053+0000 7fd6d8000640 1 --2- 172.21.15.60:0/1854056203 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd6b8003b40 0x7fd6d00cee60 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd6b800c280 tx=0x7fd6c80012b0 comp rx=0 tx=0).stop 2024-04-14T23:48:52.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.053+0000 7fd6d8000640 1 -- 172.21.15.60:0/1854056203 shutdown_connections 2024-04-14T23:48:52.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.053+0000 7fd6d8000640 1 --2- 172.21.15.60:0/1854056203 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fd6b006b9c0 0x7fd6b006de80 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:52.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.053+0000 7fd6d8000640 1 --2- 172.21.15.60:0/1854056203 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6d0158480 0x7fd6d0158a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:52.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.053+0000 7fd6d8000640 1 --2- 172.21.15.60:0/1854056203 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd6d0158040 0x7fd6d01561b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:52.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.053+0000 7fd6d8000640 1 --2- 172.21.15.60:0/1854056203 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd6b8003b40 0x7fd6d00cee60 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:52.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.053+0000 7fd6d8000640 1 -- 172.21.15.60:0/1854056203 >> 172.21.15.60:0/1854056203 conn(0x7fd6d0066f10 msgr2=0x7fd6d015c250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:52.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.053+0000 7fd6d8000640 1 -- 172.21.15.60:0/1854056203 shutdown_connections 2024-04-14T23:48:52.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:52.053+0000 7fd6d8000640 1 -- 172.21.15.60:0/1854056203 wait complete. 2024-04-14T23:48:52.064 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574852 for osd.7 2024-04-14T23:48:52.075 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.1 2024-04-14T23:48:52.075 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574852 for osd.5 2024-04-14T23:48:52.077 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574853 for osd.6 2024-04-14T23:48:52.078 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574853 for osd.4 2024-04-14T23:48:53.051 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-14T23:48:53.052 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-14T23:48:53.060 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-14T23:48:53.066 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-14T23:48:53.076 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-14T23:48:53.077 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-14T23:48:53.078 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-14T23:48:53.078 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-14T23:48:53.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.324+0000 7fde18400640 1 Processor -- start 2024-04-14T23:48:53.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.324+0000 7fde18400640 1 -- start start 2024-04-14T23:48:53.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.326+0000 7fde18400640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fde08005e90 0x7fde080f6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:53.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.327+0000 7fde18400640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde08006990 con 0x7fde08045930 2024-04-14T23:48:53.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.327+0000 7fde18400640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde08006c70 con 0x7fde0804b090 2024-04-14T23:48:53.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.327+0000 7fde18400640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fde080075a0 con 0x7fde08005e90 2024-04-14T23:48:53.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.327+0000 7fde16400640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fde08045930 0x7fde08107300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53248/0 (socket says 172.21.15.60:53248) 2024-04-14T23:48:53.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.327+0000 7fde16400640 1 -- 172.21.15.60:0/3059217877 learned_addr learned my addr 172.21.15.60:0/3059217877 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:48:53.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.328+0000 7fde15a00640 1 --2- 172.21.15.60:0/3059217877 >> v2:172.21.15.186:3300/0 conn(0x7fde08005e90 0x7fde080f6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:53.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.329+0000 7fde15a00640 1 -- 172.21.15.60:0/3059217877 >> v1:172.21.15.186:6789/0 conn(0x7fde0804b090 legacy=0x7fde08045220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:53.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.329+0000 7fd99b000640 1 Processor -- start 2024-04-14T23:48:53.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.329+0000 7fd99b000640 1 -- start start 2024-04-14T23:48:53.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.330+0000 7fde15a00640 1 -- 172.21.15.60:0/3059217877 >> v1:172.21.15.60:6789/0 conn(0x7fde08045930 legacy=0x7fde08107300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:53.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.330+0000 7fde15a00640 1 -- 172.21.15.60:0/3059217877 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde08006f50 con 0x7fde08005e90 2024-04-14T23:48:53.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.331+0000 7fd99b000640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd9940c8360 0x7fd99415ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:53.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.331+0000 7fd99b000640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd9940c75c0 0x7fd994158e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:53.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.332+0000 7fde15a00640 1 --2- 172.21.15.60:0/3059217877 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fde08005e90 0x7fde080f6ae0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fddfc0097a0 tx=0x7fddfc02b680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=501034e1efc0480d server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:53.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.332+0000 7fd99b000640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd99415f7c0 con 0x7fd9940c8740 2024-04-14T23:48:53.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.332+0000 7fde0fe00640 1 -- 172.21.15.60:0/3059217877 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddfc037070 con 0x7fde08005e90 2024-04-14T23:48:53.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.332+0000 7fd99b000640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fd9940c6b70 con 0x7fd9940c8360 2024-04-14T23:48:53.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.332+0000 7fd99b000640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fd99406fa20 con 0x7fd9940c75c0 2024-04-14T23:48:53.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.333+0000 7fde0fe00640 1 -- 172.21.15.60:0/3059217877 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fddfc030710 con 0x7fde08005e90 2024-04-14T23:48:53.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.333+0000 7fde0fe00640 1 -- 172.21.15.60:0/3059217877 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddfc030ae0 con 0x7fde08005e90 2024-04-14T23:48:53.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.332+0000 7fd993e00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd9940c8360 0x7fd99415ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:53.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.332+0000 7fd993400640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fd9940c8740 0x7fd99406f310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.60:33936/0 (socket says 172.21.15.60:33936) 2024-04-14T23:48:53.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.332+0000 7fd993400640 1 -- 172.21.15.60:0/1364273875 learned_addr learned my addr 172.21.15.60:0/1364273875 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:48:53.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.332+0000 7fd999000640 1 --2- 172.21.15.60:0/1364273875 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd9940c75c0 0x7fd994158e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:53.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.333+0000 7fd992a00640 1 -- 172.21.15.60:0/1364273875 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4195139390 0 0) 0x7fd99415f7c0 con 0x7fd9940c8740 2024-04-14T23:48:53.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.333+0000 7fd992a00640 1 -- 172.21.15.60:0/1364273875 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd978003520 con 0x7fd9940c8740 2024-04-14T23:48:53.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.333+0000 7fd993e00640 1 -- 172.21.15.60:0/1364273875 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd9940c75c0 msgr2=0x7fd994158e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:53.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.333+0000 7fd993e00640 1 --2- 172.21.15.60:0/1364273875 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd9940c75c0 0x7fd994158e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.333+0000 7fd993e00640 1 -- 172.21.15.60:0/1364273875 >> v1:172.21.15.186:6789/0 conn(0x7fd9940c8740 legacy=0x7fd99406f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:53.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.333+0000 7fd993e00640 1 -- 172.21.15.60:0/1364273875 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd99415faa0 con 0x7fd9940c8360 2024-04-14T23:48:53.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.333+0000 7fd918a00640 1 Processor -- start 2024-04-14T23:48:53.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.334+0000 7fd918a00640 1 -- start start 2024-04-14T23:48:53.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.334+0000 7fde18400640 1 -- 172.21.15.60:0/3059217877 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fde08005e90 msgr2=0x7fde080f6ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:53.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.334+0000 7fde18400640 1 --2- 172.21.15.60:0/3059217877 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fde08005e90 0x7fde080f6ae0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fddfc0097a0 tx=0x7fddfc02b680 comp rx=0 tx=0).stop 2024-04-14T23:48:53.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.334+0000 7fd993e00640 1 --2- 172.21.15.60:0/1364273875 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9940c8360 0x7fd99415ef80 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fd9840097a0 tx=0x7fd98402b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=91786acd2b782704 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:53.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.335+0000 7fd992a00640 1 -- 172.21.15.60:0/1364273875 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd984035020 con 0x7fd9940c8360 2024-04-14T23:48:53.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.335+0000 7fd992a00640 1 -- 172.21.15.60:0/1364273875 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd984031810 con 0x7fd9940c8360 2024-04-14T23:48:53.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.335+0000 7fd992a00640 1 -- 172.21.15.60:0/1364273875 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd984031be0 con 0x7fd9940c8360 2024-04-14T23:48:53.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.336+0000 7fde18400640 1 -- 172.21.15.60:0/3059217877 shutdown_connections 2024-04-14T23:48:53.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.336+0000 7fde18400640 1 --2- 172.21.15.60:0/3059217877 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fde08005e90 0x7fde080f6ae0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.336+0000 7fde18400640 1 -- 172.21.15.60:0/3059217877 >> 172.21.15.60:0/3059217877 conn(0x7fde08050180 msgr2=0x7fde08050590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:53.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.336+0000 7fde18400640 1 -- 172.21.15.60:0/3059217877 shutdown_connections 2024-04-14T23:48:53.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.336+0000 7fde18400640 1 -- 172.21.15.60:0/3059217877 wait complete. 2024-04-14T23:48:53.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.337+0000 7fd99b000640 1 -- 172.21.15.60:0/1364273875 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9940c8360 msgr2=0x7fd99415ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:53.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.337+0000 7fd99b000640 1 --2- 172.21.15.60:0/1364273875 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9940c8360 0x7fd99415ef80 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fd9840097a0 tx=0x7fd98402b8b0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.337+0000 7fd99b000640 1 -- 172.21.15.60:0/1364273875 shutdown_connections 2024-04-14T23:48:53.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.337+0000 7fd99b000640 1 --2- 172.21.15.60:0/1364273875 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd9940c75c0 0x7fd994158e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.337+0000 7fd99b000640 1 --2- 172.21.15.60:0/1364273875 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9940c8360 0x7fd99415ef80 secure :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fd9840097a0 tx=0x7fd98402b8b0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.337+0000 7fd99b000640 1 -- 172.21.15.60:0/1364273875 >> 172.21.15.60:0/1364273875 conn(0x7fd9940cc170 msgr2=0x7fd9940cc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:53.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.335+0000 7fd918a00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd91406f0b0 0x7fd91415aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:53.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.335+0000 7fd918a00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd9140c9680 0x7fd9140c89f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:53.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.335+0000 7fd918a00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9140720b0 con 0x7fd91406ecd0 2024-04-14T23:48:53.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.335+0000 7fd918a00640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fd9140c8f30 con 0x7fd9140c9680 2024-04-14T23:48:53.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.335+0000 7fd918a00640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fd914072350 con 0x7fd91406f0b0 2024-04-14T23:48:53.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.336+0000 7fd911400640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd91406f0b0 0x7fd91415aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:53.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.336+0000 7fd911400640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd91406f0b0 0x7fd91415aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:57830/0 (socket says 172.21.15.60:57830) 2024-04-14T23:48:53.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.336+0000 7fd911400640 1 -- 172.21.15.60:0/4216087083 learned_addr learned my addr 172.21.15.60:0/4216087083 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:53.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.336+0000 7fd912800640 1 --2- 172.21.15.60:0/4216087083 >> v2:172.21.15.60:3300/0 conn(0x7fd9140c9680 0x7fd9140c89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:53.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.336+0000 7fd911400640 1 -- 172.21.15.60:0/4216087083 >> v2:172.21.15.60:3300/0 conn(0x7fd9140c9680 msgr2=0x7fd9140c89f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:53.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.337+0000 7fd910a00640 1 -- 172.21.15.60:0/4216087083 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2855102313 0 0) 0x7fd9140720b0 con 0x7fd91406ecd0 2024-04-14T23:48:53.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.337+0000 7fd911400640 1 --2- 172.21.15.60:0/4216087083 >> v2:172.21.15.60:3300/0 conn(0x7fd9140c9680 0x7fd9140c89f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.337+0000 7fd911400640 1 -- 172.21.15.60:0/4216087083 >> v1:172.21.15.60:6789/0 conn(0x7fd91406ecd0 legacy=0x7fd9140c7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:53.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.337+0000 7fd911400640 1 -- 172.21.15.60:0/4216087083 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd914072630 con 0x7fd91406f0b0 2024-04-14T23:48:53.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.338+0000 7fd912800640 1 --2- 172.21.15.60:0/4216087083 >> v2:172.21.15.60:3300/0 conn(0x7fd9140c9680 0x7fd9140c89f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-14T23:48:53.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.338+0000 7fd99b000640 1 -- 172.21.15.60:0/1364273875 shutdown_connections 2024-04-14T23:48:53.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.338+0000 7fd911400640 1 --2- 172.21.15.60:0/4216087083 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd91406f0b0 0x7fd91415aba0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd9140c7410 tx=0x7fd90c02c010 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f335018c6d253e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:53.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.339+0000 7fd99b000640 1 -- 172.21.15.60:0/1364273875 wait complete. 2024-04-14T23:48:53.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.339+0000 7fd910a00640 1 -- 172.21.15.60:0/4216087083 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd90c02cbb0 con 0x7fd91406f0b0 2024-04-14T23:48:53.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.339+0000 7fd910a00640 1 -- 172.21.15.60:0/4216087083 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd90c02cd50 con 0x7fd91406f0b0 2024-04-14T23:48:53.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.339+0000 7fd910a00640 1 -- 172.21.15.60:0/4216087083 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd90c034610 con 0x7fd91406f0b0 2024-04-14T23:48:53.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.339+0000 7fde18400640 1 Processor -- start 2024-04-14T23:48:53.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.340+0000 7fd918a00640 1 -- 172.21.15.60:0/4216087083 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd91406f0b0 msgr2=0x7fd91415aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:53.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.340+0000 7fd918a00640 1 --2- 172.21.15.60:0/4216087083 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd91406f0b0 0x7fd91415aba0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd9140c7410 tx=0x7fd90c02c010 comp rx=0 tx=0).stop 2024-04-14T23:48:53.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.340+0000 7fde18400640 1 -- start start 2024-04-14T23:48:53.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.340+0000 7fd918a00640 1 -- 172.21.15.60:0/4216087083 shutdown_connections 2024-04-14T23:48:53.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.340+0000 7fd918a00640 1 --2- 172.21.15.60:0/4216087083 >> v2:172.21.15.60:3300/0 conn(0x7fd9140c9680 0x7fd9140c89f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.340+0000 7fd918a00640 1 --2- 172.21.15.60:0/4216087083 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd91406f0b0 0x7fd91415aba0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.340+0000 7fd918a00640 1 -- 172.21.15.60:0/4216087083 >> 172.21.15.60:0/4216087083 conn(0x7fd914066f10 msgr2=0x7fd914067320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:53.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.341+0000 7fd99b000640 1 Processor -- start 2024-04-14T23:48:53.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.342+0000 7fd99b000640 1 -- start start 2024-04-14T23:48:53.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.343+0000 7fd99b000640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd9940c75c0 0x7fd9941f8870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:53.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.343+0000 7fd99b000640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9940c8740 0x7fd99414a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:53.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.343+0000 7fd99b000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd99415cdf0 0x7fd9941f4f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:53.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.343+0000 7fd99b000640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fd9940c70a0 con 0x7fd9940c8740 2024-04-14T23:48:53.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.343+0000 7fd99b000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd9940c6dc0 con 0x7fd99415cdf0 2024-04-14T23:48:53.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.343+0000 7fd99b000640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fd99415f4c0 con 0x7fd9940c75c0 2024-04-14T23:48:53.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.344+0000 7fd993e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd9940c75c0 0x7fd9941f8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:53.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.344+0000 7fd993e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd9940c75c0 0x7fd9941f8870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.60:51042/0 (socket says 172.21.15.60:51042) 2024-04-14T23:48:53.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.344+0000 7fd993e00640 1 -- 172.21.15.60:0/3323363356 learned_addr learned my addr 172.21.15.60:0/3323363356 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:53.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.344+0000 7fd993400640 1 --2- 172.21.15.60:0/3323363356 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9940c8740 0x7fd99414a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:53.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.344+0000 7fd999000640 1 --2- 172.21.15.60:0/3323363356 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd99415cdf0 0x7fd9941f4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:53.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.340+0000 7fd918a00640 1 -- 172.21.15.60:0/4216087083 shutdown_connections 2024-04-14T23:48:53.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.344+0000 7fd999000640 1 -- 172.21.15.60:0/3323363356 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd9940c75c0 msgr2=0x7fd9941f8870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:53.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.344+0000 7fde18400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fde08005e90 0x7fde080f93b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:53.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.344+0000 7fde18400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fde08045930 0x7fde080f2f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:53.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.344+0000 7fde18400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fde0804b090 0x7fde080f3450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:53.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.344+0000 7fde18400640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fde0804c720 con 0x7fde0804b090 2024-04-14T23:48:53.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.344+0000 7fde18400640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fde080f7020 con 0x7fde08005e90 2024-04-14T23:48:53.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.344+0000 7fde18400640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fde0804c8a0 con 0x7fde08045930 2024-04-14T23:48:53.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.345+0000 7fde15a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fde08005e90 0x7fde080f93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:53.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.345+0000 7fde15a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fde08005e90 0x7fde080f93b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:57834/0 (socket says 172.21.15.60:57834) 2024-04-14T23:48:53.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.345+0000 7fde15a00640 1 -- 172.21.15.60:0/2847892484 learned_addr learned my addr 172.21.15.60:0/2847892484 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:53.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.344+0000 7fd999000640 1 --2- 172.21.15.60:0/3323363356 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd9940c75c0 0x7fd9941f8870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.344+0000 7fd999000640 1 -- 172.21.15.60:0/3323363356 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9940c8740 msgr2=0x7fd99414a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:53.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.340+0000 7fd918a00640 1 -- 172.21.15.60:0/4216087083 wait complete. 2024-04-14T23:48:53.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.343+0000 7fd918a00640 1 Processor -- start 2024-04-14T23:48:53.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.345+0000 7fde16400640 1 --2- 172.21.15.60:0/2847892484 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fde0804b090 0x7fde080f3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:53.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.346+0000 7fd918a00640 1 -- start start 2024-04-14T23:48:53.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.345+0000 7fd999000640 1 --2- 172.21.15.60:0/3323363356 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9940c8740 0x7fd99414a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.346+0000 7fde15000640 1 --2- 172.21.15.60:0/2847892484 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fde08045930 0x7fde080f2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:53.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.345+0000 7fd999000640 1 -- 172.21.15.60:0/3323363356 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd99415c360 con 0x7fd99415cdf0 2024-04-14T23:48:53.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.345+0000 7fd999000640 1 --2- 172.21.15.60:0/3323363356 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd99415cdf0 0x7fd9941f4f20 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd98c009d90 tx=0x7fd98c00c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:53.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.346+0000 7fd98be00640 1 -- 172.21.15.60:0/3323363356 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd98c018020 con 0x7fd99415cdf0 2024-04-14T23:48:53.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.346+0000 7fd98be00640 1 -- 172.21.15.60:0/3323363356 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd98c014480 con 0x7fd99415cdf0 2024-04-14T23:48:53.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.346+0000 7fd98be00640 1 -- 172.21.15.60:0/3323363356 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd98c014870 con 0x7fd99415cdf0 2024-04-14T23:48:53.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.346+0000 7fd993e00640 1 --2- 172.21.15.60:0/3323363356 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd9940c75c0 0x7fd9941f8870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-14T23:48:53.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.346+0000 7fd99b000640 1 -- 172.21.15.60:0/3323363356 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd99415c8e0 con 0x7fd99415cdf0 2024-04-14T23:48:53.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.346+0000 7fd99b000640 1 -- 172.21.15.60:0/3323363356 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd99415c600 con 0x7fd99415cdf0 2024-04-14T23:48:53.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.347+0000 7fde15000640 1 -- 172.21.15.60:0/2847892484 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fde08005e90 msgr2=0x7fde080f93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:53.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.347+0000 7fde15000640 1 --2- 172.21.15.60:0/2847892484 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fde08005e90 0x7fde080f93b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.347+0000 7fde15000640 1 -- 172.21.15.60:0/2847892484 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fde0804b090 msgr2=0x7fde080f3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:53.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.347+0000 7fde15000640 1 --2- 172.21.15.60:0/2847892484 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fde0804b090 0x7fde080f3450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.348+0000 7fde15000640 1 -- 172.21.15.60:0/2847892484 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde0804fdf0 con 0x7fde08045930 2024-04-14T23:48:53.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.347+0000 7fd993400640 1 --2- 172.21.15.60:0/3323363356 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9940c8740 0x7fd99414a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:48:53.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.348+0000 7fd918a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd91406ecd0 0x7fd91414e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:53.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.348+0000 7fd918a00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd91406f0b0 0x7fd9141f39e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:53.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.348+0000 7fd918a00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9140c9680 0x7fd914148900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:53.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.348+0000 7fd918a00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fd9140c9b70 con 0x7fd9140c9680 2024-04-14T23:48:53.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.349+0000 7fd918a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd9140c7880 con 0x7fd91406ecd0 2024-04-14T23:48:53.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.349+0000 7fd918a00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fd9140c6c00 con 0x7fd91406f0b0 2024-04-14T23:48:53.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.349+0000 7fd911400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd91406f0b0 0x7fd9141f39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:53.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.350+0000 7fde15000640 1 --2- 172.21.15.60:0/2847892484 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fde08045930 0x7fde080f2f10 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fde0400c550 tx=0x7fde0400ca10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:53.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.349+0000 7fd99b000640 1 -- 172.21.15.60:0/3323363356 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd994076970 con 0x7fd99415cdf0 2024-04-14T23:48:53.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.349+0000 7fd911e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd91406ecd0 0x7fd91414e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:53.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.349+0000 7fd911e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd91406ecd0 0x7fd91414e7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:57836/0 (socket says 172.21.15.60:57836) 2024-04-14T23:48:53.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.349+0000 7fd911e00640 1 -- 172.21.15.60:0/3619087474 learned_addr learned my addr 172.21.15.60:0/3619087474 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:53.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.349+0000 7fd912800640 1 --2- 172.21.15.60:0/3619087474 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9140c9680 0x7fd914148900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:53.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.350+0000 7fd912800640 1 -- 172.21.15.60:0/3619087474 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd91406f0b0 msgr2=0x7fd9141f39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:53.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.350+0000 7fd912800640 1 --2- 172.21.15.60:0/3619087474 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd91406f0b0 0x7fd9141f39e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.350+0000 7fd912800640 1 -- 172.21.15.60:0/3619087474 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd91406ecd0 msgr2=0x7fd91414e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:53.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.350+0000 7fd912800640 1 --2- 172.21.15.60:0/3619087474 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd91406ecd0 0x7fd91414e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.350+0000 7fd912800640 1 -- 172.21.15.60:0/3619087474 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd914152a70 con 0x7fd9140c9680 2024-04-14T23:48:53.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.351+0000 7fd98be00640 1 -- 172.21.15.60:0/3323363356 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7fd98c005d10 con 0x7fd99415cdf0 2024-04-14T23:48:53.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.351+0000 7fd911400640 1 --2- 172.21.15.60:0/3619087474 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd91406f0b0 0x7fd9141f39e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-14T23:48:53.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.350+0000 7fde0d600640 1 -- 172.21.15.60:0/2847892484 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde04017070 con 0x7fde08045930 2024-04-14T23:48:53.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.351+0000 7fd911e00640 1 --2- 172.21.15.60:0/3619087474 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd91406ecd0 0x7fd91414e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-14T23:48:53.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.351+0000 7fd912800640 1 --2- 172.21.15.60:0/3619087474 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9140c9680 0x7fd914148900 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd90400c5c0 tx=0x7fd90400ca80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:53.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.351+0000 7fde0d600640 1 -- 172.21.15.60:0/2847892484 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde040035f0 con 0x7fde08045930 2024-04-14T23:48:53.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.351+0000 7fde0d600640 1 -- 172.21.15.60:0/2847892484 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde04017070 con 0x7fde08045930 2024-04-14T23:48:53.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.351+0000 7fd90a000640 1 -- 172.21.15.60:0/3619087474 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd904015070 con 0x7fd9140c9680 2024-04-14T23:48:53.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.352+0000 7fd918a00640 1 -- 172.21.15.60:0/3619087474 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd914152d20 con 0x7fd9140c9680 2024-04-14T23:48:53.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.352+0000 7fd918a00640 1 -- 172.21.15.60:0/3619087474 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd914152550 con 0x7fd9140c9680 2024-04-14T23:48:53.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.352+0000 7fd90a000640 1 -- 172.21.15.60:0/3619087474 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd904011800 con 0x7fd9140c9680 2024-04-14T23:48:53.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.352+0000 7fd90a000640 1 -- 172.21.15.60:0/3619087474 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd904015070 con 0x7fd9140c9680 2024-04-14T23:48:53.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.353+0000 7fde18400640 1 -- 172.21.15.60:0/2847892484 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde081a7010 con 0x7fde08045930 2024-04-14T23:48:53.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.353+0000 7fde18400640 1 -- 172.21.15.60:0/2847892484 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fde081a75d0 con 0x7fde08045930 2024-04-14T23:48:53.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.353+0000 7fd98be00640 1 --2- 172.21.15.60:0/3323363356 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fd97406ba90 0x7fd97406df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:53.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.354+0000 7fd98be00640 1 -- 172.21.15.60:0/3323363356 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fd98c08b590 con 0x7fd99415cdf0 2024-04-14T23:48:53.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.355+0000 7fd918a00640 1 -- 172.21.15.60:0/3619087474 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8ec005290 con 0x7fd9140c9680 2024-04-14T23:48:53.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.354+0000 7fd993e00640 1 --2- 172.21.15.60:0/3323363356 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fd97406ba90 0x7fd97406df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:53.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.356+0000 7fd993e00640 1 --2- 172.21.15.60:0/3323363356 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fd97406ba90 0x7fd97406df50 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd98402b8f0 tx=0x7fd984008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:53.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.357+0000 7fd98be00640 1 -- 172.21.15.60:0/3323363356 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd98c022030 con 0x7fd99415cdf0 2024-04-14T23:48:53.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.360+0000 7fde02a00640 1 -- 172.21.15.60:0/2847892484 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdde4005290 con 0x7fde08045930 2024-04-14T23:48:53.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.360+0000 7fde0d600640 1 -- 172.21.15.60:0/2847892484 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7fde040037a0 con 0x7fde08045930 2024-04-14T23:48:53.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.362+0000 7fde0d600640 1 --2- 172.21.15.60:0/2847892484 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fddf006bb90 0x7fddf006e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:53.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.362+0000 7fde0d600640 1 -- 172.21.15.60:0/2847892484 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fde040897a0 con 0x7fde08045930 2024-04-14T23:48:53.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.359+0000 7fd90a000640 1 -- 172.21.15.60:0/3619087474 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7fd904018070 con 0x7fd9140c9680 2024-04-14T23:48:53.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.360+0000 7fd90a000640 1 --2- 172.21.15.60:0/3619087474 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fd8f406bbc0 0x7fd8f406e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:53.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.360+0000 7fd90a000640 1 -- 172.21.15.60:0/3619087474 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fd90408a240 con 0x7fd9140c9680 2024-04-14T23:48:53.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.361+0000 7fd911e00640 1 --2- 172.21.15.60:0/3619087474 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fd8f406bbc0 0x7fd8f406e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:53.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.362+0000 7fd911e00640 1 --2- 172.21.15.60:0/3619087474 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fd8f406bbc0 0x7fd8f406e080 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd900001860 tx=0x7fd900009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:53.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.365+0000 7fde15a00640 1 --2- 172.21.15.60:0/2847892484 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fddf006bb90 0x7fddf006e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:53.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.366+0000 7fd90a000640 1 -- 172.21.15.60:0/3619087474 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd904058e00 con 0x7fd9140c9680 2024-04-14T23:48:53.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.367+0000 7fde15a00640 1 --2- 172.21.15.60:0/2847892484 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fddf006bb90 0x7fddf006e050 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fddfc02b6c0 tx=0x7fddfc008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:53.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.369+0000 7fde0d600640 1 -- 172.21.15.60:0/2847892484 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fde04057d50 con 0x7fde08045930 2024-04-14T23:48:53.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.413+0000 7f482aa00640 1 Processor -- start 2024-04-14T23:48:53.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.413+0000 7f1cc7e00640 1 Processor -- start 2024-04-14T23:48:53.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.413+0000 7f1cc7e00640 1 -- start start 2024-04-14T23:48:53.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.415+0000 7f1cc7e00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1cc80c5470 0x7f1cc8071ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:53.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.415+0000 7f1cc7e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1cc80c5ae0 0x7f1cc8162d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:53.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.415+0000 7f1cc7e00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1cc80c7ab0 con 0x7f1cc80c6800 2024-04-14T23:48:53.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.415+0000 7f1cc7e00640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f1cc80c7240 con 0x7f1cc80c5470 2024-04-14T23:48:53.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.415+0000 7f1cc7e00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f1cc8163400 con 0x7f1cc80c5ae0 2024-04-14T23:48:53.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.416+0000 7f1cc6a00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1cc80c5470 0x7f1cc8071ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:53.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.416+0000 7f1cc6000640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f1cc80c6800 0x7f1cc8162610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53272/0 (socket says 172.21.15.60:53272) 2024-04-14T23:48:53.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.416+0000 7f1cc6000640 1 -- 172.21.15.60:0/2540194074 learned_addr learned my addr 172.21.15.60:0/2540194074 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:48:53.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.414+0000 7f482aa00640 1 -- start start 2024-04-14T23:48:53.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.415+0000 7f482aa00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f481c0dd3c0 0x7f481c046460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:53.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.416+0000 7f482aa00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f481c0e37a0 0x7f481c04be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:53.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.416+0000 7f482a000640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f481c0e37a0 0x7f481c04be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:53.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.416+0000 7f482a000640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f481c0e37a0 0x7f481c04be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41766/0 (socket says 172.21.15.60:41766) 2024-04-14T23:48:53.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.416+0000 7f482a000640 1 -- 172.21.15.60:0/3623017654 learned_addr learned my addr 172.21.15.60:0/3623017654 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:53.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.416+0000 7f482aa00640 1 -- 172.21.15.60:0/3623017654 --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f481c0dd7a0 con 0x7f481c04ba60 2024-04-14T23:48:53.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.417+0000 7f1cc7400640 1 --2- 172.21.15.60:0/2540194074 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1cc80c5ae0 0x7f1cc8162d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:53.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.417+0000 7f1cc5600640 1 -- 172.21.15.60:0/2540194074 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2333998127 0 0) 0x7f1cc80c7ab0 con 0x7f1cc80c6800 2024-04-14T23:48:53.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.417+0000 7f1cc5600640 1 -- 172.21.15.60:0/2540194074 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1cac003520 con 0x7f1cc80c6800 2024-04-14T23:48:53.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.417+0000 7f1cc6a00640 1 -- 172.21.15.60:0/2540194074 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1cc80c5ae0 msgr2=0x7f1cc8162d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:53.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.417+0000 7f1cc6a00640 1 --2- 172.21.15.60:0/2540194074 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1cc80c5ae0 0x7f1cc8162d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.417+0000 7f1cc6a00640 1 -- 172.21.15.60:0/2540194074 >> v1:172.21.15.60:6789/0 conn(0x7f1cc80c6800 legacy=0x7f1cc8162610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:53.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.417+0000 7f1cc6a00640 1 -- 172.21.15.60:0/2540194074 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cc80c7d70 con 0x7f1cc80c5470 2024-04-14T23:48:53.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.418+0000 7f4829600640 1 --2- 172.21.15.60:0/3623017654 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f481c0dd3c0 0x7f481c046460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:53.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.417+0000 7f482aa00640 1 -- 172.21.15.60:0/3623017654 --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f481c045450 con 0x7f481c0e37a0 2024-04-14T23:48:53.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.418+0000 7f482aa00640 1 -- 172.21.15.60:0/3623017654 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f481c045fd0 con 0x7f481c0dd3c0 2024-04-14T23:48:53.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.419+0000 7f4823e00640 1 -- 172.21.15.60:0/3623017654 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3543818239 0 0) 0x7f4818001110 con 0x7f481c04ba60 2024-04-14T23:48:53.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.420+0000 7f1cc6a00640 1 --2- 172.21.15.60:0/2540194074 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cc80c5470 0x7f1cc8071ec0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1cc0009d90 tx=0x7f1cc002bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92872bb90674b052 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:53.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.420+0000 7f1cc5600640 1 -- 172.21.15.60:0/2540194074 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cc0039070 con 0x7f1cc80c5470 2024-04-14T23:48:53.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.421+0000 7f1cc5600640 1 -- 172.21.15.60:0/2540194074 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1cc002ccf0 con 0x7f1cc80c5470 2024-04-14T23:48:53.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.421+0000 7f1cc5600640 1 -- 172.21.15.60:0/2540194074 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cc0034660 con 0x7f1cc80c5470 2024-04-14T23:48:53.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.419+0000 7f4823e00640 1 -- 172.21.15.60:0/3623017654 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f480c003520 con 0x7f481c04ba60 2024-04-14T23:48:53.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.419+0000 7f482a000640 1 -- 172.21.15.60:0/3623017654 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f481c0dd3c0 msgr2=0x7f481c046460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:53.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.419+0000 7f482a000640 1 --2- 172.21.15.60:0/3623017654 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f481c0dd3c0 0x7f481c046460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.419+0000 7f482a000640 1 -- 172.21.15.60:0/3623017654 >> v1:172.21.15.60:6789/0 conn(0x7f481c04ba60 legacy=0x7f481c0e3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:53.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.419+0000 7f482a000640 1 -- 172.21.15.60:0/3623017654 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f481c0e3e90 con 0x7f481c0e37a0 2024-04-14T23:48:53.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.421+0000 7f482a000640 1 --2- 172.21.15.60:0/3623017654 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f481c0e37a0 0x7f481c04be80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4810009870 tx=0x7f481002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=730d5a279700042d server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:53.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.422+0000 7f4823e00640 1 -- 172.21.15.60:0/3623017654 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4810035020 con 0x7f481c0e37a0 2024-04-14T23:48:53.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.422+0000 7f4823e00640 1 -- 172.21.15.60:0/3623017654 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4810031870 con 0x7f481c0e37a0 2024-04-14T23:48:53.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.422+0000 7f4823e00640 1 -- 172.21.15.60:0/3623017654 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4810031c80 con 0x7f481c0e37a0 2024-04-14T23:48:53.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.422+0000 7f1cc7e00640 1 -- 172.21.15.60:0/2540194074 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cc80c5470 msgr2=0x7f1cc8071ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:53.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.422+0000 7f1cc7e00640 1 --2- 172.21.15.60:0/2540194074 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cc80c5470 0x7f1cc8071ec0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1cc0009d90 tx=0x7f1cc002bf40 comp rx=0 tx=0).stop 2024-04-14T23:48:53.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.423+0000 7f482aa00640 1 -- 172.21.15.60:0/3623017654 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f481c0e37a0 msgr2=0x7f481c04be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:53.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.423+0000 7f482aa00640 1 --2- 172.21.15.60:0/3623017654 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f481c0e37a0 0x7f481c04be80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4810009870 tx=0x7f481002b910 comp rx=0 tx=0).stop 2024-04-14T23:48:53.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.423+0000 7f482aa00640 1 -- 172.21.15.60:0/3623017654 shutdown_connections 2024-04-14T23:48:53.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.423+0000 7f482aa00640 1 --2- 172.21.15.60:0/3623017654 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f481c0e37a0 0x7f481c04be80 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.423+0000 7f482aa00640 1 --2- 172.21.15.60:0/3623017654 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f481c0dd3c0 0x7f481c046460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.423+0000 7f482aa00640 1 -- 172.21.15.60:0/3623017654 >> 172.21.15.60:0/3623017654 conn(0x7f481c050180 msgr2=0x7f481c050590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:53.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.423+0000 7f482aa00640 1 -- 172.21.15.60:0/3623017654 shutdown_connections 2024-04-14T23:48:53.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.424+0000 7f1cc7e00640 1 -- 172.21.15.60:0/2540194074 shutdown_connections 2024-04-14T23:48:53.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.424+0000 7f1cc7e00640 1 --2- 172.21.15.60:0/2540194074 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1cc80c5ae0 0x7f1cc8162d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.424+0000 7f1cc7e00640 1 --2- 172.21.15.60:0/2540194074 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cc80c5470 0x7f1cc8071ec0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.424+0000 7f1cc7e00640 1 -- 172.21.15.60:0/2540194074 >> 172.21.15.60:0/2540194074 conn(0x7f1cc80cc2f0 msgr2=0x7f1cc80cc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:53.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.423+0000 7f482aa00640 1 -- 172.21.15.60:0/3623017654 wait complete. 2024-04-14T23:48:53.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.425+0000 7f1cc7e00640 1 -- 172.21.15.60:0/2540194074 shutdown_connections 2024-04-14T23:48:53.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.425+0000 7f1cc7e00640 1 -- 172.21.15.60:0/2540194074 wait complete. 2024-04-14T23:48:53.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.426+0000 7f482aa00640 1 Processor -- start 2024-04-14T23:48:53.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.426+0000 7f482aa00640 1 -- start start 2024-04-14T23:48:53.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.428+0000 7f482aa00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f481c04ba60 0x7f481c178de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:53.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.428+0000 7f482aa00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f481c0dd3c0 0x7f481c04ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:53.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.429+0000 7f4829600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f481c04ba60 0x7f481c178de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:53.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.429+0000 7f4829600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f481c04ba60 0x7f481c178de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:57838/0 (socket says 172.21.15.60:57838) 2024-04-14T23:48:53.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.428+0000 7f1cc7e00640 1 Processor -- start 2024-04-14T23:48:53.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.428+0000 7f1cc7e00640 1 -- start start 2024-04-14T23:48:53.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.429+0000 7f4828c00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f481c0dd3c0 0x7f481c04ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:53.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.429+0000 7f219e600640 1 Processor -- start 2024-04-14T23:48:53.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.429+0000 7f4829600640 1 -- 172.21.15.60:0/1603187010 learned_addr learned my addr 172.21.15.60:0/1603187010 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:53.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.429+0000 7f482aa00640 1 --2- 172.21.15.60:0/1603187010 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f481c0e37a0 0x7f481c04b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:53.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.429+0000 7f219e600640 1 -- start start 2024-04-14T23:48:53.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.430+0000 7ff6c3600640 1 Processor -- start 2024-04-14T23:48:53.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.430+0000 7f482aa00640 1 -- 172.21.15.60:0/1603187010 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f481c045980 con 0x7f481c0dd3c0 2024-04-14T23:48:53.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.430+0000 7f482aa00640 1 -- 172.21.15.60:0/1603187010 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f481c0e3c30 con 0x7f481c04ba60 2024-04-14T23:48:53.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.430+0000 7f482a000640 1 --2- 172.21.15.60:0/1603187010 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f481c0e37a0 0x7f481c04b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:53.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.429+0000 7f1cc7e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1cc80c5470 0x7f1cc80c2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:53.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.429+0000 7f1cc7e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cc80c5ae0 0x7f1cc80c0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:53.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.429+0000 7f1cc7e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1cc80c6800 0x7f1cc815a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:53.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.430+0000 7f482aa00640 1 -- 172.21.15.60:0/1603187010 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f481c045c60 con 0x7f481c0e37a0 2024-04-14T23:48:53.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.430+0000 7f1cc7e00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f1cc80c86a0 con 0x7f1cc80c5ae0 2024-04-14T23:48:53.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.430+0000 7ff6c3600640 1 -- start start 2024-04-14T23:48:53.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.430+0000 7f1cc7e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1cc80c5850 con 0x7f1cc80c5470 2024-04-14T23:48:53.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.430+0000 7f1cc6a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1cc80c5470 0x7f1cc80c2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:53.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.430+0000 7f1cc6a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1cc80c5470 0x7f1cc80c2ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:57852/0 (socket says 172.21.15.60:57852) 2024-04-14T23:48:53.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.430+0000 7f1cc6a00640 1 -- 172.21.15.60:0/3126452961 learned_addr learned my addr 172.21.15.60:0/3126452961 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:53.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.430+0000 7f1cc7e00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f1cc80c8820 con 0x7f1cc80c6800 2024-04-14T23:48:53.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.430+0000 7fdb21e00640 1 Processor -- start 2024-04-14T23:48:53.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.430+0000 7f1cc7400640 1 --2- 172.21.15.60:0/3126452961 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1cc80c6800 0x7f1cc815a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:53.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.431+0000 7f1cc7400640 1 -- 172.21.15.60:0/3126452961 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1cc80c5470 msgr2=0x7f1cc80c2ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:53.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.431+0000 7f1cc7400640 1 --2- 172.21.15.60:0/3126452961 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1cc80c5470 0x7f1cc80c2ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.431+0000 7f1cc7400640 1 -- 172.21.15.60:0/3126452961 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cc80c5ae0 msgr2=0x7f1cc80c0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:53.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.431+0000 7fdb21e00640 1 -- start start 2024-04-14T23:48:53.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.431+0000 7f1cc6000640 1 --2- 172.21.15.60:0/3126452961 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cc80c5ae0 0x7f1cc80c0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:53.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.431+0000 7f219e600640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f21980c5ce0 0x7f2198158f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:53.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.431+0000 7f219e600640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f219806c720 0x7f21981594a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:53.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.431+0000 7f482a000640 1 -- 172.21.15.60:0/1603187010 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f481c04ba60 msgr2=0x7f481c178de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:53.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.431+0000 7f482a000640 1 --2- 172.21.15.60:0/1603187010 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f481c04ba60 0x7f481c178de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.431+0000 7f482a000640 1 -- 172.21.15.60:0/1603187010 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f481c0dd3c0 msgr2=0x7f481c04ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:53.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.431+0000 7f482a000640 1 --2- 172.21.15.60:0/1603187010 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f481c0dd3c0 0x7f481c04ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.431+0000 7f482a000640 1 -- 172.21.15.60:0/1603187010 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f481c0d8c10 con 0x7f481c0e37a0 2024-04-14T23:48:53.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.431+0000 7f219e600640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21980c6220 con 0x7f2198152be0 2024-04-14T23:48:53.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.431+0000 7f219e600640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f2198158ad0 con 0x7f21980c5ce0 2024-04-14T23:48:53.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.431+0000 7f219e600640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f21980c8160 con 0x7f219806c720 2024-04-14T23:48:53.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.432+0000 7f219d200640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f2198152be0 0x7f219806ebc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.60:33948/0 (socket says 172.21.15.60:33948) 2024-04-14T23:48:53.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.432+0000 7f219d200640 1 -- 172.21.15.60:0/1143501439 learned_addr learned my addr 172.21.15.60:0/1143501439 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:48:53.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.432+0000 7f219dc00640 1 --2- 172.21.15.60:0/1143501439 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f219806c720 0x7f21981594a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:53.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.432+0000 7f482a000640 1 --2- 172.21.15.60:0/1603187010 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f481c0e37a0 0x7f481c04b060 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4810033000 tx=0x7f48100046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:53.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.432+0000 7f2197e00640 1 --2- 172.21.15.60:0/1143501439 >> v2:172.21.15.60:3300/0 conn(0x7f21980c5ce0 0x7f2198158f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:53.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.431+0000 7f1cc7400640 1 --2- 172.21.15.60:0/3126452961 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cc80c5ae0 0x7f1cc80c0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.431+0000 7f1cc7400640 1 -- 172.21.15.60:0/3126452961 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cc8160190 con 0x7f1cc80c6800 2024-04-14T23:48:53.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.432+0000 7f4821600640 1 -- 172.21.15.60:0/1603187010 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4810040070 con 0x7f481c0e37a0 2024-04-14T23:48:53.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.432+0000 7fdb21e00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fdb1c0cd160 0x7fdb1c15ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:53.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.432+0000 7f1cc6a00640 1 --2- 172.21.15.60:0/3126452961 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1cc80c5470 0x7f1cc80c2ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-14T23:48:53.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.433+0000 7f4821600640 1 -- 172.21.15.60:0/1603187010 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4810031e50 con 0x7f481c0e37a0 2024-04-14T23:48:53.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.433+0000 7fdb21e00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdb1c0cd540 0x7fdb1c0b3190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:53.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.433+0000 7fdb1aa00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fdb1c0cd160 0x7fdb1c15ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:53.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.433+0000 7fdb1aa00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fdb1c0cd160 0x7fdb1c15ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:57868/0 (socket says 172.21.15.60:57868) 2024-04-14T23:48:53.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.432+0000 7f1cc6000640 1 --2- 172.21.15.60:0/3126452961 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cc80c5ae0 0x7f1cc80c0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-14T23:48:53.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.433+0000 7f482aa00640 1 -- 172.21.15.60:0/1603187010 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f481c0e30d0 con 0x7f481c0e37a0 2024-04-14T23:48:53.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.433+0000 7f482aa00640 1 -- 172.21.15.60:0/1603187010 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f481c17c4e0 con 0x7f481c0e37a0 2024-04-14T23:48:53.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.433+0000 7f219dc00640 1 -- 172.21.15.60:0/1143501439 >> v2:172.21.15.60:3300/0 conn(0x7f21980c5ce0 msgr2=0x7f2198158f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:53.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.433+0000 7f219dc00640 1 --2- 172.21.15.60:0/1143501439 >> v2:172.21.15.60:3300/0 conn(0x7f21980c5ce0 0x7f2198158f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.433+0000 7f219dc00640 1 -- 172.21.15.60:0/1143501439 >> v1:172.21.15.186:6789/0 conn(0x7f2198152be0 legacy=0x7f219806ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:53.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.433+0000 7f219dc00640 1 -- 172.21.15.60:0/1143501439 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21980c8440 con 0x7f219806c720 2024-04-14T23:48:53.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.432+0000 7f1cc7400640 1 --2- 172.21.15.60:0/3126452961 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1cc80c6800 0x7f1cc815a8a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1cb000d560 tx=0x7f1cb000da20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:53.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.433+0000 7fdb1aa00640 1 -- 172.21.15.60:0/1216392317 learned_addr learned my addr 172.21.15.60:0/1216392317 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:53.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.433+0000 7fdb21e00640 1 -- 172.21.15.60:0/1216392317 --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb1c0b3f50 con 0x7fdb1c0cb750 2024-04-14T23:48:53.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.433+0000 7fdb21e00640 1 -- 172.21.15.60:0/1216392317 --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fdb1c15b370 con 0x7fdb1c0cd540 2024-04-14T23:48:53.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.433+0000 7fdb21e00640 1 -- 172.21.15.60:0/1216392317 --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fdb1c0b39b0 con 0x7fdb1c0cd160 2024-04-14T23:48:53.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.433+0000 7f4821600640 1 -- 172.21.15.60:0/1603187010 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4810035040 con 0x7f481c0e37a0 2024-04-14T23:48:53.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.434+0000 7f1cbea00640 1 -- 172.21.15.60:0/3126452961 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cb00125f0 con 0x7f1cc80c6800 2024-04-14T23:48:53.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.434+0000 7f1cbea00640 1 -- 172.21.15.60:0/3126452961 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1cb0019070 con 0x7f1cc80c6800 2024-04-14T23:48:53.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.434+0000 7f1cbea00640 1 -- 172.21.15.60:0/3126452961 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cb00129b0 con 0x7f1cc80c6800 2024-04-14T23:48:53.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.435+0000 7fdb1aa00640 1 -- 172.21.15.60:0/1216392317 >> v2:172.21.15.60:3300/0 conn(0x7fdb1c0cd540 msgr2=0x7fdb1c0b3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:53.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.435+0000 7fdb1aa00640 1 --2- 172.21.15.60:0/1216392317 >> v2:172.21.15.60:3300/0 conn(0x7fdb1c0cd540 0x7fdb1c0b3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.435+0000 7fdb1aa00640 1 -- 172.21.15.60:0/1216392317 >> v1:172.21.15.60:6789/0 conn(0x7fdb1c0cb750 legacy=0x7fdb1c154b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:53.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.435+0000 7fdb1aa00640 1 -- 172.21.15.60:0/1216392317 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb1c0b3c70 con 0x7fdb1c0cd160 2024-04-14T23:48:53.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.435+0000 7f1cc7e00640 1 -- 172.21.15.60:0/3126452961 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cc820ef60 con 0x7f1cc80c6800 2024-04-14T23:48:53.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.435+0000 7f482aa00640 1 -- 172.21.15.60:0/1603187010 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4800005290 con 0x7f481c0e37a0 2024-04-14T23:48:53.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.436+0000 7fdb1aa00640 1 --2- 172.21.15.60:0/1216392317 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb1c0cd160 0x7fdb1c15ae30 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c067ef0 tx=0x7fdb0c02b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f5885eadc1e58bc server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:53.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.436+0000 7f1cc7e00640 1 -- 172.21.15.60:0/3126452961 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cc8157a00 con 0x7f1cc80c6800 2024-04-14T23:48:53.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.436+0000 7f219dc00640 1 --2- 172.21.15.60:0/1143501439 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f219806c720 0x7f21981594a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2198158c50 tx=0x7f219002be20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17fb069362edbb63 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:53.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.436+0000 7fdb19600640 1 -- 172.21.15.60:0/1216392317 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb0c034070 con 0x7fdb1c0cd160 2024-04-14T23:48:53.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.437+0000 7fdb19600640 1 -- 172.21.15.60:0/1216392317 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb0c030860 con 0x7fdb1c0cd160 2024-04-14T23:48:53.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.437+0000 7fdb19600640 1 -- 172.21.15.60:0/1216392317 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb0c030c30 con 0x7fdb1c0cd160 2024-04-14T23:48:53.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.437+0000 7f4821600640 1 -- 172.21.15.60:0/1603187010 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f481004b060 con 0x7f481c0e37a0 2024-04-14T23:48:53.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.437+0000 7f1cc7e00640 1 -- 172.21.15.60:0/3126452961 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1cc80c4940 con 0x7f1cc80c6800 2024-04-14T23:48:53.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.438+0000 7ff6c3600640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff6bc072ec0 0x7ff6bc0c6b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:53.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.438+0000 7fdb21e00640 1 -- 172.21.15.60:0/1216392317 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb1c0cd160 msgr2=0x7fdb1c15ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:53.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.438+0000 7fdb21e00640 1 --2- 172.21.15.60:0/1216392317 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb1c0cd160 0x7fdb1c15ae30 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c067ef0 tx=0x7fdb0c02b9a0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.438+0000 7fdb21e00640 1 -- 172.21.15.60:0/1216392317 shutdown_connections 2024-04-14T23:48:53.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.438+0000 7fdb21e00640 1 --2- 172.21.15.60:0/1216392317 >> v2:172.21.15.60:3300/0 conn(0x7fdb1c0cd540 0x7fdb1c0b3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.438+0000 7fdb21e00640 1 --2- 172.21.15.60:0/1216392317 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb1c0cd160 0x7fdb1c15ae30 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.438+0000 7fdb21e00640 1 -- 172.21.15.60:0/1216392317 >> 172.21.15.60:0/1216392317 conn(0x7fdb1c066f10 msgr2=0x7fdb1c0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:53.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.438+0000 7ff6c3600640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff6bc0c70b0 0x7ff6bc15a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:53.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.438+0000 7fdb21e00640 1 -- 172.21.15.60:0/1216392317 shutdown_connections 2024-04-14T23:48:53.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.438+0000 7fdb21e00640 1 -- 172.21.15.60:0/1216392317 wait complete. 2024-04-14T23:48:53.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.438+0000 7ff6c3600640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6bc06f690 con 0x7ff6bc150e00 2024-04-14T23:48:53.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.438+0000 7ff6c3600640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7ff6bc0c3a00 con 0x7ff6bc072ec0 2024-04-14T23:48:53.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.438+0000 7ff6c3600640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7ff6bc06fa10 con 0x7ff6bc0c70b0 2024-04-14T23:48:53.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.438+0000 7f4821600640 1 --2- 172.21.15.60:0/1603187010 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f480406ba10 0x7f480406ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:53.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.439+0000 7f4821600640 1 -- 172.21.15.60:0/1603187010 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f48100ac310 con 0x7f481c0e37a0 2024-04-14T23:48:53.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.439+0000 7ff6c1600640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff6bc0c70b0 0x7ff6bc15a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:53.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.439+0000 7ff6bbe00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff6bc072ec0 0x7ff6bc0c6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:53.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.438+0000 7f2197400640 1 -- 172.21.15.60:0/1143501439 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2190039070 con 0x7f219806c720 2024-04-14T23:48:53.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.439+0000 7f2197400640 1 -- 172.21.15.60:0/1143501439 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f219002cbd0 con 0x7f219806c720 2024-04-14T23:48:53.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.439+0000 7ff6c1600640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff6bc0c70b0 0x7ff6bc15a490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:57872/0 (socket says 172.21.15.60:57872) 2024-04-14T23:48:53.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.439+0000 7ff6c1600640 1 -- 172.21.15.60:0/178138993 learned_addr learned my addr 172.21.15.60:0/178138993 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:53.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.439+0000 7f2197400640 1 -- 172.21.15.60:0/1143501439 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2190034440 con 0x7f219806c720 2024-04-14T23:48:53.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.439+0000 7ff6c0c00640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7ff6bc150e00 0x7ff6bc06ef80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53300/0 (socket says 172.21.15.60:53300) 2024-04-14T23:48:53.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.439+0000 7f4829600640 1 --2- 172.21.15.60:0/1603187010 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f480406ba10 0x7f480406ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:53.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.440+0000 7f4829600640 1 --2- 172.21.15.60:0/1603187010 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f480406ba10 0x7f480406ded0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f48240768b0 tx=0x7f482408e040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:53.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.440+0000 7f219e600640 1 -- 172.21.15.60:0/1143501439 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f219806c720 msgr2=0x7f21981594a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:53.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.440+0000 7f219e600640 1 --2- 172.21.15.60:0/1143501439 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f219806c720 0x7f21981594a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2198158c50 tx=0x7f219002be20 comp rx=0 tx=0).stop 2024-04-14T23:48:53.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.440+0000 7ff6c1600640 1 -- 172.21.15.60:0/178138993 >> v2:172.21.15.60:3300/0 conn(0x7ff6bc072ec0 msgr2=0x7ff6bc0c6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:53.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.440+0000 7ff6c1600640 1 --2- 172.21.15.60:0/178138993 >> v2:172.21.15.60:3300/0 conn(0x7ff6bc072ec0 0x7ff6bc0c6b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.440+0000 7ff6c1600640 1 -- 172.21.15.60:0/178138993 >> v1:172.21.15.60:6789/0 conn(0x7ff6bc150e00 legacy=0x7ff6bc06ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:53.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.440+0000 7ff6c1600640 1 -- 172.21.15.60:0/178138993 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6bc156ce0 con 0x7ff6bc0c70b0 2024-04-14T23:48:53.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.440+0000 7f1cbea00640 1 -- 172.21.15.60:0/3126452961 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f1cb0012b50 con 0x7f1cc80c6800 2024-04-14T23:48:53.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.440+0000 7f219e600640 1 -- 172.21.15.60:0/1143501439 shutdown_connections 2024-04-14T23:48:53.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.441+0000 7f219e600640 1 --2- 172.21.15.60:0/1143501439 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f219806c720 0x7f21981594a0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.441+0000 7f219e600640 1 --2- 172.21.15.60:0/1143501439 >> v2:172.21.15.60:3300/0 conn(0x7f21980c5ce0 0x7f2198158f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.441+0000 7f219e600640 1 -- 172.21.15.60:0/1143501439 >> 172.21.15.60:0/1143501439 conn(0x7f21980cc2f0 msgr2=0x7f21980cc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:53.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.441+0000 7f219e600640 1 -- 172.21.15.60:0/1143501439 shutdown_connections 2024-04-14T23:48:53.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.441+0000 7f219e600640 1 -- 172.21.15.60:0/1143501439 wait complete. 2024-04-14T23:48:53.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.441+0000 7ff6c1600640 1 --2- 172.21.15.60:0/178138993 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6bc0c70b0 0x7ff6bc15a490 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff6bc0ca020 tx=0x7ff6ac030430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c75a8c0275d51d33 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:53.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.442+0000 7fdb21e00640 1 Processor -- start 2024-04-14T23:48:53.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.442+0000 7ff6bb400640 1 -- 172.21.15.60:0/178138993 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6ac03c070 con 0x7ff6bc0c70b0 2024-04-14T23:48:53.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.442+0000 7ff6bb400640 1 -- 172.21.15.60:0/178138993 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6ac037600 con 0x7ff6bc0c70b0 2024-04-14T23:48:53.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.442+0000 7ff6bb400640 1 -- 172.21.15.60:0/178138993 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6ac0379f0 con 0x7ff6bc0c70b0 2024-04-14T23:48:53.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.442+0000 7f1cbea00640 1 --2- 172.21.15.60:0/3126452961 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f1ca406ba10 0x7f1ca406ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:53.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.442+0000 7f1cbea00640 1 -- 172.21.15.60:0/3126452961 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f1cb008b3c0 con 0x7f1cc80c6800 2024-04-14T23:48:53.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.442+0000 7fdb21e00640 1 -- start start 2024-04-14T23:48:53.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.443+0000 7f1cc6a00640 1 --2- 172.21.15.60:0/3126452961 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f1ca406ba10 0x7f1ca406ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:53.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.443+0000 7f4821600640 1 -- 172.21.15.60:0/1603187010 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f481007ae70 con 0x7f481c0e37a0 2024-04-14T23:48:53.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.443+0000 7fdb21e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdb1c0cb750 0x7fdb1c1e52a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:53.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.443+0000 7fdb21e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb1c0cd160 0x7fdb1c0c9d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:53.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.443+0000 7fdb21e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb1c0cd540 0x7fdb1c0ca270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:53.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.443+0000 7fdb21e00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fdb1c15ea70 con 0x7fdb1c0cd160 2024-04-14T23:48:53.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.443+0000 7fdb21e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdb1c15e790 con 0x7fdb1c0cd540 2024-04-14T23:48:53.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.443+0000 7fdb21e00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fdb1c0b36d0 con 0x7fdb1c0cb750 2024-04-14T23:48:53.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.443+0000 7ff6c3600640 1 -- 172.21.15.60:0/178138993 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6bc0c70b0 msgr2=0x7ff6bc15a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:53.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.443+0000 7ff6c3600640 1 --2- 172.21.15.60:0/178138993 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6bc0c70b0 0x7ff6bc15a490 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff6bc0ca020 tx=0x7ff6ac030430 comp rx=0 tx=0).stop 2024-04-14T23:48:53.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.443+0000 7f1cc6a00640 1 --2- 172.21.15.60:0/3126452961 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f1ca406ba10 0x7f1ca406ded0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1cc002bf80 tx=0x7f1cc0030000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:53.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.444+0000 7fdb1aa00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdb1c0cb750 0x7fdb1c1e52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:53.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.444+0000 7fdb1aa00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdb1c0cb750 0x7fdb1c1e52a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.60:51134/0 (socket says 172.21.15.60:51134) 2024-04-14T23:48:53.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.444+0000 7fdb1b400640 1 --2- 172.21.15.60:0/51267926 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb1c0cd540 0x7fdb1c0ca270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:53.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.444+0000 7ff6c3600640 1 -- 172.21.15.60:0/178138993 shutdown_connections 2024-04-14T23:48:53.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.444+0000 7ff6c3600640 1 --2- 172.21.15.60:0/178138993 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6bc0c70b0 0x7ff6bc15a490 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.445+0000 7f219e600640 1 Processor -- start 2024-04-14T23:48:53.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.444+0000 7fdb1aa00640 1 -- 172.21.15.60:0/51267926 learned_addr learned my addr 172.21.15.60:0/51267926 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:53.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.444+0000 7fdb1a000640 1 --2- 172.21.15.60:0/51267926 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb1c0cd160 0x7fdb1c0c9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:53.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.444+0000 7fdb1a000640 1 -- 172.21.15.60:0/51267926 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdb1c0cb750 msgr2=0x7fdb1c1e52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:53.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.444+0000 7fdb1a000640 1 --2- 172.21.15.60:0/51267926 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdb1c0cb750 0x7fdb1c1e52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.444+0000 7fdb1a000640 1 -- 172.21.15.60:0/51267926 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb1c0cd540 msgr2=0x7fdb1c0ca270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:53.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.444+0000 7fdb1a000640 1 --2- 172.21.15.60:0/51267926 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb1c0cd540 0x7fdb1c0ca270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.444+0000 7fdb1a000640 1 -- 172.21.15.60:0/51267926 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb1c1f55e0 con 0x7fdb1c0cd160 2024-04-14T23:48:53.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.444+0000 7ff6c3600640 1 --2- 172.21.15.60:0/178138993 >> v2:172.21.15.60:3300/0 conn(0x7ff6bc072ec0 0x7ff6bc0c6b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.444+0000 7ff6c3600640 1 -- 172.21.15.60:0/178138993 >> 172.21.15.60:0/178138993 conn(0x7ff6bc0cc170 msgr2=0x7ff6bc0cc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:53.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.445+0000 7fdb1b400640 1 --2- 172.21.15.60:0/51267926 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb1c0cd540 0x7fdb1c0ca270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-14T23:48:53.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.445+0000 7fdb1aa00640 1 --2- 172.21.15.60:0/51267926 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdb1c0cb750 0x7fdb1c1e52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-14T23:48:53.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.445+0000 7fdb1a000640 1 --2- 172.21.15.60:0/51267926 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb1c0cd160 0x7fdb1c0c9d30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fdb0800b3f0 tx=0x7fdb0800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:53.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.445+0000 7f219e600640 1 -- start start 2024-04-14T23:48:53.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.445+0000 7fdb13400640 1 -- 172.21.15.60:0/51267926 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb08015020 con 0x7fdb1c0cd160 2024-04-14T23:48:53.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.446+0000 7fdb13400640 1 -- 172.21.15.60:0/51267926 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb08011830 con 0x7fdb1c0cd160 2024-04-14T23:48:53.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.446+0000 7fdb13400640 1 -- 172.21.15.60:0/51267926 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb08011c00 con 0x7fdb1c0cd160 2024-04-14T23:48:53.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.446+0000 7fdb21e00640 1 -- 172.21.15.60:0/51267926 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb1c0c7200 con 0x7fdb1c0cd160 2024-04-14T23:48:53.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.446+0000 7ff6c3600640 1 -- 172.21.15.60:0/178138993 shutdown_connections 2024-04-14T23:48:53.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.446+0000 7fdb21e00640 1 -- 172.21.15.60:0/51267926 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb1c1e4aa0 con 0x7fdb1c0cd160 2024-04-14T23:48:53.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.447+0000 7f219e600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f219806c720 0x7f2198157b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:53.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.447+0000 7f219e600640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f21980c5ce0 0x7f2198158040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:53.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.447+0000 7f219e600640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2198152be0 0x7f2198072040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:53.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.447+0000 7f219e600640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f2198152fc0 con 0x7f2198152be0 2024-04-14T23:48:53.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.447+0000 7f219e600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f21980c7190 con 0x7f219806c720 2024-04-14T23:48:53.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.447+0000 7f219e600640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f2198153140 con 0x7f21980c5ce0 2024-04-14T23:48:53.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.447+0000 7f219d200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f219806c720 0x7f2198157b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:53.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.447+0000 7f219d200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f219806c720 0x7f2198157b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:57898/0 (socket says 172.21.15.60:57898) 2024-04-14T23:48:53.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.447+0000 7f219d200640 1 -- 172.21.15.60:0/3152618802 learned_addr learned my addr 172.21.15.60:0/3152618802 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:53.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.447+0000 7f219dc00640 1 --2- 172.21.15.60:0/3152618802 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2198152be0 0x7f2198072040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:53.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.447+0000 7f2197e00640 1 --2- 172.21.15.60:0/3152618802 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f21980c5ce0 0x7f2198158040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:53.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.448+0000 7f219d200640 1 -- 172.21.15.60:0/3152618802 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f21980c5ce0 msgr2=0x7f2198158040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:53.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.448+0000 7f219d200640 1 --2- 172.21.15.60:0/3152618802 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f21980c5ce0 0x7f2198158040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.448+0000 7f219d200640 1 -- 172.21.15.60:0/3152618802 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2198152be0 msgr2=0x7f2198072040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:53.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.448+0000 7f219d200640 1 --2- 172.21.15.60:0/3152618802 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2198152be0 0x7f2198072040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.448+0000 7f219d200640 1 -- 172.21.15.60:0/3152618802 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f219814b3b0 con 0x7f219806c720 2024-04-14T23:48:53.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.448+0000 7fdb10c00640 1 -- 172.21.15.60:0/51267926 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb1c076970 con 0x7fdb1c0cd160 2024-04-14T23:48:53.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.448+0000 7f219d200640 1 --2- 172.21.15.60:0/3152618802 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f219806c720 0x7f2198157b00 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f21880094f0 tx=0x7f21880099b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:53.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.449+0000 7f2194c00640 1 -- 172.21.15.60:0/3152618802 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2188018070 con 0x7f219806c720 2024-04-14T23:48:53.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.448+0000 7ff6c3600640 1 -- 172.21.15.60:0/178138993 wait complete. 2024-04-14T23:48:53.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.449+0000 7f2194c00640 1 -- 172.21.15.60:0/3152618802 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2188009ae0 con 0x7f219806c720 2024-04-14T23:48:53.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.449+0000 7f2194c00640 1 -- 172.21.15.60:0/3152618802 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21880062e0 con 0x7f219806c720 2024-04-14T23:48:53.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.449+0000 7f219e600640 1 -- 172.21.15.60:0/3152618802 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f219814b650 con 0x7f219806c720 2024-04-14T23:48:53.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.449+0000 7f219e600640 1 -- 172.21.15.60:0/3152618802 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f219814bc10 con 0x7f219806c720 2024-04-14T23:48:53.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.450+0000 7f219e600640 1 -- 172.21.15.60:0/3152618802 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f219804f540 con 0x7f219806c720 2024-04-14T23:48:53.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.451+0000 7ff6c3600640 1 Processor -- start 2024-04-14T23:48:53.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.448+0000 7f1cbea00640 1 -- 172.21.15.60:0/3126452961 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1cc80c4940 con 0x7f1cc80c6800 2024-04-14T23:48:53.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.452+0000 7ff6c3600640 1 -- start start 2024-04-14T23:48:53.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.453+0000 7ff6c3600640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6bc072ec0 0x7ff6bc0c6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:53.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.453+0000 7f2194c00640 1 -- 172.21.15.60:0/3152618802 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f218801b070 con 0x7f219806c720 2024-04-14T23:48:53.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.453+0000 7ff6c3600640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff6bc0c70b0 0x7ff6bc142940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:53.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.453+0000 7ff6c3600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6bc150e00 0x7ff6bc1ec170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:53.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.453+0000 7ff6c3600640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff6bc157580 con 0x7ff6bc0c70b0 2024-04-14T23:48:53.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.453+0000 7ff6c3600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff6bc1572a0 con 0x7ff6bc150e00 2024-04-14T23:48:53.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.450+0000 7fdb13400640 1 -- 172.21.15.60:0/51267926 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7fdb08018020 con 0x7fdb1c0cd160 2024-04-14T23:48:53.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.453+0000 7fdb13400640 1 --2- 172.21.15.60:0/51267926 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fdafc06ba10 0x7fdafc06ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:53.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.453+0000 7fdb13400640 1 -- 172.21.15.60:0/51267926 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fdb0808a650 con 0x7fdb1c0cd160 2024-04-14T23:48:53.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.453+0000 7ff6c3600640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7ff6bc157860 con 0x7ff6bc072ec0 2024-04-14T23:48:53.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.454+0000 7ff6c1600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6bc150e00 0x7ff6bc1ec170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:53.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.454+0000 7ff6c1600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6bc150e00 0x7ff6bc1ec170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:57906/0 (socket says 172.21.15.60:57906) 2024-04-14T23:48:53.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.454+0000 7ff6c1600640 1 -- 172.21.15.60:0/3104527883 learned_addr learned my addr 172.21.15.60:0/3104527883 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:53.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.455+0000 7f2194c00640 1 --2- 172.21.15.60:0/3152618802 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f217406ba10 0x7f217406ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:53.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.455+0000 7f2194c00640 1 -- 172.21.15.60:0/3152618802 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f2188056750 con 0x7f219806c720 2024-04-14T23:48:53.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.454+0000 7ff6c1600640 1 -- 172.21.15.60:0/3104527883 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6bc072ec0 msgr2=0x7ff6bc0c6420 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-14T23:48:53.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.454+0000 7ff6c1600640 1 --2- 172.21.15.60:0/3104527883 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6bc072ec0 0x7ff6bc0c6420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.454+0000 7ff6c1600640 1 -- 172.21.15.60:0/3104527883 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff6bc0c70b0 msgr2=0x7ff6bc142940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:53.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.454+0000 7ff6c1600640 1 --2- 172.21.15.60:0/3104527883 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff6bc0c70b0 0x7ff6bc142940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.454+0000 7ff6c1600640 1 -- 172.21.15.60:0/3104527883 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6bc1ef780 con 0x7ff6bc150e00 2024-04-14T23:48:53.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.455+0000 7ff6c1600640 1 --2- 172.21.15.60:0/3104527883 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6bc150e00 0x7ff6bc1ec170 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff6ac03a040 tx=0x7ff6ac004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:53.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.456+0000 7ff6b8c00640 1 -- 172.21.15.60:0/3104527883 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6ac03c030 con 0x7ff6bc150e00 2024-04-14T23:48:53.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.456+0000 7ff6b8c00640 1 -- 172.21.15.60:0/3104527883 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6ac03d040 con 0x7ff6bc150e00 2024-04-14T23:48:53.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.456+0000 7ff6b8c00640 1 -- 172.21.15.60:0/3104527883 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6ac009510 con 0x7ff6bc150e00 2024-04-14T23:48:53.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.456+0000 7f2194c00640 1 -- 172.21.15.60:0/3152618802 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2188022020 con 0x7f219806c720 2024-04-14T23:48:53.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.457+0000 7fdb1aa00640 1 --2- 172.21.15.60:0/51267926 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fdafc06ba10 0x7fdafc06ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:53.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.457+0000 7f2197e00640 1 --2- 172.21.15.60:0/3152618802 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f217406ba10 0x7f217406ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:53.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.457+0000 7ff6c3600640 1 -- 172.21.15.60:0/3104527883 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6bc0c5dc0 con 0x7ff6bc150e00 2024-04-14T23:48:53.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.457+0000 7ff6c3600640 1 -- 172.21.15.60:0/3104527883 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6bc1efd00 con 0x7ff6bc150e00 2024-04-14T23:48:53.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.458+0000 7f2197e00640 1 --2- 172.21.15.60:0/3152618802 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f217406ba10 0x7f217406ded0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2180006fd0 tx=0x7f2180008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:53.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.459+0000 7ff6c3600640 1 -- 172.21.15.60:0/3104527883 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6bc076970 con 0x7ff6bc150e00 2024-04-14T23:48:53.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.460+0000 7fdb1aa00640 1 --2- 172.21.15.60:0/51267926 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fdafc06ba10 0x7fdafc06ded0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fdb0c009630 tx=0x7fdb0c03b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:53.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.461+0000 7fdb13400640 1 -- 172.21.15.60:0/51267926 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fdb0801f020 con 0x7fdb1c0cd160 2024-04-14T23:48:53.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.462+0000 7ff6b8c00640 1 -- 172.21.15.60:0/3104527883 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7ff6ac048020 con 0x7ff6bc150e00 2024-04-14T23:48:53.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.464+0000 7ff6b8c00640 1 --2- 172.21.15.60:0/3104527883 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff69c06b9e0 0x7ff69c06dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:53.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.464+0000 7ff6b8c00640 1 -- 172.21.15.60:0/3104527883 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ff6ac0b0ff0 con 0x7ff6bc150e00 2024-04-14T23:48:53.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.465+0000 7ff6c0c00640 1 --2- 172.21.15.60:0/3104527883 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff69c06b9e0 0x7ff69c06dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:53.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.466+0000 7ff6b8c00640 1 -- 172.21.15.60:0/3104527883 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff6ac07fbe0 con 0x7ff6bc150e00 2024-04-14T23:48:53.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.466+0000 7ff6c0c00640 1 --2- 172.21.15.60:0/3104527883 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff69c06b9e0 0x7ff69c06dea0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff6b4000f60 tx=0x7ff6b4007a00 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:53.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.791+0000 7fd99b000640 1 -- 172.21.15.60:0/3323363356 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fd9941583f0 con 0x7fd99415cdf0 2024-04-14T23:48:53.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.804+0000 7fd98be00640 1 -- 172.21.15.60:0/3323363356 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd98c010040 con 0x7fd99415cdf0 2024-04-14T23:48:53.809 INFO:teuthology.orchestra.run.smithi060.stdout:55834574853 2024-04-14T23:48:53.812 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.804+0000 7fde02a00640 1 -- 172.21.15.60:0/2847892484 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fdde4005580 con 0x7fde08045930 2024-04-14T23:48:53.812 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.807+0000 7fd99b000640 1 -- 172.21.15.60:0/3323363356 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fd97406ba90 msgr2=0x7fd97406df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:53.813 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.807+0000 7fd99b000640 1 --2- 172.21.15.60:0/3323363356 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fd97406ba90 0x7fd97406df50 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd98402b8f0 tx=0x7fd984008030 comp rx=0 tx=0).stop 2024-04-14T23:48:53.813 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.807+0000 7fd99b000640 1 -- 172.21.15.60:0/3323363356 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd99415cdf0 msgr2=0x7fd9941f4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:53.813 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.807+0000 7fd99b000640 1 --2- 172.21.15.60:0/3323363356 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd99415cdf0 0x7fd9941f4f20 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd98c009d90 tx=0x7fd98c00c490 comp rx=0 tx=0).stop 2024-04-14T23:48:53.813 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.807+0000 7fd99b000640 1 -- 172.21.15.60:0/3323363356 shutdown_connections 2024-04-14T23:48:53.813 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.807+0000 7fd99b000640 1 --2- 172.21.15.60:0/3323363356 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fd97406ba90 0x7fd97406df50 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.813 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.807+0000 7fd99b000640 1 --2- 172.21.15.60:0/3323363356 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd99415cdf0 0x7fd9941f4f20 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.813 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.807+0000 7fd99b000640 1 --2- 172.21.15.60:0/3323363356 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9940c8740 0x7fd99414a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.807+0000 7fd99b000640 1 --2- 172.21.15.60:0/3323363356 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd9940c75c0 0x7fd9941f8870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.807+0000 7fd99b000640 1 -- 172.21.15.60:0/3323363356 >> 172.21.15.60:0/3323363356 conn(0x7fd9940cc170 msgr2=0x7fd994158c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:53.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.809+0000 7fd99b000640 1 -- 172.21.15.60:0/3323363356 shutdown_connections 2024-04-14T23:48:53.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.809+0000 7fd99b000640 1 -- 172.21.15.60:0/3323363356 wait complete. 2024-04-14T23:48:53.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.809+0000 7fde0d600640 1 -- 172.21.15.60:0/2847892484 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fde0405bdf0 con 0x7fde08045930 2024-04-14T23:48:53.818 INFO:teuthology.orchestra.run.smithi060.stdout:55834574854 2024-04-14T23:48:53.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.812+0000 7fde02a00640 1 -- 172.21.15.60:0/2847892484 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fddf006bb90 msgr2=0x7fddf006e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:53.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.812+0000 7fde02a00640 1 --2- 172.21.15.60:0/2847892484 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fddf006bb90 0x7fddf006e050 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fddfc02b6c0 tx=0x7fddfc008030 comp rx=0 tx=0).stop 2024-04-14T23:48:53.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.812+0000 7fde02a00640 1 -- 172.21.15.60:0/2847892484 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fde08045930 msgr2=0x7fde080f2f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:53.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.812+0000 7fde02a00640 1 --2- 172.21.15.60:0/2847892484 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fde08045930 0x7fde080f2f10 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fde0400c550 tx=0x7fde0400ca10 comp rx=0 tx=0).stop 2024-04-14T23:48:53.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.813+0000 7fde02a00640 1 -- 172.21.15.60:0/2847892484 shutdown_connections 2024-04-14T23:48:53.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.813+0000 7fde02a00640 1 --2- 172.21.15.60:0/2847892484 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fddf006bb90 0x7fddf006e050 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.813+0000 7fde02a00640 1 --2- 172.21.15.60:0/2847892484 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fde0804b090 0x7fde080f3450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.813+0000 7fde02a00640 1 --2- 172.21.15.60:0/2847892484 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fde08045930 0x7fde080f2f10 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.813+0000 7fde02a00640 1 --2- 172.21.15.60:0/2847892484 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fde08005e90 0x7fde080f93b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.813+0000 7fde02a00640 1 -- 172.21.15.60:0/2847892484 >> 172.21.15.60:0/2847892484 conn(0x7fde08050180 msgr2=0x7fde080e5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:53.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.813+0000 7fde02a00640 1 -- 172.21.15.60:0/2847892484 shutdown_connections 2024-04-14T23:48:53.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.813+0000 7fde02a00640 1 -- 172.21.15.60:0/2847892484 wait complete. 2024-04-14T23:48:53.846 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.0 2024-04-14T23:48:53.846 DEBUG:teuthology.parallel:result is None 2024-04-14T23:48:53.849 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.2 2024-04-14T23:48:53.849 DEBUG:teuthology.parallel:result is None 2024-04-14T23:48:53.862 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.856+0000 7fd918a00640 1 -- 172.21.15.60:0/3619087474 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fd8ec005580 con 0x7fd9140c9680 2024-04-14T23:48:53.862 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.857+0000 7fd90a000640 1 -- 172.21.15.60:0/3619087474 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd904058700 con 0x7fd9140c9680 2024-04-14T23:48:53.863 INFO:teuthology.orchestra.run.smithi060.stdout:55834574853 2024-04-14T23:48:53.863 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.857+0000 7f1cc7e00640 1 -- 172.21.15.60:0/3126452961 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f1cc8071ea0 con 0x7f1cc80c6800 2024-04-14T23:48:53.863 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.858+0000 7f1cbea00640 1 -- 172.21.15.60:0/3126452961 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f1cb0059f20 con 0x7f1cc80c6800 2024-04-14T23:48:53.865 INFO:teuthology.orchestra.run.smithi060.stdout:55834574854 2024-04-14T23:48:53.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.860+0000 7f1cc7e00640 1 -- 172.21.15.60:0/3126452961 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f1ca406ba10 msgr2=0x7f1ca406ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:53.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.860+0000 7f1cc7e00640 1 --2- 172.21.15.60:0/3126452961 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f1ca406ba10 0x7f1ca406ded0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1cc002bf80 tx=0x7f1cc0030000 comp rx=0 tx=0).stop 2024-04-14T23:48:53.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.860+0000 7f1cc7e00640 1 -- 172.21.15.60:0/3126452961 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1cc80c6800 msgr2=0x7f1cc815a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:53.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.860+0000 7f1cc7e00640 1 --2- 172.21.15.60:0/3126452961 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1cc80c6800 0x7f1cc815a8a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1cb000d560 tx=0x7f1cb000da20 comp rx=0 tx=0).stop 2024-04-14T23:48:53.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.860+0000 7f1cc7e00640 1 -- 172.21.15.60:0/3126452961 shutdown_connections 2024-04-14T23:48:53.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.860+0000 7f1cc7e00640 1 --2- 172.21.15.60:0/3126452961 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f1ca406ba10 0x7f1ca406ded0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.860+0000 7f1cc7e00640 1 --2- 172.21.15.60:0/3126452961 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1cc80c6800 0x7f1cc815a8a0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.860+0000 7f1cc7e00640 1 --2- 172.21.15.60:0/3126452961 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cc80c5ae0 0x7f1cc80c0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.860+0000 7f1cc7e00640 1 --2- 172.21.15.60:0/3126452961 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1cc80c5470 0x7f1cc80c2ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.867 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.860+0000 7f1cc7e00640 1 -- 172.21.15.60:0/3126452961 >> 172.21.15.60:0/3126452961 conn(0x7f1cc80cc2f0 msgr2=0x7f1cc80c9d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:53.867 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.861+0000 7f1cc7e00640 1 -- 172.21.15.60:0/3126452961 shutdown_connections 2024-04-14T23:48:53.867 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.861+0000 7f1cc7e00640 1 -- 172.21.15.60:0/3126452961 wait complete. 2024-04-14T23:48:53.868 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.861+0000 7fd918a00640 1 -- 172.21.15.60:0/3619087474 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fd8f406bbc0 msgr2=0x7fd8f406e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:53.868 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.861+0000 7fd918a00640 1 --2- 172.21.15.60:0/3619087474 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fd8f406bbc0 0x7fd8f406e080 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd900001860 tx=0x7fd900009040 comp rx=0 tx=0).stop 2024-04-14T23:48:53.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.861+0000 7fd918a00640 1 -- 172.21.15.60:0/3619087474 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9140c9680 msgr2=0x7fd914148900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:53.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.861+0000 7fd918a00640 1 --2- 172.21.15.60:0/3619087474 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9140c9680 0x7fd914148900 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd90400c5c0 tx=0x7fd90400ca80 comp rx=0 tx=0).stop 2024-04-14T23:48:53.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.861+0000 7fd918a00640 1 -- 172.21.15.60:0/3619087474 shutdown_connections 2024-04-14T23:48:53.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.861+0000 7fd918a00640 1 --2- 172.21.15.60:0/3619087474 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fd8f406bbc0 0x7fd8f406e080 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.861+0000 7fd918a00640 1 --2- 172.21.15.60:0/3619087474 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9140c9680 0x7fd914148900 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.861+0000 7fd918a00640 1 --2- 172.21.15.60:0/3619087474 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd91406f0b0 0x7fd9141f39e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.861+0000 7fd918a00640 1 --2- 172.21.15.60:0/3619087474 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd91406ecd0 0x7fd91414e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.861+0000 7fd918a00640 1 -- 172.21.15.60:0/3619087474 >> 172.21.15.60:0/3619087474 conn(0x7fd914066f10 msgr2=0x7fd9140c84d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:53.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.862+0000 7fd918a00640 1 -- 172.21.15.60:0/3619087474 shutdown_connections 2024-04-14T23:48:53.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.862+0000 7fd918a00640 1 -- 172.21.15.60:0/3619087474 wait complete. 2024-04-14T23:48:53.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.877+0000 7f482aa00640 1 -- 172.21.15.60:0/1603187010 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f4800005580 con 0x7f481c0e37a0 2024-04-14T23:48:53.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.878+0000 7f4821600640 1 -- 172.21.15.60:0/1603187010 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f481007a770 con 0x7f481c0e37a0 2024-04-14T23:48:53.886 INFO:teuthology.orchestra.run.smithi060.stdout:55834574854 2024-04-14T23:48:53.887 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.7 2024-04-14T23:48:53.887 DEBUG:teuthology.parallel:result is None 2024-04-14T23:48:53.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.880+0000 7f482aa00640 1 -- 172.21.15.60:0/1603187010 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f480406ba10 msgr2=0x7f480406ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:53.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.880+0000 7f482aa00640 1 --2- 172.21.15.60:0/1603187010 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f480406ba10 0x7f480406ded0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f48240768b0 tx=0x7f482408e040 comp rx=0 tx=0).stop 2024-04-14T23:48:53.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.880+0000 7f482aa00640 1 -- 172.21.15.60:0/1603187010 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f481c0e37a0 msgr2=0x7f481c04b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:53.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.880+0000 7f482aa00640 1 --2- 172.21.15.60:0/1603187010 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f481c0e37a0 0x7f481c04b060 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4810033000 tx=0x7f48100046c0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.880+0000 7f482aa00640 1 -- 172.21.15.60:0/1603187010 shutdown_connections 2024-04-14T23:48:53.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.880+0000 7f482aa00640 1 --2- 172.21.15.60:0/1603187010 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f480406ba10 0x7f480406ded0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.880+0000 7f482aa00640 1 --2- 172.21.15.60:0/1603187010 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f481c0e37a0 0x7f481c04b060 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.880+0000 7f482aa00640 1 --2- 172.21.15.60:0/1603187010 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f481c0dd3c0 0x7f481c04ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.880+0000 7f482aa00640 1 --2- 172.21.15.60:0/1603187010 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f481c04ba60 0x7f481c178de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.880+0000 7f482aa00640 1 -- 172.21.15.60:0/1603187010 >> 172.21.15.60:0/1603187010 conn(0x7f481c050180 msgr2=0x7f481c006810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:53.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.881+0000 7f482aa00640 1 -- 172.21.15.60:0/1603187010 shutdown_connections 2024-04-14T23:48:53.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.881+0000 7f482aa00640 1 -- 172.21.15.60:0/1603187010 wait complete. 2024-04-14T23:48:53.892 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.3 2024-04-14T23:48:53.892 DEBUG:teuthology.parallel:result is None 2024-04-14T23:48:53.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.891+0000 7ff6c3600640 1 -- 172.21.15.60:0/3104527883 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7ff6bc072a90 con 0x7ff6bc150e00 2024-04-14T23:48:53.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.892+0000 7ff6b8c00640 1 -- 172.21.15.60:0/3104527883 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ff6ac07f4e0 con 0x7ff6bc150e00 2024-04-14T23:48:53.898 INFO:teuthology.orchestra.run.smithi060.stdout:55834574855 2024-04-14T23:48:53.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.894+0000 7ff6c3600640 1 -- 172.21.15.60:0/3104527883 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff69c06b9e0 msgr2=0x7ff69c06dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:53.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.894+0000 7ff6c3600640 1 --2- 172.21.15.60:0/3104527883 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff69c06b9e0 0x7ff69c06dea0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff6b4000f60 tx=0x7ff6b4007a00 comp rx=0 tx=0).stop 2024-04-14T23:48:53.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.894+0000 7ff6c3600640 1 -- 172.21.15.60:0/3104527883 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6bc150e00 msgr2=0x7ff6bc1ec170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:53.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.894+0000 7ff6c3600640 1 --2- 172.21.15.60:0/3104527883 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6bc150e00 0x7ff6bc1ec170 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff6ac03a040 tx=0x7ff6ac004a40 comp rx=0 tx=0).stop 2024-04-14T23:48:53.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.895+0000 7ff6c3600640 1 -- 172.21.15.60:0/3104527883 shutdown_connections 2024-04-14T23:48:53.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.895+0000 7ff6c3600640 1 --2- 172.21.15.60:0/3104527883 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff69c06b9e0 0x7ff69c06dea0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.895+0000 7ff6c3600640 1 --2- 172.21.15.60:0/3104527883 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6bc150e00 0x7ff6bc1ec170 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.895+0000 7ff6c3600640 1 --2- 172.21.15.60:0/3104527883 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff6bc0c70b0 0x7ff6bc142940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.895+0000 7ff6c3600640 1 --2- 172.21.15.60:0/3104527883 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6bc072ec0 0x7ff6bc0c6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.895+0000 7ff6c3600640 1 -- 172.21.15.60:0/3104527883 >> 172.21.15.60:0/3104527883 conn(0x7ff6bc0cc170 msgr2=0x7ff6bc14e580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:53.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.895+0000 7ff6c3600640 1 -- 172.21.15.60:0/3104527883 shutdown_connections 2024-04-14T23:48:53.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.895+0000 7ff6c3600640 1 -- 172.21.15.60:0/3104527883 wait complete. 2024-04-14T23:48:53.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.899+0000 7f219e600640 1 -- 172.21.15.60:0/3152618802 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f21980c6ee0 con 0x7f219806c720 2024-04-14T23:48:53.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.900+0000 7f2194c00640 1 -- 172.21.15.60:0/3152618802 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f2188058db0 con 0x7f219806c720 2024-04-14T23:48:53.905 INFO:teuthology.orchestra.run.smithi060.stdout:55834574854 2024-04-14T23:48:53.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.902+0000 7f219e600640 1 -- 172.21.15.60:0/3152618802 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f217406ba10 msgr2=0x7f217406ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:53.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.902+0000 7f219e600640 1 --2- 172.21.15.60:0/3152618802 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f217406ba10 0x7f217406ded0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2180006fd0 tx=0x7f2180008040 comp rx=0 tx=0).stop 2024-04-14T23:48:53.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.902+0000 7f219e600640 1 -- 172.21.15.60:0/3152618802 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f219806c720 msgr2=0x7f2198157b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:53.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.902+0000 7f219e600640 1 --2- 172.21.15.60:0/3152618802 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f219806c720 0x7f2198157b00 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f21880094f0 tx=0x7f21880099b0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.902+0000 7f219e600640 1 -- 172.21.15.60:0/3152618802 shutdown_connections 2024-04-14T23:48:53.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.902+0000 7f219e600640 1 --2- 172.21.15.60:0/3152618802 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f217406ba10 0x7f217406ded0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.902+0000 7f219e600640 1 --2- 172.21.15.60:0/3152618802 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2198152be0 0x7f2198072040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.902+0000 7f219e600640 1 --2- 172.21.15.60:0/3152618802 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f21980c5ce0 0x7f2198158040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.902+0000 7f219e600640 1 --2- 172.21.15.60:0/3152618802 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f219806c720 0x7f2198157b00 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.902+0000 7f219e600640 1 -- 172.21.15.60:0/3152618802 >> 172.21.15.60:0/3152618802 conn(0x7f21980cc2f0 msgr2=0x7f2198156ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:53.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.902+0000 7f219e600640 1 -- 172.21.15.60:0/3152618802 shutdown_connections 2024-04-14T23:48:53.912 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.1 2024-04-14T23:48:53.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.902+0000 7f219e600640 1 -- 172.21.15.60:0/3152618802 wait complete. 2024-04-14T23:48:53.913 DEBUG:teuthology.parallel:result is None 2024-04-14T23:48:53.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.905+0000 7fdb10c00640 1 -- 172.21.15.60:0/51267926 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fdb1c0b5210 con 0x7fdb1c0cd160 2024-04-14T23:48:53.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.905+0000 7fdb13400640 1 -- 172.21.15.60:0/51267926 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fdb1c1f55e0 con 0x7fdb1c0cd160 2024-04-14T23:48:53.914 INFO:teuthology.orchestra.run.smithi060.stdout:55834574855 2024-04-14T23:48:53.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.907+0000 7fdb10c00640 1 -- 172.21.15.60:0/51267926 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fdafc06ba10 msgr2=0x7fdafc06ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:53.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.907+0000 7fdb10c00640 1 --2- 172.21.15.60:0/51267926 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fdafc06ba10 0x7fdafc06ded0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fdb0c009630 tx=0x7fdb0c03b040 comp rx=0 tx=0).stop 2024-04-14T23:48:53.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.907+0000 7fdb10c00640 1 -- 172.21.15.60:0/51267926 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb1c0cd160 msgr2=0x7fdb1c0c9d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:53.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.907+0000 7fdb10c00640 1 --2- 172.21.15.60:0/51267926 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb1c0cd160 0x7fdb1c0c9d30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fdb0800b3f0 tx=0x7fdb0800b8b0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.908+0000 7fdb10c00640 1 -- 172.21.15.60:0/51267926 shutdown_connections 2024-04-14T23:48:53.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.908+0000 7fdb10c00640 1 --2- 172.21.15.60:0/51267926 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fdafc06ba10 0x7fdafc06ded0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.908+0000 7fdb10c00640 1 --2- 172.21.15.60:0/51267926 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb1c0cd540 0x7fdb1c0ca270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.908+0000 7fdb10c00640 1 --2- 172.21.15.60:0/51267926 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb1c0cd160 0x7fdb1c0c9d30 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.908+0000 7fdb10c00640 1 --2- 172.21.15.60:0/51267926 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdb1c0cb750 0x7fdb1c1e52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:53.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.908+0000 7fdb10c00640 1 -- 172.21.15.60:0/51267926 >> 172.21.15.60:0/51267926 conn(0x7fdb1c066f10 msgr2=0x7fdb1c0cfc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:53.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.908+0000 7fdb10c00640 1 -- 172.21.15.60:0/51267926 shutdown_connections 2024-04-14T23:48:53.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:53.908+0000 7fdb10c00640 1 -- 172.21.15.60:0/51267926 wait complete. 2024-04-14T23:48:53.919 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.4 2024-04-14T23:48:53.919 DEBUG:teuthology.parallel:result is None 2024-04-14T23:48:53.931 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.5 2024-04-14T23:48:53.931 DEBUG:teuthology.parallel:result is None 2024-04-14T23:48:53.933 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.6 2024-04-14T23:48:53.933 DEBUG:teuthology.parallel:result is None 2024-04-14T23:48:53.933 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-14T23:48:53.933 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-14T23:48:54.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.169+0000 7f2c05400640 1 Processor -- start 2024-04-14T23:48:54.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.169+0000 7f2c05400640 1 -- start start 2024-04-14T23:48:54.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.170+0000 7f2c05400640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2c000b1e00 0x7f2c000c6950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:54.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.170+0000 7f2c05400640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c00152fb0 con 0x7f2c00073320 2024-04-14T23:48:54.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.170+0000 7f2c05400640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c00159310 con 0x7f2c000c3d90 2024-04-14T23:48:54.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.170+0000 7f2c05400640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f2c000c6540 con 0x7f2c000b1e00 2024-04-14T23:48:54.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.171+0000 7f2bffe00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2c000b1e00 0x7f2c000c6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:54.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.171+0000 7f2bffe00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2c000b1e00 0x7f2c000c6950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:57910/0 (socket says 172.21.15.60:57910) 2024-04-14T23:48:54.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.171+0000 7f2bffe00640 1 -- 172.21.15.60:0/3126415431 learned_addr learned my addr 172.21.15.60:0/3126415431 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:54.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.171+0000 7f2bffe00640 1 -- 172.21.15.60:0/3126415431 >> v1:172.21.15.186:6789/0 conn(0x7f2c000c3d90 legacy=0x7f2c000c41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:54.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.171+0000 7f2bffe00640 1 -- 172.21.15.60:0/3126415431 >> v1:172.21.15.60:6789/0 conn(0x7f2c00073320 legacy=0x7f2c000c7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:54.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.171+0000 7f2bff400640 1 -- 172.21.15.60:0/3126415431 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 937739202 0 0) 0x7f2c00159310 con 0x7f2c000c3d90 2024-04-14T23:48:54.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.171+0000 7f2bffe00640 1 -- 172.21.15.60:0/3126415431 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c000c38b0 con 0x7f2c000b1e00 2024-04-14T23:48:54.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.172+0000 7f2bffe00640 1 --2- 172.21.15.60:0/3126415431 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c000b1e00 0x7f2c000c6950 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2c001594e0 tx=0x7f2bf002b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b3a141c5f1481aae server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:54.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.173+0000 7f2bff400640 1 -- 172.21.15.60:0/3126415431 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bf00317d0 con 0x7f2c000b1e00 2024-04-14T23:48:54.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.173+0000 7f2bff400640 1 -- 172.21.15.60:0/3126415431 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2bf0037070 con 0x7f2c000b1e00 2024-04-14T23:48:54.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.173+0000 7f2bff400640 1 -- 172.21.15.60:0/3126415431 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bf0031b90 con 0x7f2c000b1e00 2024-04-14T23:48:54.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.173+0000 7f2c05400640 1 -- 172.21.15.60:0/3126415431 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c000b1e00 msgr2=0x7f2c000c6950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:54.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.173+0000 7f2c05400640 1 --2- 172.21.15.60:0/3126415431 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c000b1e00 0x7f2c000c6950 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2c001594e0 tx=0x7f2bf002b8f0 comp rx=0 tx=0).stop 2024-04-14T23:48:54.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.173+0000 7f2c05400640 1 -- 172.21.15.60:0/3126415431 shutdown_connections 2024-04-14T23:48:54.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.173+0000 7f2c05400640 1 --2- 172.21.15.60:0/3126415431 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c000b1e00 0x7f2c000c6950 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:54.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.173+0000 7f2c05400640 1 -- 172.21.15.60:0/3126415431 >> 172.21.15.60:0/3126415431 conn(0x7f2c000c8b50 msgr2=0x7f2c000c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:54.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.173+0000 7f2c05400640 1 -- 172.21.15.60:0/3126415431 shutdown_connections 2024-04-14T23:48:54.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.173+0000 7f2c05400640 1 -- 172.21.15.60:0/3126415431 wait complete. 2024-04-14T23:48:54.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.175+0000 7f2c05400640 1 Processor -- start 2024-04-14T23:48:54.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.175+0000 7f2c05400640 1 -- start start 2024-04-14T23:48:54.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.176+0000 7f2c05400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c00073320 0x7f2c0014b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:54.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.176+0000 7f2c05400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2c000b1e00 0x7f2c001ee470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:54.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.176+0000 7f2c05400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c000c3d90 0x7f2c001552d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:54.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.176+0000 7f2c05400640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f2c00159890 con 0x7f2c00073320 2024-04-14T23:48:54.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.176+0000 7f2c05400640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2c000c5f80 con 0x7f2c000c3d90 2024-04-14T23:48:54.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.176+0000 7f2c05400640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f2c000c6260 con 0x7f2c000b1e00 2024-04-14T23:48:54.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.176+0000 7f2bffe00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c00073320 0x7f2c0014b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:54.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.176+0000 7f2bffe00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c00073320 0x7f2c0014b4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41864/0 (socket says 172.21.15.60:41864) 2024-04-14T23:48:54.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.176+0000 7f2bffe00640 1 -- 172.21.15.60:0/1026076635 learned_addr learned my addr 172.21.15.60:0/1026076635 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:54.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.176+0000 7f2bf7e00640 1 --2- 172.21.15.60:0/1026076635 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2c000b1e00 0x7f2c001ee470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:54.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.176+0000 7f2c04a00640 1 --2- 172.21.15.60:0/1026076635 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c000c3d90 0x7f2c001552d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:54.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.177+0000 7f2bffe00640 1 -- 172.21.15.60:0/1026076635 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2c000b1e00 msgr2=0x7f2c001ee470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:54.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.177+0000 7f2bffe00640 1 --2- 172.21.15.60:0/1026076635 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2c000b1e00 0x7f2c001ee470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:54.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.177+0000 7f2bffe00640 1 -- 172.21.15.60:0/1026076635 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c000c3d90 msgr2=0x7f2c001552d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:54.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.177+0000 7f2bffe00640 1 --2- 172.21.15.60:0/1026076635 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c000c3d90 0x7f2c001552d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:54.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.177+0000 7f2bffe00640 1 -- 172.21.15.60:0/1026076635 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c001514e0 con 0x7f2c00073320 2024-04-14T23:48:54.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.177+0000 7f2bffe00640 1 --2- 172.21.15.60:0/1026076635 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c00073320 0x7f2c0014b4c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2bf0000c00 tx=0x7f2bf00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:54.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.177+0000 7f2bfcc00640 1 -- 172.21.15.60:0/1026076635 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bf0032870 con 0x7f2c00073320 2024-04-14T23:48:54.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.177+0000 7f2bfcc00640 1 -- 172.21.15.60:0/1026076635 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2bf0032a10 con 0x7f2c00073320 2024-04-14T23:48:54.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.177+0000 7f2bfcc00640 1 -- 172.21.15.60:0/1026076635 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bf0032e00 con 0x7f2c00073320 2024-04-14T23:48:54.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.177+0000 7f2c05400640 1 -- 172.21.15.60:0/1026076635 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c001517c0 con 0x7f2c00073320 2024-04-14T23:48:54.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.177+0000 7f2c05400640 1 -- 172.21.15.60:0/1026076635 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c001f1df0 con 0x7f2c00073320 2024-04-14T23:48:54.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.179+0000 7f2c05400640 1 -- 172.21.15.60:0/1026076635 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c0004f540 con 0x7f2c00073320 2024-04-14T23:48:54.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.179+0000 7f2bfcc00640 1 -- 172.21.15.60:0/1026076635 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f2bf00453f0 con 0x7f2c00073320 2024-04-14T23:48:54.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.180+0000 7f2bfcc00640 1 --2- 172.21.15.60:0/1026076635 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f2bdc06b990 0x7f2bdc06de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:54.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.180+0000 7f2bfcc00640 1 -- 172.21.15.60:0/1026076635 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f2bf00ac9c0 con 0x7f2c00073320 2024-04-14T23:48:54.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.180+0000 7f2bf7e00640 1 --2- 172.21.15.60:0/1026076635 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f2bdc06b990 0x7f2bdc06de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:54.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.181+0000 7f2bf7e00640 1 --2- 172.21.15.60:0/1026076635 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f2bdc06b990 0x7f2bdc06de50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2c0014a870 tx=0x7f2be8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:54.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.184+0000 7f2bfcc00640 1 -- 172.21.15.60:0/1026076635 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2bf0040070 con 0x7f2c00073320 2024-04-14T23:48:54.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.375+0000 7f2c05400640 1 -- 172.21.15.60:0/1026076635 --> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2c00151aa0 con 0x7f2bdc06b990 2024-04-14T23:48:54.391 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:48:54.394 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-04-14T23:48:54.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.382+0000 7f2bfcc00640 1 -- 172.21.15.60:0/1026076635 <== mgr.4100 v2:172.21.15.186:6832/714104835 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251915 (secure 0 0 0) 0x7f2c00151aa0 con 0x7f2bdc06b990 2024-04-14T23:48:54.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.387+0000 7f2c05400640 1 -- 172.21.15.60:0/1026076635 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f2bdc06b990 msgr2=0x7f2bdc06de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:54.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.387+0000 7f2c05400640 1 --2- 172.21.15.60:0/1026076635 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f2bdc06b990 0x7f2bdc06de50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2c0014a870 tx=0x7f2be8009040 comp rx=0 tx=0).stop 2024-04-14T23:48:54.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.387+0000 7f2c05400640 1 -- 172.21.15.60:0/1026076635 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c00073320 msgr2=0x7f2c0014b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:54.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.387+0000 7f2c05400640 1 --2- 172.21.15.60:0/1026076635 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c00073320 0x7f2c0014b4c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2bf0000c00 tx=0x7f2bf00046c0 comp rx=0 tx=0).stop 2024-04-14T23:48:54.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.387+0000 7f2c05400640 1 -- 172.21.15.60:0/1026076635 shutdown_connections 2024-04-14T23:48:54.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.387+0000 7f2c05400640 1 --2- 172.21.15.60:0/1026076635 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f2bdc06b990 0x7f2bdc06de50 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:54.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.387+0000 7f2c05400640 1 --2- 172.21.15.60:0/1026076635 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c000c3d90 0x7f2c001552d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:54.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.388+0000 7f2c05400640 1 --2- 172.21.15.60:0/1026076635 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2c000b1e00 0x7f2c001ee470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:54.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.388+0000 7f2c05400640 1 --2- 172.21.15.60:0/1026076635 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c00073320 0x7f2c0014b4c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:54.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.388+0000 7f2c05400640 1 -- 172.21.15.60:0/1026076635 >> 172.21.15.60:0/1026076635 conn(0x7f2c000c8b50 msgr2=0x7f2c000c41c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:54.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.388+0000 7f2c05400640 1 -- 172.21.15.60:0/1026076635 shutdown_connections 2024-04-14T23:48:54.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.388+0000 7f2c05400640 1 -- 172.21.15.60:0/1026076635 wait complete. 2024-04-14T23:48:54.474 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-04-14T23:48:54.169000+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218016,"kb_used_data":1888,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756704,"statfs":{"total":773094113280,"available":772870864896,"internally_reserved":0,"allocated":1933312,"data_stored":1790396,"data_compressed":65128,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":58,"num_write_kb":276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002593"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.549104+0000","last_change":"2024-04-14T23:48:43.222542+0000","last_active":"2024-04-14T23:48:45.549104+0000","last_peered":"2024-04-14T23:48:45.549104+0000","last_clean":"2024-04-14T23:48:45.549104+0000","last_became_active":"2024-04-14T23:48:43.222278+0000","last_became_peered":"2024-04-14T23:48:43.222278+0000","last_unstale":"2024-04-14T23:48:45.549104+0000","last_undegraded":"2024-04-14T23:48:45.549104+0000","last_fullsized":"2024-04-14T23:48:45.549104+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:13:06.877485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.049059+0000","last_change":"2024-04-14T23:48:43.226781+0000","last_active":"2024-04-14T23:48:45.049059+0000","last_peered":"2024-04-14T23:48:45.049059+0000","last_clean":"2024-04-14T23:48:45.049059+0000","last_became_active":"2024-04-14T23:48:43.226426+0000","last_became_peered":"2024-04-14T23:48:43.226426+0000","last_unstale":"2024-04-14T23:48:45.049059+0000","last_undegraded":"2024-04-14T23:48:45.049059+0000","last_fullsized":"2024-04-14T23:48:45.049059+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:35:24.780859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.235142+0000","last_change":"2024-04-14T23:48:43.236939+0000","last_active":"2024-04-14T23:48:44.235142+0000","last_peered":"2024-04-14T23:48:44.235142+0000","last_clean":"2024-04-14T23:48:44.235142+0000","last_became_active":"2024-04-14T23:48:43.236616+0000","last_became_peered":"2024-04-14T23:48:43.236616+0000","last_unstale":"2024-04-14T23:48:44.235142+0000","last_undegraded":"2024-04-14T23:48:44.235142+0000","last_fullsized":"2024-04-14T23:48:44.235142+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:35:42.619139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230420+0000","last_change":"2024-04-14T23:48:43.242349+0000","last_active":"2024-04-14T23:48:44.230420+0000","last_peered":"2024-04-14T23:48:44.230420+0000","last_clean":"2024-04-14T23:48:44.230420+0000","last_became_active":"2024-04-14T23:48:43.242000+0000","last_became_peered":"2024-04-14T23:48:43.242000+0000","last_unstale":"2024-04-14T23:48:44.230420+0000","last_undegraded":"2024-04-14T23:48:44.230420+0000","last_fullsized":"2024-04-14T23:48:44.230420+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:33:52.458016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548549+0000","last_change":"2024-04-14T23:48:43.233937+0000","last_active":"2024-04-14T23:48:45.548549+0000","last_peered":"2024-04-14T23:48:45.548549+0000","last_clean":"2024-04-14T23:48:45.548549+0000","last_became_active":"2024-04-14T23:48:43.233674+0000","last_became_peered":"2024-04-14T23:48:43.233674+0000","last_unstale":"2024-04-14T23:48:45.548549+0000","last_undegraded":"2024-04-14T23:48:45.548549+0000","last_fullsized":"2024-04-14T23:48:45.548549+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:00:07.838600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230260+0000","last_change":"2024-04-14T23:48:43.230925+0000","last_active":"2024-04-14T23:48:44.230260+0000","last_peered":"2024-04-14T23:48:44.230260+0000","last_clean":"2024-04-14T23:48:44.230260+0000","last_became_active":"2024-04-14T23:48:43.230405+0000","last_became_peered":"2024-04-14T23:48:43.230405+0000","last_unstale":"2024-04-14T23:48:44.230260+0000","last_undegraded":"2024-04-14T23:48:44.230260+0000","last_fullsized":"2024-04-14T23:48:44.230260+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:15:12.452964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548354+0000","last_change":"2024-04-14T23:48:43.231088+0000","last_active":"2024-04-14T23:48:45.548354+0000","last_peered":"2024-04-14T23:48:45.548354+0000","last_clean":"2024-04-14T23:48:45.548354+0000","last_became_active":"2024-04-14T23:48:43.230784+0000","last_became_peered":"2024-04-14T23:48:43.230784+0000","last_unstale":"2024-04-14T23:48:45.548354+0000","last_undegraded":"2024-04-14T23:48:45.548354+0000","last_fullsized":"2024-04-14T23:48:45.548354+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:23:10.461088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.231230+0000","last_change":"2024-04-14T23:48:43.236019+0000","last_active":"2024-04-14T23:48:44.231230+0000","last_peered":"2024-04-14T23:48:44.231230+0000","last_clean":"2024-04-14T23:48:44.231230+0000","last_became_active":"2024-04-14T23:48:43.235719+0000","last_became_peered":"2024-04-14T23:48:43.235719+0000","last_unstale":"2024-04-14T23:48:44.231230+0000","last_undegraded":"2024-04-14T23:48:44.231230+0000","last_fullsized":"2024-04-14T23:48:44.231230+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T04:41:16.130730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.549156+0000","last_change":"2024-04-14T23:48:43.232166+0000","last_active":"2024-04-14T23:48:45.549156+0000","last_peered":"2024-04-14T23:48:45.549156+0000","last_clean":"2024-04-14T23:48:45.549156+0000","last_became_active":"2024-04-14T23:48:43.231751+0000","last_became_peered":"2024-04-14T23:48:43.231751+0000","last_unstale":"2024-04-14T23:48:45.549156+0000","last_undegraded":"2024-04-14T23:48:45.549156+0000","last_fullsized":"2024-04-14T23:48:45.549156+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:05:47.289092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548848+0000","last_change":"2024-04-14T23:48:43.224938+0000","last_active":"2024-04-14T23:48:45.548848+0000","last_peered":"2024-04-14T23:48:45.548848+0000","last_clean":"2024-04-14T23:48:45.548848+0000","last_became_active":"2024-04-14T23:48:43.224592+0000","last_became_peered":"2024-04-14T23:48:43.224592+0000","last_unstale":"2024-04-14T23:48:45.548848+0000","last_undegraded":"2024-04-14T23:48:45.548848+0000","last_fullsized":"2024-04-14T23:48:45.548848+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:59:12.923596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548591+0000","last_change":"2024-04-14T23:48:43.225290+0000","last_active":"2024-04-14T23:48:45.548591+0000","last_peered":"2024-04-14T23:48:45.548591+0000","last_clean":"2024-04-14T23:48:45.548591+0000","last_became_active":"2024-04-14T23:48:43.224989+0000","last_became_peered":"2024-04-14T23:48:43.224989+0000","last_unstale":"2024-04-14T23:48:45.548591+0000","last_undegraded":"2024-04-14T23:48:45.548591+0000","last_fullsized":"2024-04-14T23:48:45.548591+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:44:49.116229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548742+0000","last_change":"2024-04-14T23:48:43.226194+0000","last_active":"2024-04-14T23:48:45.548742+0000","last_peered":"2024-04-14T23:48:45.548742+0000","last_clean":"2024-04-14T23:48:45.548742+0000","last_became_active":"2024-04-14T23:48:43.225892+0000","last_became_peered":"2024-04-14T23:48:43.225892+0000","last_unstale":"2024-04-14T23:48:45.548742+0000","last_undegraded":"2024-04-14T23:48:45.548742+0000","last_fullsized":"2024-04-14T23:48:45.548742+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:04:12.377770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.233039+0000","last_change":"2024-04-14T23:48:43.220571+0000","last_active":"2024-04-14T23:48:44.233039+0000","last_peered":"2024-04-14T23:48:44.233039+0000","last_clean":"2024-04-14T23:48:44.233039+0000","last_became_active":"2024-04-14T23:48:43.219614+0000","last_became_peered":"2024-04-14T23:48:43.219614+0000","last_unstale":"2024-04-14T23:48:44.233039+0000","last_undegraded":"2024-04-14T23:48:44.233039+0000","last_fullsized":"2024-04-14T23:48:44.233039+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:14:41.854957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.234686+0000","last_change":"2024-04-14T23:48:43.239403+0000","last_active":"2024-04-14T23:48:44.234686+0000","last_peered":"2024-04-14T23:48:44.234686+0000","last_clean":"2024-04-14T23:48:44.234686+0000","last_became_active":"2024-04-14T23:48:43.239136+0000","last_became_peered":"2024-04-14T23:48:43.239136+0000","last_unstale":"2024-04-14T23:48:44.234686+0000","last_undegraded":"2024-04-14T23:48:44.234686+0000","last_fullsized":"2024-04-14T23:48:44.234686+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:27:30.512861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.048083+0000","last_change":"2024-04-14T23:48:43.228157+0000","last_active":"2024-04-14T23:48:45.048083+0000","last_peered":"2024-04-14T23:48:45.048083+0000","last_clean":"2024-04-14T23:48:45.048083+0000","last_became_active":"2024-04-14T23:48:43.227898+0000","last_became_peered":"2024-04-14T23:48:43.227898+0000","last_unstale":"2024-04-14T23:48:45.048083+0000","last_undegraded":"2024-04-14T23:48:45.048083+0000","last_fullsized":"2024-04-14T23:48:45.048083+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:06:37.602935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230387+0000","last_change":"2024-04-14T23:48:43.238986+0000","last_active":"2024-04-14T23:48:44.230387+0000","last_peered":"2024-04-14T23:48:44.230387+0000","last_clean":"2024-04-14T23:48:44.230387+0000","last_became_active":"2024-04-14T23:48:43.238714+0000","last_became_peered":"2024-04-14T23:48:43.238714+0000","last_unstale":"2024-04-14T23:48:44.230387+0000","last_undegraded":"2024-04-14T23:48:44.230387+0000","last_fullsized":"2024-04-14T23:48:44.230387+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:05:30.655048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.231004+0000","last_change":"2024-04-14T23:48:43.229325+0000","last_active":"2024-04-14T23:48:44.231004+0000","last_peered":"2024-04-14T23:48:44.231004+0000","last_clean":"2024-04-14T23:48:44.231004+0000","last_became_active":"2024-04-14T23:48:43.228618+0000","last_became_peered":"2024-04-14T23:48:43.228618+0000","last_unstale":"2024-04-14T23:48:44.231004+0000","last_undegraded":"2024-04-14T23:48:44.231004+0000","last_fullsized":"2024-04-14T23:48:44.231004+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:06:43.745698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.649355+0000","last_change":"2024-04-14T23:48:43.233288+0000","last_active":"2024-04-14T23:48:44.649355+0000","last_peered":"2024-04-14T23:48:44.649355+0000","last_clean":"2024-04-14T23:48:44.649355+0000","last_became_active":"2024-04-14T23:48:43.232971+0000","last_became_peered":"2024-04-14T23:48:43.232971+0000","last_unstale":"2024-04-14T23:48:44.649355+0000","last_undegraded":"2024-04-14T23:48:44.649355+0000","last_fullsized":"2024-04-14T23:48:44.649355+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:06:23.588161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.549346+0000","last_change":"2024-04-14T23:48:43.233169+0000","last_active":"2024-04-14T23:48:45.549346+0000","last_peered":"2024-04-14T23:48:45.549346+0000","last_clean":"2024-04-14T23:48:45.549346+0000","last_became_active":"2024-04-14T23:48:43.232135+0000","last_became_peered":"2024-04-14T23:48:43.232135+0000","last_unstale":"2024-04-14T23:48:45.549346+0000","last_undegraded":"2024-04-14T23:48:45.549346+0000","last_fullsized":"2024-04-14T23:48:45.549346+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:37:06.166596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.547820+0000","last_change":"2024-04-14T23:48:43.228728+0000","last_active":"2024-04-14T23:48:45.547820+0000","last_peered":"2024-04-14T23:48:45.547820+0000","last_clean":"2024-04-14T23:48:45.547820+0000","last_became_active":"2024-04-14T23:48:43.226225+0000","last_became_peered":"2024-04-14T23:48:43.226225+0000","last_unstale":"2024-04-14T23:48:45.547820+0000","last_undegraded":"2024-04-14T23:48:45.547820+0000","last_fullsized":"2024-04-14T23:48:45.547820+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:06:03.453599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.234949+0000","last_change":"2024-04-14T23:48:43.237234+0000","last_active":"2024-04-14T23:48:44.234949+0000","last_peered":"2024-04-14T23:48:44.234949+0000","last_clean":"2024-04-14T23:48:44.234949+0000","last_became_active":"2024-04-14T23:48:43.230056+0000","last_became_peered":"2024-04-14T23:48:43.230056+0000","last_unstale":"2024-04-14T23:48:44.234949+0000","last_undegraded":"2024-04-14T23:48:44.234949+0000","last_fullsized":"2024-04-14T23:48:44.234949+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:43:44.966384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230697+0000","last_change":"2024-04-14T23:48:43.229387+0000","last_active":"2024-04-14T23:48:44.230697+0000","last_peered":"2024-04-14T23:48:44.230697+0000","last_clean":"2024-04-14T23:48:44.230697+0000","last_became_active":"2024-04-14T23:48:43.229110+0000","last_became_peered":"2024-04-14T23:48:43.229110+0000","last_unstale":"2024-04-14T23:48:44.230697+0000","last_undegraded":"2024-04-14T23:48:44.230697+0000","last_fullsized":"2024-04-14T23:48:44.230697+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T09:18:46.939940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.232750+0000","last_change":"2024-04-14T23:48:43.234154+0000","last_active":"2024-04-14T23:48:44.232750+0000","last_peered":"2024-04-14T23:48:44.232750+0000","last_clean":"2024-04-14T23:48:44.232750+0000","last_became_active":"2024-04-14T23:48:43.233853+0000","last_became_peered":"2024-04-14T23:48:43.233853+0000","last_unstale":"2024-04-14T23:48:44.232750+0000","last_undegraded":"2024-04-14T23:48:44.232750+0000","last_fullsized":"2024-04-14T23:48:44.232750+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T23:54:02.200200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230330+0000","last_change":"2024-04-14T23:48:43.242154+0000","last_active":"2024-04-14T23:48:44.230330+0000","last_peered":"2024-04-14T23:48:44.230330+0000","last_clean":"2024-04-14T23:48:44.230330+0000","last_became_active":"2024-04-14T23:48:43.241808+0000","last_became_peered":"2024-04-14T23:48:43.241808+0000","last_unstale":"2024-04-14T23:48:44.230330+0000","last_undegraded":"2024-04-14T23:48:44.230330+0000","last_fullsized":"2024-04-14T23:48:44.230330+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:48:42.680808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.232996+0000","last_change":"2024-04-14T23:48:43.233530+0000","last_active":"2024-04-14T23:48:44.232996+0000","last_peered":"2024-04-14T23:48:44.232996+0000","last_clean":"2024-04-14T23:48:44.232996+0000","last_became_active":"2024-04-14T23:48:43.233238+0000","last_became_peered":"2024-04-14T23:48:43.233238+0000","last_unstale":"2024-04-14T23:48:44.232996+0000","last_undegraded":"2024-04-14T23:48:44.232996+0000","last_fullsized":"2024-04-14T23:48:44.232996+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:10:58.525655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.549193+0000","last_change":"2024-04-14T23:48:43.222945+0000","last_active":"2024-04-14T23:48:45.549193+0000","last_peered":"2024-04-14T23:48:45.549193+0000","last_clean":"2024-04-14T23:48:45.549193+0000","last_became_active":"2024-04-14T23:48:43.222624+0000","last_became_peered":"2024-04-14T23:48:43.222624+0000","last_unstale":"2024-04-14T23:48:45.549193+0000","last_undegraded":"2024-04-14T23:48:45.549193+0000","last_fullsized":"2024-04-14T23:48:45.549193+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:22:08.289006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.048719+0000","last_change":"2024-04-14T23:48:43.232567+0000","last_active":"2024-04-14T23:48:45.048719+0000","last_peered":"2024-04-14T23:48:45.048719+0000","last_clean":"2024-04-14T23:48:45.048719+0000","last_became_active":"2024-04-14T23:48:43.232277+0000","last_became_peered":"2024-04-14T23:48:43.232277+0000","last_unstale":"2024-04-14T23:48:45.048719+0000","last_undegraded":"2024-04-14T23:48:45.048719+0000","last_fullsized":"2024-04-14T23:48:45.048719+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T04:59:39.714566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.232426+0000","last_change":"2024-04-14T23:48:43.229443+0000","last_active":"2024-04-14T23:48:44.232426+0000","last_peered":"2024-04-14T23:48:44.232426+0000","last_clean":"2024-04-14T23:48:44.232426+0000","last_became_active":"2024-04-14T23:48:43.229172+0000","last_became_peered":"2024-04-14T23:48:43.229172+0000","last_unstale":"2024-04-14T23:48:44.232426+0000","last_undegraded":"2024-04-14T23:48:44.232426+0000","last_fullsized":"2024-04-14T23:48:44.232426+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:41:31.405287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.234812+0000","last_change":"2024-04-14T23:48:43.242202+0000","last_active":"2024-04-14T23:48:44.234812+0000","last_peered":"2024-04-14T23:48:44.234812+0000","last_clean":"2024-04-14T23:48:44.234812+0000","last_became_active":"2024-04-14T23:48:43.241879+0000","last_became_peered":"2024-04-14T23:48:43.241879+0000","last_unstale":"2024-04-14T23:48:44.234812+0000","last_undegraded":"2024-04-14T23:48:44.234812+0000","last_fullsized":"2024-04-14T23:48:44.234812+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:11:27.552454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230871+0000","last_change":"2024-04-14T23:48:43.242068+0000","last_active":"2024-04-14T23:48:44.230871+0000","last_peered":"2024-04-14T23:48:44.230871+0000","last_clean":"2024-04-14T23:48:44.230871+0000","last_became_active":"2024-04-14T23:48:43.241734+0000","last_became_peered":"2024-04-14T23:48:43.241734+0000","last_unstale":"2024-04-14T23:48:44.230871+0000","last_undegraded":"2024-04-14T23:48:44.230871+0000","last_fullsized":"2024-04-14T23:48:44.230871+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:32:04.579906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.648997+0000","last_change":"2024-04-14T23:48:43.236034+0000","last_active":"2024-04-14T23:48:44.648997+0000","last_peered":"2024-04-14T23:48:44.648997+0000","last_clean":"2024-04-14T23:48:44.648997+0000","last_became_active":"2024-04-14T23:48:43.233567+0000","last_became_peered":"2024-04-14T23:48:43.233567+0000","last_unstale":"2024-04-14T23:48:44.648997+0000","last_undegraded":"2024-04-14T23:48:44.648997+0000","last_fullsized":"2024-04-14T23:48:44.648997+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:07:15.904075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.547675+0000","last_change":"2024-04-14T23:48:43.231045+0000","last_active":"2024-04-14T23:48:45.547675+0000","last_peered":"2024-04-14T23:48:45.547675+0000","last_clean":"2024-04-14T23:48:45.547675+0000","last_became_active":"2024-04-14T23:48:43.230626+0000","last_became_peered":"2024-04-14T23:48:43.230626+0000","last_unstale":"2024-04-14T23:48:45.547675+0000","last_undegraded":"2024-04-14T23:48:45.547675+0000","last_fullsized":"2024-04-14T23:48:45.547675+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:24:03.654061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548613+0000","last_change":"2024-04-14T23:48:43.227543+0000","last_active":"2024-04-14T23:48:45.548613+0000","last_peered":"2024-04-14T23:48:45.548613+0000","last_clean":"2024-04-14T23:48:45.548613+0000","last_became_active":"2024-04-14T23:48:43.226773+0000","last_became_peered":"2024-04-14T23:48:43.226773+0000","last_unstale":"2024-04-14T23:48:45.548613+0000","last_undegraded":"2024-04-14T23:48:45.548613+0000","last_fullsized":"2024-04-14T23:48:45.548613+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:44:56.691708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.211656+0000","last_change":"2024-04-14T23:48:42.239254+0000","last_active":"2024-04-14T23:48:45.211656+0000","last_peered":"2024-04-14T23:48:45.211656+0000","last_clean":"2024-04-14T23:48:45.211656+0000","last_became_active":"2024-04-14T23:48:42.238847+0000","last_became_peered":"2024-04-14T23:48:42.238847+0000","last_unstale":"2024-04-14T23:48:45.211656+0000","last_undegraded":"2024-04-14T23:48:45.211656+0000","last_fullsized":"2024-04-14T23:48:45.211656+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:46:16.301900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.235039+0000","last_change":"2024-04-14T23:48:43.237720+0000","last_active":"2024-04-14T23:48:44.235039+0000","last_peered":"2024-04-14T23:48:44.235039+0000","last_clean":"2024-04-14T23:48:44.235039+0000","last_became_active":"2024-04-14T23:48:43.237391+0000","last_became_peered":"2024-04-14T23:48:43.237391+0000","last_unstale":"2024-04-14T23:48:44.235039+0000","last_undegraded":"2024-04-14T23:48:44.235039+0000","last_fullsized":"2024-04-14T23:48:44.235039+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:38:26.879906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548161+0000","last_change":"2024-04-14T23:48:42.219277+0000","last_active":"2024-04-14T23:48:45.548161+0000","last_peered":"2024-04-14T23:48:45.548161+0000","last_clean":"2024-04-14T23:48:45.548161+0000","last_became_active":"2024-04-14T23:48:42.218995+0000","last_became_peered":"2024-04-14T23:48:42.218995+0000","last_unstale":"2024-04-14T23:48:45.548161+0000","last_undegraded":"2024-04-14T23:48:45.548161+0000","last_fullsized":"2024-04-14T23:48:45.548161+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T23:51:33.370857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.232590+0000","last_change":"2024-04-14T23:48:42.236803+0000","last_active":"2024-04-14T23:48:44.232590+0000","last_peered":"2024-04-14T23:48:44.232590+0000","last_clean":"2024-04-14T23:48:44.232590+0000","last_became_active":"2024-04-14T23:48:42.225937+0000","last_became_peered":"2024-04-14T23:48:42.225937+0000","last_unstale":"2024-04-14T23:48:44.232590+0000","last_undegraded":"2024-04-14T23:48:44.232590+0000","last_fullsized":"2024-04-14T23:48:44.232590+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:52:32.609126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.231732+0000","last_change":"2024-04-14T23:48:43.229506+0000","last_active":"2024-04-14T23:48:44.231732+0000","last_peered":"2024-04-14T23:48:44.231732+0000","last_clean":"2024-04-14T23:48:44.231732+0000","last_became_active":"2024-04-14T23:48:43.227356+0000","last_became_peered":"2024-04-14T23:48:43.227356+0000","last_unstale":"2024-04-14T23:48:44.231732+0000","last_undegraded":"2024-04-14T23:48:44.231732+0000","last_fullsized":"2024-04-14T23:48:44.231732+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:13:04.840655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230660+0000","last_change":"2024-04-14T23:48:42.210883+0000","last_active":"2024-04-14T23:48:44.230660+0000","last_peered":"2024-04-14T23:48:44.230660+0000","last_clean":"2024-04-14T23:48:44.230660+0000","last_became_active":"2024-04-14T23:48:42.210607+0000","last_became_peered":"2024-04-14T23:48:42.210607+0000","last_unstale":"2024-04-14T23:48:44.230660+0000","last_undegraded":"2024-04-14T23:48:44.230660+0000","last_fullsized":"2024-04-14T23:48:44.230660+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:33:32.526223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.048141+0000","last_change":"2024-04-14T23:48:43.218259+0000","last_active":"2024-04-14T23:48:45.048141+0000","last_peered":"2024-04-14T23:48:45.048141+0000","last_clean":"2024-04-14T23:48:45.048141+0000","last_became_active":"2024-04-14T23:48:43.217801+0000","last_became_peered":"2024-04-14T23:48:43.217801+0000","last_unstale":"2024-04-14T23:48:45.048141+0000","last_undegraded":"2024-04-14T23:48:45.048141+0000","last_fullsized":"2024-04-14T23:48:45.048141+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T04:13:42.425034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.547845+0000","last_change":"2024-04-14T23:48:42.218652+0000","last_active":"2024-04-14T23:48:45.547845+0000","last_peered":"2024-04-14T23:48:45.547845+0000","last_clean":"2024-04-14T23:48:45.547845+0000","last_became_active":"2024-04-14T23:48:42.218321+0000","last_became_peered":"2024-04-14T23:48:42.218321+0000","last_unstale":"2024-04-14T23:48:45.547845+0000","last_undegraded":"2024-04-14T23:48:45.547845+0000","last_fullsized":"2024-04-14T23:48:45.547845+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T04:22:06.957747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230879+0000","last_change":"2024-04-14T23:48:43.229117+0000","last_active":"2024-04-14T23:48:44.230879+0000","last_peered":"2024-04-14T23:48:44.230879+0000","last_clean":"2024-04-14T23:48:44.230879+0000","last_became_active":"2024-04-14T23:48:43.228805+0000","last_became_peered":"2024-04-14T23:48:43.228805+0000","last_unstale":"2024-04-14T23:48:44.230879+0000","last_undegraded":"2024-04-14T23:48:44.230879+0000","last_fullsized":"2024-04-14T23:48:44.230879+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:10:39.043818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230930+0000","last_change":"2024-04-14T23:48:42.221104+0000","last_active":"2024-04-14T23:48:44.230930+0000","last_peered":"2024-04-14T23:48:44.230930+0000","last_clean":"2024-04-14T23:48:44.230930+0000","last_became_active":"2024-04-14T23:48:42.220767+0000","last_became_peered":"2024-04-14T23:48:42.220767+0000","last_unstale":"2024-04-14T23:48:44.230930+0000","last_undegraded":"2024-04-14T23:48:44.230930+0000","last_fullsized":"2024-04-14T23:48:44.230930+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:23:57.213764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.235822+0000","last_change":"2024-04-14T23:48:43.236916+0000","last_active":"2024-04-14T23:48:44.235822+0000","last_peered":"2024-04-14T23:48:44.235822+0000","last_clean":"2024-04-14T23:48:44.235822+0000","last_became_active":"2024-04-14T23:48:43.236168+0000","last_became_peered":"2024-04-14T23:48:43.236168+0000","last_unstale":"2024-04-14T23:48:44.235822+0000","last_undegraded":"2024-04-14T23:48:44.235822+0000","last_fullsized":"2024-04-14T23:48:44.235822+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:45:52.057070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:48.220199+0000","last_change":"2024-04-14T23:48:42.231329+0000","last_active":"2024-04-14T23:48:48.220199+0000","last_peered":"2024-04-14T23:48:48.220199+0000","last_clean":"2024-04-14T23:48:48.220199+0000","last_became_active":"2024-04-14T23:48:42.230969+0000","last_became_peered":"2024-04-14T23:48:42.230969+0000","last_unstale":"2024-04-14T23:48:48.220199+0000","last_undegraded":"2024-04-14T23:48:48.220199+0000","last_fullsized":"2024-04-14T23:48:48.220199+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:06:16.557589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548833+0000","last_change":"2024-04-14T23:48:43.230531+0000","last_active":"2024-04-14T23:48:45.548833+0000","last_peered":"2024-04-14T23:48:45.548833+0000","last_clean":"2024-04-14T23:48:45.548833+0000","last_became_active":"2024-04-14T23:48:43.230253+0000","last_became_peered":"2024-04-14T23:48:43.230253+0000","last_unstale":"2024-04-14T23:48:45.548833+0000","last_undegraded":"2024-04-14T23:48:45.548833+0000","last_fullsized":"2024-04-14T23:48:45.548833+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:34:37.449824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:48.230834+0000","last_change":"2024-04-14T23:48:42.237069+0000","last_active":"2024-04-14T23:48:48.230834+0000","last_peered":"2024-04-14T23:48:48.230834+0000","last_clean":"2024-04-14T23:48:48.230834+0000","last_became_active":"2024-04-14T23:48:42.236804+0000","last_became_peered":"2024-04-14T23:48:42.236804+0000","last_unstale":"2024-04-14T23:48:48.230834+0000","last_undegraded":"2024-04-14T23:48:48.230834+0000","last_fullsized":"2024-04-14T23:48:48.230834+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:25:37.318379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548904+0000","last_change":"2024-04-14T23:48:43.221771+0000","last_active":"2024-04-14T23:48:45.548904+0000","last_peered":"2024-04-14T23:48:45.548904+0000","last_clean":"2024-04-14T23:48:45.548904+0000","last_became_active":"2024-04-14T23:48:43.221462+0000","last_became_peered":"2024-04-14T23:48:43.221462+0000","last_unstale":"2024-04-14T23:48:45.548904+0000","last_undegraded":"2024-04-14T23:48:45.548904+0000","last_fullsized":"2024-04-14T23:48:45.548904+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:29:52.737196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:48.227204+0000","last_change":"2024-04-14T23:48:42.230518+0000","last_active":"2024-04-14T23:48:48.227204+0000","last_peered":"2024-04-14T23:48:48.227204+0000","last_clean":"2024-04-14T23:48:48.227204+0000","last_became_active":"2024-04-14T23:48:42.229758+0000","last_became_peered":"2024-04-14T23:48:42.229758+0000","last_unstale":"2024-04-14T23:48:48.227204+0000","last_undegraded":"2024-04-14T23:48:48.227204+0000","last_fullsized":"2024-04-14T23:48:48.227204+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:01:41.332553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.234816+0000","last_change":"2024-04-14T23:48:43.229873+0000","last_active":"2024-04-14T23:48:44.234816+0000","last_peered":"2024-04-14T23:48:44.234816+0000","last_clean":"2024-04-14T23:48:44.234816+0000","last_became_active":"2024-04-14T23:48:43.229476+0000","last_became_peered":"2024-04-14T23:48:43.229476+0000","last_unstale":"2024-04-14T23:48:44.234816+0000","last_undegraded":"2024-04-14T23:48:44.234816+0000","last_fullsized":"2024-04-14T23:48:44.234816+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:54:22.874344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230858+0000","last_change":"2024-04-14T23:48:42.225404+0000","last_active":"2024-04-14T23:48:44.230858+0000","last_peered":"2024-04-14T23:48:44.230858+0000","last_clean":"2024-04-14T23:48:44.230858+0000","last_became_active":"2024-04-14T23:48:42.225211+0000","last_became_peered":"2024-04-14T23:48:42.225211+0000","last_unstale":"2024-04-14T23:48:44.230858+0000","last_undegraded":"2024-04-14T23:48:44.230858+0000","last_fullsized":"2024-04-14T23:48:44.230858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:49:45.282259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548573+0000","last_change":"2024-04-14T23:48:43.229543+0000","last_active":"2024-04-14T23:48:45.548573+0000","last_peered":"2024-04-14T23:48:45.548573+0000","last_clean":"2024-04-14T23:48:45.548573+0000","last_became_active":"2024-04-14T23:48:43.229297+0000","last_became_peered":"2024-04-14T23:48:43.229297+0000","last_unstale":"2024-04-14T23:48:45.548573+0000","last_undegraded":"2024-04-14T23:48:45.548573+0000","last_fullsized":"2024-04-14T23:48:45.548573+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:00:17.923013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.547633+0000","last_change":"2024-04-14T23:48:42.220492+0000","last_active":"2024-04-14T23:48:45.547633+0000","last_peered":"2024-04-14T23:48:45.547633+0000","last_clean":"2024-04-14T23:48:45.547633+0000","last_became_active":"2024-04-14T23:48:42.220145+0000","last_became_peered":"2024-04-14T23:48:42.220145+0000","last_unstale":"2024-04-14T23:48:45.547633+0000","last_undegraded":"2024-04-14T23:48:45.547633+0000","last_fullsized":"2024-04-14T23:48:45.547633+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:17:45.278993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.547774+0000","last_change":"2024-04-14T23:48:43.235676+0000","last_active":"2024-04-14T23:48:45.547774+0000","last_peered":"2024-04-14T23:48:45.547774+0000","last_clean":"2024-04-14T23:48:45.547774+0000","last_became_active":"2024-04-14T23:48:43.235319+0000","last_became_peered":"2024-04-14T23:48:43.235319+0000","last_unstale":"2024-04-14T23:48:45.547774+0000","last_undegraded":"2024-04-14T23:48:45.547774+0000","last_fullsized":"2024-04-14T23:48:45.547774+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:32:06.283229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:48.230880+0000","last_change":"2024-04-14T23:48:42.231178+0000","last_active":"2024-04-14T23:48:48.230880+0000","last_peered":"2024-04-14T23:48:48.230880+0000","last_clean":"2024-04-14T23:48:48.230880+0000","last_became_active":"2024-04-14T23:48:42.230792+0000","last_became_peered":"2024-04-14T23:48:42.230792+0000","last_unstale":"2024-04-14T23:48:48.230880+0000","last_undegraded":"2024-04-14T23:48:48.230880+0000","last_fullsized":"2024-04-14T23:48:48.230880+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:22:40.744314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548527+0000","last_change":"2024-04-14T23:48:43.228290+0000","last_active":"2024-04-14T23:48:45.548527+0000","last_peered":"2024-04-14T23:48:45.548527+0000","last_clean":"2024-04-14T23:48:45.548527+0000","last_became_active":"2024-04-14T23:48:43.228004+0000","last_became_peered":"2024-04-14T23:48:43.228004+0000","last_unstale":"2024-04-14T23:48:45.548527+0000","last_undegraded":"2024-04-14T23:48:45.548527+0000","last_fullsized":"2024-04-14T23:48:45.548527+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:20:06.572625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548424+0000","last_change":"2024-04-14T23:48:42.231936+0000","last_active":"2024-04-14T23:48:45.548424+0000","last_peered":"2024-04-14T23:48:45.548424+0000","last_clean":"2024-04-14T23:48:45.548424+0000","last_became_active":"2024-04-14T23:48:42.231612+0000","last_became_peered":"2024-04-14T23:48:42.231612+0000","last_unstale":"2024-04-14T23:48:45.548424+0000","last_undegraded":"2024-04-14T23:48:45.548424+0000","last_fullsized":"2024-04-14T23:48:45.548424+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:45:03.098024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.549200+0000","last_change":"2024-04-14T23:48:43.228933+0000","last_active":"2024-04-14T23:48:45.549200+0000","last_peered":"2024-04-14T23:48:45.549200+0000","last_clean":"2024-04-14T23:48:45.549200+0000","last_became_active":"2024-04-14T23:48:43.228665+0000","last_became_peered":"2024-04-14T23:48:43.228665+0000","last_unstale":"2024-04-14T23:48:45.549200+0000","last_undegraded":"2024-04-14T23:48:45.549200+0000","last_fullsized":"2024-04-14T23:48:45.549200+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:43:04.747711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:48.230335+0000","last_change":"2024-04-14T23:48:42.239794+0000","last_active":"2024-04-14T23:48:48.230335+0000","last_peered":"2024-04-14T23:48:48.230335+0000","last_clean":"2024-04-14T23:48:48.230335+0000","last_became_active":"2024-04-14T23:48:42.239505+0000","last_became_peered":"2024-04-14T23:48:42.239505+0000","last_unstale":"2024-04-14T23:48:48.230335+0000","last_undegraded":"2024-04-14T23:48:48.230335+0000","last_fullsized":"2024-04-14T23:48:48.230335+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T09:43:16.582708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.549265+0000","last_change":"2024-04-14T23:48:43.230251+0000","last_active":"2024-04-14T23:48:45.549265+0000","last_peered":"2024-04-14T23:48:45.549265+0000","last_clean":"2024-04-14T23:48:45.549265+0000","last_became_active":"2024-04-14T23:48:43.229949+0000","last_became_peered":"2024-04-14T23:48:43.229949+0000","last_unstale":"2024-04-14T23:48:45.549265+0000","last_undegraded":"2024-04-14T23:48:45.549265+0000","last_fullsized":"2024-04-14T23:48:45.549265+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T09:54:42.581135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.048901+0000","last_change":"2024-04-14T23:48:42.227708+0000","last_active":"2024-04-14T23:48:45.048901+0000","last_peered":"2024-04-14T23:48:45.048901+0000","last_clean":"2024-04-14T23:48:45.048901+0000","last_became_active":"2024-04-14T23:48:42.226845+0000","last_became_peered":"2024-04-14T23:48:42.226845+0000","last_unstale":"2024-04-14T23:48:45.048901+0000","last_undegraded":"2024-04-14T23:48:45.048901+0000","last_fullsized":"2024-04-14T23:48:45.048901+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:02:54.468796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230624+0000","last_change":"2024-04-14T23:48:43.227439+0000","last_active":"2024-04-14T23:48:44.230624+0000","last_peered":"2024-04-14T23:48:44.230624+0000","last_clean":"2024-04-14T23:48:44.230624+0000","last_became_active":"2024-04-14T23:48:43.226717+0000","last_became_peered":"2024-04-14T23:48:43.226717+0000","last_unstale":"2024-04-14T23:48:44.230624+0000","last_undegraded":"2024-04-14T23:48:44.230624+0000","last_fullsized":"2024-04-14T23:48:44.230624+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:01:11.249999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.649211+0000","last_change":"2024-04-14T23:48:42.232164+0000","last_active":"2024-04-14T23:48:44.649211+0000","last_peered":"2024-04-14T23:48:44.649211+0000","last_clean":"2024-04-14T23:48:44.649211+0000","last_became_active":"2024-04-14T23:48:42.229785+0000","last_became_peered":"2024-04-14T23:48:42.229785+0000","last_unstale":"2024-04-14T23:48:44.649211+0000","last_undegraded":"2024-04-14T23:48:44.649211+0000","last_fullsized":"2024-04-14T23:48:44.649211+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T04:21:37.538735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.549097+0000","last_change":"2024-04-14T23:48:43.232949+0000","last_active":"2024-04-14T23:48:45.549097+0000","last_peered":"2024-04-14T23:48:45.549097+0000","last_clean":"2024-04-14T23:48:45.549097+0000","last_became_active":"2024-04-14T23:48:43.232612+0000","last_became_peered":"2024-04-14T23:48:43.232612+0000","last_unstale":"2024-04-14T23:48:45.549097+0000","last_undegraded":"2024-04-14T23:48:45.549097+0000","last_fullsized":"2024-04-14T23:48:45.549097+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:01:00.524663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548069+0000","last_change":"2024-04-14T23:48:38.150200+0000","last_active":"2024-04-14T23:48:45.548069+0000","last_peered":"2024-04-14T23:48:45.548069+0000","last_clean":"2024-04-14T23:48:45.548069+0000","last_became_active":"2024-04-14T23:48:38.149301+0000","last_became_peered":"2024-04-14T23:48:38.149301+0000","last_unstale":"2024-04-14T23:48:45.548069+0000","last_undegraded":"2024-04-14T23:48:45.548069+0000","last_fullsized":"2024-04-14T23:48:45.548069+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:37.129663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:37.129663+0000","last_clean_scrub_stamp":"2024-04-14T23:48:37.129663+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T23:58:01.303851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230439+0000","last_change":"2024-04-14T23:48:42.229238+0000","last_active":"2024-04-14T23:48:44.230439+0000","last_peered":"2024-04-14T23:48:44.230439+0000","last_clean":"2024-04-14T23:48:44.230439+0000","last_became_active":"2024-04-14T23:48:42.228994+0000","last_became_peered":"2024-04-14T23:48:42.228994+0000","last_unstale":"2024-04-14T23:48:44.230439+0000","last_undegraded":"2024-04-14T23:48:44.230439+0000","last_fullsized":"2024-04-14T23:48:44.230439+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:47:15.406228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.232256+0000","last_change":"2024-04-14T23:48:43.223973+0000","last_active":"2024-04-14T23:48:44.232256+0000","last_peered":"2024-04-14T23:48:44.232256+0000","last_clean":"2024-04-14T23:48:44.232256+0000","last_became_active":"2024-04-14T23:48:43.223665+0000","last_became_peered":"2024-04-14T23:48:43.223665+0000","last_unstale":"2024-04-14T23:48:44.232256+0000","last_undegraded":"2024-04-14T23:48:44.232256+0000","last_fullsized":"2024-04-14T23:48:44.232256+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:23:28.495821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.547518+0000","last_change":"2024-04-14T23:48:42.230404+0000","last_active":"2024-04-14T23:48:45.547518+0000","last_peered":"2024-04-14T23:48:45.547518+0000","last_clean":"2024-04-14T23:48:45.547518+0000","last_became_active":"2024-04-14T23:48:42.230173+0000","last_became_peered":"2024-04-14T23:48:42.230173+0000","last_unstale":"2024-04-14T23:48:45.547518+0000","last_undegraded":"2024-04-14T23:48:45.547518+0000","last_fullsized":"2024-04-14T23:48:45.547518+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:23:45.666680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.234372+0000","last_change":"2024-04-14T23:48:43.231958+0000","last_active":"2024-04-14T23:48:44.234372+0000","last_peered":"2024-04-14T23:48:44.234372+0000","last_clean":"2024-04-14T23:48:44.234372+0000","last_became_active":"2024-04-14T23:48:43.231025+0000","last_became_peered":"2024-04-14T23:48:43.231025+0000","last_unstale":"2024-04-14T23:48:44.234372+0000","last_undegraded":"2024-04-14T23:48:44.234372+0000","last_fullsized":"2024-04-14T23:48:44.234372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T23:51:47.947116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.649047+0000","last_change":"2024-04-14T23:48:42.240651+0000","last_active":"2024-04-14T23:48:44.649047+0000","last_peered":"2024-04-14T23:48:44.649047+0000","last_clean":"2024-04-14T23:48:44.649047+0000","last_became_active":"2024-04-14T23:48:42.240351+0000","last_became_peered":"2024-04-14T23:48:42.240351+0000","last_unstale":"2024-04-14T23:48:44.649047+0000","last_undegraded":"2024-04-14T23:48:44.649047+0000","last_fullsized":"2024-04-14T23:48:44.649047+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:03:02.804444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.048231+0000","last_change":"2024-04-14T23:48:43.232415+0000","last_active":"2024-04-14T23:48:45.048231+0000","last_peered":"2024-04-14T23:48:45.048231+0000","last_clean":"2024-04-14T23:48:45.048231+0000","last_became_active":"2024-04-14T23:48:43.232120+0000","last_became_peered":"2024-04-14T23:48:43.232120+0000","last_unstale":"2024-04-14T23:48:45.048231+0000","last_undegraded":"2024-04-14T23:48:45.048231+0000","last_fullsized":"2024-04-14T23:48:45.048231+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:05:27.562738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230988+0000","last_change":"2024-04-14T23:48:42.233743+0000","last_active":"2024-04-14T23:48:44.230988+0000","last_peered":"2024-04-14T23:48:44.230988+0000","last_clean":"2024-04-14T23:48:44.230988+0000","last_became_active":"2024-04-14T23:48:42.233354+0000","last_became_peered":"2024-04-14T23:48:42.233354+0000","last_unstale":"2024-04-14T23:48:44.230988+0000","last_undegraded":"2024-04-14T23:48:44.230988+0000","last_fullsized":"2024-04-14T23:48:44.230988+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:49:23.045968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548229+0000","last_change":"2024-04-14T23:48:43.232468+0000","last_active":"2024-04-14T23:48:45.548229+0000","last_peered":"2024-04-14T23:48:45.548229+0000","last_clean":"2024-04-14T23:48:45.548229+0000","last_became_active":"2024-04-14T23:48:43.232119+0000","last_became_peered":"2024-04-14T23:48:43.232119+0000","last_unstale":"2024-04-14T23:48:45.548229+0000","last_undegraded":"2024-04-14T23:48:45.548229+0000","last_fullsized":"2024-04-14T23:48:45.548229+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:24:12.553325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.048630+0000","last_change":"2024-04-14T23:48:42.242249+0000","last_active":"2024-04-14T23:48:45.048630+0000","last_peered":"2024-04-14T23:48:45.048630+0000","last_clean":"2024-04-14T23:48:45.048630+0000","last_became_active":"2024-04-14T23:48:42.242039+0000","last_became_peered":"2024-04-14T23:48:42.242039+0000","last_unstale":"2024-04-14T23:48:45.048630+0000","last_undegraded":"2024-04-14T23:48:45.048630+0000","last_fullsized":"2024-04-14T23:48:45.048630+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:46:16.301900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230960+0000","last_change":"2024-04-14T23:48:43.242228+0000","last_active":"2024-04-14T23:48:44.230960+0000","last_peered":"2024-04-14T23:48:44.230960+0000","last_clean":"2024-04-14T23:48:44.230960+0000","last_became_active":"2024-04-14T23:48:43.241776+0000","last_became_peered":"2024-04-14T23:48:43.241776+0000","last_unstale":"2024-04-14T23:48:44.230960+0000","last_undegraded":"2024-04-14T23:48:44.230960+0000","last_fullsized":"2024-04-14T23:48:44.230960+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:48:23.653866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.549314+0000","last_change":"2024-04-14T23:48:42.232537+0000","last_active":"2024-04-14T23:48:45.549314+0000","last_peered":"2024-04-14T23:48:45.549314+0000","last_clean":"2024-04-14T23:48:45.549314+0000","last_became_active":"2024-04-14T23:48:42.232325+0000","last_became_peered":"2024-04-14T23:48:42.232325+0000","last_unstale":"2024-04-14T23:48:45.549314+0000","last_undegraded":"2024-04-14T23:48:45.549314+0000","last_fullsized":"2024-04-14T23:48:45.549314+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:34:37.448301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.048333+0000","last_change":"2024-04-14T23:48:43.234445+0000","last_active":"2024-04-14T23:48:45.048333+0000","last_peered":"2024-04-14T23:48:45.048333+0000","last_clean":"2024-04-14T23:48:45.048333+0000","last_became_active":"2024-04-14T23:48:43.234150+0000","last_became_peered":"2024-04-14T23:48:43.234150+0000","last_unstale":"2024-04-14T23:48:45.048333+0000","last_undegraded":"2024-04-14T23:48:45.048333+0000","last_fullsized":"2024-04-14T23:48:45.048333+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:50:59.708087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.049088+0000","last_change":"2024-04-14T23:48:42.230014+0000","last_active":"2024-04-14T23:48:45.049088+0000","last_peered":"2024-04-14T23:48:45.049088+0000","last_clean":"2024-04-14T23:48:45.049088+0000","last_became_active":"2024-04-14T23:48:42.229712+0000","last_became_peered":"2024-04-14T23:48:42.229712+0000","last_unstale":"2024-04-14T23:48:45.049088+0000","last_undegraded":"2024-04-14T23:48:45.049088+0000","last_fullsized":"2024-04-14T23:48:45.049088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:54:20.595391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.233392+0000","last_change":"2024-04-14T23:48:43.229128+0000","last_active":"2024-04-14T23:48:44.233392+0000","last_peered":"2024-04-14T23:48:44.233392+0000","last_clean":"2024-04-14T23:48:44.233392+0000","last_became_active":"2024-04-14T23:48:43.228739+0000","last_became_peered":"2024-04-14T23:48:43.228739+0000","last_unstale":"2024-04-14T23:48:44.233392+0000","last_undegraded":"2024-04-14T23:48:44.233392+0000","last_fullsized":"2024-04-14T23:48:44.233392+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:08:08.148345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:48.223491+0000","last_change":"2024-04-14T23:48:42.220035+0000","last_active":"2024-04-14T23:48:48.223491+0000","last_peered":"2024-04-14T23:48:48.223491+0000","last_clean":"2024-04-14T23:48:48.223491+0000","last_became_active":"2024-04-14T23:48:42.219750+0000","last_became_peered":"2024-04-14T23:48:42.219750+0000","last_unstale":"2024-04-14T23:48:48.223491+0000","last_undegraded":"2024-04-14T23:48:48.223491+0000","last_fullsized":"2024-04-14T23:48:48.223491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:00:58.928166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.235426+0000","last_change":"2024-04-14T23:48:43.232997+0000","last_active":"2024-04-14T23:48:44.235426+0000","last_peered":"2024-04-14T23:48:44.235426+0000","last_clean":"2024-04-14T23:48:44.235426+0000","last_became_active":"2024-04-14T23:48:43.232730+0000","last_became_peered":"2024-04-14T23:48:43.232730+0000","last_unstale":"2024-04-14T23:48:44.235426+0000","last_undegraded":"2024-04-14T23:48:44.235426+0000","last_fullsized":"2024-04-14T23:48:44.235426+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:38:44.068804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.547928+0000","last_change":"2024-04-14T23:48:42.232087+0000","last_active":"2024-04-14T23:48:45.547928+0000","last_peered":"2024-04-14T23:48:45.547928+0000","last_clean":"2024-04-14T23:48:45.547928+0000","last_became_active":"2024-04-14T23:48:42.231796+0000","last_became_peered":"2024-04-14T23:48:42.231796+0000","last_unstale":"2024-04-14T23:48:45.547928+0000","last_undegraded":"2024-04-14T23:48:45.547928+0000","last_fullsized":"2024-04-14T23:48:45.547928+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:14:35.050093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548007+0000","last_change":"2024-04-14T23:48:43.232871+0000","last_active":"2024-04-14T23:48:45.548007+0000","last_peered":"2024-04-14T23:48:45.548007+0000","last_clean":"2024-04-14T23:48:45.548007+0000","last_became_active":"2024-04-14T23:48:43.232594+0000","last_became_peered":"2024-04-14T23:48:43.232594+0000","last_unstale":"2024-04-14T23:48:45.548007+0000","last_undegraded":"2024-04-14T23:48:45.548007+0000","last_fullsized":"2024-04-14T23:48:45.548007+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:27:37.320994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.048370+0000","last_change":"2024-04-14T23:48:42.231237+0000","last_active":"2024-04-14T23:48:45.048370+0000","last_peered":"2024-04-14T23:48:45.048370+0000","last_clean":"2024-04-14T23:48:45.048370+0000","last_became_active":"2024-04-14T23:48:42.230902+0000","last_became_peered":"2024-04-14T23:48:42.230902+0000","last_unstale":"2024-04-14T23:48:45.048370+0000","last_undegraded":"2024-04-14T23:48:45.048370+0000","last_fullsized":"2024-04-14T23:48:45.048370+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:28:37.228039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.232080+0000","last_change":"2024-04-14T23:48:43.233035+0000","last_active":"2024-04-14T23:48:44.232080+0000","last_peered":"2024-04-14T23:48:44.232080+0000","last_clean":"2024-04-14T23:48:44.232080+0000","last_became_active":"2024-04-14T23:48:43.232747+0000","last_became_peered":"2024-04-14T23:48:43.232747+0000","last_unstale":"2024-04-14T23:48:44.232080+0000","last_undegraded":"2024-04-14T23:48:44.232080+0000","last_fullsized":"2024-04-14T23:48:44.232080+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:38:57.324408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.048301+0000","last_change":"2024-04-14T23:48:42.240496+0000","last_active":"2024-04-14T23:48:45.048301+0000","last_peered":"2024-04-14T23:48:45.048301+0000","last_clean":"2024-04-14T23:48:45.048301+0000","last_became_active":"2024-04-14T23:48:42.238524+0000","last_became_peered":"2024-04-14T23:48:42.238524+0000","last_unstale":"2024-04-14T23:48:45.048301+0000","last_undegraded":"2024-04-14T23:48:45.048301+0000","last_fullsized":"2024-04-14T23:48:45.048301+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:29:35.218759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.233225+0000","last_change":"2024-04-14T23:48:43.234025+0000","last_active":"2024-04-14T23:48:44.233225+0000","last_peered":"2024-04-14T23:48:44.233225+0000","last_clean":"2024-04-14T23:48:44.233225+0000","last_became_active":"2024-04-14T23:48:43.233651+0000","last_became_peered":"2024-04-14T23:48:43.233651+0000","last_unstale":"2024-04-14T23:48:44.233225+0000","last_undegraded":"2024-04-14T23:48:44.233225+0000","last_fullsized":"2024-04-14T23:48:44.233225+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:26:58.094545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:48.217913+0000","last_change":"2024-04-14T23:48:42.218897+0000","last_active":"2024-04-14T23:48:48.217913+0000","last_peered":"2024-04-14T23:48:48.217913+0000","last_clean":"2024-04-14T23:48:48.217913+0000","last_became_active":"2024-04-14T23:48:42.218580+0000","last_became_peered":"2024-04-14T23:48:42.218580+0000","last_unstale":"2024-04-14T23:48:48.217913+0000","last_undegraded":"2024-04-14T23:48:48.217913+0000","last_fullsized":"2024-04-14T23:48:48.217913+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:26:44.477638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548008+0000","last_change":"2024-04-14T23:48:43.219871+0000","last_active":"2024-04-14T23:48:45.548008+0000","last_peered":"2024-04-14T23:48:45.548008+0000","last_clean":"2024-04-14T23:48:45.548008+0000","last_became_active":"2024-04-14T23:48:43.219601+0000","last_became_peered":"2024-04-14T23:48:43.219601+0000","last_unstale":"2024-04-14T23:48:45.548008+0000","last_undegraded":"2024-04-14T23:48:45.548008+0000","last_fullsized":"2024-04-14T23:48:45.548008+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:36:20.379172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230971+0000","last_change":"2024-04-14T23:48:43.222259+0000","last_active":"2024-04-14T23:48:44.230971+0000","last_peered":"2024-04-14T23:48:44.230971+0000","last_clean":"2024-04-14T23:48:44.230971+0000","last_became_active":"2024-04-14T23:48:43.221842+0000","last_became_peered":"2024-04-14T23:48:43.221842+0000","last_unstale":"2024-04-14T23:48:44.230971+0000","last_undegraded":"2024-04-14T23:48:44.230971+0000","last_fullsized":"2024-04-14T23:48:44.230971+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:12:20.038987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:48.217880+0000","last_change":"2024-04-14T23:48:42.221314+0000","last_active":"2024-04-14T23:48:48.217880+0000","last_peered":"2024-04-14T23:48:48.217880+0000","last_clean":"2024-04-14T23:48:48.217880+0000","last_became_active":"2024-04-14T23:48:42.221028+0000","last_became_peered":"2024-04-14T23:48:42.221028+0000","last_unstale":"2024-04-14T23:48:48.217880+0000","last_undegraded":"2024-04-14T23:48:48.217880+0000","last_fullsized":"2024-04-14T23:48:48.217880+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T09:58:18.195400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230606+0000","last_change":"2024-04-14T23:48:43.224577+0000","last_active":"2024-04-14T23:48:44.230606+0000","last_peered":"2024-04-14T23:48:44.230606+0000","last_clean":"2024-04-14T23:48:44.230606+0000","last_became_active":"2024-04-14T23:48:43.224312+0000","last_became_peered":"2024-04-14T23:48:43.224312+0000","last_unstale":"2024-04-14T23:48:44.230606+0000","last_undegraded":"2024-04-14T23:48:44.230606+0000","last_fullsized":"2024-04-14T23:48:44.230606+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:52:44.955773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:48.217170+0000","last_change":"2024-04-14T23:48:42.221000+0000","last_active":"2024-04-14T23:48:48.217170+0000","last_peered":"2024-04-14T23:48:48.217170+0000","last_clean":"2024-04-14T23:48:48.217170+0000","last_became_active":"2024-04-14T23:48:42.219813+0000","last_became_peered":"2024-04-14T23:48:42.219813+0000","last_unstale":"2024-04-14T23:48:48.217170+0000","last_undegraded":"2024-04-14T23:48:48.217170+0000","last_fullsized":"2024-04-14T23:48:48.217170+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:00:38.107953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.235379+0000","last_change":"2024-04-14T23:48:43.236946+0000","last_active":"2024-04-14T23:48:44.235379+0000","last_peered":"2024-04-14T23:48:44.235379+0000","last_clean":"2024-04-14T23:48:44.235379+0000","last_became_active":"2024-04-14T23:48:43.236595+0000","last_became_peered":"2024-04-14T23:48:43.236595+0000","last_unstale":"2024-04-14T23:48:44.235379+0000","last_undegraded":"2024-04-14T23:48:44.235379+0000","last_fullsized":"2024-04-14T23:48:44.235379+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:56:46.037478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.211743+0000","last_change":"2024-04-14T23:48:42.239255+0000","last_active":"2024-04-14T23:48:45.211743+0000","last_peered":"2024-04-14T23:48:45.211743+0000","last_clean":"2024-04-14T23:48:45.211743+0000","last_became_active":"2024-04-14T23:48:42.238892+0000","last_became_peered":"2024-04-14T23:48:42.238892+0000","last_unstale":"2024-04-14T23:48:45.211743+0000","last_undegraded":"2024-04-14T23:48:45.211743+0000","last_fullsized":"2024-04-14T23:48:45.211743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:44:39.632155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.233893+0000","last_change":"2024-04-14T23:48:43.227240+0000","last_active":"2024-04-14T23:48:44.233893+0000","last_peered":"2024-04-14T23:48:44.233893+0000","last_clean":"2024-04-14T23:48:44.233893+0000","last_became_active":"2024-04-14T23:48:43.222707+0000","last_became_peered":"2024-04-14T23:48:43.222707+0000","last_unstale":"2024-04-14T23:48:44.233893+0000","last_undegraded":"2024-04-14T23:48:44.233893+0000","last_fullsized":"2024-04-14T23:48:44.233893+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:02:17.231763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:48.228047+0000","last_change":"2024-04-14T23:48:42.226563+0000","last_active":"2024-04-14T23:48:48.228047+0000","last_peered":"2024-04-14T23:48:48.228047+0000","last_clean":"2024-04-14T23:48:48.228047+0000","last_became_active":"2024-04-14T23:48:42.226238+0000","last_became_peered":"2024-04-14T23:48:42.226238+0000","last_unstale":"2024-04-14T23:48:48.228047+0000","last_undegraded":"2024-04-14T23:48:48.228047+0000","last_fullsized":"2024-04-14T23:48:48.228047+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:01:36.516898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65128,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":286720,"data_stored":666555,"data_compressed":32564,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":221184,"data_stored":75695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":75731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":225280,"data_stored":75765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":294912,"data_stored":666611,"data_compressed":32564,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32564,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32564,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-14T23:48:54.478 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-14T23:48:54.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.675+0000 7f9f31600640 1 Processor -- start 2024-04-14T23:48:54.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.675+0000 7f9f31600640 1 -- start start 2024-04-14T23:48:54.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.677+0000 7f9f31600640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f9f2c0c75c0 0x7f9f2c158e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:54.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.677+0000 7f9f31600640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f2c15f7c0 con 0x7f9f2c0c8740 2024-04-14T23:48:54.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.677+0000 7f9f31600640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f2c15faa0 con 0x7f9f2c0c8360 2024-04-14T23:48:54.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.677+0000 7f9f31600640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f9f2c0c6b70 con 0x7f9f2c0c75c0 2024-04-14T23:48:54.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.678+0000 7f9f22000640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f9f2c0c8740 0x7f9f2c06f310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53316/0 (socket says 172.21.15.60:53316) 2024-04-14T23:48:54.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.678+0000 7f9f22000640 1 -- 172.21.15.60:0/2125891583 learned_addr learned my addr 172.21.15.60:0/2125891583 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:48:54.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.678+0000 7f9f2aa00640 1 --1- 172.21.15.60:0/2125891583 >> v1:172.21.15.186:6789/0 conn(0x7f9f2c0c8360 0x7f9f2c15ef80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.60:33960/0 (socket says 172.21.15.60:33960) 2024-04-14T23:48:54.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.678+0000 7f9f2b400640 1 --2- 172.21.15.60:0/2125891583 >> v2:172.21.15.186:3300/0 conn(0x7f9f2c0c75c0 0x7f9f2c158e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:54.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.678+0000 7f9f2a000640 1 -- 172.21.15.60:0/2125891583 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3708493414 0 0) 0x7f9f2c15faa0 con 0x7f9f2c0c8360 2024-04-14T23:48:54.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.678+0000 7f9f2a000640 1 -- 172.21.15.60:0/2125891583 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f10003520 con 0x7f9f2c0c8360 2024-04-14T23:48:54.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.678+0000 7f9f2a000640 1 -- 172.21.15.60:0/2125891583 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 241308568 0 0) 0x7f9f2c15f7c0 con 0x7f9f2c0c8740 2024-04-14T23:48:54.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.678+0000 7f9f2a000640 1 -- 172.21.15.60:0/2125891583 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f2c15faa0 con 0x7f9f2c0c8740 2024-04-14T23:48:54.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.678+0000 7f9f2b400640 1 -- 172.21.15.60:0/2125891583 >> v1:172.21.15.186:6789/0 conn(0x7f9f2c0c8360 legacy=0x7f9f2c15ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:54.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.678+0000 7f9f2b400640 1 -- 172.21.15.60:0/2125891583 >> v1:172.21.15.60:6789/0 conn(0x7f9f2c0c8740 legacy=0x7f9f2c06f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:54.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.678+0000 7f9f2b400640 1 -- 172.21.15.60:0/2125891583 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f2c15fd80 con 0x7f9f2c0c75c0 2024-04-14T23:48:54.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.680+0000 7f9f2b400640 1 --2- 172.21.15.60:0/2125891583 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f2c0c75c0 0x7f9f2c158e30 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9f24009d90 tx=0x7f9f2402bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea7be97f30513e5e server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:54.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.680+0000 7f9f2a000640 1 -- 172.21.15.60:0/2125891583 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f24039070 con 0x7f9f2c0c75c0 2024-04-14T23:48:54.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.680+0000 7f9f2a000640 1 -- 172.21.15.60:0/2125891583 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f2402cd00 con 0x7f9f2c0c75c0 2024-04-14T23:48:54.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.680+0000 7f9f2a000640 1 -- 172.21.15.60:0/2125891583 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f24034640 con 0x7f9f2c0c75c0 2024-04-14T23:48:54.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.681+0000 7f9f31600640 1 -- 172.21.15.60:0/2125891583 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f2c0c75c0 msgr2=0x7f9f2c158e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:54.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.681+0000 7f9f31600640 1 --2- 172.21.15.60:0/2125891583 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f2c0c75c0 0x7f9f2c158e30 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9f24009d90 tx=0x7f9f2402bf50 comp rx=0 tx=0).stop 2024-04-14T23:48:54.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.681+0000 7f9f31600640 1 -- 172.21.15.60:0/2125891583 shutdown_connections 2024-04-14T23:48:54.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.681+0000 7f9f31600640 1 --2- 172.21.15.60:0/2125891583 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f2c0c75c0 0x7f9f2c158e30 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:54.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.681+0000 7f9f31600640 1 -- 172.21.15.60:0/2125891583 >> 172.21.15.60:0/2125891583 conn(0x7f9f2c0cc170 msgr2=0x7f9f2c0cc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:54.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.681+0000 7f9f31600640 1 -- 172.21.15.60:0/2125891583 shutdown_connections 2024-04-14T23:48:54.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.681+0000 7f9f31600640 1 -- 172.21.15.60:0/2125891583 wait complete. 2024-04-14T23:48:54.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.683+0000 7f9f31600640 1 Processor -- start 2024-04-14T23:48:54.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.683+0000 7f9f31600640 1 -- start start 2024-04-14T23:48:54.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.684+0000 7f9f31600640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9f2c0c75c0 0x7f9f2c15de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:54.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.684+0000 7f9f31600640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f2c0c8360 0x7f9f2c15e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:54.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.684+0000 7f9f31600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f2c0c8740 0x7f9f2c13fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:54.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.684+0000 7f9f31600640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f9f2c0c70a0 con 0x7f9f2c0c8360 2024-04-14T23:48:54.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.684+0000 7f9f31600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9f2c06fa20 con 0x7f9f2c0c8740 2024-04-14T23:48:54.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.684+0000 7f9f31600640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f9f2c071030 con 0x7f9f2c0c75c0 2024-04-14T23:48:54.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.685+0000 7f9f22000640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f2c0c8360 0x7f9f2c15e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:54.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.685+0000 7f9f22000640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f2c0c8360 0x7f9f2c15e380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41866/0 (socket says 172.21.15.60:41866) 2024-04-14T23:48:54.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.685+0000 7f9f22000640 1 -- 172.21.15.60:0/4275193892 learned_addr learned my addr 172.21.15.60:0/4275193892 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:54.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.685+0000 7f9f2aa00640 1 --2- 172.21.15.60:0/4275193892 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9f2c0c75c0 0x7f9f2c15de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:54.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.685+0000 7f9f2b400640 1 --2- 172.21.15.60:0/4275193892 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f2c0c8740 0x7f9f2c13fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:54.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.685+0000 7f9f22000640 1 -- 172.21.15.60:0/4275193892 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9f2c0c75c0 msgr2=0x7f9f2c15de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:54.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.685+0000 7f9f22000640 1 --2- 172.21.15.60:0/4275193892 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9f2c0c75c0 0x7f9f2c15de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:54.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.685+0000 7f9f22000640 1 -- 172.21.15.60:0/4275193892 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f2c0c8740 msgr2=0x7f9f2c13fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:54.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.685+0000 7f9f22000640 1 --2- 172.21.15.60:0/4275193892 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f2c0c8740 0x7f9f2c13fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:54.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.685+0000 7f9f22000640 1 -- 172.21.15.60:0/4275193892 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f2c1f4f90 con 0x7f9f2c0c8360 2024-04-14T23:48:54.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.686+0000 7f9f22000640 1 --2- 172.21.15.60:0/4275193892 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f2c0c8360 0x7f9f2c15e380 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9f1400c3c0 tx=0x7f9f1400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:54.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.686+0000 7f9f23400640 1 -- 172.21.15.60:0/4275193892 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f14017070 con 0x7f9f2c0c8360 2024-04-14T23:48:54.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.686+0000 7f9f23400640 1 -- 172.21.15.60:0/4275193892 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f14005ce0 con 0x7f9f2c0c8360 2024-04-14T23:48:54.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.686+0000 7f9f23400640 1 -- 172.21.15.60:0/4275193892 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f140072f0 con 0x7f9f2c0c8360 2024-04-14T23:48:54.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.686+0000 7f9f31600640 1 -- 172.21.15.60:0/4275193892 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f2c15ef00 con 0x7f9f2c0c8360 2024-04-14T23:48:54.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.686+0000 7f9f31600640 1 -- 172.21.15.60:0/4275193892 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f2c1f5510 con 0x7f9f2c0c8360 2024-04-14T23:48:54.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.687+0000 7f9f31600640 1 -- 172.21.15.60:0/4275193892 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f2c068dd0 con 0x7f9f2c0c8360 2024-04-14T23:48:54.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.688+0000 7f9f23400640 1 -- 172.21.15.60:0/4275193892 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f9f1401a070 con 0x7f9f2c0c8360 2024-04-14T23:48:54.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.689+0000 7f9f23400640 1 --2- 172.21.15.60:0/4275193892 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f9f0806b990 0x7f9f0806de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:54.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.689+0000 7f9f23400640 1 -- 172.21.15.60:0/4275193892 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f9f14089c10 con 0x7f9f2c0c8360 2024-04-14T23:48:54.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.689+0000 7f9f2aa00640 1 --2- 172.21.15.60:0/4275193892 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f9f0806b990 0x7f9f0806de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:54.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.690+0000 7f9f2aa00640 1 --2- 172.21.15.60:0/4275193892 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f9f0806b990 0x7f9f0806de50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f9f1c0011e0 tx=0x7f9f1c007d30 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:54.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.695+0000 7f9f23400640 1 -- 172.21.15.60:0/4275193892 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f9f2c068dd0 con 0x7f9f2c0c8360 2024-04-14T23:48:54.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.888+0000 7f9f31600640 1 -- 172.21.15.60:0/4275193892 --> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9f2c15f7c0 con 0x7f9f0806b990 2024-04-14T23:48:54.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.895+0000 7f9f23400640 1 -- 172.21.15.60:0/4275193892 <== mgr.4100 v2:172.21.15.186:6832/714104835 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251915 (secure 0 0 0) 0x7f9f2c15f7c0 con 0x7f9f0806b990 2024-04-14T23:48:54.904 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:48:54.905 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-04-14T23:48:54.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.900+0000 7f9f31600640 1 -- 172.21.15.60:0/4275193892 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f9f0806b990 msgr2=0x7f9f0806de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:54.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.900+0000 7f9f31600640 1 --2- 172.21.15.60:0/4275193892 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f9f0806b990 0x7f9f0806de50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f9f1c0011e0 tx=0x7f9f1c007d30 comp rx=0 tx=0).stop 2024-04-14T23:48:54.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.900+0000 7f9f31600640 1 -- 172.21.15.60:0/4275193892 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f2c0c8360 msgr2=0x7f9f2c15e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:54.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.900+0000 7f9f31600640 1 --2- 172.21.15.60:0/4275193892 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f2c0c8360 0x7f9f2c15e380 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9f1400c3c0 tx=0x7f9f1400c880 comp rx=0 tx=0).stop 2024-04-14T23:48:54.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.900+0000 7f9f31600640 1 -- 172.21.15.60:0/4275193892 shutdown_connections 2024-04-14T23:48:54.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.900+0000 7f9f31600640 1 --2- 172.21.15.60:0/4275193892 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f9f0806b990 0x7f9f0806de50 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:54.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.900+0000 7f9f31600640 1 --2- 172.21.15.60:0/4275193892 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f2c0c8740 0x7f9f2c13fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:54.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.900+0000 7f9f31600640 1 --2- 172.21.15.60:0/4275193892 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f2c0c8360 0x7f9f2c15e380 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:54.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.900+0000 7f9f31600640 1 --2- 172.21.15.60:0/4275193892 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9f2c0c75c0 0x7f9f2c15de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:54.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.900+0000 7f9f31600640 1 -- 172.21.15.60:0/4275193892 >> 172.21.15.60:0/4275193892 conn(0x7f9f2c0cc170 msgr2=0x7f9f2c157820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:54.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.900+0000 7f9f31600640 1 -- 172.21.15.60:0/4275193892 shutdown_connections 2024-04-14T23:48:54.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:54.900+0000 7f9f31600640 1 -- 172.21.15.60:0/4275193892 wait complete. 2024-04-14T23:48:54.922 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-04-14T23:48:54.169000+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218016,"kb_used_data":1888,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756704,"statfs":{"total":773094113280,"available":772870864896,"internally_reserved":0,"allocated":1933312,"data_stored":1790396,"data_compressed":65128,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":58,"num_write_kb":276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002593"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.549104+0000","last_change":"2024-04-14T23:48:43.222542+0000","last_active":"2024-04-14T23:48:45.549104+0000","last_peered":"2024-04-14T23:48:45.549104+0000","last_clean":"2024-04-14T23:48:45.549104+0000","last_became_active":"2024-04-14T23:48:43.222278+0000","last_became_peered":"2024-04-14T23:48:43.222278+0000","last_unstale":"2024-04-14T23:48:45.549104+0000","last_undegraded":"2024-04-14T23:48:45.549104+0000","last_fullsized":"2024-04-14T23:48:45.549104+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:13:06.877485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.049059+0000","last_change":"2024-04-14T23:48:43.226781+0000","last_active":"2024-04-14T23:48:45.049059+0000","last_peered":"2024-04-14T23:48:45.049059+0000","last_clean":"2024-04-14T23:48:45.049059+0000","last_became_active":"2024-04-14T23:48:43.226426+0000","last_became_peered":"2024-04-14T23:48:43.226426+0000","last_unstale":"2024-04-14T23:48:45.049059+0000","last_undegraded":"2024-04-14T23:48:45.049059+0000","last_fullsized":"2024-04-14T23:48:45.049059+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:35:24.780859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.235142+0000","last_change":"2024-04-14T23:48:43.236939+0000","last_active":"2024-04-14T23:48:44.235142+0000","last_peered":"2024-04-14T23:48:44.235142+0000","last_clean":"2024-04-14T23:48:44.235142+0000","last_became_active":"2024-04-14T23:48:43.236616+0000","last_became_peered":"2024-04-14T23:48:43.236616+0000","last_unstale":"2024-04-14T23:48:44.235142+0000","last_undegraded":"2024-04-14T23:48:44.235142+0000","last_fullsized":"2024-04-14T23:48:44.235142+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:35:42.619139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230420+0000","last_change":"2024-04-14T23:48:43.242349+0000","last_active":"2024-04-14T23:48:44.230420+0000","last_peered":"2024-04-14T23:48:44.230420+0000","last_clean":"2024-04-14T23:48:44.230420+0000","last_became_active":"2024-04-14T23:48:43.242000+0000","last_became_peered":"2024-04-14T23:48:43.242000+0000","last_unstale":"2024-04-14T23:48:44.230420+0000","last_undegraded":"2024-04-14T23:48:44.230420+0000","last_fullsized":"2024-04-14T23:48:44.230420+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:33:52.458016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548549+0000","last_change":"2024-04-14T23:48:43.233937+0000","last_active":"2024-04-14T23:48:45.548549+0000","last_peered":"2024-04-14T23:48:45.548549+0000","last_clean":"2024-04-14T23:48:45.548549+0000","last_became_active":"2024-04-14T23:48:43.233674+0000","last_became_peered":"2024-04-14T23:48:43.233674+0000","last_unstale":"2024-04-14T23:48:45.548549+0000","last_undegraded":"2024-04-14T23:48:45.548549+0000","last_fullsized":"2024-04-14T23:48:45.548549+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:00:07.838600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230260+0000","last_change":"2024-04-14T23:48:43.230925+0000","last_active":"2024-04-14T23:48:44.230260+0000","last_peered":"2024-04-14T23:48:44.230260+0000","last_clean":"2024-04-14T23:48:44.230260+0000","last_became_active":"2024-04-14T23:48:43.230405+0000","last_became_peered":"2024-04-14T23:48:43.230405+0000","last_unstale":"2024-04-14T23:48:44.230260+0000","last_undegraded":"2024-04-14T23:48:44.230260+0000","last_fullsized":"2024-04-14T23:48:44.230260+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:15:12.452964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548354+0000","last_change":"2024-04-14T23:48:43.231088+0000","last_active":"2024-04-14T23:48:45.548354+0000","last_peered":"2024-04-14T23:48:45.548354+0000","last_clean":"2024-04-14T23:48:45.548354+0000","last_became_active":"2024-04-14T23:48:43.230784+0000","last_became_peered":"2024-04-14T23:48:43.230784+0000","last_unstale":"2024-04-14T23:48:45.548354+0000","last_undegraded":"2024-04-14T23:48:45.548354+0000","last_fullsized":"2024-04-14T23:48:45.548354+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:23:10.461088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.231230+0000","last_change":"2024-04-14T23:48:43.236019+0000","last_active":"2024-04-14T23:48:44.231230+0000","last_peered":"2024-04-14T23:48:44.231230+0000","last_clean":"2024-04-14T23:48:44.231230+0000","last_became_active":"2024-04-14T23:48:43.235719+0000","last_became_peered":"2024-04-14T23:48:43.235719+0000","last_unstale":"2024-04-14T23:48:44.231230+0000","last_undegraded":"2024-04-14T23:48:44.231230+0000","last_fullsized":"2024-04-14T23:48:44.231230+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T04:41:16.130730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.549156+0000","last_change":"2024-04-14T23:48:43.232166+0000","last_active":"2024-04-14T23:48:45.549156+0000","last_peered":"2024-04-14T23:48:45.549156+0000","last_clean":"2024-04-14T23:48:45.549156+0000","last_became_active":"2024-04-14T23:48:43.231751+0000","last_became_peered":"2024-04-14T23:48:43.231751+0000","last_unstale":"2024-04-14T23:48:45.549156+0000","last_undegraded":"2024-04-14T23:48:45.549156+0000","last_fullsized":"2024-04-14T23:48:45.549156+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:05:47.289092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548848+0000","last_change":"2024-04-14T23:48:43.224938+0000","last_active":"2024-04-14T23:48:45.548848+0000","last_peered":"2024-04-14T23:48:45.548848+0000","last_clean":"2024-04-14T23:48:45.548848+0000","last_became_active":"2024-04-14T23:48:43.224592+0000","last_became_peered":"2024-04-14T23:48:43.224592+0000","last_unstale":"2024-04-14T23:48:45.548848+0000","last_undegraded":"2024-04-14T23:48:45.548848+0000","last_fullsized":"2024-04-14T23:48:45.548848+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:59:12.923596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548591+0000","last_change":"2024-04-14T23:48:43.225290+0000","last_active":"2024-04-14T23:48:45.548591+0000","last_peered":"2024-04-14T23:48:45.548591+0000","last_clean":"2024-04-14T23:48:45.548591+0000","last_became_active":"2024-04-14T23:48:43.224989+0000","last_became_peered":"2024-04-14T23:48:43.224989+0000","last_unstale":"2024-04-14T23:48:45.548591+0000","last_undegraded":"2024-04-14T23:48:45.548591+0000","last_fullsized":"2024-04-14T23:48:45.548591+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:44:49.116229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548742+0000","last_change":"2024-04-14T23:48:43.226194+0000","last_active":"2024-04-14T23:48:45.548742+0000","last_peered":"2024-04-14T23:48:45.548742+0000","last_clean":"2024-04-14T23:48:45.548742+0000","last_became_active":"2024-04-14T23:48:43.225892+0000","last_became_peered":"2024-04-14T23:48:43.225892+0000","last_unstale":"2024-04-14T23:48:45.548742+0000","last_undegraded":"2024-04-14T23:48:45.548742+0000","last_fullsized":"2024-04-14T23:48:45.548742+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:04:12.377770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.233039+0000","last_change":"2024-04-14T23:48:43.220571+0000","last_active":"2024-04-14T23:48:44.233039+0000","last_peered":"2024-04-14T23:48:44.233039+0000","last_clean":"2024-04-14T23:48:44.233039+0000","last_became_active":"2024-04-14T23:48:43.219614+0000","last_became_peered":"2024-04-14T23:48:43.219614+0000","last_unstale":"2024-04-14T23:48:44.233039+0000","last_undegraded":"2024-04-14T23:48:44.233039+0000","last_fullsized":"2024-04-14T23:48:44.233039+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:14:41.854957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.234686+0000","last_change":"2024-04-14T23:48:43.239403+0000","last_active":"2024-04-14T23:48:44.234686+0000","last_peered":"2024-04-14T23:48:44.234686+0000","last_clean":"2024-04-14T23:48:44.234686+0000","last_became_active":"2024-04-14T23:48:43.239136+0000","last_became_peered":"2024-04-14T23:48:43.239136+0000","last_unstale":"2024-04-14T23:48:44.234686+0000","last_undegraded":"2024-04-14T23:48:44.234686+0000","last_fullsized":"2024-04-14T23:48:44.234686+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:27:30.512861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.048083+0000","last_change":"2024-04-14T23:48:43.228157+0000","last_active":"2024-04-14T23:48:45.048083+0000","last_peered":"2024-04-14T23:48:45.048083+0000","last_clean":"2024-04-14T23:48:45.048083+0000","last_became_active":"2024-04-14T23:48:43.227898+0000","last_became_peered":"2024-04-14T23:48:43.227898+0000","last_unstale":"2024-04-14T23:48:45.048083+0000","last_undegraded":"2024-04-14T23:48:45.048083+0000","last_fullsized":"2024-04-14T23:48:45.048083+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:06:37.602935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230387+0000","last_change":"2024-04-14T23:48:43.238986+0000","last_active":"2024-04-14T23:48:44.230387+0000","last_peered":"2024-04-14T23:48:44.230387+0000","last_clean":"2024-04-14T23:48:44.230387+0000","last_became_active":"2024-04-14T23:48:43.238714+0000","last_became_peered":"2024-04-14T23:48:43.238714+0000","last_unstale":"2024-04-14T23:48:44.230387+0000","last_undegraded":"2024-04-14T23:48:44.230387+0000","last_fullsized":"2024-04-14T23:48:44.230387+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:05:30.655048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.231004+0000","last_change":"2024-04-14T23:48:43.229325+0000","last_active":"2024-04-14T23:48:44.231004+0000","last_peered":"2024-04-14T23:48:44.231004+0000","last_clean":"2024-04-14T23:48:44.231004+0000","last_became_active":"2024-04-14T23:48:43.228618+0000","last_became_peered":"2024-04-14T23:48:43.228618+0000","last_unstale":"2024-04-14T23:48:44.231004+0000","last_undegraded":"2024-04-14T23:48:44.231004+0000","last_fullsized":"2024-04-14T23:48:44.231004+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:06:43.745698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.649355+0000","last_change":"2024-04-14T23:48:43.233288+0000","last_active":"2024-04-14T23:48:44.649355+0000","last_peered":"2024-04-14T23:48:44.649355+0000","last_clean":"2024-04-14T23:48:44.649355+0000","last_became_active":"2024-04-14T23:48:43.232971+0000","last_became_peered":"2024-04-14T23:48:43.232971+0000","last_unstale":"2024-04-14T23:48:44.649355+0000","last_undegraded":"2024-04-14T23:48:44.649355+0000","last_fullsized":"2024-04-14T23:48:44.649355+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:06:23.588161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.549346+0000","last_change":"2024-04-14T23:48:43.233169+0000","last_active":"2024-04-14T23:48:45.549346+0000","last_peered":"2024-04-14T23:48:45.549346+0000","last_clean":"2024-04-14T23:48:45.549346+0000","last_became_active":"2024-04-14T23:48:43.232135+0000","last_became_peered":"2024-04-14T23:48:43.232135+0000","last_unstale":"2024-04-14T23:48:45.549346+0000","last_undegraded":"2024-04-14T23:48:45.549346+0000","last_fullsized":"2024-04-14T23:48:45.549346+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:37:06.166596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.547820+0000","last_change":"2024-04-14T23:48:43.228728+0000","last_active":"2024-04-14T23:48:45.547820+0000","last_peered":"2024-04-14T23:48:45.547820+0000","last_clean":"2024-04-14T23:48:45.547820+0000","last_became_active":"2024-04-14T23:48:43.226225+0000","last_became_peered":"2024-04-14T23:48:43.226225+0000","last_unstale":"2024-04-14T23:48:45.547820+0000","last_undegraded":"2024-04-14T23:48:45.547820+0000","last_fullsized":"2024-04-14T23:48:45.547820+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:06:03.453599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.234949+0000","last_change":"2024-04-14T23:48:43.237234+0000","last_active":"2024-04-14T23:48:44.234949+0000","last_peered":"2024-04-14T23:48:44.234949+0000","last_clean":"2024-04-14T23:48:44.234949+0000","last_became_active":"2024-04-14T23:48:43.230056+0000","last_became_peered":"2024-04-14T23:48:43.230056+0000","last_unstale":"2024-04-14T23:48:44.234949+0000","last_undegraded":"2024-04-14T23:48:44.234949+0000","last_fullsized":"2024-04-14T23:48:44.234949+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:43:44.966384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230697+0000","last_change":"2024-04-14T23:48:43.229387+0000","last_active":"2024-04-14T23:48:44.230697+0000","last_peered":"2024-04-14T23:48:44.230697+0000","last_clean":"2024-04-14T23:48:44.230697+0000","last_became_active":"2024-04-14T23:48:43.229110+0000","last_became_peered":"2024-04-14T23:48:43.229110+0000","last_unstale":"2024-04-14T23:48:44.230697+0000","last_undegraded":"2024-04-14T23:48:44.230697+0000","last_fullsized":"2024-04-14T23:48:44.230697+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T09:18:46.939940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.232750+0000","last_change":"2024-04-14T23:48:43.234154+0000","last_active":"2024-04-14T23:48:44.232750+0000","last_peered":"2024-04-14T23:48:44.232750+0000","last_clean":"2024-04-14T23:48:44.232750+0000","last_became_active":"2024-04-14T23:48:43.233853+0000","last_became_peered":"2024-04-14T23:48:43.233853+0000","last_unstale":"2024-04-14T23:48:44.232750+0000","last_undegraded":"2024-04-14T23:48:44.232750+0000","last_fullsized":"2024-04-14T23:48:44.232750+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T23:54:02.200200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230330+0000","last_change":"2024-04-14T23:48:43.242154+0000","last_active":"2024-04-14T23:48:44.230330+0000","last_peered":"2024-04-14T23:48:44.230330+0000","last_clean":"2024-04-14T23:48:44.230330+0000","last_became_active":"2024-04-14T23:48:43.241808+0000","last_became_peered":"2024-04-14T23:48:43.241808+0000","last_unstale":"2024-04-14T23:48:44.230330+0000","last_undegraded":"2024-04-14T23:48:44.230330+0000","last_fullsized":"2024-04-14T23:48:44.230330+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:48:42.680808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.232996+0000","last_change":"2024-04-14T23:48:43.233530+0000","last_active":"2024-04-14T23:48:44.232996+0000","last_peered":"2024-04-14T23:48:44.232996+0000","last_clean":"2024-04-14T23:48:44.232996+0000","last_became_active":"2024-04-14T23:48:43.233238+0000","last_became_peered":"2024-04-14T23:48:43.233238+0000","last_unstale":"2024-04-14T23:48:44.232996+0000","last_undegraded":"2024-04-14T23:48:44.232996+0000","last_fullsized":"2024-04-14T23:48:44.232996+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:10:58.525655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.549193+0000","last_change":"2024-04-14T23:48:43.222945+0000","last_active":"2024-04-14T23:48:45.549193+0000","last_peered":"2024-04-14T23:48:45.549193+0000","last_clean":"2024-04-14T23:48:45.549193+0000","last_became_active":"2024-04-14T23:48:43.222624+0000","last_became_peered":"2024-04-14T23:48:43.222624+0000","last_unstale":"2024-04-14T23:48:45.549193+0000","last_undegraded":"2024-04-14T23:48:45.549193+0000","last_fullsized":"2024-04-14T23:48:45.549193+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:22:08.289006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.048719+0000","last_change":"2024-04-14T23:48:43.232567+0000","last_active":"2024-04-14T23:48:45.048719+0000","last_peered":"2024-04-14T23:48:45.048719+0000","last_clean":"2024-04-14T23:48:45.048719+0000","last_became_active":"2024-04-14T23:48:43.232277+0000","last_became_peered":"2024-04-14T23:48:43.232277+0000","last_unstale":"2024-04-14T23:48:45.048719+0000","last_undegraded":"2024-04-14T23:48:45.048719+0000","last_fullsized":"2024-04-14T23:48:45.048719+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T04:59:39.714566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.232426+0000","last_change":"2024-04-14T23:48:43.229443+0000","last_active":"2024-04-14T23:48:44.232426+0000","last_peered":"2024-04-14T23:48:44.232426+0000","last_clean":"2024-04-14T23:48:44.232426+0000","last_became_active":"2024-04-14T23:48:43.229172+0000","last_became_peered":"2024-04-14T23:48:43.229172+0000","last_unstale":"2024-04-14T23:48:44.232426+0000","last_undegraded":"2024-04-14T23:48:44.232426+0000","last_fullsized":"2024-04-14T23:48:44.232426+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:41:31.405287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.234812+0000","last_change":"2024-04-14T23:48:43.242202+0000","last_active":"2024-04-14T23:48:44.234812+0000","last_peered":"2024-04-14T23:48:44.234812+0000","last_clean":"2024-04-14T23:48:44.234812+0000","last_became_active":"2024-04-14T23:48:43.241879+0000","last_became_peered":"2024-04-14T23:48:43.241879+0000","last_unstale":"2024-04-14T23:48:44.234812+0000","last_undegraded":"2024-04-14T23:48:44.234812+0000","last_fullsized":"2024-04-14T23:48:44.234812+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:11:27.552454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230871+0000","last_change":"2024-04-14T23:48:43.242068+0000","last_active":"2024-04-14T23:48:44.230871+0000","last_peered":"2024-04-14T23:48:44.230871+0000","last_clean":"2024-04-14T23:48:44.230871+0000","last_became_active":"2024-04-14T23:48:43.241734+0000","last_became_peered":"2024-04-14T23:48:43.241734+0000","last_unstale":"2024-04-14T23:48:44.230871+0000","last_undegraded":"2024-04-14T23:48:44.230871+0000","last_fullsized":"2024-04-14T23:48:44.230871+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:32:04.579906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.648997+0000","last_change":"2024-04-14T23:48:43.236034+0000","last_active":"2024-04-14T23:48:44.648997+0000","last_peered":"2024-04-14T23:48:44.648997+0000","last_clean":"2024-04-14T23:48:44.648997+0000","last_became_active":"2024-04-14T23:48:43.233567+0000","last_became_peered":"2024-04-14T23:48:43.233567+0000","last_unstale":"2024-04-14T23:48:44.648997+0000","last_undegraded":"2024-04-14T23:48:44.648997+0000","last_fullsized":"2024-04-14T23:48:44.648997+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:07:15.904075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.547675+0000","last_change":"2024-04-14T23:48:43.231045+0000","last_active":"2024-04-14T23:48:45.547675+0000","last_peered":"2024-04-14T23:48:45.547675+0000","last_clean":"2024-04-14T23:48:45.547675+0000","last_became_active":"2024-04-14T23:48:43.230626+0000","last_became_peered":"2024-04-14T23:48:43.230626+0000","last_unstale":"2024-04-14T23:48:45.547675+0000","last_undegraded":"2024-04-14T23:48:45.547675+0000","last_fullsized":"2024-04-14T23:48:45.547675+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:24:03.654061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548613+0000","last_change":"2024-04-14T23:48:43.227543+0000","last_active":"2024-04-14T23:48:45.548613+0000","last_peered":"2024-04-14T23:48:45.548613+0000","last_clean":"2024-04-14T23:48:45.548613+0000","last_became_active":"2024-04-14T23:48:43.226773+0000","last_became_peered":"2024-04-14T23:48:43.226773+0000","last_unstale":"2024-04-14T23:48:45.548613+0000","last_undegraded":"2024-04-14T23:48:45.548613+0000","last_fullsized":"2024-04-14T23:48:45.548613+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:44:56.691708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.211656+0000","last_change":"2024-04-14T23:48:42.239254+0000","last_active":"2024-04-14T23:48:45.211656+0000","last_peered":"2024-04-14T23:48:45.211656+0000","last_clean":"2024-04-14T23:48:45.211656+0000","last_became_active":"2024-04-14T23:48:42.238847+0000","last_became_peered":"2024-04-14T23:48:42.238847+0000","last_unstale":"2024-04-14T23:48:45.211656+0000","last_undegraded":"2024-04-14T23:48:45.211656+0000","last_fullsized":"2024-04-14T23:48:45.211656+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:46:16.301900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.235039+0000","last_change":"2024-04-14T23:48:43.237720+0000","last_active":"2024-04-14T23:48:44.235039+0000","last_peered":"2024-04-14T23:48:44.235039+0000","last_clean":"2024-04-14T23:48:44.235039+0000","last_became_active":"2024-04-14T23:48:43.237391+0000","last_became_peered":"2024-04-14T23:48:43.237391+0000","last_unstale":"2024-04-14T23:48:44.235039+0000","last_undegraded":"2024-04-14T23:48:44.235039+0000","last_fullsized":"2024-04-14T23:48:44.235039+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:38:26.879906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548161+0000","last_change":"2024-04-14T23:48:42.219277+0000","last_active":"2024-04-14T23:48:45.548161+0000","last_peered":"2024-04-14T23:48:45.548161+0000","last_clean":"2024-04-14T23:48:45.548161+0000","last_became_active":"2024-04-14T23:48:42.218995+0000","last_became_peered":"2024-04-14T23:48:42.218995+0000","last_unstale":"2024-04-14T23:48:45.548161+0000","last_undegraded":"2024-04-14T23:48:45.548161+0000","last_fullsized":"2024-04-14T23:48:45.548161+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T23:51:33.370857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.232590+0000","last_change":"2024-04-14T23:48:42.236803+0000","last_active":"2024-04-14T23:48:44.232590+0000","last_peered":"2024-04-14T23:48:44.232590+0000","last_clean":"2024-04-14T23:48:44.232590+0000","last_became_active":"2024-04-14T23:48:42.225937+0000","last_became_peered":"2024-04-14T23:48:42.225937+0000","last_unstale":"2024-04-14T23:48:44.232590+0000","last_undegraded":"2024-04-14T23:48:44.232590+0000","last_fullsized":"2024-04-14T23:48:44.232590+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:52:32.609126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.231732+0000","last_change":"2024-04-14T23:48:43.229506+0000","last_active":"2024-04-14T23:48:44.231732+0000","last_peered":"2024-04-14T23:48:44.231732+0000","last_clean":"2024-04-14T23:48:44.231732+0000","last_became_active":"2024-04-14T23:48:43.227356+0000","last_became_peered":"2024-04-14T23:48:43.227356+0000","last_unstale":"2024-04-14T23:48:44.231732+0000","last_undegraded":"2024-04-14T23:48:44.231732+0000","last_fullsized":"2024-04-14T23:48:44.231732+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:13:04.840655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230660+0000","last_change":"2024-04-14T23:48:42.210883+0000","last_active":"2024-04-14T23:48:44.230660+0000","last_peered":"2024-04-14T23:48:44.230660+0000","last_clean":"2024-04-14T23:48:44.230660+0000","last_became_active":"2024-04-14T23:48:42.210607+0000","last_became_peered":"2024-04-14T23:48:42.210607+0000","last_unstale":"2024-04-14T23:48:44.230660+0000","last_undegraded":"2024-04-14T23:48:44.230660+0000","last_fullsized":"2024-04-14T23:48:44.230660+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:33:32.526223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.048141+0000","last_change":"2024-04-14T23:48:43.218259+0000","last_active":"2024-04-14T23:48:45.048141+0000","last_peered":"2024-04-14T23:48:45.048141+0000","last_clean":"2024-04-14T23:48:45.048141+0000","last_became_active":"2024-04-14T23:48:43.217801+0000","last_became_peered":"2024-04-14T23:48:43.217801+0000","last_unstale":"2024-04-14T23:48:45.048141+0000","last_undegraded":"2024-04-14T23:48:45.048141+0000","last_fullsized":"2024-04-14T23:48:45.048141+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T04:13:42.425034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.547845+0000","last_change":"2024-04-14T23:48:42.218652+0000","last_active":"2024-04-14T23:48:45.547845+0000","last_peered":"2024-04-14T23:48:45.547845+0000","last_clean":"2024-04-14T23:48:45.547845+0000","last_became_active":"2024-04-14T23:48:42.218321+0000","last_became_peered":"2024-04-14T23:48:42.218321+0000","last_unstale":"2024-04-14T23:48:45.547845+0000","last_undegraded":"2024-04-14T23:48:45.547845+0000","last_fullsized":"2024-04-14T23:48:45.547845+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T04:22:06.957747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230879+0000","last_change":"2024-04-14T23:48:43.229117+0000","last_active":"2024-04-14T23:48:44.230879+0000","last_peered":"2024-04-14T23:48:44.230879+0000","last_clean":"2024-04-14T23:48:44.230879+0000","last_became_active":"2024-04-14T23:48:43.228805+0000","last_became_peered":"2024-04-14T23:48:43.228805+0000","last_unstale":"2024-04-14T23:48:44.230879+0000","last_undegraded":"2024-04-14T23:48:44.230879+0000","last_fullsized":"2024-04-14T23:48:44.230879+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:10:39.043818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230930+0000","last_change":"2024-04-14T23:48:42.221104+0000","last_active":"2024-04-14T23:48:44.230930+0000","last_peered":"2024-04-14T23:48:44.230930+0000","last_clean":"2024-04-14T23:48:44.230930+0000","last_became_active":"2024-04-14T23:48:42.220767+0000","last_became_peered":"2024-04-14T23:48:42.220767+0000","last_unstale":"2024-04-14T23:48:44.230930+0000","last_undegraded":"2024-04-14T23:48:44.230930+0000","last_fullsized":"2024-04-14T23:48:44.230930+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:23:57.213764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.235822+0000","last_change":"2024-04-14T23:48:43.236916+0000","last_active":"2024-04-14T23:48:44.235822+0000","last_peered":"2024-04-14T23:48:44.235822+0000","last_clean":"2024-04-14T23:48:44.235822+0000","last_became_active":"2024-04-14T23:48:43.236168+0000","last_became_peered":"2024-04-14T23:48:43.236168+0000","last_unstale":"2024-04-14T23:48:44.235822+0000","last_undegraded":"2024-04-14T23:48:44.235822+0000","last_fullsized":"2024-04-14T23:48:44.235822+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:45:52.057070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:48.220199+0000","last_change":"2024-04-14T23:48:42.231329+0000","last_active":"2024-04-14T23:48:48.220199+0000","last_peered":"2024-04-14T23:48:48.220199+0000","last_clean":"2024-04-14T23:48:48.220199+0000","last_became_active":"2024-04-14T23:48:42.230969+0000","last_became_peered":"2024-04-14T23:48:42.230969+0000","last_unstale":"2024-04-14T23:48:48.220199+0000","last_undegraded":"2024-04-14T23:48:48.220199+0000","last_fullsized":"2024-04-14T23:48:48.220199+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:06:16.557589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548833+0000","last_change":"2024-04-14T23:48:43.230531+0000","last_active":"2024-04-14T23:48:45.548833+0000","last_peered":"2024-04-14T23:48:45.548833+0000","last_clean":"2024-04-14T23:48:45.548833+0000","last_became_active":"2024-04-14T23:48:43.230253+0000","last_became_peered":"2024-04-14T23:48:43.230253+0000","last_unstale":"2024-04-14T23:48:45.548833+0000","last_undegraded":"2024-04-14T23:48:45.548833+0000","last_fullsized":"2024-04-14T23:48:45.548833+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:34:37.449824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:48.230834+0000","last_change":"2024-04-14T23:48:42.237069+0000","last_active":"2024-04-14T23:48:48.230834+0000","last_peered":"2024-04-14T23:48:48.230834+0000","last_clean":"2024-04-14T23:48:48.230834+0000","last_became_active":"2024-04-14T23:48:42.236804+0000","last_became_peered":"2024-04-14T23:48:42.236804+0000","last_unstale":"2024-04-14T23:48:48.230834+0000","last_undegraded":"2024-04-14T23:48:48.230834+0000","last_fullsized":"2024-04-14T23:48:48.230834+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:25:37.318379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548904+0000","last_change":"2024-04-14T23:48:43.221771+0000","last_active":"2024-04-14T23:48:45.548904+0000","last_peered":"2024-04-14T23:48:45.548904+0000","last_clean":"2024-04-14T23:48:45.548904+0000","last_became_active":"2024-04-14T23:48:43.221462+0000","last_became_peered":"2024-04-14T23:48:43.221462+0000","last_unstale":"2024-04-14T23:48:45.548904+0000","last_undegraded":"2024-04-14T23:48:45.548904+0000","last_fullsized":"2024-04-14T23:48:45.548904+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:29:52.737196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:48.227204+0000","last_change":"2024-04-14T23:48:42.230518+0000","last_active":"2024-04-14T23:48:48.227204+0000","last_peered":"2024-04-14T23:48:48.227204+0000","last_clean":"2024-04-14T23:48:48.227204+0000","last_became_active":"2024-04-14T23:48:42.229758+0000","last_became_peered":"2024-04-14T23:48:42.229758+0000","last_unstale":"2024-04-14T23:48:48.227204+0000","last_undegraded":"2024-04-14T23:48:48.227204+0000","last_fullsized":"2024-04-14T23:48:48.227204+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:01:41.332553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.234816+0000","last_change":"2024-04-14T23:48:43.229873+0000","last_active":"2024-04-14T23:48:44.234816+0000","last_peered":"2024-04-14T23:48:44.234816+0000","last_clean":"2024-04-14T23:48:44.234816+0000","last_became_active":"2024-04-14T23:48:43.229476+0000","last_became_peered":"2024-04-14T23:48:43.229476+0000","last_unstale":"2024-04-14T23:48:44.234816+0000","last_undegraded":"2024-04-14T23:48:44.234816+0000","last_fullsized":"2024-04-14T23:48:44.234816+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:54:22.874344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230858+0000","last_change":"2024-04-14T23:48:42.225404+0000","last_active":"2024-04-14T23:48:44.230858+0000","last_peered":"2024-04-14T23:48:44.230858+0000","last_clean":"2024-04-14T23:48:44.230858+0000","last_became_active":"2024-04-14T23:48:42.225211+0000","last_became_peered":"2024-04-14T23:48:42.225211+0000","last_unstale":"2024-04-14T23:48:44.230858+0000","last_undegraded":"2024-04-14T23:48:44.230858+0000","last_fullsized":"2024-04-14T23:48:44.230858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:49:45.282259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548573+0000","last_change":"2024-04-14T23:48:43.229543+0000","last_active":"2024-04-14T23:48:45.548573+0000","last_peered":"2024-04-14T23:48:45.548573+0000","last_clean":"2024-04-14T23:48:45.548573+0000","last_became_active":"2024-04-14T23:48:43.229297+0000","last_became_peered":"2024-04-14T23:48:43.229297+0000","last_unstale":"2024-04-14T23:48:45.548573+0000","last_undegraded":"2024-04-14T23:48:45.548573+0000","last_fullsized":"2024-04-14T23:48:45.548573+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:00:17.923013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.547633+0000","last_change":"2024-04-14T23:48:42.220492+0000","last_active":"2024-04-14T23:48:45.547633+0000","last_peered":"2024-04-14T23:48:45.547633+0000","last_clean":"2024-04-14T23:48:45.547633+0000","last_became_active":"2024-04-14T23:48:42.220145+0000","last_became_peered":"2024-04-14T23:48:42.220145+0000","last_unstale":"2024-04-14T23:48:45.547633+0000","last_undegraded":"2024-04-14T23:48:45.547633+0000","last_fullsized":"2024-04-14T23:48:45.547633+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:17:45.278993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.547774+0000","last_change":"2024-04-14T23:48:43.235676+0000","last_active":"2024-04-14T23:48:45.547774+0000","last_peered":"2024-04-14T23:48:45.547774+0000","last_clean":"2024-04-14T23:48:45.547774+0000","last_became_active":"2024-04-14T23:48:43.235319+0000","last_became_peered":"2024-04-14T23:48:43.235319+0000","last_unstale":"2024-04-14T23:48:45.547774+0000","last_undegraded":"2024-04-14T23:48:45.547774+0000","last_fullsized":"2024-04-14T23:48:45.547774+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:32:06.283229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:48.230880+0000","last_change":"2024-04-14T23:48:42.231178+0000","last_active":"2024-04-14T23:48:48.230880+0000","last_peered":"2024-04-14T23:48:48.230880+0000","last_clean":"2024-04-14T23:48:48.230880+0000","last_became_active":"2024-04-14T23:48:42.230792+0000","last_became_peered":"2024-04-14T23:48:42.230792+0000","last_unstale":"2024-04-14T23:48:48.230880+0000","last_undegraded":"2024-04-14T23:48:48.230880+0000","last_fullsized":"2024-04-14T23:48:48.230880+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:22:40.744314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548527+0000","last_change":"2024-04-14T23:48:43.228290+0000","last_active":"2024-04-14T23:48:45.548527+0000","last_peered":"2024-04-14T23:48:45.548527+0000","last_clean":"2024-04-14T23:48:45.548527+0000","last_became_active":"2024-04-14T23:48:43.228004+0000","last_became_peered":"2024-04-14T23:48:43.228004+0000","last_unstale":"2024-04-14T23:48:45.548527+0000","last_undegraded":"2024-04-14T23:48:45.548527+0000","last_fullsized":"2024-04-14T23:48:45.548527+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:20:06.572625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548424+0000","last_change":"2024-04-14T23:48:42.231936+0000","last_active":"2024-04-14T23:48:45.548424+0000","last_peered":"2024-04-14T23:48:45.548424+0000","last_clean":"2024-04-14T23:48:45.548424+0000","last_became_active":"2024-04-14T23:48:42.231612+0000","last_became_peered":"2024-04-14T23:48:42.231612+0000","last_unstale":"2024-04-14T23:48:45.548424+0000","last_undegraded":"2024-04-14T23:48:45.548424+0000","last_fullsized":"2024-04-14T23:48:45.548424+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:45:03.098024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.549200+0000","last_change":"2024-04-14T23:48:43.228933+0000","last_active":"2024-04-14T23:48:45.549200+0000","last_peered":"2024-04-14T23:48:45.549200+0000","last_clean":"2024-04-14T23:48:45.549200+0000","last_became_active":"2024-04-14T23:48:43.228665+0000","last_became_peered":"2024-04-14T23:48:43.228665+0000","last_unstale":"2024-04-14T23:48:45.549200+0000","last_undegraded":"2024-04-14T23:48:45.549200+0000","last_fullsized":"2024-04-14T23:48:45.549200+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:43:04.747711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:48.230335+0000","last_change":"2024-04-14T23:48:42.239794+0000","last_active":"2024-04-14T23:48:48.230335+0000","last_peered":"2024-04-14T23:48:48.230335+0000","last_clean":"2024-04-14T23:48:48.230335+0000","last_became_active":"2024-04-14T23:48:42.239505+0000","last_became_peered":"2024-04-14T23:48:42.239505+0000","last_unstale":"2024-04-14T23:48:48.230335+0000","last_undegraded":"2024-04-14T23:48:48.230335+0000","last_fullsized":"2024-04-14T23:48:48.230335+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T09:43:16.582708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.549265+0000","last_change":"2024-04-14T23:48:43.230251+0000","last_active":"2024-04-14T23:48:45.549265+0000","last_peered":"2024-04-14T23:48:45.549265+0000","last_clean":"2024-04-14T23:48:45.549265+0000","last_became_active":"2024-04-14T23:48:43.229949+0000","last_became_peered":"2024-04-14T23:48:43.229949+0000","last_unstale":"2024-04-14T23:48:45.549265+0000","last_undegraded":"2024-04-14T23:48:45.549265+0000","last_fullsized":"2024-04-14T23:48:45.549265+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T09:54:42.581135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.048901+0000","last_change":"2024-04-14T23:48:42.227708+0000","last_active":"2024-04-14T23:48:45.048901+0000","last_peered":"2024-04-14T23:48:45.048901+0000","last_clean":"2024-04-14T23:48:45.048901+0000","last_became_active":"2024-04-14T23:48:42.226845+0000","last_became_peered":"2024-04-14T23:48:42.226845+0000","last_unstale":"2024-04-14T23:48:45.048901+0000","last_undegraded":"2024-04-14T23:48:45.048901+0000","last_fullsized":"2024-04-14T23:48:45.048901+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:02:54.468796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230624+0000","last_change":"2024-04-14T23:48:43.227439+0000","last_active":"2024-04-14T23:48:44.230624+0000","last_peered":"2024-04-14T23:48:44.230624+0000","last_clean":"2024-04-14T23:48:44.230624+0000","last_became_active":"2024-04-14T23:48:43.226717+0000","last_became_peered":"2024-04-14T23:48:43.226717+0000","last_unstale":"2024-04-14T23:48:44.230624+0000","last_undegraded":"2024-04-14T23:48:44.230624+0000","last_fullsized":"2024-04-14T23:48:44.230624+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:01:11.249999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.649211+0000","last_change":"2024-04-14T23:48:42.232164+0000","last_active":"2024-04-14T23:48:44.649211+0000","last_peered":"2024-04-14T23:48:44.649211+0000","last_clean":"2024-04-14T23:48:44.649211+0000","last_became_active":"2024-04-14T23:48:42.229785+0000","last_became_peered":"2024-04-14T23:48:42.229785+0000","last_unstale":"2024-04-14T23:48:44.649211+0000","last_undegraded":"2024-04-14T23:48:44.649211+0000","last_fullsized":"2024-04-14T23:48:44.649211+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T04:21:37.538735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.549097+0000","last_change":"2024-04-14T23:48:43.232949+0000","last_active":"2024-04-14T23:48:45.549097+0000","last_peered":"2024-04-14T23:48:45.549097+0000","last_clean":"2024-04-14T23:48:45.549097+0000","last_became_active":"2024-04-14T23:48:43.232612+0000","last_became_peered":"2024-04-14T23:48:43.232612+0000","last_unstale":"2024-04-14T23:48:45.549097+0000","last_undegraded":"2024-04-14T23:48:45.549097+0000","last_fullsized":"2024-04-14T23:48:45.549097+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:01:00.524663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548069+0000","last_change":"2024-04-14T23:48:38.150200+0000","last_active":"2024-04-14T23:48:45.548069+0000","last_peered":"2024-04-14T23:48:45.548069+0000","last_clean":"2024-04-14T23:48:45.548069+0000","last_became_active":"2024-04-14T23:48:38.149301+0000","last_became_peered":"2024-04-14T23:48:38.149301+0000","last_unstale":"2024-04-14T23:48:45.548069+0000","last_undegraded":"2024-04-14T23:48:45.548069+0000","last_fullsized":"2024-04-14T23:48:45.548069+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:37.129663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:37.129663+0000","last_clean_scrub_stamp":"2024-04-14T23:48:37.129663+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T23:58:01.303851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230439+0000","last_change":"2024-04-14T23:48:42.229238+0000","last_active":"2024-04-14T23:48:44.230439+0000","last_peered":"2024-04-14T23:48:44.230439+0000","last_clean":"2024-04-14T23:48:44.230439+0000","last_became_active":"2024-04-14T23:48:42.228994+0000","last_became_peered":"2024-04-14T23:48:42.228994+0000","last_unstale":"2024-04-14T23:48:44.230439+0000","last_undegraded":"2024-04-14T23:48:44.230439+0000","last_fullsized":"2024-04-14T23:48:44.230439+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:47:15.406228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.232256+0000","last_change":"2024-04-14T23:48:43.223973+0000","last_active":"2024-04-14T23:48:44.232256+0000","last_peered":"2024-04-14T23:48:44.232256+0000","last_clean":"2024-04-14T23:48:44.232256+0000","last_became_active":"2024-04-14T23:48:43.223665+0000","last_became_peered":"2024-04-14T23:48:43.223665+0000","last_unstale":"2024-04-14T23:48:44.232256+0000","last_undegraded":"2024-04-14T23:48:44.232256+0000","last_fullsized":"2024-04-14T23:48:44.232256+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:23:28.495821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.547518+0000","last_change":"2024-04-14T23:48:42.230404+0000","last_active":"2024-04-14T23:48:45.547518+0000","last_peered":"2024-04-14T23:48:45.547518+0000","last_clean":"2024-04-14T23:48:45.547518+0000","last_became_active":"2024-04-14T23:48:42.230173+0000","last_became_peered":"2024-04-14T23:48:42.230173+0000","last_unstale":"2024-04-14T23:48:45.547518+0000","last_undegraded":"2024-04-14T23:48:45.547518+0000","last_fullsized":"2024-04-14T23:48:45.547518+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:23:45.666680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.234372+0000","last_change":"2024-04-14T23:48:43.231958+0000","last_active":"2024-04-14T23:48:44.234372+0000","last_peered":"2024-04-14T23:48:44.234372+0000","last_clean":"2024-04-14T23:48:44.234372+0000","last_became_active":"2024-04-14T23:48:43.231025+0000","last_became_peered":"2024-04-14T23:48:43.231025+0000","last_unstale":"2024-04-14T23:48:44.234372+0000","last_undegraded":"2024-04-14T23:48:44.234372+0000","last_fullsized":"2024-04-14T23:48:44.234372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T23:51:47.947116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.649047+0000","last_change":"2024-04-14T23:48:42.240651+0000","last_active":"2024-04-14T23:48:44.649047+0000","last_peered":"2024-04-14T23:48:44.649047+0000","last_clean":"2024-04-14T23:48:44.649047+0000","last_became_active":"2024-04-14T23:48:42.240351+0000","last_became_peered":"2024-04-14T23:48:42.240351+0000","last_unstale":"2024-04-14T23:48:44.649047+0000","last_undegraded":"2024-04-14T23:48:44.649047+0000","last_fullsized":"2024-04-14T23:48:44.649047+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:03:02.804444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.048231+0000","last_change":"2024-04-14T23:48:43.232415+0000","last_active":"2024-04-14T23:48:45.048231+0000","last_peered":"2024-04-14T23:48:45.048231+0000","last_clean":"2024-04-14T23:48:45.048231+0000","last_became_active":"2024-04-14T23:48:43.232120+0000","last_became_peered":"2024-04-14T23:48:43.232120+0000","last_unstale":"2024-04-14T23:48:45.048231+0000","last_undegraded":"2024-04-14T23:48:45.048231+0000","last_fullsized":"2024-04-14T23:48:45.048231+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:05:27.562738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230988+0000","last_change":"2024-04-14T23:48:42.233743+0000","last_active":"2024-04-14T23:48:44.230988+0000","last_peered":"2024-04-14T23:48:44.230988+0000","last_clean":"2024-04-14T23:48:44.230988+0000","last_became_active":"2024-04-14T23:48:42.233354+0000","last_became_peered":"2024-04-14T23:48:42.233354+0000","last_unstale":"2024-04-14T23:48:44.230988+0000","last_undegraded":"2024-04-14T23:48:44.230988+0000","last_fullsized":"2024-04-14T23:48:44.230988+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:49:23.045968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548229+0000","last_change":"2024-04-14T23:48:43.232468+0000","last_active":"2024-04-14T23:48:45.548229+0000","last_peered":"2024-04-14T23:48:45.548229+0000","last_clean":"2024-04-14T23:48:45.548229+0000","last_became_active":"2024-04-14T23:48:43.232119+0000","last_became_peered":"2024-04-14T23:48:43.232119+0000","last_unstale":"2024-04-14T23:48:45.548229+0000","last_undegraded":"2024-04-14T23:48:45.548229+0000","last_fullsized":"2024-04-14T23:48:45.548229+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:24:12.553325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.048630+0000","last_change":"2024-04-14T23:48:42.242249+0000","last_active":"2024-04-14T23:48:45.048630+0000","last_peered":"2024-04-14T23:48:45.048630+0000","last_clean":"2024-04-14T23:48:45.048630+0000","last_became_active":"2024-04-14T23:48:42.242039+0000","last_became_peered":"2024-04-14T23:48:42.242039+0000","last_unstale":"2024-04-14T23:48:45.048630+0000","last_undegraded":"2024-04-14T23:48:45.048630+0000","last_fullsized":"2024-04-14T23:48:45.048630+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:46:16.301900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230960+0000","last_change":"2024-04-14T23:48:43.242228+0000","last_active":"2024-04-14T23:48:44.230960+0000","last_peered":"2024-04-14T23:48:44.230960+0000","last_clean":"2024-04-14T23:48:44.230960+0000","last_became_active":"2024-04-14T23:48:43.241776+0000","last_became_peered":"2024-04-14T23:48:43.241776+0000","last_unstale":"2024-04-14T23:48:44.230960+0000","last_undegraded":"2024-04-14T23:48:44.230960+0000","last_fullsized":"2024-04-14T23:48:44.230960+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:48:23.653866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.549314+0000","last_change":"2024-04-14T23:48:42.232537+0000","last_active":"2024-04-14T23:48:45.549314+0000","last_peered":"2024-04-14T23:48:45.549314+0000","last_clean":"2024-04-14T23:48:45.549314+0000","last_became_active":"2024-04-14T23:48:42.232325+0000","last_became_peered":"2024-04-14T23:48:42.232325+0000","last_unstale":"2024-04-14T23:48:45.549314+0000","last_undegraded":"2024-04-14T23:48:45.549314+0000","last_fullsized":"2024-04-14T23:48:45.549314+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:34:37.448301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.048333+0000","last_change":"2024-04-14T23:48:43.234445+0000","last_active":"2024-04-14T23:48:45.048333+0000","last_peered":"2024-04-14T23:48:45.048333+0000","last_clean":"2024-04-14T23:48:45.048333+0000","last_became_active":"2024-04-14T23:48:43.234150+0000","last_became_peered":"2024-04-14T23:48:43.234150+0000","last_unstale":"2024-04-14T23:48:45.048333+0000","last_undegraded":"2024-04-14T23:48:45.048333+0000","last_fullsized":"2024-04-14T23:48:45.048333+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:50:59.708087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.049088+0000","last_change":"2024-04-14T23:48:42.230014+0000","last_active":"2024-04-14T23:48:45.049088+0000","last_peered":"2024-04-14T23:48:45.049088+0000","last_clean":"2024-04-14T23:48:45.049088+0000","last_became_active":"2024-04-14T23:48:42.229712+0000","last_became_peered":"2024-04-14T23:48:42.229712+0000","last_unstale":"2024-04-14T23:48:45.049088+0000","last_undegraded":"2024-04-14T23:48:45.049088+0000","last_fullsized":"2024-04-14T23:48:45.049088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:54:20.595391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.233392+0000","last_change":"2024-04-14T23:48:43.229128+0000","last_active":"2024-04-14T23:48:44.233392+0000","last_peered":"2024-04-14T23:48:44.233392+0000","last_clean":"2024-04-14T23:48:44.233392+0000","last_became_active":"2024-04-14T23:48:43.228739+0000","last_became_peered":"2024-04-14T23:48:43.228739+0000","last_unstale":"2024-04-14T23:48:44.233392+0000","last_undegraded":"2024-04-14T23:48:44.233392+0000","last_fullsized":"2024-04-14T23:48:44.233392+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:08:08.148345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:48.223491+0000","last_change":"2024-04-14T23:48:42.220035+0000","last_active":"2024-04-14T23:48:48.223491+0000","last_peered":"2024-04-14T23:48:48.223491+0000","last_clean":"2024-04-14T23:48:48.223491+0000","last_became_active":"2024-04-14T23:48:42.219750+0000","last_became_peered":"2024-04-14T23:48:42.219750+0000","last_unstale":"2024-04-14T23:48:48.223491+0000","last_undegraded":"2024-04-14T23:48:48.223491+0000","last_fullsized":"2024-04-14T23:48:48.223491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:00:58.928166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.235426+0000","last_change":"2024-04-14T23:48:43.232997+0000","last_active":"2024-04-14T23:48:44.235426+0000","last_peered":"2024-04-14T23:48:44.235426+0000","last_clean":"2024-04-14T23:48:44.235426+0000","last_became_active":"2024-04-14T23:48:43.232730+0000","last_became_peered":"2024-04-14T23:48:43.232730+0000","last_unstale":"2024-04-14T23:48:44.235426+0000","last_undegraded":"2024-04-14T23:48:44.235426+0000","last_fullsized":"2024-04-14T23:48:44.235426+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:38:44.068804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.547928+0000","last_change":"2024-04-14T23:48:42.232087+0000","last_active":"2024-04-14T23:48:45.547928+0000","last_peered":"2024-04-14T23:48:45.547928+0000","last_clean":"2024-04-14T23:48:45.547928+0000","last_became_active":"2024-04-14T23:48:42.231796+0000","last_became_peered":"2024-04-14T23:48:42.231796+0000","last_unstale":"2024-04-14T23:48:45.547928+0000","last_undegraded":"2024-04-14T23:48:45.547928+0000","last_fullsized":"2024-04-14T23:48:45.547928+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:14:35.050093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548007+0000","last_change":"2024-04-14T23:48:43.232871+0000","last_active":"2024-04-14T23:48:45.548007+0000","last_peered":"2024-04-14T23:48:45.548007+0000","last_clean":"2024-04-14T23:48:45.548007+0000","last_became_active":"2024-04-14T23:48:43.232594+0000","last_became_peered":"2024-04-14T23:48:43.232594+0000","last_unstale":"2024-04-14T23:48:45.548007+0000","last_undegraded":"2024-04-14T23:48:45.548007+0000","last_fullsized":"2024-04-14T23:48:45.548007+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:27:37.320994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.048370+0000","last_change":"2024-04-14T23:48:42.231237+0000","last_active":"2024-04-14T23:48:45.048370+0000","last_peered":"2024-04-14T23:48:45.048370+0000","last_clean":"2024-04-14T23:48:45.048370+0000","last_became_active":"2024-04-14T23:48:42.230902+0000","last_became_peered":"2024-04-14T23:48:42.230902+0000","last_unstale":"2024-04-14T23:48:45.048370+0000","last_undegraded":"2024-04-14T23:48:45.048370+0000","last_fullsized":"2024-04-14T23:48:45.048370+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:28:37.228039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.232080+0000","last_change":"2024-04-14T23:48:43.233035+0000","last_active":"2024-04-14T23:48:44.232080+0000","last_peered":"2024-04-14T23:48:44.232080+0000","last_clean":"2024-04-14T23:48:44.232080+0000","last_became_active":"2024-04-14T23:48:43.232747+0000","last_became_peered":"2024-04-14T23:48:43.232747+0000","last_unstale":"2024-04-14T23:48:44.232080+0000","last_undegraded":"2024-04-14T23:48:44.232080+0000","last_fullsized":"2024-04-14T23:48:44.232080+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:38:57.324408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.048301+0000","last_change":"2024-04-14T23:48:42.240496+0000","last_active":"2024-04-14T23:48:45.048301+0000","last_peered":"2024-04-14T23:48:45.048301+0000","last_clean":"2024-04-14T23:48:45.048301+0000","last_became_active":"2024-04-14T23:48:42.238524+0000","last_became_peered":"2024-04-14T23:48:42.238524+0000","last_unstale":"2024-04-14T23:48:45.048301+0000","last_undegraded":"2024-04-14T23:48:45.048301+0000","last_fullsized":"2024-04-14T23:48:45.048301+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:29:35.218759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.233225+0000","last_change":"2024-04-14T23:48:43.234025+0000","last_active":"2024-04-14T23:48:44.233225+0000","last_peered":"2024-04-14T23:48:44.233225+0000","last_clean":"2024-04-14T23:48:44.233225+0000","last_became_active":"2024-04-14T23:48:43.233651+0000","last_became_peered":"2024-04-14T23:48:43.233651+0000","last_unstale":"2024-04-14T23:48:44.233225+0000","last_undegraded":"2024-04-14T23:48:44.233225+0000","last_fullsized":"2024-04-14T23:48:44.233225+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:26:58.094545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:48.217913+0000","last_change":"2024-04-14T23:48:42.218897+0000","last_active":"2024-04-14T23:48:48.217913+0000","last_peered":"2024-04-14T23:48:48.217913+0000","last_clean":"2024-04-14T23:48:48.217913+0000","last_became_active":"2024-04-14T23:48:42.218580+0000","last_became_peered":"2024-04-14T23:48:42.218580+0000","last_unstale":"2024-04-14T23:48:48.217913+0000","last_undegraded":"2024-04-14T23:48:48.217913+0000","last_fullsized":"2024-04-14T23:48:48.217913+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:26:44.477638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548008+0000","last_change":"2024-04-14T23:48:43.219871+0000","last_active":"2024-04-14T23:48:45.548008+0000","last_peered":"2024-04-14T23:48:45.548008+0000","last_clean":"2024-04-14T23:48:45.548008+0000","last_became_active":"2024-04-14T23:48:43.219601+0000","last_became_peered":"2024-04-14T23:48:43.219601+0000","last_unstale":"2024-04-14T23:48:45.548008+0000","last_undegraded":"2024-04-14T23:48:45.548008+0000","last_fullsized":"2024-04-14T23:48:45.548008+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:36:20.379172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230971+0000","last_change":"2024-04-14T23:48:43.222259+0000","last_active":"2024-04-14T23:48:44.230971+0000","last_peered":"2024-04-14T23:48:44.230971+0000","last_clean":"2024-04-14T23:48:44.230971+0000","last_became_active":"2024-04-14T23:48:43.221842+0000","last_became_peered":"2024-04-14T23:48:43.221842+0000","last_unstale":"2024-04-14T23:48:44.230971+0000","last_undegraded":"2024-04-14T23:48:44.230971+0000","last_fullsized":"2024-04-14T23:48:44.230971+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:12:20.038987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:48.217880+0000","last_change":"2024-04-14T23:48:42.221314+0000","last_active":"2024-04-14T23:48:48.217880+0000","last_peered":"2024-04-14T23:48:48.217880+0000","last_clean":"2024-04-14T23:48:48.217880+0000","last_became_active":"2024-04-14T23:48:42.221028+0000","last_became_peered":"2024-04-14T23:48:42.221028+0000","last_unstale":"2024-04-14T23:48:48.217880+0000","last_undegraded":"2024-04-14T23:48:48.217880+0000","last_fullsized":"2024-04-14T23:48:48.217880+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T09:58:18.195400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230606+0000","last_change":"2024-04-14T23:48:43.224577+0000","last_active":"2024-04-14T23:48:44.230606+0000","last_peered":"2024-04-14T23:48:44.230606+0000","last_clean":"2024-04-14T23:48:44.230606+0000","last_became_active":"2024-04-14T23:48:43.224312+0000","last_became_peered":"2024-04-14T23:48:43.224312+0000","last_unstale":"2024-04-14T23:48:44.230606+0000","last_undegraded":"2024-04-14T23:48:44.230606+0000","last_fullsized":"2024-04-14T23:48:44.230606+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:52:44.955773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:48.217170+0000","last_change":"2024-04-14T23:48:42.221000+0000","last_active":"2024-04-14T23:48:48.217170+0000","last_peered":"2024-04-14T23:48:48.217170+0000","last_clean":"2024-04-14T23:48:48.217170+0000","last_became_active":"2024-04-14T23:48:42.219813+0000","last_became_peered":"2024-04-14T23:48:42.219813+0000","last_unstale":"2024-04-14T23:48:48.217170+0000","last_undegraded":"2024-04-14T23:48:48.217170+0000","last_fullsized":"2024-04-14T23:48:48.217170+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:00:38.107953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.235379+0000","last_change":"2024-04-14T23:48:43.236946+0000","last_active":"2024-04-14T23:48:44.235379+0000","last_peered":"2024-04-14T23:48:44.235379+0000","last_clean":"2024-04-14T23:48:44.235379+0000","last_became_active":"2024-04-14T23:48:43.236595+0000","last_became_peered":"2024-04-14T23:48:43.236595+0000","last_unstale":"2024-04-14T23:48:44.235379+0000","last_undegraded":"2024-04-14T23:48:44.235379+0000","last_fullsized":"2024-04-14T23:48:44.235379+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:56:46.037478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.211743+0000","last_change":"2024-04-14T23:48:42.239255+0000","last_active":"2024-04-14T23:48:45.211743+0000","last_peered":"2024-04-14T23:48:45.211743+0000","last_clean":"2024-04-14T23:48:45.211743+0000","last_became_active":"2024-04-14T23:48:42.238892+0000","last_became_peered":"2024-04-14T23:48:42.238892+0000","last_unstale":"2024-04-14T23:48:45.211743+0000","last_undegraded":"2024-04-14T23:48:45.211743+0000","last_fullsized":"2024-04-14T23:48:45.211743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:44:39.632155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.233893+0000","last_change":"2024-04-14T23:48:43.227240+0000","last_active":"2024-04-14T23:48:44.233893+0000","last_peered":"2024-04-14T23:48:44.233893+0000","last_clean":"2024-04-14T23:48:44.233893+0000","last_became_active":"2024-04-14T23:48:43.222707+0000","last_became_peered":"2024-04-14T23:48:43.222707+0000","last_unstale":"2024-04-14T23:48:44.233893+0000","last_undegraded":"2024-04-14T23:48:44.233893+0000","last_fullsized":"2024-04-14T23:48:44.233893+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:02:17.231763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:48.228047+0000","last_change":"2024-04-14T23:48:42.226563+0000","last_active":"2024-04-14T23:48:48.228047+0000","last_peered":"2024-04-14T23:48:48.228047+0000","last_clean":"2024-04-14T23:48:48.228047+0000","last_became_active":"2024-04-14T23:48:42.226238+0000","last_became_peered":"2024-04-14T23:48:42.226238+0000","last_unstale":"2024-04-14T23:48:48.228047+0000","last_undegraded":"2024-04-14T23:48:48.228047+0000","last_fullsized":"2024-04-14T23:48:48.228047+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:01:36.516898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65128,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":286720,"data_stored":666555,"data_compressed":32564,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":221184,"data_stored":75695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":75731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":225280,"data_stored":75765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":294912,"data_stored":666611,"data_compressed":32564,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32564,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32564,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-14T23:48:54.927 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-14T23:48:54.927 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-14T23:48:54.927 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-14T23:48:54.928 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-14T23:48:55.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.175+0000 7f366ca00640 1 Processor -- start 2024-04-14T23:48:55.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.175+0000 7f366ca00640 1 -- start start 2024-04-14T23:48:55.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.176+0000 7f366ca00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f36680b1e00 0x7f36680c6950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:55.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.176+0000 7f366ca00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3668073320 0x7f36680c7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:55.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.176+0000 7f366ca00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f36680c84e0 0x7f36680c3d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:55.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.177+0000 7f366ca00640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f36680c6540 con 0x7f36680c84e0 2024-04-14T23:48:55.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.177+0000 7f366ca00640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f36680b21e0 con 0x7f36680b1e00 2024-04-14T23:48:55.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.177+0000 7f366ca00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f3668159890 con 0x7f3668073320 2024-04-14T23:48:55.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.177+0000 7f3667e00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f36680c84e0 0x7f36680c3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:55.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.177+0000 7f3667e00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f36680c84e0 0x7f36680c3d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41870/0 (socket says 172.21.15.60:41870) 2024-04-14T23:48:55.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.177+0000 7f3667e00640 1 -- 172.21.15.60:0/1525033517 learned_addr learned my addr 172.21.15.60:0/1525033517 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:55.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.177+0000 7f3667400640 1 --2- 172.21.15.60:0/1525033517 >> v2:172.21.15.186:3300/0 conn(0x7f36680b1e00 0x7f36680c6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:55.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.177+0000 7f365fe00640 1 --2- 172.21.15.60:0/1525033517 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3668073320 0x7f36680c7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:55.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.177+0000 7f3667e00640 1 -- 172.21.15.60:0/1525033517 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3668073320 msgr2=0x7f36680c7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:55.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.177+0000 7f3667e00640 1 --2- 172.21.15.60:0/1525033517 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3668073320 0x7f36680c7fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:55.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.177+0000 7f3667e00640 1 -- 172.21.15.60:0/1525033517 >> v2:172.21.15.186:3300/0 conn(0x7f36680b1e00 msgr2=0x7f36680c6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:55.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.177+0000 7f3667e00640 1 --2- 172.21.15.60:0/1525033517 >> v2:172.21.15.186:3300/0 conn(0x7f36680b1e00 0x7f36680c6950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:55.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.177+0000 7f3667e00640 1 -- 172.21.15.60:0/1525033517 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3668152fb0 con 0x7f36680c84e0 2024-04-14T23:48:55.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.179+0000 7f3667e00640 1 --2- 172.21.15.60:0/1525033517 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f36680c84e0 0x7f36680c3d90 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f36680c36e0 tx=0x7f366002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d356cb7e813bf756 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:55.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.179+0000 7f3666a00640 1 -- 172.21.15.60:0/1525033517 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f366002cb00 con 0x7f36680c84e0 2024-04-14T23:48:55.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.179+0000 7f3666a00640 1 -- 172.21.15.60:0/1525033517 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f366002cca0 con 0x7f36680c84e0 2024-04-14T23:48:55.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.179+0000 7f3666a00640 1 -- 172.21.15.60:0/1525033517 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3660034610 con 0x7f36680c84e0 2024-04-14T23:48:55.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.180+0000 7f366ca00640 1 -- 172.21.15.60:0/1525033517 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f36680c84e0 msgr2=0x7f36680c3d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:55.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.180+0000 7f366ca00640 1 --2- 172.21.15.60:0/1525033517 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f36680c84e0 0x7f36680c3d90 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f36680c36e0 tx=0x7f366002bf60 comp rx=0 tx=0).stop 2024-04-14T23:48:55.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.180+0000 7f366ca00640 1 -- 172.21.15.60:0/1525033517 shutdown_connections 2024-04-14T23:48:55.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.180+0000 7f366ca00640 1 --2- 172.21.15.60:0/1525033517 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f36680c84e0 0x7f36680c3d90 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:55.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.180+0000 7f366ca00640 1 --2- 172.21.15.60:0/1525033517 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3668073320 0x7f36680c7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:55.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.180+0000 7f366ca00640 1 --2- 172.21.15.60:0/1525033517 >> v2:172.21.15.186:3300/0 conn(0x7f36680b1e00 0x7f36680c6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:55.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.180+0000 7f366ca00640 1 -- 172.21.15.60:0/1525033517 >> 172.21.15.60:0/1525033517 conn(0x7f36680c8b50 msgr2=0x7f36680c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:55.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.180+0000 7f366ca00640 1 -- 172.21.15.60:0/1525033517 shutdown_connections 2024-04-14T23:48:55.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.180+0000 7f366ca00640 1 -- 172.21.15.60:0/1525033517 wait complete. 2024-04-14T23:48:55.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.182+0000 7f366ca00640 1 Processor -- start 2024-04-14T23:48:55.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.182+0000 7f366ca00640 1 -- start start 2024-04-14T23:48:55.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.183+0000 7f366ca00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3668073320 0x7f3668159a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:55.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.183+0000 7f366ca00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36680b1e00 0x7f366815c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:55.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.183+0000 7f366ca00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f36680c84e0 0x7f3668071870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:55.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.183+0000 7f366ca00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f36680c38b0 con 0x7f3668073320 2024-04-14T23:48:55.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.183+0000 7f366ca00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3668159310 con 0x7f36680b1e00 2024-04-14T23:48:55.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.183+0000 7f366ca00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f36680c5f80 con 0x7f36680c84e0 2024-04-14T23:48:55.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.183+0000 7f3667400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3668073320 0x7f3668159a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:55.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.183+0000 7f3667400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3668073320 0x7f3668159a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41886/0 (socket says 172.21.15.60:41886) 2024-04-14T23:48:55.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.183+0000 7f3667400640 1 -- 172.21.15.60:0/3563429442 learned_addr learned my addr 172.21.15.60:0/3563429442 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:55.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.183+0000 7f365fe00640 1 --2- 172.21.15.60:0/3563429442 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36680b1e00 0x7f366815c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:55.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.183+0000 7f3667e00640 1 --2- 172.21.15.60:0/3563429442 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f36680c84e0 0x7f3668071870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:55.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.183+0000 7f3667400640 1 -- 172.21.15.60:0/3563429442 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f36680c84e0 msgr2=0x7f3668071870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:55.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.183+0000 7f3667400640 1 --2- 172.21.15.60:0/3563429442 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f36680c84e0 0x7f3668071870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:55.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.183+0000 7f3667400640 1 -- 172.21.15.60:0/3563429442 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36680b1e00 msgr2=0x7f366815c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:55.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.183+0000 7f3667400640 1 --2- 172.21.15.60:0/3563429442 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36680b1e00 0x7f366815c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:55.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.184+0000 7f3667400640 1 -- 172.21.15.60:0/3563429442 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f366815bac0 con 0x7f3668073320 2024-04-14T23:48:55.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.184+0000 7f3667400640 1 --2- 172.21.15.60:0/3563429442 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3668073320 0x7f3668159a50 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f365800e9b0 tx=0x7f365800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:55.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.184+0000 7f365f400640 1 -- 172.21.15.60:0/3563429442 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3658017070 con 0x7f3668073320 2024-04-14T23:48:55.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.184+0000 7f365f400640 1 -- 172.21.15.60:0/3563429442 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f365800c9a0 con 0x7f3668073320 2024-04-14T23:48:55.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.184+0000 7f366ca00640 1 -- 172.21.15.60:0/3563429442 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f366815a070 con 0x7f3668073320 2024-04-14T23:48:55.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.184+0000 7f365f400640 1 -- 172.21.15.60:0/3563429442 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f365800cd90 con 0x7f3668073320 2024-04-14T23:48:55.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.184+0000 7f366ca00640 1 -- 172.21.15.60:0/3563429442 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f366815c010 con 0x7f3668073320 2024-04-14T23:48:55.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.186+0000 7f366ca00640 1 -- 172.21.15.60:0/3563429442 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f366804f540 con 0x7f3668073320 2024-04-14T23:48:55.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.186+0000 7f365f400640 1 -- 172.21.15.60:0/3563429442 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f365801a070 con 0x7f3668073320 2024-04-14T23:48:55.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.187+0000 7f365f400640 1 --2- 172.21.15.60:0/3563429442 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f364406b990 0x7f364406de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:55.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.187+0000 7f365f400640 1 -- 172.21.15.60:0/3563429442 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f365808acb0 con 0x7f3668073320 2024-04-14T23:48:55.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.188+0000 7f365fe00640 1 --2- 172.21.15.60:0/3563429442 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f364406b990 0x7f364406de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:55.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.188+0000 7f365fe00640 1 --2- 172.21.15.60:0/3563429442 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f364406b990 0x7f364406de50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f36681572f0 tx=0x7f3650009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:55.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.193+0000 7f365f400640 1 -- 172.21.15.60:0/3563429442 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3658059140 con 0x7f3668073320 2024-04-14T23:48:55.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.431+0000 7f366ca00640 1 -- 172.21.15.60:0/3563429442 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f36680cb760 con 0x7f3668073320 2024-04-14T23:48:55.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.432+0000 7f365f400640 1 -- 172.21.15.60:0/3563429442 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f365805d1e0 con 0x7f3668073320 2024-04-14T23:48:55.438 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:48:55.438 INFO:teuthology.orchestra.run.smithi060.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-14T23:48:55.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.435+0000 7f366ca00640 1 -- 172.21.15.60:0/3563429442 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f364406b990 msgr2=0x7f364406de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:55.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.435+0000 7f366ca00640 1 --2- 172.21.15.60:0/3563429442 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f364406b990 0x7f364406de50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f36681572f0 tx=0x7f3650009190 comp rx=0 tx=0).stop 2024-04-14T23:48:55.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.435+0000 7f366ca00640 1 -- 172.21.15.60:0/3563429442 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3668073320 msgr2=0x7f3668159a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:55.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.435+0000 7f366ca00640 1 --2- 172.21.15.60:0/3563429442 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3668073320 0x7f3668159a50 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f365800e9b0 tx=0x7f365800ee70 comp rx=0 tx=0).stop 2024-04-14T23:48:55.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.435+0000 7f366ca00640 1 -- 172.21.15.60:0/3563429442 shutdown_connections 2024-04-14T23:48:55.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.435+0000 7f366ca00640 1 --2- 172.21.15.60:0/3563429442 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f364406b990 0x7f364406de50 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:55.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.435+0000 7f366ca00640 1 --2- 172.21.15.60:0/3563429442 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f36680c84e0 0x7f3668071870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:55.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.435+0000 7f366ca00640 1 --2- 172.21.15.60:0/3563429442 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36680b1e00 0x7f366815c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:55.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.435+0000 7f366ca00640 1 --2- 172.21.15.60:0/3563429442 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3668073320 0x7f3668159a50 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:55.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.435+0000 7f366ca00640 1 -- 172.21.15.60:0/3563429442 >> 172.21.15.60:0/3563429442 conn(0x7f36680c8b50 msgr2=0x7f3668156700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:55.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.435+0000 7f366ca00640 1 -- 172.21.15.60:0/3563429442 shutdown_connections 2024-04-14T23:48:55.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.435+0000 7f366ca00640 1 -- 172.21.15.60:0/3563429442 wait complete. 2024-04-14T23:48:55.458 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-14T23:48:55.459 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-14T23:48:55.673 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.668+0000 7f049dc00640 1 Processor -- start 2024-04-14T23:48:55.674 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.668+0000 7f049dc00640 1 -- start start 2024-04-14T23:48:55.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.669+0000 7f049dc00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f049806f2b0 0x7f0498158940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:55.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.669+0000 7f049dc00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f049806fa80 0x7f0498158e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:55.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.669+0000 7f049dc00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04980c9eb0 con 0x7f0498152bf0 2024-04-14T23:48:55.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.669+0000 7f049dc00640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f04980cb500 con 0x7f049806fa80 2024-04-14T23:48:55.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.669+0000 7f049dc00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f04980cb7c0 con 0x7f049806f2b0 2024-04-14T23:48:55.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.670+0000 7f0496a00640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f0498152bf0 0x7f049806eba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53324/0 (socket says 172.21.15.60:53324) 2024-04-14T23:48:55.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.670+0000 7f0496a00640 1 -- 172.21.15.60:0/2160509827 learned_addr learned my addr 172.21.15.60:0/2160509827 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:48:55.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.670+0000 7f0496000640 1 --2- 172.21.15.60:0/2160509827 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f049806f2b0 0x7f0498158940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:55.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.670+0000 7f0497400640 1 --2- 172.21.15.60:0/2160509827 >> v2:172.21.15.186:3300/0 conn(0x7f049806fa80 0x7f0498158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:55.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.670+0000 7f0495600640 1 -- 172.21.15.60:0/2160509827 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2201228844 0 0) 0x7f04980c9eb0 con 0x7f0498152bf0 2024-04-14T23:48:55.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.670+0000 7f0495600640 1 -- 172.21.15.60:0/2160509827 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f047c003520 con 0x7f0498152bf0 2024-04-14T23:48:55.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.670+0000 7f0497400640 1 -- 172.21.15.60:0/2160509827 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f049806f2b0 msgr2=0x7f0498158940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:55.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.670+0000 7f0497400640 1 --2- 172.21.15.60:0/2160509827 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f049806f2b0 0x7f0498158940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:55.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.670+0000 7f0497400640 1 -- 172.21.15.60:0/2160509827 >> v1:172.21.15.60:6789/0 conn(0x7f0498152bf0 legacy=0x7f049806eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:55.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.670+0000 7f0497400640 1 -- 172.21.15.60:0/2160509827 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04980c98f0 con 0x7f049806fa80 2024-04-14T23:48:55.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.671+0000 7f0497400640 1 --2- 172.21.15.60:0/2160509827 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f049806fa80 0x7f0498158e80 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0490009d90 tx=0x7f049002beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6dd38ca680219487 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:55.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.672+0000 7f0495600640 1 -- 172.21.15.60:0/2160509827 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0490039070 con 0x7f049806fa80 2024-04-14T23:48:55.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.672+0000 7f0495600640 1 -- 172.21.15.60:0/2160509827 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f049002cc60 con 0x7f049806fa80 2024-04-14T23:48:55.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.672+0000 7f0495600640 1 -- 172.21.15.60:0/2160509827 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0490034660 con 0x7f049806fa80 2024-04-14T23:48:55.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.672+0000 7f049dc00640 1 -- 172.21.15.60:0/2160509827 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f049806fa80 msgr2=0x7f0498158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:55.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.672+0000 7f049dc00640 1 --2- 172.21.15.60:0/2160509827 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f049806fa80 0x7f0498158e80 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0490009d90 tx=0x7f049002beb0 comp rx=0 tx=0).stop 2024-04-14T23:48:55.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.672+0000 7f049dc00640 1 -- 172.21.15.60:0/2160509827 shutdown_connections 2024-04-14T23:48:55.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.672+0000 7f049dc00640 1 --2- 172.21.15.60:0/2160509827 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f049806fa80 0x7f0498158e80 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:55.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.672+0000 7f049dc00640 1 --2- 172.21.15.60:0/2160509827 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f049806f2b0 0x7f0498158940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:55.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.672+0000 7f049dc00640 1 -- 172.21.15.60:0/2160509827 >> 172.21.15.60:0/2160509827 conn(0x7f0498070d20 msgr2=0x7f0498071100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:55.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.672+0000 7f049dc00640 1 -- 172.21.15.60:0/2160509827 shutdown_connections 2024-04-14T23:48:55.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.672+0000 7f049dc00640 1 -- 172.21.15.60:0/2160509827 wait complete. 2024-04-14T23:48:55.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.674+0000 7f049dc00640 1 Processor -- start 2024-04-14T23:48:55.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.674+0000 7f049dc00640 1 -- start start 2024-04-14T23:48:55.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.675+0000 7f049dc00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f049806f2b0 0x7f04981510a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:55.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.675+0000 7f049dc00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f049806fa80 0x7f04980b3040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:55.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.675+0000 7f049dc00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0498152bf0 0x7f049814bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:55.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.675+0000 7f049dc00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f0498152fd0 con 0x7f049806f2b0 2024-04-14T23:48:55.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.675+0000 7f049dc00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f04980c9030 con 0x7f049806fa80 2024-04-14T23:48:55.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.675+0000 7f049dc00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f04981593c0 con 0x7f0498152bf0 2024-04-14T23:48:55.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.675+0000 7f0496a00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f049806f2b0 0x7f04981510a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:55.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.676+0000 7f0496000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f049806fa80 0x7f04980b3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:55.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.676+0000 7f0496a00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f049806f2b0 0x7f04981510a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41888/0 (socket says 172.21.15.60:41888) 2024-04-14T23:48:55.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.676+0000 7f0496a00640 1 -- 172.21.15.60:0/3439819267 learned_addr learned my addr 172.21.15.60:0/3439819267 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:55.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.676+0000 7f0497400640 1 --2- 172.21.15.60:0/3439819267 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0498152bf0 0x7f049814bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:55.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.676+0000 7f0497400640 1 -- 172.21.15.60:0/3439819267 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f049806fa80 msgr2=0x7f04980b3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:55.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.676+0000 7f0497400640 1 --2- 172.21.15.60:0/3439819267 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f049806fa80 0x7f04980b3040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:55.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.676+0000 7f0497400640 1 -- 172.21.15.60:0/3439819267 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f049806f2b0 msgr2=0x7f04981510a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:55.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.676+0000 7f0497400640 1 --2- 172.21.15.60:0/3439819267 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f049806f2b0 0x7f04981510a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:55.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.676+0000 7f0497400640 1 -- 172.21.15.60:0/3439819267 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f049814fd60 con 0x7f0498152bf0 2024-04-14T23:48:55.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.676+0000 7f0496a00640 1 --2- 172.21.15.60:0/3439819267 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f049806f2b0 0x7f04981510a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:48:55.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.676+0000 7f0496000640 1 --2- 172.21.15.60:0/3439819267 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f049806fa80 0x7f04980b3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:48:55.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.676+0000 7f0497400640 1 --2- 172.21.15.60:0/3439819267 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0498152bf0 0x7f049814bc60 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0498071ac0 tx=0x7f0490004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:55.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.677+0000 7f048ea00640 1 -- 172.21.15.60:0/3439819267 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0490039050 con 0x7f0498152bf0 2024-04-14T23:48:55.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.677+0000 7f048ea00640 1 -- 172.21.15.60:0/3439819267 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0490045070 con 0x7f0498152bf0 2024-04-14T23:48:55.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.677+0000 7f048ea00640 1 -- 172.21.15.60:0/3439819267 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0490009450 con 0x7f0498152bf0 2024-04-14T23:48:55.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.677+0000 7f049dc00640 1 -- 172.21.15.60:0/3439819267 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0498150000 con 0x7f0498152bf0 2024-04-14T23:48:55.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.677+0000 7f049dc00640 1 -- 172.21.15.60:0/3439819267 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f04981505c0 con 0x7f0498152bf0 2024-04-14T23:48:55.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.678+0000 7f049dc00640 1 -- 172.21.15.60:0/3439819267 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04980cf3e0 con 0x7f0498152bf0 2024-04-14T23:48:55.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.679+0000 7f048ea00640 1 -- 172.21.15.60:0/3439819267 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f049003e850 con 0x7f0498152bf0 2024-04-14T23:48:55.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.680+0000 7f048ea00640 1 --2- 172.21.15.60:0/3439819267 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f047806b990 0x7f047806de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:55.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.680+0000 7f048ea00640 1 -- 172.21.15.60:0/3439819267 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f04900ad760 con 0x7f0498152bf0 2024-04-14T23:48:55.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.680+0000 7f0496a00640 1 --2- 172.21.15.60:0/3439819267 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f047806b990 0x7f047806de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:55.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.681+0000 7f0496a00640 1 --2- 172.21.15.60:0/3439819267 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f047806b990 0x7f047806de50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0488005880 tx=0x7f0488009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:55.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.684+0000 7f048ea00640 1 -- 172.21.15.60:0/3439819267 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f04980cf3e0 con 0x7f0498152bf0 2024-04-14T23:48:55.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.877+0000 7f049dc00640 1 -- 172.21.15.60:0/3439819267 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f04980b44f0 con 0x7f0498152bf0 2024-04-14T23:48:55.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.879+0000 7f048ea00640 1 -- 172.21.15.60:0/3439819267 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16605 (secure 0 0 0) 0x7f04980b44f0 con 0x7f0498152bf0 2024-04-14T23:48:55.885 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:48:55.885 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":21,"fsid":"5847db56-a76d-4486-b563-d83f1186af92","created":"2024-04-14T23:48:23.900003+0000","modified":"2024-04-14T23:48:44.219352+0000","last_up_change":"2024-04-14T23:48:34.203573+0000","last_in_change":"2024-04-14T23:48:26.345499+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-14T23:48:36.247372+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-14T23:48:40.274195+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-14T23:48:41.722183+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b2b2b755-7f42-4f9b-af9a-a178f7cc5a52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6817","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6818","nonce":2536528895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6822","nonce":2536528895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6827","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6828","nonce":2536528895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6825","nonce":2536528895}]},"public_addr":"172.21.15.60:6818/2536528895","cluster_addr":"172.21.15.60:6822/2536528895","heartbeat_back_addr":"172.21.15.60:6828/2536528895","heartbeat_front_addr":"172.21.15.60:6825/2536528895","state":["exists","up"]},{"osd":1,"uuid":"321418cd-5ab7-41a5-b9c8-b3fc12f68ebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6801","nonce":3115695676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6803","nonce":3115695676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6807","nonce":3115695676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6805","nonce":3115695676}]},"public_addr":"172.21.15.60:6801/3115695676","cluster_addr":"172.21.15.60:6803/3115695676","heartbeat_back_addr":"172.21.15.60:6807/3115695676","heartbeat_front_addr":"172.21.15.60:6805/3115695676","state":["exists","up"]},{"osd":2,"uuid":"07874e9f-3a38-41dc-be9f-2b75e84e53d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6819","nonce":3475168698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6824","nonce":3475168698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6831","nonce":3475168698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6829","nonce":3475168698}]},"public_addr":"172.21.15.60:6819/3475168698","cluster_addr":"172.21.15.60:6824/3475168698","heartbeat_back_addr":"172.21.15.60:6831/3475168698","heartbeat_front_addr":"172.21.15.60:6829/3475168698","state":["exists","up"]},{"osd":3,"uuid":"7e4ef6c9-b17a-47c6-a320-f5d16a6fe05e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6809","nonce":1730813674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6811","nonce":1730813674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6815","nonce":1730813674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6813","nonce":1730813674}]},"public_addr":"172.21.15.60:6809/1730813674","cluster_addr":"172.21.15.60:6811/1730813674","heartbeat_back_addr":"172.21.15.60:6815/1730813674","heartbeat_front_addr":"172.21.15.60:6813/1730813674","state":["exists","up"]},{"osd":4,"uuid":"759a3efc-dade-44c5-b57a-1623499aba50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6805","nonce":1464587038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6810","nonce":1464587038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6819","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6822","nonce":1464587038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6813","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6816","nonce":1464587038}]},"public_addr":"172.21.15.186:6805/1464587038","cluster_addr":"172.21.15.186:6810/1464587038","heartbeat_back_addr":"172.21.15.186:6822/1464587038","heartbeat_front_addr":"172.21.15.186:6816/1464587038","state":["exists","up"]},{"osd":5,"uuid":"24cd58c1-0cf1-4077-ab33-84c39c6907ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6801","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6803","nonce":1697098067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6809","nonce":1697098067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6820","nonce":1697098067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6815","nonce":1697098067}]},"public_addr":"172.21.15.186:6803/1697098067","cluster_addr":"172.21.15.186:6809/1697098067","heartbeat_back_addr":"172.21.15.186:6820/1697098067","heartbeat_front_addr":"172.21.15.186:6815/1697098067","state":["exists","up"]},{"osd":6,"uuid":"6add38f1-65e9-4b0d-927a-4ab77f5e2323","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6804","nonce":1837108559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6807","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6811","nonce":1837108559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6823","nonce":1837108559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6817","nonce":1837108559}]},"public_addr":"172.21.15.186:6804/1837108559","cluster_addr":"172.21.15.186:6811/1837108559","heartbeat_back_addr":"172.21.15.186:6823/1837108559","heartbeat_front_addr":"172.21.15.186:6817/1837108559","state":["exists","up"]},{"osd":7,"uuid":"a2326b6f-7601-4dda-a456-67c443ea5929","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6825","nonce":842537875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6827","nonce":842537875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6831","nonce":842537875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6829","nonce":842537875}]},"public_addr":"172.21.15.186:6825/842537875","cluster_addr":"172.21.15.186:6827/842537875","heartbeat_back_addr":"172.21.15.186:6831/842537875","heartbeat_front_addr":"172.21.15.186:6829/842537875","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.975583+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.976806+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.722725+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.763705+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.502140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.528972+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.517961+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.513350+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-14T23:48:55.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.881+0000 7f049dc00640 1 -- 172.21.15.60:0/3439819267 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f047806b990 msgr2=0x7f047806de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:55.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.881+0000 7f049dc00640 1 --2- 172.21.15.60:0/3439819267 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f047806b990 0x7f047806de50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0488005880 tx=0x7f0488009040 comp rx=0 tx=0).stop 2024-04-14T23:48:55.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.881+0000 7f049dc00640 1 -- 172.21.15.60:0/3439819267 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0498152bf0 msgr2=0x7f049814bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:55.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.881+0000 7f049dc00640 1 --2- 172.21.15.60:0/3439819267 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0498152bf0 0x7f049814bc60 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0498071ac0 tx=0x7f0490004a40 comp rx=0 tx=0).stop 2024-04-14T23:48:55.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.881+0000 7f049dc00640 1 -- 172.21.15.60:0/3439819267 shutdown_connections 2024-04-14T23:48:55.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.881+0000 7f049dc00640 1 --2- 172.21.15.60:0/3439819267 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f047806b990 0x7f047806de50 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:55.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.881+0000 7f049dc00640 1 --2- 172.21.15.60:0/3439819267 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0498152bf0 0x7f049814bc60 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:55.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.881+0000 7f049dc00640 1 --2- 172.21.15.60:0/3439819267 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f049806fa80 0x7f04980b3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:55.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.881+0000 7f049dc00640 1 --2- 172.21.15.60:0/3439819267 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f049806f2b0 0x7f04981510a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:55.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.881+0000 7f049dc00640 1 -- 172.21.15.60:0/3439819267 >> 172.21.15.60:0/3439819267 conn(0x7f0498070d20 msgr2=0x7f04980c42e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:55.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.881+0000 7f049dc00640 1 -- 172.21.15.60:0/3439819267 shutdown_connections 2024-04-14T23:48:55.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:55.881+0000 7f049dc00640 1 -- 172.21.15.60:0/3439819267 wait complete. 2024-04-14T23:48:55.905 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-14T23:48:36.247372+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-14T23:48:40.274195+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-14T23:48:41.722183+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-14T23:48:55.905 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-14T23:48:56.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.112+0000 7fdfb9200640 1 Processor -- start 2024-04-14T23:48:56.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.112+0000 7fdfb9200640 1 -- start start 2024-04-14T23:48:56.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.114+0000 7fdfb9200640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fdfb40c6d70 0x7fdfb40c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:56.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.114+0000 7fdfb9200640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfb40c76f0 con 0x7fdfb415b740 2024-04-14T23:48:56.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.114+0000 7fdfb9200640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfb415bb20 con 0x7fdfb40711a0 2024-04-14T23:48:56.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.114+0000 7fdfb9200640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fdfb4070d10 con 0x7fdfb40c6d70 2024-04-14T23:48:56.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.114+0000 7fdfb3400640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fdfb415b740 0x7fdfb406eb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53336/0 (socket says 172.21.15.60:53336) 2024-04-14T23:48:56.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.114+0000 7fdfb3400640 1 -- 172.21.15.60:0/2764958373 learned_addr learned my addr 172.21.15.60:0/2764958373 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:48:56.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.114+0000 7fdfb2a00640 1 --2- 172.21.15.60:0/2764958373 >> v2:172.21.15.186:3300/0 conn(0x7fdfb40c6d70 0x7fdfb40c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:56.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.115+0000 7fdfb2000640 1 -- 172.21.15.60:0/2764958373 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 296169796 0 0) 0x7fdfb415bb20 con 0x7fdfb40711a0 2024-04-14T23:48:56.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.115+0000 7fdfb2a00640 1 -- 172.21.15.60:0/2764958373 >> v1:172.21.15.186:6789/0 conn(0x7fdfb40711a0 legacy=0x7fdfb4071610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:56.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.115+0000 7fdfb2a00640 1 -- 172.21.15.60:0/2764958373 >> v1:172.21.15.60:6789/0 conn(0x7fdfb415b740 legacy=0x7fdfb406eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:56.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.115+0000 7fdfb2a00640 1 -- 172.21.15.60:0/2764958373 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfb415e310 con 0x7fdfb40c6d70 2024-04-14T23:48:56.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.116+0000 7fdfb2a00640 1 --2- 172.21.15.60:0/2764958373 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdfb40c6d70 0x7fdfb40c71b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c0097a0 tx=0x7fdf9c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7af160a70e3d2ba0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:56.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.116+0000 7fdfb2000640 1 -- 172.21.15.60:0/2764958373 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf9c034070 con 0x7fdfb40c6d70 2024-04-14T23:48:56.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.116+0000 7fdfb2000640 1 -- 172.21.15.60:0/2764958373 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf9c030610 con 0x7fdfb40c6d70 2024-04-14T23:48:56.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.116+0000 7fdfb2000640 1 -- 172.21.15.60:0/2764958373 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf9c0309e0 con 0x7fdfb40c6d70 2024-04-14T23:48:56.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.117+0000 7fdfb9200640 1 -- 172.21.15.60:0/2764958373 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdfb40c6d70 msgr2=0x7fdfb40c71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:56.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.117+0000 7fdfb9200640 1 --2- 172.21.15.60:0/2764958373 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdfb40c6d70 0x7fdfb40c71b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c0097a0 tx=0x7fdf9c02b960 comp rx=0 tx=0).stop 2024-04-14T23:48:56.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.117+0000 7fdfb9200640 1 -- 172.21.15.60:0/2764958373 shutdown_connections 2024-04-14T23:48:56.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.117+0000 7fdfb9200640 1 --2- 172.21.15.60:0/2764958373 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdfb40c6d70 0x7fdfb40c71b0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:56.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.117+0000 7fdfb9200640 1 -- 172.21.15.60:0/2764958373 >> 172.21.15.60:0/2764958373 conn(0x7fdfb40cc2f0 msgr2=0x7fdfb40cc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:56.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.117+0000 7fdfb9200640 1 -- 172.21.15.60:0/2764958373 shutdown_connections 2024-04-14T23:48:56.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.117+0000 7fdfb9200640 1 -- 172.21.15.60:0/2764958373 wait complete. 2024-04-14T23:48:56.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.119+0000 7fdfb9200640 1 Processor -- start 2024-04-14T23:48:56.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.119+0000 7fdfb9200640 1 -- start start 2024-04-14T23:48:56.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.120+0000 7fdfb9200640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdfb40711a0 0x7fdfb41533b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:56.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.120+0000 7fdfb9200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdfb40c6d70 0x7fdfb41e5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:56.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.120+0000 7fdfb9200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdfb415b740 0x7fdfb41f4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:56.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.120+0000 7fdfb9200640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fdfb415ebb0 con 0x7fdfb40711a0 2024-04-14T23:48:56.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.120+0000 7fdfb9200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdfb415e8d0 con 0x7fdfb40c6d70 2024-04-14T23:48:56.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.120+0000 7fdfb9200640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fdfb415ee90 con 0x7fdfb415b740 2024-04-14T23:48:56.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.120+0000 7fdfb3400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdfb40711a0 0x7fdfb41533b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:56.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.120+0000 7fdfb2a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdfb40c6d70 0x7fdfb41e5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:56.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.120+0000 7fdfb2a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdfb40c6d70 0x7fdfb41e5300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:58010/0 (socket says 172.21.15.60:58010) 2024-04-14T23:48:56.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.120+0000 7fdfb2a00640 1 -- 172.21.15.60:0/2141648092 learned_addr learned my addr 172.21.15.60:0/2141648092 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:56.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.120+0000 7fdfb3e00640 1 --2- 172.21.15.60:0/2141648092 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdfb415b740 0x7fdfb41f4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:56.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.121+0000 7fdfb2a00640 1 -- 172.21.15.60:0/2141648092 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdfb415b740 msgr2=0x7fdfb41f4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:56.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.121+0000 7fdfb2a00640 1 --2- 172.21.15.60:0/2141648092 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdfb415b740 0x7fdfb41f4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:56.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.121+0000 7fdfb2a00640 1 -- 172.21.15.60:0/2141648092 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdfb40711a0 msgr2=0x7fdfb41533b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:56.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.121+0000 7fdfb2a00640 1 --2- 172.21.15.60:0/2141648092 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdfb40711a0 0x7fdfb41533b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:56.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.121+0000 7fdfb2a00640 1 -- 172.21.15.60:0/2141648092 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfb41e5840 con 0x7fdfb40c6d70 2024-04-14T23:48:56.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.121+0000 7fdfb2a00640 1 --2- 172.21.15.60:0/2141648092 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdfb40c6d70 0x7fdfb41e5300 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c0098c0 tx=0x7fdf9c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:56.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.121+0000 7fdfab400640 1 -- 172.21.15.60:0/2141648092 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf9c034040 con 0x7fdfb40c6d70 2024-04-14T23:48:56.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.121+0000 7fdfab400640 1 -- 172.21.15.60:0/2141648092 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf9c039a10 con 0x7fdfb40c6d70 2024-04-14T23:48:56.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.121+0000 7fdfab400640 1 -- 172.21.15.60:0/2141648092 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf9c039de0 con 0x7fdfb40c6d70 2024-04-14T23:48:56.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.121+0000 7fdfb9200640 1 -- 172.21.15.60:0/2141648092 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfb41f4d40 con 0x7fdfb40c6d70 2024-04-14T23:48:56.129 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.122+0000 7fdfb9200640 1 -- 172.21.15.60:0/2141648092 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfb415db80 con 0x7fdfb40c6d70 2024-04-14T23:48:56.129 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.123+0000 7fdfb9200640 1 -- 172.21.15.60:0/2141648092 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfb406f440 con 0x7fdfb40c6d70 2024-04-14T23:48:56.129 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.124+0000 7fdfab400640 1 -- 172.21.15.60:0/2141648092 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7fdf9c037070 con 0x7fdfb40c6d70 2024-04-14T23:48:56.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.125+0000 7fdfab400640 1 --2- 172.21.15.60:0/2141648092 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fdf9006b9c0 0x7fdf9006de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:56.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.125+0000 7fdfab400640 1 -- 172.21.15.60:0/2141648092 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fdf9c0ac820 con 0x7fdfb40c6d70 2024-04-14T23:48:56.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.125+0000 7fdfb3400640 1 --2- 172.21.15.60:0/2141648092 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fdf9006b9c0 0x7fdf9006de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:56.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.126+0000 7fdfb3400640 1 --2- 172.21.15.60:0/2141648092 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fdf9006b9c0 0x7fdf9006de80 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fdfa4005880 tx=0x7fdfa4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:56.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.129+0000 7fdfab400640 1 -- 172.21.15.60:0/2141648092 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fdfb406f440 con 0x7fdfb40c6d70 2024-04-14T23:48:56.146 INFO:tasks.ceph.mds.c.smithi060.stderr:2024-04-14T23:48:56.140+0000 7f00df200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-14T23:48:56.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.330+0000 7fdfb9200640 1 -- 172.21.15.60:0/2141648092 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fdfb404f540 con 0x7fdfb40c6d70 2024-04-14T23:48:56.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.331+0000 7fdfab400640 1 -- 172.21.15.60:0/2141648092 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7fdf9c07add0 con 0x7fdfb40c6d70 2024-04-14T23:48:56.337 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2024-04-14T23:48:56.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.334+0000 7fdfb9200640 1 -- 172.21.15.60:0/2141648092 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fdf9006b9c0 msgr2=0x7fdf9006de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:56.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.334+0000 7fdfb9200640 1 --2- 172.21.15.60:0/2141648092 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fdf9006b9c0 0x7fdf9006de80 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fdfa4005880 tx=0x7fdfa4009040 comp rx=0 tx=0).stop 2024-04-14T23:48:56.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.334+0000 7fdfb9200640 1 -- 172.21.15.60:0/2141648092 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdfb40c6d70 msgr2=0x7fdfb41e5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:56.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.334+0000 7fdfb9200640 1 --2- 172.21.15.60:0/2141648092 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdfb40c6d70 0x7fdfb41e5300 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c0098c0 tx=0x7fdf9c0046c0 comp rx=0 tx=0).stop 2024-04-14T23:48:56.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.334+0000 7fdfb9200640 1 -- 172.21.15.60:0/2141648092 shutdown_connections 2024-04-14T23:48:56.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.334+0000 7fdfb9200640 1 --2- 172.21.15.60:0/2141648092 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fdf9006b9c0 0x7fdf9006de80 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:56.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.334+0000 7fdfb9200640 1 --2- 172.21.15.60:0/2141648092 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdfb415b740 0x7fdfb41f4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:56.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.334+0000 7fdfb9200640 1 --2- 172.21.15.60:0/2141648092 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdfb40c6d70 0x7fdfb41e5300 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:56.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.334+0000 7fdfb9200640 1 --2- 172.21.15.60:0/2141648092 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdfb40711a0 0x7fdfb41533b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:56.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.334+0000 7fdfb9200640 1 -- 172.21.15.60:0/2141648092 >> 172.21.15.60:0/2141648092 conn(0x7fdfb40cc2f0 msgr2=0x7fdfb4159be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:56.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.334+0000 7fdfb9200640 1 -- 172.21.15.60:0/2141648092 shutdown_connections 2024-04-14T23:48:56.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.334+0000 7fdfb9200640 1 -- 172.21.15.60:0/2141648092 wait complete. 2024-04-14T23:48:56.359 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-14T23:48:56.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.577+0000 7f4e95400640 1 Processor -- start 2024-04-14T23:48:56.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.577+0000 7f4e95400640 1 -- start start 2024-04-14T23:48:56.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.578+0000 7f4e95400640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f4e90152cf0 0x7f4e9015c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:56.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.579+0000 7f4e95400640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4e90159450 0x7f4e9015c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:56.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.579+0000 7f4e95400640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e901530d0 con 0x7f4e900c63c0 2024-04-14T23:48:56.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.579+0000 7f4e95400640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f4e900751f0 con 0x7f4e90159450 2024-04-14T23:48:56.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.579+0000 7f4e95400640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f4e900c8370 con 0x7f4e90152cf0 2024-04-14T23:48:56.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.579+0000 7f4e8f200640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4e90159450 0x7f4e9015c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:56.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.579+0000 7f4e8e800640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f4e90152cf0 0x7f4e9015c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:56.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.579+0000 7f4e8f200640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4e90159450 0x7f4e9015c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41914/0 (socket says 172.21.15.60:41914) 2024-04-14T23:48:56.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.579+0000 7f4e8f200640 1 -- 172.21.15.60:0/2315397808 learned_addr learned my addr 172.21.15.60:0/2315397808 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:56.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.579+0000 7f4e8de00640 1 --1- 172.21.15.60:0/2315397808 >> v1:172.21.15.186:6789/0 conn(0x7f4e900c63c0 0x7f4e900c3f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.60:33984/0 (socket says 172.21.15.60:33984) 2024-04-14T23:48:56.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.579+0000 7f4e8f200640 1 -- 172.21.15.60:0/2315397808 >> v2:172.21.15.186:3300/0 conn(0x7f4e90152cf0 msgr2=0x7f4e9015c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:56.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.579+0000 7f4e8f200640 1 --2- 172.21.15.60:0/2315397808 >> v2:172.21.15.186:3300/0 conn(0x7f4e90152cf0 0x7f4e9015c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:56.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.579+0000 7f4e8f200640 1 -- 172.21.15.60:0/2315397808 >> v1:172.21.15.186:6789/0 conn(0x7f4e900c63c0 legacy=0x7f4e900c3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:56.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.579+0000 7f4e8f200640 1 -- 172.21.15.60:0/2315397808 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e900c3810 con 0x7f4e90159450 2024-04-14T23:48:56.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.581+0000 7f4e8f200640 1 --2- 172.21.15.60:0/2315397808 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e90159450 0x7f4e9015c7e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4e9006fbe0 tx=0x7f4e8802ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=297713e96b9cebd4 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:56.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.581+0000 7f4e8d400640 1 -- 172.21.15.60:0/2315397808 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e8802f850 con 0x7f4e90159450 2024-04-14T23:48:56.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.581+0000 7f4e8d400640 1 -- 172.21.15.60:0/2315397808 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e8802f9f0 con 0x7f4e90159450 2024-04-14T23:48:56.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.581+0000 7f4e8d400640 1 -- 172.21.15.60:0/2315397808 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e8802fdc0 con 0x7f4e90159450 2024-04-14T23:48:56.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.582+0000 7f4e95400640 1 -- 172.21.15.60:0/2315397808 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e90159450 msgr2=0x7f4e9015c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:56.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.582+0000 7f4e95400640 1 --2- 172.21.15.60:0/2315397808 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e90159450 0x7f4e9015c7e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4e9006fbe0 tx=0x7f4e8802ba30 comp rx=0 tx=0).stop 2024-04-14T23:48:56.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.582+0000 7f4e95400640 1 -- 172.21.15.60:0/2315397808 shutdown_connections 2024-04-14T23:48:56.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.582+0000 7f4e95400640 1 --2- 172.21.15.60:0/2315397808 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e90159450 0x7f4e9015c7e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:56.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.582+0000 7f4e95400640 1 --2- 172.21.15.60:0/2315397808 >> v2:172.21.15.186:3300/0 conn(0x7f4e90152cf0 0x7f4e9015c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:56.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.582+0000 7f4e95400640 1 -- 172.21.15.60:0/2315397808 >> 172.21.15.60:0/2315397808 conn(0x7f4e900cc170 msgr2=0x7f4e900cc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:56.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.582+0000 7f4e95400640 1 -- 172.21.15.60:0/2315397808 shutdown_connections 2024-04-14T23:48:56.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.582+0000 7f4e95400640 1 -- 172.21.15.60:0/2315397808 wait complete. 2024-04-14T23:48:56.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.583+0000 7f4e95400640 1 Processor -- start 2024-04-14T23:48:56.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.584+0000 7f4e95400640 1 -- start start 2024-04-14T23:48:56.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.584+0000 7f4e95400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e900c63c0 0x7f4e90145940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:56.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.584+0000 7f4e95400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e90152cf0 0x7f4e90150470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:56.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.584+0000 7f4e95400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e90159450 0x7f4e9015b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:56.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.584+0000 7f4e95400640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f4e900c88f0 con 0x7f4e90152cf0 2024-04-14T23:48:56.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.584+0000 7f4e95400640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4e900c8610 con 0x7f4e900c63c0 2024-04-14T23:48:56.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.584+0000 7f4e95400640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f4e900c4690 con 0x7f4e90159450 2024-04-14T23:48:56.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.585+0000 7f4e8de00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e90152cf0 0x7f4e90150470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:56.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.585+0000 7f4e8de00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e90152cf0 0x7f4e90150470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41924/0 (socket says 172.21.15.60:41924) 2024-04-14T23:48:56.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.585+0000 7f4e8de00640 1 -- 172.21.15.60:0/1505584453 learned_addr learned my addr 172.21.15.60:0/1505584453 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:56.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.585+0000 7f4e8e800640 1 --2- 172.21.15.60:0/1505584453 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e900c63c0 0x7f4e90145940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:56.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.585+0000 7f4e8f200640 1 --2- 172.21.15.60:0/1505584453 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e90159450 0x7f4e9015b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:56.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.585+0000 7f4e8de00640 1 -- 172.21.15.60:0/1505584453 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e90159450 msgr2=0x7f4e9015b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:56.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.585+0000 7f4e8de00640 1 --2- 172.21.15.60:0/1505584453 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e90159450 0x7f4e9015b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:56.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.585+0000 7f4e8de00640 1 -- 172.21.15.60:0/1505584453 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e900c63c0 msgr2=0x7f4e90145940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:56.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.585+0000 7f4e8de00640 1 --2- 172.21.15.60:0/1505584453 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e900c63c0 0x7f4e90145940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:56.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.585+0000 7f4e8de00640 1 -- 172.21.15.60:0/1505584453 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e901f4dc0 con 0x7f4e90152cf0 2024-04-14T23:48:56.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.585+0000 7f4e8de00640 1 --2- 172.21.15.60:0/1505584453 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e90152cf0 0x7f4e90150470 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f4e7800c440 tx=0x7f4e78001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:56.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.586+0000 7f4e86a00640 1 -- 172.21.15.60:0/1505584453 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e780074e0 con 0x7f4e90152cf0 2024-04-14T23:48:56.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.586+0000 7f4e86a00640 1 -- 172.21.15.60:0/1505584453 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e78007680 con 0x7f4e90152cf0 2024-04-14T23:48:56.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.586+0000 7f4e86a00640 1 -- 172.21.15.60:0/1505584453 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e78004a20 con 0x7f4e90152cf0 2024-04-14T23:48:56.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.586+0000 7f4e95400640 1 -- 172.21.15.60:0/1505584453 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e901f5080 con 0x7f4e90152cf0 2024-04-14T23:48:56.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.586+0000 7f4e95400640 1 -- 172.21.15.60:0/1505584453 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e900718d0 con 0x7f4e90152cf0 2024-04-14T23:48:56.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.587+0000 7f4e95400640 1 -- 172.21.15.60:0/1505584453 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e90076970 con 0x7f4e90152cf0 2024-04-14T23:48:56.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.588+0000 7f4e86a00640 1 -- 172.21.15.60:0/1505584453 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f4e7801a020 con 0x7f4e90152cf0 2024-04-14T23:48:56.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.589+0000 7f4e86a00640 1 --2- 172.21.15.60:0/1505584453 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f4e6c06b9c0 0x7f4e6c06de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:56.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.589+0000 7f4e86a00640 1 -- 172.21.15.60:0/1505584453 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f4e7808a0e0 con 0x7f4e90152cf0 2024-04-14T23:48:56.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.589+0000 7f4e8e800640 1 --2- 172.21.15.60:0/1505584453 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f4e6c06b9c0 0x7f4e6c06de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:56.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.590+0000 7f4e8e800640 1 --2- 172.21.15.60:0/1505584453 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f4e6c06b9c0 0x7f4e6c06de80 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4e800074a0 tx=0x7f4e8000b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:56.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.593+0000 7f4e86a00640 1 -- 172.21.15.60:0/1505584453 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4e78010300 con 0x7f4e90152cf0 2024-04-14T23:48:56.793 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.787+0000 7f4e95400640 1 -- 172.21.15.60:0/1505584453 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f4e901562f0 con 0x7f4e90152cf0 2024-04-14T23:48:56.793 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.788+0000 7f4e86a00640 1 -- 172.21.15.60:0/1505584453 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7f4e78058c50 con 0x7f4e90152cf0 2024-04-14T23:48:56.794 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2024-04-14T23:48:56.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.791+0000 7f4e95400640 1 -- 172.21.15.60:0/1505584453 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f4e6c06b9c0 msgr2=0x7f4e6c06de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:56.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.791+0000 7f4e95400640 1 --2- 172.21.15.60:0/1505584453 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f4e6c06b9c0 0x7f4e6c06de80 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4e800074a0 tx=0x7f4e8000b040 comp rx=0 tx=0).stop 2024-04-14T23:48:56.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.791+0000 7f4e95400640 1 -- 172.21.15.60:0/1505584453 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e90152cf0 msgr2=0x7f4e90150470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:56.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.791+0000 7f4e95400640 1 --2- 172.21.15.60:0/1505584453 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e90152cf0 0x7f4e90150470 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f4e7800c440 tx=0x7f4e78001e70 comp rx=0 tx=0).stop 2024-04-14T23:48:56.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.791+0000 7f4e95400640 1 -- 172.21.15.60:0/1505584453 shutdown_connections 2024-04-14T23:48:56.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.791+0000 7f4e95400640 1 --2- 172.21.15.60:0/1505584453 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f4e6c06b9c0 0x7f4e6c06de80 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:56.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.791+0000 7f4e95400640 1 --2- 172.21.15.60:0/1505584453 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e90159450 0x7f4e9015b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:56.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.791+0000 7f4e95400640 1 --2- 172.21.15.60:0/1505584453 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e90152cf0 0x7f4e90150470 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:56.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.791+0000 7f4e95400640 1 --2- 172.21.15.60:0/1505584453 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e900c63c0 0x7f4e90145940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:56.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.791+0000 7f4e95400640 1 -- 172.21.15.60:0/1505584453 >> 172.21.15.60:0/1505584453 conn(0x7f4e900cc170 msgr2=0x7f4e900cc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:56.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.791+0000 7f4e95400640 1 -- 172.21.15.60:0/1505584453 shutdown_connections 2024-04-14T23:48:56.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:56.791+0000 7f4e95400640 1 -- 172.21.15.60:0/1505584453 wait complete. 2024-04-14T23:48:56.814 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-14T23:48:57.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.035+0000 7f48bf200640 1 Processor -- start 2024-04-14T23:48:57.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.035+0000 7f48bf200640 1 -- start start 2024-04-14T23:48:57.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.037+0000 7f48bf200640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f48b8154bc0 0x7f48b815b550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:57.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.037+0000 7f48bf200640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f48b80cc7d0 0x7f48b80b45a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:57.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.037+0000 7f48bf200640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48b815ba90 con 0x7f48b80b4ae0 2024-04-14T23:48:57.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.037+0000 7f48bf200640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f48b806c270 con 0x7f48b80cc7d0 2024-04-14T23:48:57.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.037+0000 7f48bf200640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f48b8154fa0 con 0x7f48b8154bc0 2024-04-14T23:48:57.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.037+0000 7f48bd400640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f48b80cc7d0 0x7f48b80b45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:57.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.037+0000 7f48bd400640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f48b80cc7d0 0x7f48b80b45a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41932/0 (socket says 172.21.15.60:41932) 2024-04-14T23:48:57.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.037+0000 7f48bd400640 1 -- 172.21.15.60:0/2251222360 learned_addr learned my addr 172.21.15.60:0/2251222360 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:57.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.037+0000 7f48be800640 1 --1- 172.21.15.60:0/2251222360 >> v1:172.21.15.60:6789/0 conn(0x7f48b80b4ae0 0x7f48b80cd110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53350/0 (socket says 172.21.15.60:53350) 2024-04-14T23:48:57.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.037+0000 7f48bde00640 1 --2- 172.21.15.60:0/2251222360 >> v2:172.21.15.186:3300/0 conn(0x7f48b8154bc0 0x7f48b815b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:57.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.038+0000 7f48bd400640 1 -- 172.21.15.60:0/2251222360 >> v2:172.21.15.186:3300/0 conn(0x7f48b8154bc0 msgr2=0x7f48b815b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:57.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.038+0000 7f48bd400640 1 --2- 172.21.15.60:0/2251222360 >> v2:172.21.15.186:3300/0 conn(0x7f48b8154bc0 0x7f48b815b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:57.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.038+0000 7f48bd400640 1 -- 172.21.15.60:0/2251222360 >> v1:172.21.15.60:6789/0 conn(0x7f48b80b4ae0 legacy=0x7f48b80cd110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:57.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.038+0000 7f48bd400640 1 -- 172.21.15.60:0/2251222360 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48b80cb6a0 con 0x7f48b80cc7d0 2024-04-14T23:48:57.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.038+0000 7f48bca00640 1 -- 172.21.15.60:0/2251222360 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4041051473 0 0) 0x7f48b4001110 con 0x7f48b80b4ae0 2024-04-14T23:48:57.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.038+0000 7f48bde00640 1 --2- 172.21.15.60:0/2251222360 >> v2:172.21.15.186:3300/0 conn(0x7f48b8154bc0 0x7f48b815b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:48:57.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.039+0000 7f48bd400640 1 --2- 172.21.15.60:0/2251222360 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48b80cc7d0 0x7f48b80b45a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f48b80cb2a0 tx=0x7f48a802ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c68a58892f4f0ebe server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:57.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.040+0000 7f48bca00640 1 -- 172.21.15.60:0/2251222360 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48a802f850 con 0x7f48b80cc7d0 2024-04-14T23:48:57.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.040+0000 7f48bca00640 1 -- 172.21.15.60:0/2251222360 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48a802f9f0 con 0x7f48b80cc7d0 2024-04-14T23:48:57.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.040+0000 7f48bca00640 1 -- 172.21.15.60:0/2251222360 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48a802fdc0 con 0x7f48b80cc7d0 2024-04-14T23:48:57.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.040+0000 7f48bf200640 1 -- 172.21.15.60:0/2251222360 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48b80cc7d0 msgr2=0x7f48b80b45a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:57.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.040+0000 7f48bf200640 1 --2- 172.21.15.60:0/2251222360 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48b80cc7d0 0x7f48b80b45a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f48b80cb2a0 tx=0x7f48a802ba30 comp rx=0 tx=0).stop 2024-04-14T23:48:57.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.041+0000 7f48bf200640 1 -- 172.21.15.60:0/2251222360 shutdown_connections 2024-04-14T23:48:57.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.041+0000 7f48bf200640 1 --2- 172.21.15.60:0/2251222360 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48b80cc7d0 0x7f48b80b45a0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:57.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.041+0000 7f48bf200640 1 --2- 172.21.15.60:0/2251222360 >> v2:172.21.15.186:3300/0 conn(0x7f48b8154bc0 0x7f48b815b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:57.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.041+0000 7f48bf200640 1 -- 172.21.15.60:0/2251222360 >> 172.21.15.60:0/2251222360 conn(0x7f48b80670f0 msgr2=0x7f48b8067500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:57.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.041+0000 7f48bf200640 1 -- 172.21.15.60:0/2251222360 shutdown_connections 2024-04-14T23:48:57.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.041+0000 7f48bf200640 1 -- 172.21.15.60:0/2251222360 wait complete. 2024-04-14T23:48:57.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.042+0000 7f48bf200640 1 Processor -- start 2024-04-14T23:48:57.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.043+0000 7f48bf200640 1 -- start start 2024-04-14T23:48:57.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.043+0000 7f48bf200640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48b80b4ae0 0x7f48b815bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:57.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.043+0000 7f48bf200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48b80cc7d0 0x7f48b8156410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:57.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.043+0000 7f48bf200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48b8154bc0 0x7f48b814abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:57.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.043+0000 7f48bf200640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f48b806bf00 con 0x7f48b80b4ae0 2024-04-14T23:48:57.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.043+0000 7f48bf200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f48b806bc20 con 0x7f48b8154bc0 2024-04-14T23:48:57.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.043+0000 7f48bf200640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f48b806c700 con 0x7f48b80cc7d0 2024-04-14T23:48:57.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.044+0000 7f48bde00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48b80b4ae0 0x7f48b815bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:57.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.044+0000 7f48bde00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48b80b4ae0 0x7f48b815bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41940/0 (socket says 172.21.15.60:41940) 2024-04-14T23:48:57.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.044+0000 7f48bde00640 1 -- 172.21.15.60:0/320517183 learned_addr learned my addr 172.21.15.60:0/320517183 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:57.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.044+0000 7f48be800640 1 --2- 172.21.15.60:0/320517183 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48b8154bc0 0x7f48b814abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:57.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.044+0000 7f48bd400640 1 --2- 172.21.15.60:0/320517183 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48b80cc7d0 0x7f48b8156410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:57.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.044+0000 7f48bde00640 1 -- 172.21.15.60:0/320517183 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48b80cc7d0 msgr2=0x7f48b8156410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:57.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.044+0000 7f48bde00640 1 --2- 172.21.15.60:0/320517183 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48b80cc7d0 0x7f48b8156410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:57.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.044+0000 7f48bde00640 1 -- 172.21.15.60:0/320517183 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48b8154bc0 msgr2=0x7f48b814abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:57.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.044+0000 7f48bde00640 1 --2- 172.21.15.60:0/320517183 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48b8154bc0 0x7f48b814abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:57.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.044+0000 7f48bde00640 1 -- 172.21.15.60:0/320517183 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48b81f4ae0 con 0x7f48b80b4ae0 2024-04-14T23:48:57.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.044+0000 7f48bde00640 1 --2- 172.21.15.60:0/320517183 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48b80b4ae0 0x7f48b815bf00 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f48b0009ec0 tx=0x7f48b000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:57.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.045+0000 7f48ae000640 1 -- 172.21.15.60:0/320517183 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48b0018020 con 0x7f48b80b4ae0 2024-04-14T23:48:57.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.045+0000 7f48ae000640 1 -- 172.21.15.60:0/320517183 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48b0014680 con 0x7f48b80b4ae0 2024-04-14T23:48:57.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.045+0000 7f48ae000640 1 -- 172.21.15.60:0/320517183 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48b0014a50 con 0x7f48b80b4ae0 2024-04-14T23:48:57.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.045+0000 7f48bf200640 1 -- 172.21.15.60:0/320517183 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48b815b6b0 con 0x7f48b80b4ae0 2024-04-14T23:48:57.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.045+0000 7f48bf200640 1 -- 172.21.15.60:0/320517183 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48b81f4000 con 0x7f48b80b4ae0 2024-04-14T23:48:57.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.046+0000 7f48bf200640 1 -- 172.21.15.60:0/320517183 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48b804f540 con 0x7f48b80b4ae0 2024-04-14T23:48:57.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.047+0000 7f48ae000640 1 -- 172.21.15.60:0/320517183 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f48b0005d10 con 0x7f48b80b4ae0 2024-04-14T23:48:57.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.048+0000 7f48ae000640 1 --2- 172.21.15.60:0/320517183 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f489c06b990 0x7f489c06de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:57.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.048+0000 7f48ae000640 1 -- 172.21.15.60:0/320517183 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f48b008b4c0 con 0x7f48b80b4ae0 2024-04-14T23:48:57.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.048+0000 7f48bd400640 1 --2- 172.21.15.60:0/320517183 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f489c06b990 0x7f489c06de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:57.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.049+0000 7f48bd400640 1 --2- 172.21.15.60:0/320517183 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f489c06b990 0x7f489c06de50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f48a8036000 tx=0x7f48a8008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:57.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.052+0000 7f48ae000640 1 -- 172.21.15.60:0/320517183 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f48b0022030 con 0x7f48b80b4ae0 2024-04-14T23:48:57.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.252+0000 7f48bf200640 1 -- 172.21.15.60:0/320517183 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f48b80cbe30 con 0x7f48b80b4ae0 2024-04-14T23:48:57.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.253+0000 7f48ae000640 1 -- 172.21.15.60:0/320517183 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7f48b0010040 con 0x7f48b80b4ae0 2024-04-14T23:48:57.259 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2024-04-14T23:48:57.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.255+0000 7f48bf200640 1 -- 172.21.15.60:0/320517183 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f489c06b990 msgr2=0x7f489c06de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:57.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.255+0000 7f48bf200640 1 --2- 172.21.15.60:0/320517183 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f489c06b990 0x7f489c06de50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f48a8036000 tx=0x7f48a8008030 comp rx=0 tx=0).stop 2024-04-14T23:48:57.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.255+0000 7f48bf200640 1 -- 172.21.15.60:0/320517183 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48b80b4ae0 msgr2=0x7f48b815bf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:57.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.255+0000 7f48bf200640 1 --2- 172.21.15.60:0/320517183 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48b80b4ae0 0x7f48b815bf00 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f48b0009ec0 tx=0x7f48b000c490 comp rx=0 tx=0).stop 2024-04-14T23:48:57.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.255+0000 7f48bf200640 1 -- 172.21.15.60:0/320517183 shutdown_connections 2024-04-14T23:48:57.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.255+0000 7f48bf200640 1 --2- 172.21.15.60:0/320517183 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f489c06b990 0x7f489c06de50 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:57.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.255+0000 7f48bf200640 1 --2- 172.21.15.60:0/320517183 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48b8154bc0 0x7f48b814abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:57.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.255+0000 7f48bf200640 1 --2- 172.21.15.60:0/320517183 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48b80cc7d0 0x7f48b8156410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:57.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.255+0000 7f48bf200640 1 --2- 172.21.15.60:0/320517183 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48b80b4ae0 0x7f48b815bf00 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:57.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.255+0000 7f48bf200640 1 -- 172.21.15.60:0/320517183 >> 172.21.15.60:0/320517183 conn(0x7f48b80670f0 msgr2=0x7f48b81523e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:57.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.256+0000 7f48bf200640 1 -- 172.21.15.60:0/320517183 shutdown_connections 2024-04-14T23:48:57.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.256+0000 7f48bf200640 1 -- 172.21.15.60:0/320517183 wait complete. 2024-04-14T23:48:57.279 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:48:57.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.474+0000 7f6810600640 1 Processor -- start 2024-04-14T23:48:57.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.475+0000 7f6810600640 1 -- start start 2024-04-14T23:48:57.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.476+0000 7f6810600640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f680806f5d0 0x7f6808154980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:57.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.476+0000 7f6810600640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f680815b0d0 con 0x7f68080731e0 2024-04-14T23:48:57.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.476+0000 7f6810600640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f680815b370 con 0x7f680806f1f0 2024-04-14T23:48:57.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.476+0000 7f6810600640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f68080c6e10 con 0x7f680806f5d0 2024-04-14T23:48:57.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.476+0000 7f680e600640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f68080731e0 0x7f68081654a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53352/0 (socket says 172.21.15.60:53352) 2024-04-14T23:48:57.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.476+0000 7f680d200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f680806f5d0 0x7f6808154980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:57.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.476+0000 7f680e600640 1 -- 172.21.15.60:0/705901203 learned_addr learned my addr 172.21.15.60:0/705901203 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:48:57.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.476+0000 7f680dc00640 1 --1- 172.21.15.60:0/705901203 >> v1:172.21.15.186:6789/0 conn(0x7f680806f1f0 0x7f68080c72a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.60:33992/0 (socket says 172.21.15.60:33992) 2024-04-14T23:48:57.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.477+0000 7f6807e00640 1 -- 172.21.15.60:0/705901203 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2505248176 0 0) 0x7f680815b370 con 0x7f680806f1f0 2024-04-14T23:48:57.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.477+0000 7f680d200640 1 -- 172.21.15.60:0/705901203 >> v1:172.21.15.186:6789/0 conn(0x7f680806f1f0 legacy=0x7f68080c72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:57.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.477+0000 7f680d200640 1 -- 172.21.15.60:0/705901203 >> v1:172.21.15.60:6789/0 conn(0x7f68080731e0 legacy=0x7f68081654a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:57.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.477+0000 7f680d200640 1 -- 172.21.15.60:0/705901203 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f680815b650 con 0x7f680806f5d0 2024-04-14T23:48:57.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.478+0000 7f680d200640 1 --2- 172.21.15.60:0/705901203 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f680806f5d0 0x7f6808154980 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f67f40097d0 tx=0x7f67f402b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6bf55591bfb5e78a server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:57.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.478+0000 7f6807e00640 1 -- 172.21.15.60:0/705901203 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67f4037070 con 0x7f680806f5d0 2024-04-14T23:48:57.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.478+0000 7f6807e00640 1 -- 172.21.15.60:0/705901203 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67f4030710 con 0x7f680806f5d0 2024-04-14T23:48:57.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.478+0000 7f6807e00640 1 -- 172.21.15.60:0/705901203 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67f4030ae0 con 0x7f680806f5d0 2024-04-14T23:48:57.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.479+0000 7f6810600640 1 -- 172.21.15.60:0/705901203 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f680806f5d0 msgr2=0x7f6808154980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:57.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.479+0000 7f6810600640 1 --2- 172.21.15.60:0/705901203 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f680806f5d0 0x7f6808154980 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f67f40097d0 tx=0x7f67f402b800 comp rx=0 tx=0).stop 2024-04-14T23:48:57.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.479+0000 7f6810600640 1 -- 172.21.15.60:0/705901203 shutdown_connections 2024-04-14T23:48:57.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.479+0000 7f6810600640 1 --2- 172.21.15.60:0/705901203 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f680806f5d0 0x7f6808154980 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:57.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.479+0000 7f6810600640 1 -- 172.21.15.60:0/705901203 >> 172.21.15.60:0/705901203 conn(0x7f68080cc170 msgr2=0x7f68080cc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:57.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.479+0000 7f6810600640 1 -- 172.21.15.60:0/705901203 shutdown_connections 2024-04-14T23:48:57.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.479+0000 7f6810600640 1 -- 172.21.15.60:0/705901203 wait complete. 2024-04-14T23:48:57.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.480+0000 7f6810600640 1 Processor -- start 2024-04-14T23:48:57.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.481+0000 7f6810600640 1 -- start start 2024-04-14T23:48:57.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.481+0000 7f6810600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f680806f1f0 0x7f680815a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:57.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.481+0000 7f6810600640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f680806f5d0 0x7f680815ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:57.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.482+0000 7f6810600640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f68080731e0 0x7f6808153270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:57.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.482+0000 7f6810600640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f68080c79b0 con 0x7f680806f5d0 2024-04-14T23:48:57.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.482+0000 7f6810600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6808071180 con 0x7f680806f1f0 2024-04-14T23:48:57.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.482+0000 7f6810600640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f68080c7b30 con 0x7f68080731e0 2024-04-14T23:48:57.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.482+0000 7f680d200640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f680806f5d0 0x7f680815ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:57.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.482+0000 7f680d200640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f680806f5d0 0x7f680815ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41944/0 (socket says 172.21.15.60:41944) 2024-04-14T23:48:57.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.482+0000 7f680d200640 1 -- 172.21.15.60:0/3048279838 learned_addr learned my addr 172.21.15.60:0/3048279838 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:57.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.482+0000 7f680e600640 1 --2- 172.21.15.60:0/3048279838 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f68080731e0 0x7f6808153270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:57.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.482+0000 7f680dc00640 1 --2- 172.21.15.60:0/3048279838 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f680806f1f0 0x7f680815a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:57.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.482+0000 7f680d200640 1 -- 172.21.15.60:0/3048279838 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f68080731e0 msgr2=0x7f6808153270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:57.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.482+0000 7f680d200640 1 --2- 172.21.15.60:0/3048279838 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f68080731e0 0x7f6808153270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:57.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.482+0000 7f680d200640 1 -- 172.21.15.60:0/3048279838 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f680806f1f0 msgr2=0x7f680815a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:57.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.482+0000 7f680d200640 1 --2- 172.21.15.60:0/3048279838 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f680806f1f0 0x7f680815a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:57.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.482+0000 7f680d200640 1 -- 172.21.15.60:0/3048279838 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68081f4310 con 0x7f680806f5d0 2024-04-14T23:48:57.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.483+0000 7f680d200640 1 --2- 172.21.15.60:0/3048279838 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f680806f5d0 0x7f680815ebc0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f67f4009720 tx=0x7f67f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:57.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.483+0000 7f6805600640 1 -- 172.21.15.60:0/3048279838 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67f4042070 con 0x7f680806f5d0 2024-04-14T23:48:57.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.483+0000 7f6805600640 1 -- 172.21.15.60:0/3048279838 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67f4032a40 con 0x7f680806f5d0 2024-04-14T23:48:57.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.483+0000 7f6805600640 1 -- 172.21.15.60:0/3048279838 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67f4037040 con 0x7f680806f5d0 2024-04-14T23:48:57.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.483+0000 7f6810600640 1 -- 172.21.15.60:0/3048279838 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f680815c7c0 con 0x7f680806f5d0 2024-04-14T23:48:57.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.483+0000 7f6810600640 1 -- 172.21.15.60:0/3048279838 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68081597e0 con 0x7f680806f5d0 2024-04-14T23:48:57.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.484+0000 7f6810600640 1 -- 172.21.15.60:0/3048279838 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6808076970 con 0x7f680806f5d0 2024-04-14T23:48:57.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.485+0000 7f6805600640 1 -- 172.21.15.60:0/3048279838 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f67f4045070 con 0x7f680806f5d0 2024-04-14T23:48:57.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.486+0000 7f6805600640 1 --2- 172.21.15.60:0/3048279838 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f67e806b990 0x7f67e806de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:57.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.486+0000 7f6805600640 1 -- 172.21.15.60:0/3048279838 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f67f40ac7f0 con 0x7f680806f5d0 2024-04-14T23:48:57.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.486+0000 7f680dc00640 1 --2- 172.21.15.60:0/3048279838 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f67e806b990 0x7f67e806de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:57.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.487+0000 7f680dc00640 1 --2- 172.21.15.60:0/3048279838 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f67e806b990 0x7f67e806de50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f67fc000f60 tx=0x7f67fc007e40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:57.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.490+0000 7f6805600640 1 -- 172.21.15.60:0/3048279838 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f67f407aec0 con 0x7f680806f5d0 2024-04-14T23:48:57.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.729+0000 7f6810600640 1 -- 172.21.15.60:0/3048279838 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f68080c7440 con 0x7f680806f5d0 2024-04-14T23:48:57.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.731+0000 7f6805600640 1 -- 172.21.15.60:0/3048279838 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6827 (secure 0 0 0) 0x7f67f407ef60 con 0x7f680806f5d0 2024-04-14T23:48:57.737 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:48:57.737 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/78303888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":78303888},{"type":"v1","addr":"172.21.15.60:6833","nonce":78303888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:48:49.256965+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.60:6835/2677133122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2677133122},{"type":"v1","addr":"172.21.15.60:6835","nonce":2677133122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4279,"qdb_cluster":[4279]},"id":1}]} 2024-04-14T23:48:57.737 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 8 2024-04-14T23:48:57.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.734+0000 7f6810600640 1 -- 172.21.15.60:0/3048279838 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f67e806b990 msgr2=0x7f67e806de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:57.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.734+0000 7f6810600640 1 --2- 172.21.15.60:0/3048279838 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f67e806b990 0x7f67e806de50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f67fc000f60 tx=0x7f67fc007e40 comp rx=0 tx=0).stop 2024-04-14T23:48:57.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.734+0000 7f6810600640 1 -- 172.21.15.60:0/3048279838 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f680806f5d0 msgr2=0x7f680815ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:57.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.734+0000 7f6810600640 1 --2- 172.21.15.60:0/3048279838 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f680806f5d0 0x7f680815ebc0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f67f4009720 tx=0x7f67f40046c0 comp rx=0 tx=0).stop 2024-04-14T23:48:57.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.734+0000 7f6810600640 1 -- 172.21.15.60:0/3048279838 shutdown_connections 2024-04-14T23:48:57.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.734+0000 7f6810600640 1 --2- 172.21.15.60:0/3048279838 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f67e806b990 0x7f67e806de50 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:57.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.734+0000 7f6810600640 1 --2- 172.21.15.60:0/3048279838 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f68080731e0 0x7f6808153270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:57.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.734+0000 7f6810600640 1 --2- 172.21.15.60:0/3048279838 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f680806f5d0 0x7f680815ebc0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:57.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.734+0000 7f6810600640 1 --2- 172.21.15.60:0/3048279838 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f680806f1f0 0x7f680815a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:57.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.734+0000 7f6810600640 1 -- 172.21.15.60:0/3048279838 >> 172.21.15.60:0/3048279838 conn(0x7f68080cc170 msgr2=0x7f68080cabe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:57.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.734+0000 7f6810600640 1 -- 172.21.15.60:0/3048279838 shutdown_connections 2024-04-14T23:48:57.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.734+0000 7f6810600640 1 -- 172.21.15.60:0/3048279838 wait complete. 2024-04-14T23:48:57.760 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-14T23:48:57.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.961+0000 7fd2c8c00640 1 Processor -- start 2024-04-14T23:48:57.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.961+0000 7fd2c8c00640 1 -- start start 2024-04-14T23:48:57.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.963+0000 7fd2c8c00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd2c4073320 0x7fd2c40c6950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:57.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.963+0000 7fd2c8c00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd2c40c3d90 0x7fd2c40c41e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:57.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.963+0000 7fd2c8c00640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2c4152fb0 con 0x7fd2c40b1e00 2024-04-14T23:48:57.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.963+0000 7fd2c8c00640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fd2c40c6540 con 0x7fd2c40c3d90 2024-04-14T23:48:57.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.963+0000 7fd2c8c00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fd2c40b21e0 con 0x7fd2c4073320 2024-04-14T23:48:57.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.963+0000 7fd2c3e00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd2c40c3d90 0x7fd2c40c41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:57.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.963+0000 7fd2c2a00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd2c4073320 0x7fd2c40c6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:57.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.963+0000 7fd2c3400640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fd2c40b1e00 0x7fd2c40c7fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.60:34002/0 (socket says 172.21.15.60:34002) 2024-04-14T23:48:57.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.963+0000 7fd2c3400640 1 -- 172.21.15.60:0/2735098475 learned_addr learned my addr 172.21.15.60:0/2735098475 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:48:57.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.964+0000 7fd2c3e00640 1 -- 172.21.15.60:0/2735098475 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd2c4073320 msgr2=0x7fd2c40c6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:57.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.964+0000 7fd2c3e00640 1 --2- 172.21.15.60:0/2735098475 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd2c4073320 0x7fd2c40c6950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:57.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.964+0000 7fd2c3e00640 1 -- 172.21.15.60:0/2735098475 >> v1:172.21.15.186:6789/0 conn(0x7fd2c40b1e00 legacy=0x7fd2c40c7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:57.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.964+0000 7fd2c3e00640 1 -- 172.21.15.60:0/2735098475 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2c4159310 con 0x7fd2c40c3d90 2024-04-14T23:48:57.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.964+0000 7fd2c2000640 1 -- 172.21.15.60:0/2735098475 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1729972196 0 0) 0x7fd2c4152fb0 con 0x7fd2c40b1e00 2024-04-14T23:48:57.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.965+0000 7fd2c3e00640 1 --2- 172.21.15.60:0/2735098475 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2c40c3d90 0x7fd2c40c41e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc009d90 tx=0x7fd2bc02bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc77ffb4028c6841 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:57.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.965+0000 7fd2c2000640 1 -- 172.21.15.60:0/2735098475 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2bc039070 con 0x7fd2c40c3d90 2024-04-14T23:48:57.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.965+0000 7fd2c2000640 1 -- 172.21.15.60:0/2735098475 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2bc02cce0 con 0x7fd2c40c3d90 2024-04-14T23:48:57.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.965+0000 7fd2c2000640 1 -- 172.21.15.60:0/2735098475 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2bc034660 con 0x7fd2c40c3d90 2024-04-14T23:48:57.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.966+0000 7fd2c8c00640 1 -- 172.21.15.60:0/2735098475 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2c40c3d90 msgr2=0x7fd2c40c41e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:57.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.966+0000 7fd2c8c00640 1 --2- 172.21.15.60:0/2735098475 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2c40c3d90 0x7fd2c40c41e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc009d90 tx=0x7fd2bc02bf30 comp rx=0 tx=0).stop 2024-04-14T23:48:57.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.966+0000 7fd2c8c00640 1 -- 172.21.15.60:0/2735098475 shutdown_connections 2024-04-14T23:48:57.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.966+0000 7fd2c8c00640 1 --2- 172.21.15.60:0/2735098475 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd2c4073320 0x7fd2c40c6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:57.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.966+0000 7fd2c8c00640 1 --2- 172.21.15.60:0/2735098475 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2c40c3d90 0x7fd2c40c41e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:57.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.966+0000 7fd2c8c00640 1 -- 172.21.15.60:0/2735098475 >> 172.21.15.60:0/2735098475 conn(0x7fd2c40c8b50 msgr2=0x7fd2c40c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:57.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.966+0000 7fd2c8c00640 1 -- 172.21.15.60:0/2735098475 shutdown_connections 2024-04-14T23:48:57.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.966+0000 7fd2c8c00640 1 -- 172.21.15.60:0/2735098475 wait complete. 2024-04-14T23:48:57.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.967+0000 7fd2c8c00640 1 Processor -- start 2024-04-14T23:48:57.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.968+0000 7fd2c8c00640 1 -- start start 2024-04-14T23:48:57.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.968+0000 7fd2c8c00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2c4073320 0x7fd2c415b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:57.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.968+0000 7fd2c8c00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd2c40b1e00 0x7fd2c40af950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:57.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.968+0000 7fd2c8c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2c40c3d90 0x7fd2c40aea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:57.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.969+0000 7fd2c8c00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fd2c40c38b0 con 0x7fd2c4073320 2024-04-14T23:48:57.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.969+0000 7fd2c8c00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd2c4159890 con 0x7fd2c40c3d90 2024-04-14T23:48:57.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.969+0000 7fd2c8c00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fd2c41595b0 con 0x7fd2c40b1e00 2024-04-14T23:48:57.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.969+0000 7fd2c3400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2c4073320 0x7fd2c415b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:57.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.969+0000 7fd2c3400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2c4073320 0x7fd2c415b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41948/0 (socket says 172.21.15.60:41948) 2024-04-14T23:48:57.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.969+0000 7fd2c3400640 1 -- 172.21.15.60:0/2465916349 learned_addr learned my addr 172.21.15.60:0/2465916349 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:57.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.969+0000 7fd2c2a00640 1 --2- 172.21.15.60:0/2465916349 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd2c40b1e00 0x7fd2c40af950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:57.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.969+0000 7fd2c3e00640 1 --2- 172.21.15.60:0/2465916349 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2c40c3d90 0x7fd2c40aea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:57.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.969+0000 7fd2c3400640 1 -- 172.21.15.60:0/2465916349 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd2c40b1e00 msgr2=0x7fd2c40af950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:57.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.969+0000 7fd2c3400640 1 --2- 172.21.15.60:0/2465916349 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd2c40b1e00 0x7fd2c40af950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:57.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.969+0000 7fd2c3400640 1 -- 172.21.15.60:0/2465916349 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2c40c3d90 msgr2=0x7fd2c40aea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:57.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.969+0000 7fd2c3400640 1 --2- 172.21.15.60:0/2465916349 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2c40c3d90 0x7fd2c40aea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:57.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.969+0000 7fd2c3400640 1 -- 172.21.15.60:0/2465916349 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2c41f0980 con 0x7fd2c4073320 2024-04-14T23:48:57.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.969+0000 7fd2c3e00640 1 --2- 172.21.15.60:0/2465916349 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2c40c3d90 0x7fd2c40aea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:48:57.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.970+0000 7fd2c3400640 1 --2- 172.21.15.60:0/2465916349 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2c4073320 0x7fd2c415b7f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd2b400c3f0 tx=0x7fd2b400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:57.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.970+0000 7fd2bb400640 1 -- 172.21.15.60:0/2465916349 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2b40074e0 con 0x7fd2c4073320 2024-04-14T23:48:57.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.970+0000 7fd2bb400640 1 -- 172.21.15.60:0/2465916349 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2b40047e0 con 0x7fd2c4073320 2024-04-14T23:48:57.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.970+0000 7fd2bb400640 1 -- 172.21.15.60:0/2465916349 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2b4004bd0 con 0x7fd2c4073320 2024-04-14T23:48:57.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.970+0000 7fd2c8c00640 1 -- 172.21.15.60:0/2465916349 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2c41f0c60 con 0x7fd2c4073320 2024-04-14T23:48:57.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.970+0000 7fd2c8c00640 1 -- 172.21.15.60:0/2465916349 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2c41f1220 con 0x7fd2c4073320 2024-04-14T23:48:57.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.971+0000 7fd2c8c00640 1 -- 172.21.15.60:0/2465916349 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2c404f540 con 0x7fd2c4073320 2024-04-14T23:48:57.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.972+0000 7fd2bb400640 1 -- 172.21.15.60:0/2465916349 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7fd2b4011070 con 0x7fd2c4073320 2024-04-14T23:48:57.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.974+0000 7fd2bb400640 1 --2- 172.21.15.60:0/2465916349 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fd2a406b9c0 0x7fd2a406de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:57.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.974+0000 7fd2bb400640 1 -- 172.21.15.60:0/2465916349 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fd2b408a300 con 0x7fd2c4073320 2024-04-14T23:48:57.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.974+0000 7fd2c2a00640 1 --2- 172.21.15.60:0/2465916349 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fd2a406b9c0 0x7fd2a406de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:57.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.975+0000 7fd2c2a00640 1 --2- 172.21.15.60:0/2465916349 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fd2a406b9c0 0x7fd2a406de80 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd2c4158ad0 tx=0x7fd2b0009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:57.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:57.978+0000 7fd2bb400640 1 -- 172.21.15.60:0/2465916349 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd2b4058820 con 0x7fd2c4073320 2024-04-14T23:48:58.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.181+0000 7fd2c8c00640 1 -- 172.21.15.60:0/2465916349 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd2c41f13f0 con 0x7fd2c4073320 2024-04-14T23:48:58.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.183+0000 7fd2bb400640 1 -- 172.21.15.60:0/2465916349 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16605 (secure 0 0 0) 0x7fd2b405c8c0 con 0x7fd2c4073320 2024-04-14T23:48:58.190 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:48:58.191 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":21,"fsid":"5847db56-a76d-4486-b563-d83f1186af92","created":"2024-04-14T23:48:23.900003+0000","modified":"2024-04-14T23:48:44.219352+0000","last_up_change":"2024-04-14T23:48:34.203573+0000","last_in_change":"2024-04-14T23:48:26.345499+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-14T23:48:36.247372+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-14T23:48:40.274195+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-14T23:48:41.722183+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b2b2b755-7f42-4f9b-af9a-a178f7cc5a52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6817","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6818","nonce":2536528895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6822","nonce":2536528895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6827","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6828","nonce":2536528895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6825","nonce":2536528895}]},"public_addr":"172.21.15.60:6818/2536528895","cluster_addr":"172.21.15.60:6822/2536528895","heartbeat_back_addr":"172.21.15.60:6828/2536528895","heartbeat_front_addr":"172.21.15.60:6825/2536528895","state":["exists","up"]},{"osd":1,"uuid":"321418cd-5ab7-41a5-b9c8-b3fc12f68ebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6801","nonce":3115695676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6803","nonce":3115695676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6807","nonce":3115695676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6805","nonce":3115695676}]},"public_addr":"172.21.15.60:6801/3115695676","cluster_addr":"172.21.15.60:6803/3115695676","heartbeat_back_addr":"172.21.15.60:6807/3115695676","heartbeat_front_addr":"172.21.15.60:6805/3115695676","state":["exists","up"]},{"osd":2,"uuid":"07874e9f-3a38-41dc-be9f-2b75e84e53d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6819","nonce":3475168698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6824","nonce":3475168698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6831","nonce":3475168698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6829","nonce":3475168698}]},"public_addr":"172.21.15.60:6819/3475168698","cluster_addr":"172.21.15.60:6824/3475168698","heartbeat_back_addr":"172.21.15.60:6831/3475168698","heartbeat_front_addr":"172.21.15.60:6829/3475168698","state":["exists","up"]},{"osd":3,"uuid":"7e4ef6c9-b17a-47c6-a320-f5d16a6fe05e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6809","nonce":1730813674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6811","nonce":1730813674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6815","nonce":1730813674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6813","nonce":1730813674}]},"public_addr":"172.21.15.60:6809/1730813674","cluster_addr":"172.21.15.60:6811/1730813674","heartbeat_back_addr":"172.21.15.60:6815/1730813674","heartbeat_front_addr":"172.21.15.60:6813/1730813674","state":["exists","up"]},{"osd":4,"uuid":"759a3efc-dade-44c5-b57a-1623499aba50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6805","nonce":1464587038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6810","nonce":1464587038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6819","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6822","nonce":1464587038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6813","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6816","nonce":1464587038}]},"public_addr":"172.21.15.186:6805/1464587038","cluster_addr":"172.21.15.186:6810/1464587038","heartbeat_back_addr":"172.21.15.186:6822/1464587038","heartbeat_front_addr":"172.21.15.186:6816/1464587038","state":["exists","up"]},{"osd":5,"uuid":"24cd58c1-0cf1-4077-ab33-84c39c6907ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6801","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6803","nonce":1697098067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6809","nonce":1697098067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6820","nonce":1697098067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6815","nonce":1697098067}]},"public_addr":"172.21.15.186:6803/1697098067","cluster_addr":"172.21.15.186:6809/1697098067","heartbeat_back_addr":"172.21.15.186:6820/1697098067","heartbeat_front_addr":"172.21.15.186:6815/1697098067","state":["exists","up"]},{"osd":6,"uuid":"6add38f1-65e9-4b0d-927a-4ab77f5e2323","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6804","nonce":1837108559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6807","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6811","nonce":1837108559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6823","nonce":1837108559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6817","nonce":1837108559}]},"public_addr":"172.21.15.186:6804/1837108559","cluster_addr":"172.21.15.186:6811/1837108559","heartbeat_back_addr":"172.21.15.186:6823/1837108559","heartbeat_front_addr":"172.21.15.186:6817/1837108559","state":["exists","up"]},{"osd":7,"uuid":"a2326b6f-7601-4dda-a456-67c443ea5929","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6825","nonce":842537875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6827","nonce":842537875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6831","nonce":842537875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6829","nonce":842537875}]},"public_addr":"172.21.15.186:6825/842537875","cluster_addr":"172.21.15.186:6827/842537875","heartbeat_back_addr":"172.21.15.186:6831/842537875","heartbeat_front_addr":"172.21.15.186:6829/842537875","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.975583+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.976806+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.722725+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.763705+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.502140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.528972+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.517961+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.513350+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-14T23:48:58.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.185+0000 7fd2c8c00640 1 -- 172.21.15.60:0/2465916349 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fd2a406b9c0 msgr2=0x7fd2a406de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:58.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.185+0000 7fd2c8c00640 1 --2- 172.21.15.60:0/2465916349 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fd2a406b9c0 0x7fd2a406de80 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd2c4158ad0 tx=0x7fd2b0009190 comp rx=0 tx=0).stop 2024-04-14T23:48:58.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.185+0000 7fd2c8c00640 1 -- 172.21.15.60:0/2465916349 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2c4073320 msgr2=0x7fd2c415b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:58.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.185+0000 7fd2c8c00640 1 --2- 172.21.15.60:0/2465916349 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2c4073320 0x7fd2c415b7f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd2b400c3f0 tx=0x7fd2b400c8b0 comp rx=0 tx=0).stop 2024-04-14T23:48:58.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.186+0000 7fd2c8c00640 1 -- 172.21.15.60:0/2465916349 shutdown_connections 2024-04-14T23:48:58.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.186+0000 7fd2c8c00640 1 --2- 172.21.15.60:0/2465916349 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fd2a406b9c0 0x7fd2a406de80 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:58.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.186+0000 7fd2c8c00640 1 --2- 172.21.15.60:0/2465916349 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2c40c3d90 0x7fd2c40aea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:58.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.186+0000 7fd2c8c00640 1 --2- 172.21.15.60:0/2465916349 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd2c40b1e00 0x7fd2c40af950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:58.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.186+0000 7fd2c8c00640 1 --2- 172.21.15.60:0/2465916349 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2c4073320 0x7fd2c415b7f0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:58.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.186+0000 7fd2c8c00640 1 -- 172.21.15.60:0/2465916349 >> 172.21.15.60:0/2465916349 conn(0x7fd2c40c8b50 msgr2=0x7fd2c40c6740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:58.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.186+0000 7fd2c8c00640 1 -- 172.21.15.60:0/2465916349 shutdown_connections 2024-04-14T23:48:58.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.186+0000 7fd2c8c00640 1 -- 172.21.15.60:0/2465916349 wait complete. 2024-04-14T23:48:58.211 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-14T23:48:36.247372+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-14T23:48:40.274195+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-14T23:48:41.722183+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-14T23:48:58.211 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-14T23:48:58.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.434+0000 7f6cd1800640 1 Processor -- start 2024-04-14T23:48:58.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.434+0000 7f6cd1800640 1 -- start start 2024-04-14T23:48:58.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.436+0000 7f6cd1800640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6ccc154d60 0x7f6ccc0cbb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:58.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.436+0000 7f6cd1800640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6ccc0cd800 0x7f6ccc06ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:58.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.436+0000 7f6cd1800640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ccc15aa50 con 0x7f6ccc06bf70 2024-04-14T23:48:58.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.436+0000 7f6cd1800640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f6ccc15e040 con 0x7f6ccc154d60 2024-04-14T23:48:58.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.436+0000 7f6cd1800640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f6ccc0cc070 con 0x7f6ccc0cd800 2024-04-14T23:48:58.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.437+0000 7f6ccbe00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6ccc154d60 0x7f6ccc0cbb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:58.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.437+0000 7f6ccbe00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6ccc154d60 0x7f6ccc0cbb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41960/0 (socket says 172.21.15.60:41960) 2024-04-14T23:48:58.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.437+0000 7f6ccbe00640 1 -- 172.21.15.60:0/208457128 learned_addr learned my addr 172.21.15.60:0/208457128 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:58.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.437+0000 7f6cd0e00640 1 --1- 172.21.15.60:0/208457128 >> v1:172.21.15.60:6789/0 conn(0x7f6ccc06bf70 0x7f6ccc15af50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53362/0 (socket says 172.21.15.60:53362) 2024-04-14T23:48:58.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.437+0000 7f6ccb400640 1 --2- 172.21.15.60:0/208457128 >> v2:172.21.15.186:3300/0 conn(0x7f6ccc0cd800 0x7f6ccc06ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:58.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.437+0000 7f6ccbe00640 1 -- 172.21.15.60:0/208457128 >> v2:172.21.15.186:3300/0 conn(0x7f6ccc0cd800 msgr2=0x7f6ccc06ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:58.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.437+0000 7f6ccbe00640 1 --2- 172.21.15.60:0/208457128 >> v2:172.21.15.186:3300/0 conn(0x7f6ccc0cd800 0x7f6ccc06ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:58.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.437+0000 7f6ccbe00640 1 -- 172.21.15.60:0/208457128 >> v1:172.21.15.60:6789/0 conn(0x7f6ccc06bf70 legacy=0x7f6ccc15af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:58.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.437+0000 7f6ccbe00640 1 -- 172.21.15.60:0/208457128 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ccc15e270 con 0x7f6ccc154d60 2024-04-14T23:48:58.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.438+0000 7f6ccbe00640 1 --2- 172.21.15.60:0/208457128 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ccc154d60 0x7f6ccc0cbb30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc0733a0 tx=0x7f6cbc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6be5c20ec2b93a9c server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:58.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.439+0000 7f6ccaa00640 1 -- 172.21.15.60:0/208457128 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cbc034070 con 0x7f6ccc154d60 2024-04-14T23:48:58.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.439+0000 7f6ccaa00640 1 -- 172.21.15.60:0/208457128 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cbc030860 con 0x7f6ccc154d60 2024-04-14T23:48:58.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.439+0000 7f6ccaa00640 1 -- 172.21.15.60:0/208457128 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cbc030c30 con 0x7f6ccc154d60 2024-04-14T23:48:58.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.439+0000 7f6cd1800640 1 -- 172.21.15.60:0/208457128 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ccc154d60 msgr2=0x7f6ccc0cbb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:58.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.439+0000 7f6cd1800640 1 --2- 172.21.15.60:0/208457128 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ccc154d60 0x7f6ccc0cbb30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc0733a0 tx=0x7f6cbc02b9a0 comp rx=0 tx=0).stop 2024-04-14T23:48:58.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.440+0000 7f6cd1800640 1 -- 172.21.15.60:0/208457128 shutdown_connections 2024-04-14T23:48:58.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.440+0000 7f6cd1800640 1 --2- 172.21.15.60:0/208457128 >> v2:172.21.15.186:3300/0 conn(0x7f6ccc0cd800 0x7f6ccc06ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:58.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.440+0000 7f6cd1800640 1 --2- 172.21.15.60:0/208457128 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ccc154d60 0x7f6ccc0cbb30 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:58.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.440+0000 7f6cd1800640 1 -- 172.21.15.60:0/208457128 >> 172.21.15.60:0/208457128 conn(0x7f6ccc072f20 msgr2=0x7f6ccc06d8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:58.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.440+0000 7f6cd1800640 1 -- 172.21.15.60:0/208457128 shutdown_connections 2024-04-14T23:48:58.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.440+0000 7f6cd1800640 1 -- 172.21.15.60:0/208457128 wait complete. 2024-04-14T23:48:58.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.442+0000 7f6cd1800640 1 Processor -- start 2024-04-14T23:48:58.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.442+0000 7f6cd1800640 1 -- start start 2024-04-14T23:48:58.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.443+0000 7f6cd1800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6ccc06bf70 0x7f6ccc1580b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:58.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.443+0000 7f6cd1800640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ccc0cd800 0x7f6ccc1f3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:58.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.443+0000 7f6cd1800640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6ccc154d60 0x7f6ccc1f2890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:58.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.443+0000 7f6cd1800640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f6ccc15eb10 con 0x7f6ccc0cd800 2024-04-14T23:48:58.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.443+0000 7f6cd1800640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6ccc15e830 con 0x7f6ccc06bf70 2024-04-14T23:48:58.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.443+0000 7f6cd1800640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f6ccc15edf0 con 0x7f6ccc154d60 2024-04-14T23:48:58.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.444+0000 7f6ccb400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ccc0cd800 0x7f6ccc1f3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:58.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.444+0000 7f6ccb400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ccc0cd800 0x7f6ccc1f3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41966/0 (socket says 172.21.15.60:41966) 2024-04-14T23:48:58.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.444+0000 7f6ccb400640 1 -- 172.21.15.60:0/1788122351 learned_addr learned my addr 172.21.15.60:0/1788122351 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:58.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.444+0000 7f6ccbe00640 1 --2- 172.21.15.60:0/1788122351 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6ccc06bf70 0x7f6ccc1580b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:58.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.444+0000 7f6cd0e00640 1 --2- 172.21.15.60:0/1788122351 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6ccc154d60 0x7f6ccc1f2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:58.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.444+0000 7f6ccb400640 1 -- 172.21.15.60:0/1788122351 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6ccc154d60 msgr2=0x7f6ccc1f2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:58.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.444+0000 7f6ccb400640 1 --2- 172.21.15.60:0/1788122351 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6ccc154d60 0x7f6ccc1f2890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:58.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.444+0000 7f6ccb400640 1 -- 172.21.15.60:0/1788122351 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6ccc06bf70 msgr2=0x7f6ccc1580b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:58.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.444+0000 7f6ccb400640 1 --2- 172.21.15.60:0/1788122351 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6ccc06bf70 0x7f6ccc1580b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:58.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.444+0000 7f6ccb400640 1 -- 172.21.15.60:0/1788122351 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ccc1f4700 con 0x7f6ccc0cd800 2024-04-14T23:48:58.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.445+0000 7f6ccb400640 1 --2- 172.21.15.60:0/1788122351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ccc0cd800 0x7f6ccc1f3e40 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6cb800e9b0 tx=0x7f6cb800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:58.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.445+0000 7f6cc3e00640 1 -- 172.21.15.60:0/1788122351 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cb8017070 con 0x7f6ccc0cd800 2024-04-14T23:48:58.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.445+0000 7f6cc3e00640 1 -- 172.21.15.60:0/1788122351 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cb800c9a0 con 0x7f6ccc0cd800 2024-04-14T23:48:58.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.445+0000 7f6cc3e00640 1 -- 172.21.15.60:0/1788122351 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cb800cd90 con 0x7f6ccc0cd800 2024-04-14T23:48:58.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.445+0000 7f6cd1800640 1 -- 172.21.15.60:0/1788122351 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ccc1f49e0 con 0x7f6ccc0cd800 2024-04-14T23:48:58.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.445+0000 7f6cd1800640 1 -- 172.21.15.60:0/1788122351 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ccc1f1db0 con 0x7f6ccc0cd800 2024-04-14T23:48:58.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.446+0000 7f6cd1800640 1 -- 172.21.15.60:0/1788122351 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ccc04f540 con 0x7f6ccc0cd800 2024-04-14T23:48:58.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.447+0000 7f6cc3e00640 1 -- 172.21.15.60:0/1788122351 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f6cb801a070 con 0x7f6ccc0cd800 2024-04-14T23:48:58.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.449+0000 7f6cc3e00640 1 --2- 172.21.15.60:0/1788122351 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f6cac06b990 0x7f6cac06de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:58.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.449+0000 7f6cc3e00640 1 -- 172.21.15.60:0/1788122351 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f6cb808abe0 con 0x7f6ccc0cd800 2024-04-14T23:48:58.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.449+0000 7f6ccbe00640 1 --2- 172.21.15.60:0/1788122351 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f6cac06b990 0x7f6cac06de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:58.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.450+0000 7f6ccbe00640 1 --2- 172.21.15.60:0/1788122351 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f6cac06b990 0x7f6cac06de50 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc036000 tx=0x7f6cbc03b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:58.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.453+0000 7f6cc3e00640 1 -- 172.21.15.60:0/1788122351 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6cb8059100 con 0x7f6ccc0cd800 2024-04-14T23:48:58.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.652+0000 7f6cd1800640 1 -- 172.21.15.60:0/1788122351 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6ccc0ca660 con 0x7f6ccc0cd800 2024-04-14T23:48:58.658 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.653+0000 7f6cc3e00640 1 -- 172.21.15.60:0/1788122351 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7f6cb805d1a0 con 0x7f6ccc0cd800 2024-04-14T23:48:58.658 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2024-04-14T23:48:58.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.656+0000 7f6cd1800640 1 -- 172.21.15.60:0/1788122351 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f6cac06b990 msgr2=0x7f6cac06de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:58.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.656+0000 7f6cd1800640 1 --2- 172.21.15.60:0/1788122351 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f6cac06b990 0x7f6cac06de50 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc036000 tx=0x7f6cbc03b040 comp rx=0 tx=0).stop 2024-04-14T23:48:58.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.656+0000 7f6cd1800640 1 -- 172.21.15.60:0/1788122351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ccc0cd800 msgr2=0x7f6ccc1f3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:58.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.656+0000 7f6cd1800640 1 --2- 172.21.15.60:0/1788122351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ccc0cd800 0x7f6ccc1f3e40 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6cb800e9b0 tx=0x7f6cb800ee70 comp rx=0 tx=0).stop 2024-04-14T23:48:58.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.656+0000 7f6cd1800640 1 -- 172.21.15.60:0/1788122351 shutdown_connections 2024-04-14T23:48:58.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.656+0000 7f6cd1800640 1 --2- 172.21.15.60:0/1788122351 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f6cac06b990 0x7f6cac06de50 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:58.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.656+0000 7f6cd1800640 1 --2- 172.21.15.60:0/1788122351 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6ccc154d60 0x7f6ccc1f2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:58.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.656+0000 7f6cd1800640 1 --2- 172.21.15.60:0/1788122351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ccc0cd800 0x7f6ccc1f3e40 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:58.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.656+0000 7f6cd1800640 1 --2- 172.21.15.60:0/1788122351 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6ccc06bf70 0x7f6ccc1580b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:58.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.656+0000 7f6cd1800640 1 -- 172.21.15.60:0/1788122351 >> 172.21.15.60:0/1788122351 conn(0x7f6ccc072f20 msgr2=0x7f6ccc159010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:58.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.656+0000 7f6cd1800640 1 -- 172.21.15.60:0/1788122351 shutdown_connections 2024-04-14T23:48:58.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.656+0000 7f6cd1800640 1 -- 172.21.15.60:0/1788122351 wait complete. 2024-04-14T23:48:58.679 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-14T23:48:58.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.895+0000 7f463fe00640 1 Processor -- start 2024-04-14T23:48:58.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.896+0000 7f463fe00640 1 -- start start 2024-04-14T23:48:58.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.898+0000 7f463fe00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f46400cd110 0x7f46400cd520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:58.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.898+0000 7f463fe00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4640154fa0 con 0x7f46400cc7d0 2024-04-14T23:48:58.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.898+0000 7f463fe00640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46400b4cb0 con 0x7f4640154bc0 2024-04-14T23:48:58.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.898+0000 7f463fe00640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f464006c270 con 0x7f46400cd110 2024-04-14T23:48:58.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.898+0000 7f4636000640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f46400cc7d0 0x7f46400b45a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53376/0 (socket says 172.21.15.60:53376) 2024-04-14T23:48:58.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.898+0000 7f4636000640 1 -- 172.21.15.60:0/3919612481 learned_addr learned my addr 172.21.15.60:0/3919612481 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:48:58.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.898+0000 7f463f400640 1 --2- 172.21.15.60:0/3919612481 >> v2:172.21.15.186:3300/0 conn(0x7f46400cd110 0x7f46400cd520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:58.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.898+0000 7f463ea00640 1 --1- 172.21.15.60:0/3919612481 >> v1:172.21.15.186:6789/0 conn(0x7f4640154bc0 0x7f464015b550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.60:34010/0 (socket says 172.21.15.60:34010) 2024-04-14T23:48:58.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.898+0000 7f463f400640 1 -- 172.21.15.60:0/3919612481 >> v1:172.21.15.186:6789/0 conn(0x7f4640154bc0 legacy=0x7f464015b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:58.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.898+0000 7f463f400640 1 -- 172.21.15.60:0/3919612481 >> v1:172.21.15.60:6789/0 conn(0x7f46400cc7d0 legacy=0x7f46400b45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:58.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.898+0000 7f463f400640 1 -- 172.21.15.60:0/3919612481 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46400cb6a0 con 0x7f46400cd110 2024-04-14T23:48:58.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.899+0000 7f463f400640 1 --2- 172.21.15.60:0/3919612481 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46400cd110 0x7f46400cd520 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f4638009d90 tx=0x7f463802bfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=95da78c91cf82a5b server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:58.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.900+0000 7f463e000640 1 -- 172.21.15.60:0/3919612481 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4638039070 con 0x7f46400cd110 2024-04-14T23:48:58.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.900+0000 7f463e000640 1 -- 172.21.15.60:0/3919612481 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f463802cd80 con 0x7f46400cd110 2024-04-14T23:48:58.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.900+0000 7f463e000640 1 -- 172.21.15.60:0/3919612481 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4638034660 con 0x7f46400cd110 2024-04-14T23:48:58.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.901+0000 7f463fe00640 1 -- 172.21.15.60:0/3919612481 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46400cd110 msgr2=0x7f46400cd520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:58.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.901+0000 7f463fe00640 1 --2- 172.21.15.60:0/3919612481 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46400cd110 0x7f46400cd520 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f4638009d90 tx=0x7f463802bfd0 comp rx=0 tx=0).stop 2024-04-14T23:48:58.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.901+0000 7f463fe00640 1 -- 172.21.15.60:0/3919612481 shutdown_connections 2024-04-14T23:48:58.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.901+0000 7f463fe00640 1 --2- 172.21.15.60:0/3919612481 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46400cd110 0x7f46400cd520 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:58.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.901+0000 7f463fe00640 1 -- 172.21.15.60:0/3919612481 >> 172.21.15.60:0/3919612481 conn(0x7f46400670f0 msgr2=0x7f4640067500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:58.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.901+0000 7f463fe00640 1 -- 172.21.15.60:0/3919612481 shutdown_connections 2024-04-14T23:48:58.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.901+0000 7f463fe00640 1 -- 172.21.15.60:0/3919612481 wait complete. 2024-04-14T23:48:58.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.903+0000 7f463fe00640 1 Processor -- start 2024-04-14T23:48:58.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.903+0000 7f463fe00640 1 -- start start 2024-04-14T23:48:58.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.905+0000 7f463fe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46400cc7d0 0x7f464014b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:58.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.905+0000 7f463fe00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f46400cd110 0x7f46401f12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:58.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.905+0000 7f463fe00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4640154bc0 0x7f46401e7850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:58.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.905+0000 7f463fe00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f464006bf00 con 0x7f46400cd110 2024-04-14T23:48:58.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.905+0000 7f463fe00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f464006bc20 con 0x7f46400cc7d0 2024-04-14T23:48:58.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.905+0000 7f463fe00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f464006c700 con 0x7f4640154bc0 2024-04-14T23:48:58.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.905+0000 7f463ea00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46400cc7d0 0x7f464014b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:58.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.905+0000 7f4636000640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f46400cd110 0x7f46401f12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:58.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.905+0000 7f463f400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4640154bc0 0x7f46401e7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:58.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.905+0000 7f463f400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4640154bc0 0x7f46401e7850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.60:51232/0 (socket says 172.21.15.60:51232) 2024-04-14T23:48:58.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.905+0000 7f463f400640 1 -- 172.21.15.60:0/3357069423 learned_addr learned my addr 172.21.15.60:0/3357069423 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:58.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.905+0000 7f463f400640 1 -- 172.21.15.60:0/3357069423 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46400cc7d0 msgr2=0x7f464014b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:58.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.905+0000 7f463f400640 1 --2- 172.21.15.60:0/3357069423 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46400cc7d0 0x7f464014b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:58.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.905+0000 7f463f400640 1 -- 172.21.15.60:0/3357069423 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f46400cd110 msgr2=0x7f46401f12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:58.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.905+0000 7f463f400640 1 --2- 172.21.15.60:0/3357069423 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f46400cd110 0x7f46401f12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:58.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.905+0000 7f463f400640 1 -- 172.21.15.60:0/3357069423 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46401f3f30 con 0x7f4640154bc0 2024-04-14T23:48:58.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.906+0000 7f463f400640 1 --2- 172.21.15.60:0/3357069423 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4640154bc0 0x7f46401e7850 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4638009eb0 tx=0x7f4638004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:58.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.906+0000 7f4637400640 1 -- 172.21.15.60:0/3357069423 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4638039050 con 0x7f4640154bc0 2024-04-14T23:48:58.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.906+0000 7f4637400640 1 -- 172.21.15.60:0/3357069423 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4638045070 con 0x7f4640154bc0 2024-04-14T23:48:58.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.906+0000 7f4637400640 1 -- 172.21.15.60:0/3357069423 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f463803e830 con 0x7f4640154bc0 2024-04-14T23:48:58.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.906+0000 7f463fe00640 1 -- 172.21.15.60:0/3357069423 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46401f4210 con 0x7f4640154bc0 2024-04-14T23:48:58.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.906+0000 7f463fe00640 1 -- 172.21.15.60:0/3357069423 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f46401f47d0 con 0x7f4640154bc0 2024-04-14T23:48:58.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.907+0000 7f463fe00640 1 -- 172.21.15.60:0/3357069423 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46400685b0 con 0x7f4640154bc0 2024-04-14T23:48:58.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.908+0000 7f4637400640 1 -- 172.21.15.60:0/3357069423 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f463803e9d0 con 0x7f4640154bc0 2024-04-14T23:48:58.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.909+0000 7f4637400640 1 --2- 172.21.15.60:0/3357069423 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f461c06b990 0x7f461c06de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:58.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.909+0000 7f4637400640 1 -- 172.21.15.60:0/3357069423 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f46380ad650 con 0x7f4640154bc0 2024-04-14T23:48:58.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.909+0000 7f463ea00640 1 --2- 172.21.15.60:0/3357069423 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f461c06b990 0x7f461c06de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:58.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.910+0000 7f463ea00640 1 --2- 172.21.15.60:0/3357069423 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f461c06b990 0x7f461c06de50 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4630000f60 tx=0x7f4630000ed0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:58.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:58.914+0000 7f4637400640 1 -- 172.21.15.60:0/3357069423 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f46400685b0 con 0x7f4640154bc0 2024-04-14T23:48:59.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.117+0000 7f463fe00640 1 -- 172.21.15.60:0/3357069423 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f464004f540 con 0x7f4640154bc0 2024-04-14T23:48:59.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.118+0000 7f4637400640 1 -- 172.21.15.60:0/3357069423 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7f463807bc90 con 0x7f4640154bc0 2024-04-14T23:48:59.124 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2024-04-14T23:48:59.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.120+0000 7f463fe00640 1 -- 172.21.15.60:0/3357069423 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f461c06b990 msgr2=0x7f461c06de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:59.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.120+0000 7f463fe00640 1 --2- 172.21.15.60:0/3357069423 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f461c06b990 0x7f461c06de50 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4630000f60 tx=0x7f4630000ed0 comp rx=0 tx=0).stop 2024-04-14T23:48:59.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.120+0000 7f463fe00640 1 -- 172.21.15.60:0/3357069423 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4640154bc0 msgr2=0x7f46401e7850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:59.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.120+0000 7f463fe00640 1 --2- 172.21.15.60:0/3357069423 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4640154bc0 0x7f46401e7850 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4638009eb0 tx=0x7f4638004a40 comp rx=0 tx=0).stop 2024-04-14T23:48:59.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.121+0000 7f463fe00640 1 -- 172.21.15.60:0/3357069423 shutdown_connections 2024-04-14T23:48:59.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.121+0000 7f463fe00640 1 --2- 172.21.15.60:0/3357069423 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f461c06b990 0x7f461c06de50 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:59.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.121+0000 7f463fe00640 1 --2- 172.21.15.60:0/3357069423 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4640154bc0 0x7f46401e7850 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:59.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.121+0000 7f463fe00640 1 --2- 172.21.15.60:0/3357069423 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f46400cd110 0x7f46401f12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:59.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.121+0000 7f463fe00640 1 --2- 172.21.15.60:0/3357069423 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46400cc7d0 0x7f464014b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:59.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.121+0000 7f463fe00640 1 -- 172.21.15.60:0/3357069423 >> 172.21.15.60:0/3357069423 conn(0x7f46400670f0 msgr2=0x7f46400c5940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:59.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.121+0000 7f463fe00640 1 -- 172.21.15.60:0/3357069423 shutdown_connections 2024-04-14T23:48:59.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.121+0000 7f463fe00640 1 -- 172.21.15.60:0/3357069423 wait complete. 2024-04-14T23:48:59.147 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-14T23:48:59.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.373+0000 7f84ad400640 1 Processor -- start 2024-04-14T23:48:59.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.373+0000 7f84ad400640 1 -- start start 2024-04-14T23:48:59.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.375+0000 7f84ad400640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f84a8158610 0x7f84a80b1740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:59.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.375+0000 7f84ad400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84a80c3400 0x7f84a806eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:59.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.375+0000 7f84ad400640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84a80b1fd0 con 0x7f84a806f0a0 2024-04-14T23:48:59.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.375+0000 7f84ad400640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f84a815b070 con 0x7f84a8158610 2024-04-14T23:48:59.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.375+0000 7f84ad400640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f84a815b350 con 0x7f84a80c3400 2024-04-14T23:48:59.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.375+0000 7f84aca00640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f84a806f0a0 0x7f84a80c6800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53386/0 (socket says 172.21.15.60:53386) 2024-04-14T23:48:59.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.375+0000 7f84aca00640 1 -- 172.21.15.60:0/2641785852 learned_addr learned my addr 172.21.15.60:0/2641785852 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:48:59.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.375+0000 7f84a7400640 1 --2- 172.21.15.60:0/2641785852 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84a80c3400 0x7f84a806eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:59.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.375+0000 7f84a7e00640 1 --2- 172.21.15.60:0/2641785852 >> v2:172.21.15.186:3300/0 conn(0x7f84a8158610 0x7f84a80b1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:59.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.376+0000 7f84a7e00640 1 -- 172.21.15.60:0/2641785852 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84a80c3400 msgr2=0x7f84a806eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:48:59.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.376+0000 7f84a7e00640 1 --2- 172.21.15.60:0/2641785852 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84a80c3400 0x7f84a806eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:59.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.376+0000 7f84a7e00640 1 -- 172.21.15.60:0/2641785852 >> v1:172.21.15.60:6789/0 conn(0x7f84a806f0a0 legacy=0x7f84a80c6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:59.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.376+0000 7f84a6a00640 1 -- 172.21.15.60:0/2641785852 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4065764162 0 0) 0x7f84a80b1fd0 con 0x7f84a806f0a0 2024-04-14T23:48:59.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.376+0000 7f84a7e00640 1 -- 172.21.15.60:0/2641785852 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84a815b630 con 0x7f84a8158610 2024-04-14T23:48:59.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.376+0000 7f84a7400640 1 --2- 172.21.15.60:0/2641785852 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84a80c3400 0x7f84a806eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:48:59.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.377+0000 7f84a7e00640 1 --2- 172.21.15.60:0/2641785852 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84a8158610 0x7f84a80b1740 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f84980097a0 tx=0x7f849802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d0c77cd66ea8201 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:59.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.377+0000 7f84a6a00640 1 -- 172.21.15.60:0/2641785852 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8498034070 con 0x7f84a8158610 2024-04-14T23:48:59.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.377+0000 7f84a6a00640 1 -- 172.21.15.60:0/2641785852 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8498030610 con 0x7f84a8158610 2024-04-14T23:48:59.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.377+0000 7f84a6a00640 1 -- 172.21.15.60:0/2641785852 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84980309e0 con 0x7f84a8158610 2024-04-14T23:48:59.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.378+0000 7f84ad400640 1 -- 172.21.15.60:0/2641785852 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84a8158610 msgr2=0x7f84a80b1740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:59.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.378+0000 7f84ad400640 1 --2- 172.21.15.60:0/2641785852 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84a8158610 0x7f84a80b1740 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f84980097a0 tx=0x7f849802b960 comp rx=0 tx=0).stop 2024-04-14T23:48:59.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.378+0000 7f84ad400640 1 -- 172.21.15.60:0/2641785852 shutdown_connections 2024-04-14T23:48:59.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.378+0000 7f84ad400640 1 --2- 172.21.15.60:0/2641785852 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84a80c3400 0x7f84a806eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:59.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.378+0000 7f84ad400640 1 --2- 172.21.15.60:0/2641785852 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84a8158610 0x7f84a80b1740 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:59.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.378+0000 7f84ad400640 1 -- 172.21.15.60:0/2641785852 >> 172.21.15.60:0/2641785852 conn(0x7f84a80cc2f0 msgr2=0x7f84a80cc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:59.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.378+0000 7f84ad400640 1 -- 172.21.15.60:0/2641785852 shutdown_connections 2024-04-14T23:48:59.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.378+0000 7f84ad400640 1 -- 172.21.15.60:0/2641785852 wait complete. 2024-04-14T23:48:59.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.379+0000 7f84ad400640 1 Processor -- start 2024-04-14T23:48:59.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.380+0000 7f84ad400640 1 -- start start 2024-04-14T23:48:59.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.380+0000 7f84ad400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f84a806f0a0 0x7f84a814ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:59.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.380+0000 7f84ad400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84a80c3400 0x7f84a81542b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:59.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.380+0000 7f84ad400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84a8158610 0x7f84a8150be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:59.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.380+0000 7f84ad400640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f84a80b1c80 con 0x7f84a806f0a0 2024-04-14T23:48:59.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.380+0000 7f84ad400640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f84a815bbf0 con 0x7f84a80c3400 2024-04-14T23:48:59.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.380+0000 7f84ad400640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f84a8071730 con 0x7f84a8158610 2024-04-14T23:48:59.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.381+0000 7f84a7e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f84a806f0a0 0x7f84a814ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:59.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.381+0000 7f84a7e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f84a806f0a0 0x7f84a814ac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41976/0 (socket says 172.21.15.60:41976) 2024-04-14T23:48:59.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.381+0000 7f84a7e00640 1 -- 172.21.15.60:0/3396252076 learned_addr learned my addr 172.21.15.60:0/3396252076 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:59.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.381+0000 7f84a7400640 1 --2- 172.21.15.60:0/3396252076 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84a80c3400 0x7f84a81542b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:59.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.381+0000 7f84aca00640 1 --2- 172.21.15.60:0/3396252076 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84a8158610 0x7f84a8150be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:59.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.381+0000 7f84a7e00640 1 -- 172.21.15.60:0/3396252076 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84a8158610 msgr2=0x7f84a8150be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:59.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.381+0000 7f84a7e00640 1 --2- 172.21.15.60:0/3396252076 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84a8158610 0x7f84a8150be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:59.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.381+0000 7f84a7e00640 1 -- 172.21.15.60:0/3396252076 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84a80c3400 msgr2=0x7f84a81542b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:59.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.381+0000 7f84a7e00640 1 --2- 172.21.15.60:0/3396252076 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84a80c3400 0x7f84a81542b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:59.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.381+0000 7f84a7e00640 1 -- 172.21.15.60:0/3396252076 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84a81f0eb0 con 0x7f84a806f0a0 2024-04-14T23:48:59.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.382+0000 7f84a7e00640 1 --2- 172.21.15.60:0/3396252076 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f84a806f0a0 0x7f84a814ac50 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8498000c00 tx=0x7f84980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:59.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.382+0000 7f849fe00640 1 -- 172.21.15.60:0/3396252076 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8498047070 con 0x7f84a806f0a0 2024-04-14T23:48:59.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.382+0000 7f849fe00640 1 -- 172.21.15.60:0/3396252076 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8498030410 con 0x7f84a806f0a0 2024-04-14T23:48:59.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.382+0000 7f849fe00640 1 -- 172.21.15.60:0/3396252076 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8498034040 con 0x7f84a806f0a0 2024-04-14T23:48:59.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.382+0000 7f84ad400640 1 -- 172.21.15.60:0/3396252076 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84a81f1120 con 0x7f84a806f0a0 2024-04-14T23:48:59.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.382+0000 7f84ad400640 1 -- 172.21.15.60:0/3396252076 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84a8150320 con 0x7f84a806f0a0 2024-04-14T23:48:59.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.383+0000 7f84ad400640 1 -- 172.21.15.60:0/3396252076 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84a804f540 con 0x7f84a806f0a0 2024-04-14T23:48:59.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.384+0000 7f849fe00640 1 -- 172.21.15.60:0/3396252076 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f849804a070 con 0x7f84a806f0a0 2024-04-14T23:48:59.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.385+0000 7f849fe00640 1 --2- 172.21.15.60:0/3396252076 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f848806b990 0x7f848806de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:59.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.385+0000 7f849fe00640 1 -- 172.21.15.60:0/3396252076 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f84980abfa0 con 0x7f84a806f0a0 2024-04-14T23:48:59.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.385+0000 7f84a7400640 1 --2- 172.21.15.60:0/3396252076 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f848806b990 0x7f848806de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:59.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.386+0000 7f84a7400640 1 --2- 172.21.15.60:0/3396252076 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f848806b990 0x7f848806de50 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8494009560 tx=0x7f8494009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:59.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.390+0000 7f849fe00640 1 -- 172.21.15.60:0/3396252076 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f849807a5f0 con 0x7f84a806f0a0 2024-04-14T23:48:59.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.589+0000 7f84ad400640 1 -- 172.21.15.60:0/3396252076 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f84a8070d10 con 0x7f84a806f0a0 2024-04-14T23:48:59.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.590+0000 7f849fe00640 1 -- 172.21.15.60:0/3396252076 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7f849807e690 con 0x7f84a806f0a0 2024-04-14T23:48:59.595 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2024-04-14T23:48:59.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.592+0000 7f84ad400640 1 -- 172.21.15.60:0/3396252076 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f848806b990 msgr2=0x7f848806de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:59.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.592+0000 7f84ad400640 1 --2- 172.21.15.60:0/3396252076 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f848806b990 0x7f848806de50 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8494009560 tx=0x7f8494009190 comp rx=0 tx=0).stop 2024-04-14T23:48:59.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.592+0000 7f84ad400640 1 -- 172.21.15.60:0/3396252076 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f84a806f0a0 msgr2=0x7f84a814ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:59.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.592+0000 7f84ad400640 1 --2- 172.21.15.60:0/3396252076 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f84a806f0a0 0x7f84a814ac50 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8498000c00 tx=0x7f84980046c0 comp rx=0 tx=0).stop 2024-04-14T23:48:59.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.592+0000 7f84ad400640 1 -- 172.21.15.60:0/3396252076 shutdown_connections 2024-04-14T23:48:59.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.592+0000 7f84ad400640 1 --2- 172.21.15.60:0/3396252076 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f848806b990 0x7f848806de50 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:59.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.592+0000 7f84ad400640 1 --2- 172.21.15.60:0/3396252076 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84a8158610 0x7f84a8150be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:59.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.592+0000 7f84ad400640 1 --2- 172.21.15.60:0/3396252076 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84a80c3400 0x7f84a81542b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:59.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.592+0000 7f84ad400640 1 --2- 172.21.15.60:0/3396252076 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f84a806f0a0 0x7f84a814ac50 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:59.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.592+0000 7f84ad400640 1 -- 172.21.15.60:0/3396252076 >> 172.21.15.60:0/3396252076 conn(0x7f84a80cc2f0 msgr2=0x7f84a80ca2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:59.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.593+0000 7f84ad400640 1 -- 172.21.15.60:0/3396252076 shutdown_connections 2024-04-14T23:48:59.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.593+0000 7f84ad400640 1 -- 172.21.15.60:0/3396252076 wait complete. 2024-04-14T23:48:59.619 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:48:59.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.833+0000 7fb402800640 1 Processor -- start 2024-04-14T23:48:59.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.833+0000 7fb402800640 1 -- start start 2024-04-14T23:48:59.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.835+0000 7fb402800640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb3fc154cc0 0x7fb3fc15b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:59.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.835+0000 7fb402800640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3fc06c450 con 0x7fb3fc0c3b20 2024-04-14T23:48:59.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.835+0000 7fb402800640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3fc06c730 con 0x7fb3fc071290 2024-04-14T23:48:59.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.835+0000 7fb402800640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fb3fc0c9180 con 0x7fb3fc154cc0 2024-04-14T23:48:59.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.835+0000 7fb401400640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb3fc154cc0 0x7fb3fc15b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:59.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.835+0000 7fb401400640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb3fc154cc0 0x7fb3fc15b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41992/0 (socket says 172.21.15.60:41992) 2024-04-14T23:48:59.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.835+0000 7fb400a00640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fb3fc071290 0x7fb3fc0ce380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.60:34022/0 (socket says 172.21.15.60:34022) 2024-04-14T23:48:59.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.835+0000 7fb401400640 1 -- 172.21.15.60:0/3052676477 learned_addr learned my addr 172.21.15.60:0/3052676477 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:59.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.835+0000 7fb401e00640 1 --1- 172.21.15.60:0/3052676477 >> v1:172.21.15.60:6789/0 conn(0x7fb3fc0c3b20 0x7fb3fc0c3f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53388/0 (socket says 172.21.15.60:53388) 2024-04-14T23:48:59.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.836+0000 7fb3fbe00640 1 -- 172.21.15.60:0/3052676477 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1586688451 0 0) 0x7fb3fc06c730 con 0x7fb3fc071290 2024-04-14T23:48:59.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.836+0000 7fb3fbe00640 1 -- 172.21.15.60:0/3052676477 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb3e4003520 con 0x7fb3fc071290 2024-04-14T23:48:59.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.836+0000 7fb401400640 1 -- 172.21.15.60:0/3052676477 >> v1:172.21.15.186:6789/0 conn(0x7fb3fc071290 legacy=0x7fb3fc0ce380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:59.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.836+0000 7fb3fbe00640 1 -- 172.21.15.60:0/3052676477 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3384128835 0 0) 0x7fb3fc06c450 con 0x7fb3fc0c3b20 2024-04-14T23:48:59.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.836+0000 7fb401400640 1 -- 172.21.15.60:0/3052676477 >> v1:172.21.15.60:6789/0 conn(0x7fb3fc0c3b20 legacy=0x7fb3fc0c3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:59.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.836+0000 7fb401400640 1 -- 172.21.15.60:0/3052676477 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3fc06ca10 con 0x7fb3fc154cc0 2024-04-14T23:48:59.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.838+0000 7fb401400640 1 --2- 172.21.15.60:0/3052676477 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3fc154cc0 0x7fb3fc15b5d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb3f0009870 tx=0x7fb3f002bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84135559124589bf server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:59.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.838+0000 7fb3fbe00640 1 -- 172.21.15.60:0/3052676477 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f002d870 con 0x7fb3fc154cc0 2024-04-14T23:48:59.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.838+0000 7fb3fbe00640 1 -- 172.21.15.60:0/3052676477 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3f002da10 con 0x7fb3fc154cc0 2024-04-14T23:48:59.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.838+0000 7fb3fbe00640 1 -- 172.21.15.60:0/3052676477 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f002dde0 con 0x7fb3fc154cc0 2024-04-14T23:48:59.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.839+0000 7fb402800640 1 -- 172.21.15.60:0/3052676477 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3fc154cc0 msgr2=0x7fb3fc15b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:59.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.839+0000 7fb402800640 1 --2- 172.21.15.60:0/3052676477 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3fc154cc0 0x7fb3fc15b5d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb3f0009870 tx=0x7fb3f002bab0 comp rx=0 tx=0).stop 2024-04-14T23:48:59.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.839+0000 7fb402800640 1 -- 172.21.15.60:0/3052676477 shutdown_connections 2024-04-14T23:48:59.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.839+0000 7fb402800640 1 --2- 172.21.15.60:0/3052676477 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3fc154cc0 0x7fb3fc15b5d0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:59.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.839+0000 7fb402800640 1 -- 172.21.15.60:0/3052676477 >> 172.21.15.60:0/3052676477 conn(0x7fb3fc066f10 msgr2=0x7fb3fc067320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:48:59.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.839+0000 7fb402800640 1 -- 172.21.15.60:0/3052676477 shutdown_connections 2024-04-14T23:48:59.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.839+0000 7fb402800640 1 -- 172.21.15.60:0/3052676477 wait complete. 2024-04-14T23:48:59.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.841+0000 7fb402800640 1 Processor -- start 2024-04-14T23:48:59.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.841+0000 7fb402800640 1 -- start start 2024-04-14T23:48:59.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.842+0000 7fb402800640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb3fc071290 0x7fb3fc0cd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:59.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.842+0000 7fb402800640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3fc0c3b20 0x7fb3fc15d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:59.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.842+0000 7fb402800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb3fc154cc0 0x7fb3fc15d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:59.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.842+0000 7fb402800640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb3fc06cfd0 con 0x7fb3fc0c3b20 2024-04-14T23:48:59.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.842+0000 7fb402800640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb3fc15bb10 con 0x7fb3fc154cc0 2024-04-14T23:48:59.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.842+0000 7fb402800640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fb3fc06d2b0 con 0x7fb3fc071290 2024-04-14T23:48:59.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.842+0000 7fb400a00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3fc0c3b20 0x7fb3fc15d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:59.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.842+0000 7fb400a00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3fc0c3b20 0x7fb3fc15d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41998/0 (socket says 172.21.15.60:41998) 2024-04-14T23:48:59.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.842+0000 7fb400a00640 1 -- 172.21.15.60:0/917301949 learned_addr learned my addr 172.21.15.60:0/917301949 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:48:59.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.842+0000 7fb401400640 1 --2- 172.21.15.60:0/917301949 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb3fc071290 0x7fb3fc0cd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:59.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.842+0000 7fb401e00640 1 --2- 172.21.15.60:0/917301949 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb3fc154cc0 0x7fb3fc15d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:59.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.842+0000 7fb400a00640 1 -- 172.21.15.60:0/917301949 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb3fc071290 msgr2=0x7fb3fc0cd4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:59.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.842+0000 7fb400a00640 1 --2- 172.21.15.60:0/917301949 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb3fc071290 0x7fb3fc0cd4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:59.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.842+0000 7fb400a00640 1 -- 172.21.15.60:0/917301949 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb3fc154cc0 msgr2=0x7fb3fc15d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:48:59.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.842+0000 7fb400a00640 1 --2- 172.21.15.60:0/917301949 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb3fc154cc0 0x7fb3fc15d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:48:59.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.842+0000 7fb400a00640 1 -- 172.21.15.60:0/917301949 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3fc0c8cc0 con 0x7fb3fc0c3b20 2024-04-14T23:48:59.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.843+0000 7fb400a00640 1 --2- 172.21.15.60:0/917301949 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3fc0c3b20 0x7fb3fc15d3b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec00c870 tx=0x7fb3ec00cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:59.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.843+0000 7fb3f9600640 1 -- 172.21.15.60:0/917301949 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3ec017070 con 0x7fb3fc0c3b20 2024-04-14T23:48:59.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.843+0000 7fb3f9600640 1 -- 172.21.15.60:0/917301949 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3ec00a640 con 0x7fb3fc0c3b20 2024-04-14T23:48:59.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.843+0000 7fb402800640 1 -- 172.21.15.60:0/917301949 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3fc15bd90 con 0x7fb3fc0c3b20 2024-04-14T23:48:59.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.843+0000 7fb3f9600640 1 -- 172.21.15.60:0/917301949 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3ec00aa10 con 0x7fb3fc0c3b20 2024-04-14T23:48:59.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.843+0000 7fb402800640 1 -- 172.21.15.60:0/917301949 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3fc15c350 con 0x7fb3fc0c3b20 2024-04-14T23:48:59.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.844+0000 7fb402800640 1 -- 172.21.15.60:0/917301949 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3fc04f540 con 0x7fb3fc0c3b20 2024-04-14T23:48:59.851 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.845+0000 7fb3f9600640 1 -- 172.21.15.60:0/917301949 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7fb3ec01a070 con 0x7fb3fc0c3b20 2024-04-14T23:48:59.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.846+0000 7fb3f9600640 1 --2- 172.21.15.60:0/917301949 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fb3e006b9c0 0x7fb3e006de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:48:59.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.847+0000 7fb3f9600640 1 -- 172.21.15.60:0/917301949 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb3ec08a040 con 0x7fb3fc0c3b20 2024-04-14T23:48:59.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.847+0000 7fb401400640 1 --2- 172.21.15.60:0/917301949 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fb3e006b9c0 0x7fb3e006de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:48:59.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.848+0000 7fb401400640 1 --2- 172.21.15.60:0/917301949 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fb3e006b9c0 0x7fb3e006de80 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb3f0036000 tx=0x7fb3f0008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:48:59.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:48:59.853+0000 7fb3f9600640 1 -- 172.21.15.60:0/917301949 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb3ec0584d0 con 0x7fb3fc0c3b20 2024-04-14T23:49:00.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.094+0000 7fb402800640 1 -- 172.21.15.60:0/917301949 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb3fc06bfe0 con 0x7fb3fc0c3b20 2024-04-14T23:49:00.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.096+0000 7fb3f9600640 1 -- 172.21.15.60:0/917301949 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6827 (secure 0 0 0) 0x7fb3ec05c570 con 0x7fb3fc0c3b20 2024-04-14T23:49:00.101 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:49:00.101 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/78303888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":78303888},{"type":"v1","addr":"172.21.15.60:6833","nonce":78303888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:48:49.256965+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.60:6835/2677133122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2677133122},{"type":"v1","addr":"172.21.15.60:6835","nonce":2677133122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4279,"qdb_cluster":[4279]},"id":1}]} 2024-04-14T23:49:00.102 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 8 2024-04-14T23:49:00.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.098+0000 7fb402800640 1 -- 172.21.15.60:0/917301949 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fb3e006b9c0 msgr2=0x7fb3e006de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:00.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.098+0000 7fb402800640 1 --2- 172.21.15.60:0/917301949 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fb3e006b9c0 0x7fb3e006de80 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb3f0036000 tx=0x7fb3f0008030 comp rx=0 tx=0).stop 2024-04-14T23:49:00.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.098+0000 7fb402800640 1 -- 172.21.15.60:0/917301949 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3fc0c3b20 msgr2=0x7fb3fc15d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:00.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.098+0000 7fb402800640 1 --2- 172.21.15.60:0/917301949 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3fc0c3b20 0x7fb3fc15d3b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec00c870 tx=0x7fb3ec00cd30 comp rx=0 tx=0).stop 2024-04-14T23:49:00.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.098+0000 7fb402800640 1 -- 172.21.15.60:0/917301949 shutdown_connections 2024-04-14T23:49:00.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.098+0000 7fb402800640 1 --2- 172.21.15.60:0/917301949 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fb3e006b9c0 0x7fb3e006de80 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:00.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.098+0000 7fb402800640 1 --2- 172.21.15.60:0/917301949 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb3fc154cc0 0x7fb3fc15d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:00.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.098+0000 7fb402800640 1 --2- 172.21.15.60:0/917301949 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3fc0c3b20 0x7fb3fc15d3b0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:00.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.098+0000 7fb402800640 1 --2- 172.21.15.60:0/917301949 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb3fc071290 0x7fb3fc0cd4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:00.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.098+0000 7fb402800640 1 -- 172.21.15.60:0/917301949 >> 172.21.15.60:0/917301949 conn(0x7fb3fc066f10 msgr2=0x7fb3fc153160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:49:00.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.099+0000 7fb402800640 1 -- 172.21.15.60:0/917301949 shutdown_connections 2024-04-14T23:49:00.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.099+0000 7fb402800640 1 -- 172.21.15.60:0/917301949 wait complete. 2024-04-14T23:49:00.125 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-14T23:49:00.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.343+0000 7f0401000640 1 Processor -- start 2024-04-14T23:49:00.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.343+0000 7f0401000640 1 -- start start 2024-04-14T23:49:00.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.345+0000 7f0401000640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f03fc154cf0 0x7f03fc15a940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:00.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.345+0000 7f0401000640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f03fc06c310 0x7f03fc06c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:00.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.345+0000 7f0401000640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f03fc0cb120 0x7f03fc0cbd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:00.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.345+0000 7f0401000640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f03fc1550d0 con 0x7f03fc154cf0 2024-04-14T23:49:00.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.345+0000 7f0401000640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f03fc0c5520 con 0x7f03fc0cb120 2024-04-14T23:49:00.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.345+0000 7f0401000640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f03fc15b040 con 0x7f03fc06c310 2024-04-14T23:49:00.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.345+0000 7f03fb400640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f03fc154cf0 0x7f03fc15a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:00.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.345+0000 7f03fb400640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f03fc154cf0 0x7f03fc15a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42008/0 (socket says 172.21.15.60:42008) 2024-04-14T23:49:00.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.345+0000 7f03fb400640 1 -- 172.21.15.60:0/3276416775 learned_addr learned my addr 172.21.15.60:0/3276416775 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:49:00.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.345+0000 7f03fbe00640 1 --2- 172.21.15.60:0/3276416775 >> v2:172.21.15.186:3300/0 conn(0x7f03fc0cb120 0x7f03fc0cbd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:00.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.345+0000 7f03f3e00640 1 --2- 172.21.15.60:0/3276416775 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f03fc06c310 0x7f03fc06c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:00.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.346+0000 7f03fb400640 1 -- 172.21.15.60:0/3276416775 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f03fc06c310 msgr2=0x7f03fc06c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:49:00.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.346+0000 7f03fb400640 1 --2- 172.21.15.60:0/3276416775 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f03fc06c310 0x7f03fc06c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:00.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.346+0000 7f03fb400640 1 -- 172.21.15.60:0/3276416775 >> v2:172.21.15.186:3300/0 conn(0x7f03fc0cb120 msgr2=0x7f03fc0cbd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:49:00.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.346+0000 7f03fb400640 1 --2- 172.21.15.60:0/3276416775 >> v2:172.21.15.186:3300/0 conn(0x7f03fc0cb120 0x7f03fc0cbd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:00.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.346+0000 7f03fb400640 1 -- 172.21.15.60:0/3276416775 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03fc15b300 con 0x7f03fc154cf0 2024-04-14T23:49:00.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.346+0000 7f03fbe00640 1 --2- 172.21.15.60:0/3276416775 >> v2:172.21.15.186:3300/0 conn(0x7f03fc0cb120 0x7f03fc0cbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-14T23:49:00.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.347+0000 7f03fb400640 1 --2- 172.21.15.60:0/3276416775 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f03fc154cf0 0x7f03fc15a940 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f03fc0ce130 tx=0x7f03ec02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b41b0d9767c93a3a server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:00.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.347+0000 7f03faa00640 1 -- 172.21.15.60:0/3276416775 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03ec034070 con 0x7f03fc154cf0 2024-04-14T23:49:00.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.347+0000 7f03faa00640 1 -- 172.21.15.60:0/3276416775 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03ec030860 con 0x7f03fc154cf0 2024-04-14T23:49:00.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.347+0000 7f03faa00640 1 -- 172.21.15.60:0/3276416775 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03ec030c30 con 0x7f03fc154cf0 2024-04-14T23:49:00.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.348+0000 7f0401000640 1 -- 172.21.15.60:0/3276416775 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f03fc154cf0 msgr2=0x7f03fc15a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:00.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.348+0000 7f0401000640 1 --2- 172.21.15.60:0/3276416775 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f03fc154cf0 0x7f03fc15a940 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f03fc0ce130 tx=0x7f03ec02b9a0 comp rx=0 tx=0).stop 2024-04-14T23:49:00.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.348+0000 7f0401000640 1 -- 172.21.15.60:0/3276416775 shutdown_connections 2024-04-14T23:49:00.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.348+0000 7f0401000640 1 --2- 172.21.15.60:0/3276416775 >> v2:172.21.15.186:3300/0 conn(0x7f03fc0cb120 0x7f03fc0cbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:00.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.348+0000 7f0401000640 1 --2- 172.21.15.60:0/3276416775 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f03fc06c310 0x7f03fc06c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:00.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.348+0000 7f0401000640 1 --2- 172.21.15.60:0/3276416775 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f03fc154cf0 0x7f03fc15a940 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:00.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.348+0000 7f0401000640 1 -- 172.21.15.60:0/3276416775 >> 172.21.15.60:0/3276416775 conn(0x7f03fc0670f0 msgr2=0x7f03fc067500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:49:00.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.348+0000 7f0401000640 1 -- 172.21.15.60:0/3276416775 shutdown_connections 2024-04-14T23:49:00.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.348+0000 7f0401000640 1 -- 172.21.15.60:0/3276416775 wait complete. 2024-04-14T23:49:00.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.350+0000 7f0401000640 1 Processor -- start 2024-04-14T23:49:00.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.350+0000 7f0401000640 1 -- start start 2024-04-14T23:49:00.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.351+0000 7f0401000640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f03fc06c310 0x7f03fc153b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:00.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.351+0000 7f0401000640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f03fc0cb120 0x7f03fc1f3da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:00.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.351+0000 7f0401000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f03fc154cf0 0x7f03fc14ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:00.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.351+0000 7f0401000640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f03fc06bb10 con 0x7f03fc06c310 2024-04-14T23:49:00.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.351+0000 7f0401000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f03fc15b8c0 con 0x7f03fc154cf0 2024-04-14T23:49:00.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.351+0000 7f0401000640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f03fc06bdf0 con 0x7f03fc0cb120 2024-04-14T23:49:00.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.351+0000 7f03fb400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f03fc06c310 0x7f03fc153b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:00.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.351+0000 7f03fb400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f03fc06c310 0x7f03fc153b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42020/0 (socket says 172.21.15.60:42020) 2024-04-14T23:49:00.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.351+0000 7f03fb400640 1 -- 172.21.15.60:0/1256292659 learned_addr learned my addr 172.21.15.60:0/1256292659 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:49:00.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.351+0000 7f03fbe00640 1 --2- 172.21.15.60:0/1256292659 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f03fc154cf0 0x7f03fc14ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:00.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.351+0000 7f03f3e00640 1 --2- 172.21.15.60:0/1256292659 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f03fc0cb120 0x7f03fc1f3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:00.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.352+0000 7f03fb400640 1 -- 172.21.15.60:0/1256292659 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f03fc0cb120 msgr2=0x7f03fc1f3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:00.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.352+0000 7f03fb400640 1 --2- 172.21.15.60:0/1256292659 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f03fc0cb120 0x7f03fc1f3da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:00.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.352+0000 7f03fb400640 1 -- 172.21.15.60:0/1256292659 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f03fc154cf0 msgr2=0x7f03fc14ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:00.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.352+0000 7f03fb400640 1 --2- 172.21.15.60:0/1256292659 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f03fc154cf0 0x7f03fc14ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:00.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.352+0000 7f03fb400640 1 -- 172.21.15.60:0/1256292659 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03fc0cf610 con 0x7f03fc06c310 2024-04-14T23:49:00.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.352+0000 7f03fbe00640 1 --2- 172.21.15.60:0/1256292659 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f03fc154cf0 0x7f03fc14ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-14T23:49:00.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.352+0000 7f03fb400640 1 --2- 172.21.15.60:0/1256292659 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f03fc06c310 0x7f03fc153b60 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f03ec02b9e0 tx=0x7f03ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:00.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.352+0000 7f03f3400640 1 -- 172.21.15.60:0/1256292659 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03ec034040 con 0x7f03fc06c310 2024-04-14T23:49:00.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.352+0000 7f03f3400640 1 -- 172.21.15.60:0/1256292659 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03ec039d90 con 0x7f03fc06c310 2024-04-14T23:49:00.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.352+0000 7f0401000640 1 -- 172.21.15.60:0/1256292659 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03fc0cf8f0 con 0x7f03fc06c310 2024-04-14T23:49:00.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.352+0000 7f03f3400640 1 -- 172.21.15.60:0/1256292659 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03ec043610 con 0x7f03fc06c310 2024-04-14T23:49:00.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.353+0000 7f0401000640 1 -- 172.21.15.60:0/1256292659 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03fc1f0d20 con 0x7f03fc06c310 2024-04-14T23:49:00.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.354+0000 7f0401000640 1 -- 172.21.15.60:0/1256292659 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03fc04f540 con 0x7f03fc06c310 2024-04-14T23:49:00.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.354+0000 7f03f3400640 1 -- 172.21.15.60:0/1256292659 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f03ec037070 con 0x7f03fc06c310 2024-04-14T23:49:00.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.356+0000 7f03f3400640 1 --2- 172.21.15.60:0/1256292659 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f03d806b990 0x7f03d806de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:00.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.356+0000 7f03f3400640 1 -- 172.21.15.60:0/1256292659 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f03ec0ac9f0 con 0x7f03fc06c310 2024-04-14T23:49:00.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.356+0000 7f03f3e00640 1 --2- 172.21.15.60:0/1256292659 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f03d806b990 0x7f03d806de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:00.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.357+0000 7f03f3e00640 1 --2- 172.21.15.60:0/1256292659 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f03d806b990 0x7f03d806de50 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f03fc0c8600 tx=0x7f03e4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:00.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.360+0000 7f03f3400640 1 -- 172.21.15.60:0/1256292659 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f03ec03e070 con 0x7f03fc06c310 2024-04-14T23:49:00.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.565+0000 7f0401000640 1 -- 172.21.15.60:0/1256292659 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f03fc06c880 con 0x7f03fc06c310 2024-04-14T23:49:00.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.568+0000 7f03f3400640 1 -- 172.21.15.60:0/1256292659 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16605 (secure 0 0 0) 0x7f03ec07b030 con 0x7f03fc06c310 2024-04-14T23:49:00.573 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:49:00.574 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":21,"fsid":"5847db56-a76d-4486-b563-d83f1186af92","created":"2024-04-14T23:48:23.900003+0000","modified":"2024-04-14T23:48:44.219352+0000","last_up_change":"2024-04-14T23:48:34.203573+0000","last_in_change":"2024-04-14T23:48:26.345499+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-14T23:48:36.247372+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-14T23:48:40.274195+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-14T23:48:41.722183+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b2b2b755-7f42-4f9b-af9a-a178f7cc5a52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6817","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6818","nonce":2536528895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6822","nonce":2536528895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6827","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6828","nonce":2536528895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6825","nonce":2536528895}]},"public_addr":"172.21.15.60:6818/2536528895","cluster_addr":"172.21.15.60:6822/2536528895","heartbeat_back_addr":"172.21.15.60:6828/2536528895","heartbeat_front_addr":"172.21.15.60:6825/2536528895","state":["exists","up"]},{"osd":1,"uuid":"321418cd-5ab7-41a5-b9c8-b3fc12f68ebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6801","nonce":3115695676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6803","nonce":3115695676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6807","nonce":3115695676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6805","nonce":3115695676}]},"public_addr":"172.21.15.60:6801/3115695676","cluster_addr":"172.21.15.60:6803/3115695676","heartbeat_back_addr":"172.21.15.60:6807/3115695676","heartbeat_front_addr":"172.21.15.60:6805/3115695676","state":["exists","up"]},{"osd":2,"uuid":"07874e9f-3a38-41dc-be9f-2b75e84e53d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6819","nonce":3475168698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6824","nonce":3475168698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6831","nonce":3475168698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6829","nonce":3475168698}]},"public_addr":"172.21.15.60:6819/3475168698","cluster_addr":"172.21.15.60:6824/3475168698","heartbeat_back_addr":"172.21.15.60:6831/3475168698","heartbeat_front_addr":"172.21.15.60:6829/3475168698","state":["exists","up"]},{"osd":3,"uuid":"7e4ef6c9-b17a-47c6-a320-f5d16a6fe05e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6809","nonce":1730813674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6811","nonce":1730813674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6815","nonce":1730813674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6813","nonce":1730813674}]},"public_addr":"172.21.15.60:6809/1730813674","cluster_addr":"172.21.15.60:6811/1730813674","heartbeat_back_addr":"172.21.15.60:6815/1730813674","heartbeat_front_addr":"172.21.15.60:6813/1730813674","state":["exists","up"]},{"osd":4,"uuid":"759a3efc-dade-44c5-b57a-1623499aba50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6805","nonce":1464587038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6810","nonce":1464587038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6819","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6822","nonce":1464587038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6813","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6816","nonce":1464587038}]},"public_addr":"172.21.15.186:6805/1464587038","cluster_addr":"172.21.15.186:6810/1464587038","heartbeat_back_addr":"172.21.15.186:6822/1464587038","heartbeat_front_addr":"172.21.15.186:6816/1464587038","state":["exists","up"]},{"osd":5,"uuid":"24cd58c1-0cf1-4077-ab33-84c39c6907ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6801","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6803","nonce":1697098067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6809","nonce":1697098067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6820","nonce":1697098067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6815","nonce":1697098067}]},"public_addr":"172.21.15.186:6803/1697098067","cluster_addr":"172.21.15.186:6809/1697098067","heartbeat_back_addr":"172.21.15.186:6820/1697098067","heartbeat_front_addr":"172.21.15.186:6815/1697098067","state":["exists","up"]},{"osd":6,"uuid":"6add38f1-65e9-4b0d-927a-4ab77f5e2323","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6804","nonce":1837108559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6807","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6811","nonce":1837108559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6823","nonce":1837108559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6817","nonce":1837108559}]},"public_addr":"172.21.15.186:6804/1837108559","cluster_addr":"172.21.15.186:6811/1837108559","heartbeat_back_addr":"172.21.15.186:6823/1837108559","heartbeat_front_addr":"172.21.15.186:6817/1837108559","state":["exists","up"]},{"osd":7,"uuid":"a2326b6f-7601-4dda-a456-67c443ea5929","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6825","nonce":842537875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6827","nonce":842537875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6831","nonce":842537875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6829","nonce":842537875}]},"public_addr":"172.21.15.186:6825/842537875","cluster_addr":"172.21.15.186:6827/842537875","heartbeat_back_addr":"172.21.15.186:6831/842537875","heartbeat_front_addr":"172.21.15.186:6829/842537875","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.975583+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.976806+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.722725+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.763705+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.502140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.528972+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.517961+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.513350+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-14T23:49:00.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.570+0000 7f0401000640 1 -- 172.21.15.60:0/1256292659 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f03d806b990 msgr2=0x7f03d806de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:00.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.570+0000 7f0401000640 1 --2- 172.21.15.60:0/1256292659 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f03d806b990 0x7f03d806de50 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f03fc0c8600 tx=0x7f03e4008040 comp rx=0 tx=0).stop 2024-04-14T23:49:00.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.570+0000 7f0401000640 1 -- 172.21.15.60:0/1256292659 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f03fc06c310 msgr2=0x7f03fc153b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:00.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.570+0000 7f0401000640 1 --2- 172.21.15.60:0/1256292659 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f03fc06c310 0x7f03fc153b60 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f03ec02b9e0 tx=0x7f03ec0046c0 comp rx=0 tx=0).stop 2024-04-14T23:49:00.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.570+0000 7f0401000640 1 -- 172.21.15.60:0/1256292659 shutdown_connections 2024-04-14T23:49:00.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.570+0000 7f0401000640 1 --2- 172.21.15.60:0/1256292659 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f03d806b990 0x7f03d806de50 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:00.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.570+0000 7f0401000640 1 --2- 172.21.15.60:0/1256292659 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f03fc154cf0 0x7f03fc14ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:00.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.570+0000 7f0401000640 1 --2- 172.21.15.60:0/1256292659 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f03fc0cb120 0x7f03fc1f3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:00.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.570+0000 7f0401000640 1 --2- 172.21.15.60:0/1256292659 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f03fc06c310 0x7f03fc153b60 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:00.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.570+0000 7f0401000640 1 -- 172.21.15.60:0/1256292659 >> 172.21.15.60:0/1256292659 conn(0x7f03fc0670f0 msgr2=0x7f03fc1502b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:49:00.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.571+0000 7f0401000640 1 -- 172.21.15.60:0/1256292659 shutdown_connections 2024-04-14T23:49:00.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.571+0000 7f0401000640 1 -- 172.21.15.60:0/1256292659 wait complete. 2024-04-14T23:49:00.595 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:49:00.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.812+0000 7fa303200640 1 Processor -- start 2024-04-14T23:49:00.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.812+0000 7fa303200640 1 -- start start 2024-04-14T23:49:00.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.814+0000 7fa303200640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa2fc0c8740 0x7fa2fc06f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:00.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.814+0000 7fa303200640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa2fc0c75c0 0x7fa2fc158e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:00.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.814+0000 7fa303200640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2fc15f7c0 con 0x7fa2fc0c8360 2024-04-14T23:49:00.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.814+0000 7fa303200640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fa2fc0c6b70 con 0x7fa2fc0c75c0 2024-04-14T23:49:00.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.814+0000 7fa303200640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fa2fc15faa0 con 0x7fa2fc0c8740 2024-04-14T23:49:00.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.814+0000 7fa301200640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa2fc0c75c0 0x7fa2fc158e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:00.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.814+0000 7fa2fbe00640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fa2fc0c8360 0x7fa2fc15ef80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53390/0 (socket says 172.21.15.60:53390) 2024-04-14T23:49:00.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.814+0000 7fa2f3400640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa2fc0c8740 0x7fa2fc06f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:00.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.814+0000 7fa2fbe00640 1 -- 172.21.15.60:0/803740573 learned_addr learned my addr 172.21.15.60:0/803740573 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:49:00.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.815+0000 7fa2f3400640 1 -- 172.21.15.60:0/803740573 >> v2:172.21.15.60:3300/0 conn(0x7fa2fc0c75c0 msgr2=0x7fa2fc158e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:49:00.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.815+0000 7fa2f3400640 1 --2- 172.21.15.60:0/803740573 >> v2:172.21.15.60:3300/0 conn(0x7fa2fc0c75c0 0x7fa2fc158e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:00.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.815+0000 7fa2f3400640 1 -- 172.21.15.60:0/803740573 >> v1:172.21.15.60:6789/0 conn(0x7fa2fc0c8360 legacy=0x7fa2fc15ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:00.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.815+0000 7fa2f3400640 1 -- 172.21.15.60:0/803740573 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2fc15fd80 con 0x7fa2fc0c8740 2024-04-14T23:49:00.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.816+0000 7fa2f3400640 1 --2- 172.21.15.60:0/803740573 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2fc0c8740 0x7fa2fc06f310 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc075690 tx=0x7fa2e402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f709c9236ac9bc0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:00.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.816+0000 7fa2fb400640 1 -- 172.21.15.60:0/803740573 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2e4035020 con 0x7fa2fc0c8740 2024-04-14T23:49:00.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.816+0000 7fa2fb400640 1 -- 172.21.15.60:0/803740573 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2e4031830 con 0x7fa2fc0c8740 2024-04-14T23:49:00.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.816+0000 7fa2fb400640 1 -- 172.21.15.60:0/803740573 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2e4031c00 con 0x7fa2fc0c8740 2024-04-14T23:49:00.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.817+0000 7fa303200640 1 -- 172.21.15.60:0/803740573 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2fc0c8740 msgr2=0x7fa2fc06f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:00.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.817+0000 7fa303200640 1 --2- 172.21.15.60:0/803740573 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2fc0c8740 0x7fa2fc06f310 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc075690 tx=0x7fa2e402b8d0 comp rx=0 tx=0).stop 2024-04-14T23:49:00.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.817+0000 7fa303200640 1 -- 172.21.15.60:0/803740573 shutdown_connections 2024-04-14T23:49:00.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.817+0000 7fa303200640 1 --2- 172.21.15.60:0/803740573 >> v2:172.21.15.60:3300/0 conn(0x7fa2fc0c75c0 0x7fa2fc158e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:00.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.817+0000 7fa303200640 1 --2- 172.21.15.60:0/803740573 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2fc0c8740 0x7fa2fc06f310 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:00.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.817+0000 7fa303200640 1 -- 172.21.15.60:0/803740573 >> 172.21.15.60:0/803740573 conn(0x7fa2fc0cc170 msgr2=0x7fa2fc0cc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:49:00.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.817+0000 7fa303200640 1 -- 172.21.15.60:0/803740573 shutdown_connections 2024-04-14T23:49:00.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.817+0000 7fa303200640 1 -- 172.21.15.60:0/803740573 wait complete. 2024-04-14T23:49:00.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.819+0000 7fa303200640 1 Processor -- start 2024-04-14T23:49:00.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.819+0000 7fa303200640 1 -- start start 2024-04-14T23:49:00.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.820+0000 7fa303200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2fc0c75c0 0x7fa2fc1f5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:00.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.820+0000 7fa303200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa2fc0c8360 0x7fa2fc155ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:00.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.820+0000 7fa303200640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2fc0c8740 0x7fa2fc153b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:00.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.820+0000 7fa303200640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fa2fc071030 con 0x7fa2fc0c8740 2024-04-14T23:49:00.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.820+0000 7fa303200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa2fc0c70a0 con 0x7fa2fc0c75c0 2024-04-14T23:49:00.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.820+0000 7fa303200640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fa2fc06f850 con 0x7fa2fc0c8360 2024-04-14T23:49:00.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.820+0000 7fa301200640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2fc0c8740 0x7fa2fc153b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:00.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.820+0000 7fa301200640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2fc0c8740 0x7fa2fc153b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42028/0 (socket says 172.21.15.60:42028) 2024-04-14T23:49:00.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.820+0000 7fa301200640 1 -- 172.21.15.60:0/4009953919 learned_addr learned my addr 172.21.15.60:0/4009953919 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:49:00.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.820+0000 7fa2fbe00640 1 --2- 172.21.15.60:0/4009953919 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2fc0c75c0 0x7fa2fc1f5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:00.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.820+0000 7fa2f3400640 1 --2- 172.21.15.60:0/4009953919 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa2fc0c8360 0x7fa2fc155ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:00.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.821+0000 7fa2f3400640 1 -- 172.21.15.60:0/4009953919 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2fc0c75c0 msgr2=0x7fa2fc1f5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:00.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.821+0000 7fa2f3400640 1 --2- 172.21.15.60:0/4009953919 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2fc0c75c0 0x7fa2fc1f5470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:00.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.821+0000 7fa2f3400640 1 -- 172.21.15.60:0/4009953919 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2fc0c8740 msgr2=0x7fa2fc153b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:00.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.821+0000 7fa2f3400640 1 --2- 172.21.15.60:0/4009953919 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2fc0c8740 0x7fa2fc153b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:00.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.821+0000 7fa2f3400640 1 -- 172.21.15.60:0/4009953919 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2fc1eb7a0 con 0x7fa2fc0c8360 2024-04-14T23:49:00.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.821+0000 7fa2f3400640 1 --2- 172.21.15.60:0/4009953919 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa2fc0c8360 0x7fa2fc155ba0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa2e4033000 tx=0x7fa2e40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:00.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.822+0000 7fa2f8c00640 1 -- 172.21.15.60:0/4009953919 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2e4041070 con 0x7fa2fc0c8360 2024-04-14T23:49:00.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.822+0000 7fa2f8c00640 1 -- 172.21.15.60:0/4009953919 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2e403b860 con 0x7fa2fc0c8360 2024-04-14T23:49:00.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.822+0000 7fa2f8c00640 1 -- 172.21.15.60:0/4009953919 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2e4035040 con 0x7fa2fc0c8360 2024-04-14T23:49:00.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.822+0000 7fa303200640 1 -- 172.21.15.60:0/4009953919 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2fc1eba80 con 0x7fa2fc0c8360 2024-04-14T23:49:00.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.822+0000 7fa303200640 1 -- 172.21.15.60:0/4009953919 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2fc1e5130 con 0x7fa2fc0c8360 2024-04-14T23:49:00.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.823+0000 7fa303200640 1 -- 172.21.15.60:0/4009953919 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2fc076970 con 0x7fa2fc0c8360 2024-04-14T23:49:00.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.824+0000 7fa2f8c00640 1 -- 172.21.15.60:0/4009953919 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7fa2e404c060 con 0x7fa2fc0c8360 2024-04-14T23:49:00.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.825+0000 7fa2f8c00640 1 --2- 172.21.15.60:0/4009953919 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fa2d806b9c0 0x7fa2d806de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:00.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.825+0000 7fa2f8c00640 1 -- 172.21.15.60:0/4009953919 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fa2e40ac220 con 0x7fa2fc0c8360 2024-04-14T23:49:00.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.825+0000 7fa2fbe00640 1 --2- 172.21.15.60:0/4009953919 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fa2d806b9c0 0x7fa2d806de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:00.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.826+0000 7fa2fbe00640 1 --2- 172.21.15.60:0/4009953919 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fa2d806b9c0 0x7fa2d806de80 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec0012e0 tx=0x7fa2ec001a30 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:00.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:00.830+0000 7fa2f8c00640 1 -- 172.21.15.60:0/4009953919 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa2e4031800 con 0x7fa2fc0c8360 2024-04-14T23:49:01.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.073+0000 7fa303200640 1 -- 172.21.15.60:0/4009953919 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa2fc0c4c10 con 0x7fa2fc0c8360 2024-04-14T23:49:01.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.075+0000 7fa2f8c00640 1 -- 172.21.15.60:0/4009953919 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6827 (secure 0 0 0) 0x7fa2e407a740 con 0x7fa2fc0c8360 2024-04-14T23:49:01.081 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:49:01.081 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/78303888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":78303888},{"type":"v1","addr":"172.21.15.60:6833","nonce":78303888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:48:49.256965+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.60:6835/2677133122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2677133122},{"type":"v1","addr":"172.21.15.60:6835","nonce":2677133122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4279,"qdb_cluster":[4279]},"id":1}]} 2024-04-14T23:49:01.081 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 8 2024-04-14T23:49:01.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.078+0000 7fa303200640 1 -- 172.21.15.60:0/4009953919 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fa2d806b9c0 msgr2=0x7fa2d806de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:01.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.078+0000 7fa303200640 1 --2- 172.21.15.60:0/4009953919 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fa2d806b9c0 0x7fa2d806de80 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec0012e0 tx=0x7fa2ec001a30 comp rx=0 tx=0).stop 2024-04-14T23:49:01.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.078+0000 7fa303200640 1 -- 172.21.15.60:0/4009953919 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa2fc0c8360 msgr2=0x7fa2fc155ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:01.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.078+0000 7fa303200640 1 --2- 172.21.15.60:0/4009953919 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa2fc0c8360 0x7fa2fc155ba0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa2e4033000 tx=0x7fa2e40046c0 comp rx=0 tx=0).stop 2024-04-14T23:49:01.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.078+0000 7fa303200640 1 -- 172.21.15.60:0/4009953919 shutdown_connections 2024-04-14T23:49:01.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.078+0000 7fa303200640 1 --2- 172.21.15.60:0/4009953919 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fa2d806b9c0 0x7fa2d806de80 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:01.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.078+0000 7fa303200640 1 --2- 172.21.15.60:0/4009953919 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2fc0c8740 0x7fa2fc153b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:01.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.078+0000 7fa303200640 1 --2- 172.21.15.60:0/4009953919 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa2fc0c8360 0x7fa2fc155ba0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:01.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.078+0000 7fa303200640 1 --2- 172.21.15.60:0/4009953919 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2fc0c75c0 0x7fa2fc1f5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:01.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.078+0000 7fa303200640 1 -- 172.21.15.60:0/4009953919 >> 172.21.15.60:0/4009953919 conn(0x7fa2fc0cc170 msgr2=0x7fa2fc06df40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:49:01.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.078+0000 7fa303200640 1 -- 172.21.15.60:0/4009953919 shutdown_connections 2024-04-14T23:49:01.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.078+0000 7fa303200640 1 -- 172.21.15.60:0/4009953919 wait complete. 2024-04-14T23:49:01.708 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-14T23:48:42.700188+0000', 'modified': '2024-04-14T23:48:49.256965+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4279}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4279': {'gid': 4279, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.60:6835/2677133122', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6834', 'nonce': 2677133122}, {'type': 'v1', 'addr': '172.21.15.60:6835', 'nonce': 2677133122}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4279, 'qdb_cluster': [4279]} 2024-04-14T23:49:01.709 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-14T23:49:01.709 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-14T23:49:01.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.922+0000 7f9fd3200640 1 Processor -- start 2024-04-14T23:49:01.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.923+0000 7f9fd3200640 1 -- start start 2024-04-14T23:49:01.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.924+0000 7f9fd3200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9fd4154d40 0x7f9fd40cb470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:01.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.924+0000 7f9fd3200640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9fd4155120 0x7f9fd40cb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:01.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.924+0000 7f9fd3200640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9fd415e580 con 0x7f9fd415b830 2024-04-14T23:49:01.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.924+0000 7f9fd3200640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f9fd415f130 con 0x7f9fd4155120 2024-04-14T23:49:01.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.924+0000 7f9fd3200640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f9fd415e320 con 0x7f9fd4154d40 2024-04-14T23:49:01.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.925+0000 7f9fd1400640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9fd4155120 0x7f9fd40cb9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:01.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.925+0000 7f9fd1e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9fd4154d40 0x7f9fd40cb470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:01.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.925+0000 7f9fd1400640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9fd4155120 0x7f9fd40cb9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38392/0 (socket says 172.21.15.60:38392) 2024-04-14T23:49:01.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.925+0000 7f9fd1400640 1 -- 172.21.15.60:0/3344421759 learned_addr learned my addr 172.21.15.60:0/3344421759 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:49:01.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.925+0000 7f9fd1e00640 1 -- 172.21.15.60:0/3344421759 >> v2:172.21.15.60:3300/0 conn(0x7f9fd4155120 msgr2=0x7f9fd40cb9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:49:01.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.925+0000 7f9fd1e00640 1 --2- 172.21.15.60:0/3344421759 >> v2:172.21.15.60:3300/0 conn(0x7f9fd4155120 0x7f9fd40cb9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:01.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.925+0000 7f9fd1e00640 1 -- 172.21.15.60:0/3344421759 >> v1:172.21.15.186:6789/0 conn(0x7f9fd415b830 legacy=0x7f9fd40d02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:01.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.925+0000 7f9fd0a00640 1 -- 172.21.15.60:0/3344421759 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 747850000 0 0) 0x7f9fd415e580 con 0x7f9fd415b830 2024-04-14T23:49:01.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.925+0000 7f9fd1e00640 1 -- 172.21.15.60:0/3344421759 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fd415e820 con 0x7f9fd4154d40 2024-04-14T23:49:01.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.927+0000 7f9fd1e00640 1 --2- 172.21.15.60:0/3344421759 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9fd4154d40 0x7f9fd40cb470 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9fc40097a0 tx=0x7f9fc402b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=727b511a32277d30 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:01.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.927+0000 7f9fd0a00640 1 -- 172.21.15.60:0/3344421759 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fc4034070 con 0x7f9fd4154d40 2024-04-14T23:49:01.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.927+0000 7f9fd0a00640 1 -- 172.21.15.60:0/3344421759 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fc4030800 con 0x7f9fd4154d40 2024-04-14T23:49:01.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.927+0000 7f9fd0a00640 1 -- 172.21.15.60:0/3344421759 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fc4030bd0 con 0x7f9fd4154d40 2024-04-14T23:49:01.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.928+0000 7f9fd3200640 1 -- 172.21.15.60:0/3344421759 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9fd4154d40 msgr2=0x7f9fd40cb470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:01.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.928+0000 7f9fd3200640 1 --2- 172.21.15.60:0/3344421759 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9fd4154d40 0x7f9fd40cb470 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9fc40097a0 tx=0x7f9fc402b940 comp rx=0 tx=0).stop 2024-04-14T23:49:01.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.928+0000 7f9fd3200640 1 -- 172.21.15.60:0/3344421759 shutdown_connections 2024-04-14T23:49:01.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.928+0000 7f9fd3200640 1 --2- 172.21.15.60:0/3344421759 >> v2:172.21.15.60:3300/0 conn(0x7f9fd4155120 0x7f9fd40cb9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:01.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.928+0000 7f9fd3200640 1 --2- 172.21.15.60:0/3344421759 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9fd4154d40 0x7f9fd40cb470 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:01.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.928+0000 7f9fd3200640 1 -- 172.21.15.60:0/3344421759 >> 172.21.15.60:0/3344421759 conn(0x7f9fd4066f10 msgr2=0x7f9fd4067320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:49:01.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.928+0000 7f9fd3200640 1 -- 172.21.15.60:0/3344421759 shutdown_connections 2024-04-14T23:49:01.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.928+0000 7f9fd3200640 1 -- 172.21.15.60:0/3344421759 wait complete. 2024-04-14T23:49:01.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.930+0000 7f9fd3200640 1 Processor -- start 2024-04-14T23:49:01.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.931+0000 7f9fd3200640 1 -- start start 2024-04-14T23:49:01.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.932+0000 7f9fd3200640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9fd4154d40 0x7f9fd40c9530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:01.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.932+0000 7f9fd3200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9fd4155120 0x7f9fd40c9a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:01.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.932+0000 7f9fd3200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9fd415b830 0x7f9fd41f1890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:01.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.932+0000 7f9fd3200640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f9fd40d09f0 con 0x7f9fd4154d40 2024-04-14T23:49:01.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.932+0000 7f9fd3200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9fd415edc0 con 0x7f9fd415b830 2024-04-14T23:49:01.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.932+0000 7f9fd3200640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f9fd40d0b70 con 0x7f9fd4155120 2024-04-14T23:49:01.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.932+0000 7f9fd1e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9fd4154d40 0x7f9fd40c9530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:01.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.932+0000 7f9fd1e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9fd4154d40 0x7f9fd40c9530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38408/0 (socket says 172.21.15.60:38408) 2024-04-14T23:49:01.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.932+0000 7f9fd1e00640 1 -- 172.21.15.60:0/2052790840 learned_addr learned my addr 172.21.15.60:0/2052790840 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:49:01.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.932+0000 7f9fd1400640 1 --2- 172.21.15.60:0/2052790840 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9fd4155120 0x7f9fd40c9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:01.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.932+0000 7f9fd2800640 1 --2- 172.21.15.60:0/2052790840 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9fd415b830 0x7f9fd41f1890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:01.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.933+0000 7f9fd1400640 1 -- 172.21.15.60:0/2052790840 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9fd415b830 msgr2=0x7f9fd41f1890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:01.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.933+0000 7f9fd1400640 1 --2- 172.21.15.60:0/2052790840 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9fd415b830 0x7f9fd41f1890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:01.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.933+0000 7f9fd1400640 1 -- 172.21.15.60:0/2052790840 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9fd4154d40 msgr2=0x7f9fd40c9530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:01.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.933+0000 7f9fd1400640 1 --2- 172.21.15.60:0/2052790840 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9fd4154d40 0x7f9fd40c9530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:01.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.933+0000 7f9fd1400640 1 -- 172.21.15.60:0/2052790840 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fd40c80b0 con 0x7f9fd4155120 2024-04-14T23:49:01.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.933+0000 7f9fd1e00640 1 --2- 172.21.15.60:0/2052790840 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9fd4154d40 0x7f9fd40c9530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:49:01.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.933+0000 7f9fd1400640 1 --2- 172.21.15.60:0/2052790840 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9fd4155120 0x7f9fd40c9a70 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc009db0 tx=0x7f9fbc00c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:01.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.933+0000 7f9fca000640 1 -- 172.21.15.60:0/2052790840 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fbc00cda0 con 0x7f9fd4155120 2024-04-14T23:49:01.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.933+0000 7f9fca000640 1 -- 172.21.15.60:0/2052790840 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fbc019070 con 0x7f9fd4155120 2024-04-14T23:49:01.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.934+0000 7f9fd3200640 1 -- 172.21.15.60:0/2052790840 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fd40c8390 con 0x7f9fd4155120 2024-04-14T23:49:01.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.934+0000 7f9fca000640 1 -- 172.21.15.60:0/2052790840 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fbc014680 con 0x7f9fd4155120 2024-04-14T23:49:01.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.934+0000 7f9fd3200640 1 -- 172.21.15.60:0/2052790840 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fd40c8950 con 0x7f9fd4155120 2024-04-14T23:49:01.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.936+0000 7f9fca000640 1 -- 172.21.15.60:0/2052790840 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f9fbc005d10 con 0x7f9fd4155120 2024-04-14T23:49:01.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.937+0000 7f9fca000640 1 --2- 172.21.15.60:0/2052790840 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f9fb006b9e0 0x7f9fb006dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:01.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.937+0000 7f9fca000640 1 -- 172.21.15.60:0/2052790840 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f9fbc08ac90 con 0x7f9fd4155120 2024-04-14T23:49:01.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.937+0000 7f9fd1e00640 1 --2- 172.21.15.60:0/2052790840 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f9fb006b9e0 0x7f9fb006dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:01.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.938+0000 7f9fdc174740 1 Processor -- start 2024-04-14T23:49:01.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.938+0000 7f9fdc174740 1 -- start start 2024-04-14T23:49:01.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.938+0000 7f9fd1e00640 1 --2- 172.21.15.60:0/2052790840 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f9fb006b9e0 0x7f9fb006dea0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9fc402b980 tx=0x7f9fc403b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:01.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.939+0000 7f9fdc174740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564900d12e30 0x564900cd2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:01.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.939+0000 7f9fdc174740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x564900cd18d0 0x564900d137f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:01.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.939+0000 7f9fdc174740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564900d13210 0x564900d12110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:01.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.939+0000 7f9fdc174740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x564900c35e80 con 0x564900d12e30 2024-04-14T23:49:01.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.939+0000 7f9fdc174740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564900bdec00 con 0x564900d13210 2024-04-14T23:49:01.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.939+0000 7f9fdc174740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x564900c31e30 con 0x564900cd18d0 2024-04-14T23:49:01.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.939+0000 7f9fd1e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564900d12e30 0x564900cd2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:01.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.939+0000 7f9fd1e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564900d12e30 0x564900cd2bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38416/0 (socket says 172.21.15.60:38416) 2024-04-14T23:49:01.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.939+0000 7f9fd1e00640 1 -- 172.21.15.60:0/2108173384 learned_addr learned my addr 172.21.15.60:0/2108173384 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:49:01.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.939+0000 7f9fd2800640 1 --2- 172.21.15.60:0/2108173384 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564900d13210 0x564900d12110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:01.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.939+0000 7f9fd1400640 1 --2- 172.21.15.60:0/2108173384 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x564900cd18d0 0x564900d137f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:01.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.940+0000 7f9fd1e00640 1 -- 172.21.15.60:0/2108173384 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x564900cd18d0 msgr2=0x564900d137f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:49:01.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.940+0000 7f9fd1e00640 1 --2- 172.21.15.60:0/2108173384 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x564900cd18d0 0x564900d137f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:01.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.940+0000 7f9fd1e00640 1 -- 172.21.15.60:0/2108173384 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564900d13210 msgr2=0x564900d12110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:49:01.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.940+0000 7f9fd1e00640 1 --2- 172.21.15.60:0/2108173384 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564900d13210 0x564900d12110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:01.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.940+0000 7f9fd1e00640 1 -- 172.21.15.60:0/2108173384 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56490094a2d0 con 0x564900d12e30 2024-04-14T23:49:01.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.940+0000 7f9fd1e00640 1 --2- 172.21.15.60:0/2108173384 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564900d12e30 0x564900cd2bd0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f9fc40094d0 tx=0x7f9fc4030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8fc50d1164d1a5a server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:01.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.940+0000 7f9fc2a00640 1 -- 172.21.15.60:0/2108173384 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fc4034040 con 0x564900d12e30 2024-04-14T23:49:01.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.940+0000 7f9fc2a00640 1 -- 172.21.15.60:0/2108173384 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fc4054070 con 0x564900d12e30 2024-04-14T23:49:01.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.940+0000 7f9fc2a00640 1 -- 172.21.15.60:0/2108173384 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fc404f7f0 con 0x564900d12e30 2024-04-14T23:49:01.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.941+0000 7f9fdc174740 1 -- 172.21.15.60:0/2108173384 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564900d12e30 msgr2=0x564900cd2bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:01.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.941+0000 7f9fdc174740 1 --2- 172.21.15.60:0/2108173384 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564900d12e30 0x564900cd2bd0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f9fc40094d0 tx=0x7f9fc4030000 comp rx=0 tx=0).stop 2024-04-14T23:49:01.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.941+0000 7f9fdc174740 1 -- 172.21.15.60:0/2108173384 shutdown_connections 2024-04-14T23:49:01.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.941+0000 7f9fdc174740 1 --2- 172.21.15.60:0/2108173384 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564900d13210 0x564900d12110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:01.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.941+0000 7f9fdc174740 1 --2- 172.21.15.60:0/2108173384 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x564900cd18d0 0x564900d137f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:01.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.941+0000 7f9fdc174740 1 --2- 172.21.15.60:0/2108173384 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564900d12e30 0x564900cd2bd0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:01.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.941+0000 7f9fdc174740 1 -- 172.21.15.60:0/2108173384 >> 172.21.15.60:0/2108173384 conn(0x564900ccea40 msgr2=0x564900cd21e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:49:01.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.941+0000 7f9fdc174740 1 -- 172.21.15.60:0/2108173384 shutdown_connections 2024-04-14T23:49:01.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.941+0000 7f9fdc174740 1 -- 172.21.15.60:0/2108173384 wait complete. 2024-04-14T23:49:01.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.942+0000 7f9fdc174740 1 -- start start 2024-04-14T23:49:01.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.942+0000 7f9fdc174740 1 Processor -- start 2024-04-14T23:49:01.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.943+0000 7f9fdc174740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564900d12e30 0x564900d15e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:01.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.943+0000 7f9fdc174740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564900d13210 0x564900d18ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:01.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.943+0000 7f9fdc174740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x564900d15690 0x564900d13e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:01.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.943+0000 7f9fdc174740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x564900bd4150 con 0x564900d13210 2024-04-14T23:49:01.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.943+0000 7f9fdc174740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564900d1c160 con 0x564900d12e30 2024-04-14T23:49:01.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.943+0000 7f9fdc174740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x564900d14810 con 0x564900d15690 2024-04-14T23:49:01.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.943+0000 7f9fdc174740 10 client.0 _ll_register_callbacks cb 0x564900c2d5a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-14T23:49:01.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.944+0000 7f9fd1400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564900d13210 0x564900d18ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:01.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.944+0000 7f9fd1e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564900d12e30 0x564900d15e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:01.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.944+0000 7f9fd1400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564900d13210 0x564900d18ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38418/0 (socket says 172.21.15.60:38418) 2024-04-14T23:49:01.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.944+0000 7f9fd1400640 1 -- 172.21.15.60:0/1082890726 learned_addr learned my addr 172.21.15.60:0/1082890726 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:49:01.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.944+0000 7f9fd2800640 1 --2- 172.21.15.60:0/1082890726 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x564900d15690 0x564900d13e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:01.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.944+0000 7f9fd1400640 1 -- 172.21.15.60:0/1082890726 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x564900d15690 msgr2=0x564900d13e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:49:01.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.944+0000 7f9fd1400640 1 --2- 172.21.15.60:0/1082890726 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x564900d15690 0x564900d13e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:01.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.944+0000 7f9fd1400640 1 -- 172.21.15.60:0/1082890726 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564900d12e30 msgr2=0x564900d15e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:49:01.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.944+0000 7f9fd1400640 1 --2- 172.21.15.60:0/1082890726 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564900d12e30 0x564900d15e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:01.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.944+0000 7f9fd1400640 1 -- 172.21.15.60:0/1082890726 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564900a146b0 con 0x564900d13210 2024-04-14T23:49:01.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.944+0000 7f9fd1400640 1 --2- 172.21.15.60:0/1082890726 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564900d13210 0x564900d18ce0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc059130 tx=0x7f9fbc058b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b112df2bd7b0f12 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:01.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.944+0000 7f9fbb400640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-04-14T23:49:01.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.944+0000 7f9fbb400640 1 -- 172.21.15.60:0/1082890726 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fbc061b00 con 0x564900d13210 2024-04-14T23:49:01.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.945+0000 7f9fbb400640 1 -- 172.21.15.60:0/1082890726 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f9fbc0585f0 con 0x564900d13210 2024-04-14T23:49:01.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.945+0000 7f9fdc174740 1 -- 172.21.15.60:0/1082890726 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9fc4054070 con 0x564900d13210 2024-04-14T23:49:01.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.945+0000 7f9fbb400640 1 -- 172.21.15.60:0/1082890726 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) ==== 24+0+0 (secure 0 0 0) 0x7f9fbc077050 con 0x564900d13210 2024-04-14T23:49:01.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.945+0000 7f9fdc174740 10 client.4578 fetch_fsmap learned FSMap version 8 2024-04-14T23:49:01.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.945+0000 7f9fdc174740 1 -- 172.21.15.60:0/1082890726 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) -- 0x564900d238d0 con 0x564900d13210 2024-04-14T23:49:01.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.945+0000 7f9fbb400640 1 -- 172.21.15.60:0/1082890726 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fbc072020 con 0x564900d13210 2024-04-14T23:49:01.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.945+0000 7f9fbb400640 1 -- 172.21.15.60:0/1082890726 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 8) ==== 4704+0+0 (secure 0 0 0) 0x7f9fbc082b00 con 0x564900d13210 2024-04-14T23:49:01.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.945+0000 7f9fdc174740 10 client.4578 fetch_fsmap finished waiting for FSMap version 8 2024-04-14T23:49:01.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.945+0000 7f9fdc174740 10 client.4578 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-14T23:49:01.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.945+0000 7f9fdc174740 20 client.4578 populate_metadata read hostname 'smithi060' 2024-04-14T23:49:01.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.945+0000 7f9fdc174740 1 --2- 172.21.15.60:0/1082890726 >> [v2:172.21.15.60:6834/2677133122,v1:172.21.15.60:6835/2677133122] conn(0x564900cd2710 0x564900d31cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:01.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.945+0000 7f9fdc174740 4 client.4578 mds_command: new command op to 4279 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-14T23:49:01.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.945+0000 7f9fdc174740 1 -- 172.21.15.60:0/1082890726 --> [v2:172.21.15.60:6834/2677133122,v1:172.21.15.60:6835/2677133122] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564900c433c0 con 0x564900cd2710 2024-04-14T23:49:01.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.946+0000 7f9fd1e00640 1 --2- 172.21.15.60:0/1082890726 >> [v2:172.21.15.60:6834/2677133122,v1:172.21.15.60:6835/2677133122] conn(0x564900cd2710 0x564900d31cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:01.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.946+0000 7f9fd1e00640 1 --2- 172.21.15.60:0/1082890726 >> [v2:172.21.15.60:6834/2677133122,v1:172.21.15.60:6835/2677133122] conn(0x564900cd2710 0x564900d31cf0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=834d387af83b96c8 server_cookie=24cc7c0446928e33 in_seq=0 out_seq=0 2024-04-14T23:49:01.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.946+0000 7f9fbb400640 10 client.4578 ms_handle_connect on v2:172.21.15.60:6834/2677133122 2024-04-14T23:49:01.957 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.947+0000 7f9fbb400640 1 -- 172.21.15.60:0/1082890726 <== mds.0 v2:172.21.15.60:6834/2677133122 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564900c433c0 con 0x564900cd2710 2024-04-14T23:49:01.957 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.947+0000 7f9fbb400640 10 client.4578 handle_command_reply: tid=0 2024-04-14T23:49:01.957 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.947+0000 7f9fdc174740 1 client.4578 shutdown 2024-04-14T23:49:01.957 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.947+0000 7f9fdc174740 1 -- 172.21.15.60:0/1082890726 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564900d13210 msgr2=0x564900d18ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:01.957 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.947+0000 7f9fdc174740 1 --2- 172.21.15.60:0/1082890726 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564900d13210 0x564900d18ce0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc059130 tx=0x7f9fbc058b70 comp rx=0 tx=0).stop 2024-04-14T23:49:01.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.947+0000 7f9fdc174740 1 -- 172.21.15.60:0/1082890726 shutdown_connections 2024-04-14T23:49:01.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.947+0000 7f9fdc174740 1 --2- 172.21.15.60:0/1082890726 >> [v2:172.21.15.60:6834/2677133122,v1:172.21.15.60:6835/2677133122] conn(0x564900cd2710 0x564900d31cf0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:01.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.947+0000 7f9fdc174740 1 --2- 172.21.15.60:0/1082890726 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x564900d15690 0x564900d13e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:01.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.947+0000 7f9fdc174740 1 --2- 172.21.15.60:0/1082890726 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564900d13210 0x564900d18ce0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:01.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.947+0000 7f9fdc174740 1 --2- 172.21.15.60:0/1082890726 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564900d12e30 0x564900d15e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:01.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.947+0000 7f9fbb400640 0 client.4578 ms_handle_reset on v2:172.21.15.60:6834/2677133122 2024-04-14T23:49:01.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.948+0000 7f9fdc174740 1 -- 172.21.15.60:0/1082890726 >> 172.21.15.60:0/1082890726 conn(0x564900cd18d0 msgr2=0x564900cfb7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:49:01.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.948+0000 7f9fdc174740 1 -- 172.21.15.60:0/1082890726 shutdown_connections 2024-04-14T23:49:01.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.948+0000 7f9fdc174740 1 -- 172.21.15.60:0/1082890726 wait complete. 2024-04-14T23:49:01.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.948+0000 7f9fdc174740 20 client.4578 trim_cache size 0 max 16384 2024-04-14T23:49:01.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.968+0000 7f9fdc174740 1 Processor -- start 2024-04-14T23:49:01.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.968+0000 7f9fdc174740 1 -- start start 2024-04-14T23:49:01.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.969+0000 7f9fdc174740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x564900d12e30 0x564900d372b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:01.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.969+0000 7f9fdc174740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564900d13210 0x564900ccc900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:01.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.969+0000 7f9fdc174740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564900d15690 0x564900ccce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:01.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.969+0000 7f9fdc174740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f9fd414a850 con 0x564900d13210 2024-04-14T23:49:01.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.969+0000 7f9fdc174740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564900d145a0 con 0x564900d15690 2024-04-14T23:49:01.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.969+0000 7f9fdc174740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x564900d1b150 con 0x564900d12e30 2024-04-14T23:49:01.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.969+0000 7f9fd1e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x564900d12e30 0x564900d372b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:01.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.969+0000 7f9fd1e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x564900d12e30 0x564900d372b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.60:48374/0 (socket says 172.21.15.60:48374) 2024-04-14T23:49:01.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.969+0000 7f9fd1e00640 1 -- 172.21.15.60:0/3456557257 learned_addr learned my addr 172.21.15.60:0/3456557257 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:49:01.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.969+0000 7f9fd2800640 1 --2- 172.21.15.60:0/3456557257 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564900d15690 0x564900ccce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:01.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.969+0000 7f9fd1400640 1 --2- 172.21.15.60:0/3456557257 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564900d13210 0x564900ccc900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:01.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.970+0000 7f9fd1e00640 1 -- 172.21.15.60:0/3456557257 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564900d15690 msgr2=0x564900ccce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:49:01.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.970+0000 7f9fd1e00640 1 --2- 172.21.15.60:0/3456557257 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564900d15690 0x564900ccce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:01.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.970+0000 7f9fd1e00640 1 -- 172.21.15.60:0/3456557257 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564900d13210 msgr2=0x564900ccc900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:49:01.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.970+0000 7f9fd1e00640 1 --2- 172.21.15.60:0/3456557257 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564900d13210 0x564900ccc900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:01.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.970+0000 7f9fd1e00640 1 -- 172.21.15.60:0/3456557257 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564900cf46f0 con 0x564900d12e30 2024-04-14T23:49:01.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.970+0000 7f9fd1e00640 1 --2- 172.21.15.60:0/3456557257 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x564900d12e30 0x564900d372b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9fc4050000 tx=0x7f9fc4031000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c26c8387bac2ef4 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:01.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.970+0000 7f9fc2000640 1 -- 172.21.15.60:0/3456557257 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fc4034040 con 0x564900d12e30 2024-04-14T23:49:01.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.970+0000 7f9fc2000640 1 -- 172.21.15.60:0/3456557257 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fc4061230 con 0x564900d12e30 2024-04-14T23:49:01.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.970+0000 7f9fc2000640 1 -- 172.21.15.60:0/3456557257 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fc40615e0 con 0x564900d12e30 2024-04-14T23:49:01.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.971+0000 7f9fdc174740 1 -- 172.21.15.60:0/3456557257 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x564900d12e30 msgr2=0x564900d372b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:01.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.971+0000 7f9fdc174740 1 --2- 172.21.15.60:0/3456557257 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x564900d12e30 0x564900d372b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9fc4050000 tx=0x7f9fc4031000 comp rx=0 tx=0).stop 2024-04-14T23:49:01.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.971+0000 7f9fdc174740 1 -- 172.21.15.60:0/3456557257 shutdown_connections 2024-04-14T23:49:01.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.971+0000 7f9fdc174740 1 --2- 172.21.15.60:0/3456557257 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564900d15690 0x564900ccce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:01.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.971+0000 7f9fdc174740 1 --2- 172.21.15.60:0/3456557257 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564900d13210 0x564900ccc900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:01.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.971+0000 7f9fdc174740 1 --2- 172.21.15.60:0/3456557257 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x564900d12e30 0x564900d372b0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:01.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.971+0000 7f9fdc174740 1 -- 172.21.15.60:0/3456557257 >> 172.21.15.60:0/3456557257 conn(0x564900cd18d0 msgr2=0x564900d13c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:49:01.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.971+0000 7f9fdc174740 1 -- 172.21.15.60:0/3456557257 shutdown_connections 2024-04-14T23:49:01.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.971+0000 7f9fdc174740 1 -- 172.21.15.60:0/3456557257 wait complete. 2024-04-14T23:49:01.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.972+0000 7f9fdc174740 1 -- start start 2024-04-14T23:49:01.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.972+0000 7f9fdc174740 1 Processor -- start 2024-04-14T23:49:01.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.973+0000 7f9fdc174740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x564900d12e30 0x564900cccd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:01.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.973+0000 7f9fdc174740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564900d13210 0x564900d8dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:01.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.973+0000 7f9fdc174740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564900d15690 0x564900d2c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:01.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.973+0000 7f9fdc174740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x564900d262d0 con 0x564900d13210 2024-04-14T23:49:01.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.973+0000 7f9fdc174740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564900cd2590 con 0x564900d15690 2024-04-14T23:49:01.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.973+0000 7f9fdc174740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x564900d26960 con 0x564900d12e30 2024-04-14T23:49:01.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.973+0000 7f9fdc174740 10 client.0 _ll_register_callbacks cb 0x7f9fbc0554d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-14T23:49:01.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.973+0000 7f9fd1400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564900d13210 0x564900d8dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:01.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.973+0000 7f9fd1e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x564900d12e30 0x564900cccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:01.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.973+0000 7f9fd2800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564900d15690 0x564900d2c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:01.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.973+0000 7f9fd1400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564900d13210 0x564900d8dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38436/0 (socket says 172.21.15.60:38436) 2024-04-14T23:49:01.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.973+0000 7f9fd1400640 1 -- 172.21.15.60:0/1800974248 learned_addr learned my addr 172.21.15.60:0/1800974248 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:49:01.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.974+0000 7f9fd1400640 1 -- 172.21.15.60:0/1800974248 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x564900d12e30 msgr2=0x564900cccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:49:01.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.974+0000 7f9fd1400640 1 --2- 172.21.15.60:0/1800974248 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x564900d12e30 0x564900cccd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:01.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.974+0000 7f9fd1400640 1 -- 172.21.15.60:0/1800974248 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564900d15690 msgr2=0x564900d2c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:49:01.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.974+0000 7f9fd1400640 1 --2- 172.21.15.60:0/1800974248 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564900d15690 0x564900d2c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:01.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.974+0000 7f9fd1400640 1 -- 172.21.15.60:0/1800974248 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564900ccb710 con 0x564900d13210 2024-04-14T23:49:01.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.974+0000 7f9fd1e00640 1 --2- 172.21.15.60:0/1800974248 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x564900d12e30 0x564900cccd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:49:01.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.974+0000 7f9fd1400640 1 --2- 172.21.15.60:0/1800974248 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564900d13210 0x564900d8dc60 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc00c990 tx=0x7f9fbc089070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5deb3cee866631d server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:01.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.974+0000 7f9fbb400640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-04-14T23:49:01.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.974+0000 7f9fbb400640 1 -- 172.21.15.60:0/1800974248 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fbc061b00 con 0x564900d13210 2024-04-14T23:49:01.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.974+0000 7f9fbb400640 1 -- 172.21.15.60:0/1800974248 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f9fbc087040 con 0x564900d13210 2024-04-14T23:49:01.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.974+0000 7f9fdc174740 1 -- 172.21.15.60:0/1800974248 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9fc4061230 con 0x564900d13210 2024-04-14T23:49:01.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.975+0000 7f9fbb400640 1 -- 172.21.15.60:0/1800974248 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) ==== 24+0+0 (secure 0 0 0) 0x7f9fbc068050 con 0x564900d13210 2024-04-14T23:49:01.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.975+0000 7f9fdc174740 10 client.4584 fetch_fsmap learned FSMap version 8 2024-04-14T23:49:01.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.975+0000 7f9fdc174740 1 -- 172.21.15.60:0/1800974248 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) -- 0x564900d150a0 con 0x564900d13210 2024-04-14T23:49:01.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.975+0000 7f9fbb400640 1 -- 172.21.15.60:0/1800974248 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fc4061230 con 0x564900d13210 2024-04-14T23:49:01.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.975+0000 7f9fbb400640 1 -- 172.21.15.60:0/1800974248 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 8) ==== 4704+0+0 (secure 0 0 0) 0x7f9fbc06b020 con 0x564900d13210 2024-04-14T23:49:01.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.975+0000 7f9fdc174740 10 client.4584 fetch_fsmap finished waiting for FSMap version 8 2024-04-14T23:49:01.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.975+0000 7f9fdc174740 10 client.4584 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-14T23:49:01.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.975+0000 7f9fdc174740 20 client.4584 populate_metadata read hostname 'smithi060' 2024-04-14T23:49:01.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.975+0000 7f9fdc174740 1 --2- 172.21.15.60:0/1800974248 >> [v2:172.21.15.60:6834/2677133122,v1:172.21.15.60:6835/2677133122] conn(0x564900cf9310 0x564900d99b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:01.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.975+0000 7f9fdc174740 4 client.4584 mds_command: new command op to 4279 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-14T23:49:01.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.975+0000 7f9fdc174740 1 -- 172.21.15.60:0/1800974248 --> [v2:172.21.15.60:6834/2677133122,v1:172.21.15.60:6835/2677133122] -- command(tid 0: {"prefix": "status"}) -- 0x564900c3d710 con 0x564900cf9310 2024-04-14T23:49:01.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.975+0000 7f9fd1e00640 1 --2- 172.21.15.60:0/1800974248 >> [v2:172.21.15.60:6834/2677133122,v1:172.21.15.60:6835/2677133122] conn(0x564900cf9310 0x564900d99b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:01.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.976+0000 7f9fd1e00640 1 --2- 172.21.15.60:0/1800974248 >> [v2:172.21.15.60:6834/2677133122,v1:172.21.15.60:6835/2677133122] conn(0x564900cf9310 0x564900d99b70 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=705aa9d0df1cc4e8 server_cookie=10e7a75b6d9c933 in_seq=0 out_seq=0 2024-04-14T23:49:01.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.976+0000 7f9fbb400640 10 client.4584 ms_handle_connect on v2:172.21.15.60:6834/2677133122 2024-04-14T23:49:01.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.976+0000 7f9fd1e00640 1 -- 172.21.15.60:0/1800974248 >> [v2:172.21.15.60:6834/2677133122,v1:172.21.15.60:6835/2677133122] conn(0x564900cf9310 msgr2=0x564900d99b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 18 2024-04-14T23:49:01.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.976+0000 7f9fd1e00640 1 -- 172.21.15.60:0/1800974248 >> [v2:172.21.15.60:6834/2677133122,v1:172.21.15.60:6835/2677133122] conn(0x564900cf9310 msgr2=0x564900d99b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-14T23:49:01.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.976+0000 7f9fd1e00640 1 --2- 172.21.15.60:0/1800974248 >> [v2:172.21.15.60:6834/2677133122,v1:172.21.15.60:6835/2677133122] conn(0x564900cf9310 0x564900d99b70 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-14T23:49:01.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.976+0000 7f9fd1e00640 1 --2- 172.21.15.60:0/1800974248 >> [v2:172.21.15.60:6834/2677133122,v1:172.21.15.60:6835/2677133122] conn(0x564900cf9310 0x564900d99b70 unknown :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault initiating reconnect 2024-04-14T23:49:01.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.976+0000 7f9fd1e00640 1 --2- 172.21.15.60:0/1800974248 >> [v2:172.21.15.60:6834/2677133122,v1:172.21.15.60:6835/2677133122] conn(0x564900cf9310 0x564900d99b70 unknown :-1 s=BANNER_CONNECTING pgs=14 cs=1 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:01.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.977+0000 7f9fd1e00640 1 --2- 172.21.15.60:0/1800974248 >> [v2:172.21.15.60:6834/2677133122,v1:172.21.15.60:6835/2677133122] conn(0x564900cf9310 0x564900d99b70 crc :-1 s=READY pgs=14 cs=1 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=705aa9d0df1cc4e8 server_cookie=10e7a75b6d9c933 in_seq=0 out_seq=1 2024-04-14T23:49:01.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.977+0000 7f9fbb400640 10 client.4584 ms_handle_connect on v2:172.21.15.60:6834/2677133122 2024-04-14T23:49:01.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.977+0000 7f9fbb400640 1 -- 172.21.15.60:0/1800974248 <== mds.0 v2:172.21.15.60:6834/2677133122 1 ==== command_reply(tid 0: 0 ) ==== 8+0+330 (crc 0 0 0) 0x564900c3d710 con 0x564900cf9310 2024-04-14T23:49:01.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.977+0000 7f9fbb400640 10 client.4584 handle_command_reply: tid=0 2024-04-14T23:49:01.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.977+0000 7f9fdc174740 1 client.4584 shutdown 2024-04-14T23:49:01.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.977+0000 7f9fdc174740 1 -- 172.21.15.60:0/1800974248 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564900d13210 msgr2=0x564900d8dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:01.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.977+0000 7f9fdc174740 1 --2- 172.21.15.60:0/1800974248 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564900d13210 0x564900d8dc60 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc00c990 tx=0x7f9fbc089070 comp rx=0 tx=0).stop 2024-04-14T23:49:01.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.977+0000 7f9fdc174740 1 -- 172.21.15.60:0/1800974248 shutdown_connections 2024-04-14T23:49:01.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.977+0000 7f9fdc174740 1 --2- 172.21.15.60:0/1800974248 >> [v2:172.21.15.60:6834/2677133122,v1:172.21.15.60:6835/2677133122] conn(0x564900cf9310 0x564900d99b70 crc :-1 s=READY pgs=14 cs=1 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:01.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.977+0000 7f9fdc174740 1 --2- 172.21.15.60:0/1800974248 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564900d15690 0x564900d2c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:01.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.977+0000 7f9fdc174740 1 --2- 172.21.15.60:0/1800974248 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564900d13210 0x564900d8dc60 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:01.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.977+0000 7f9fdc174740 1 --2- 172.21.15.60:0/1800974248 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x564900d12e30 0x564900cccd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:01.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.977+0000 7f9fdc174740 1 -- 172.21.15.60:0/1800974248 >> 172.21.15.60:0/1800974248 conn(0x564900cd18d0 msgr2=0x564900cd0a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:49:01.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.977+0000 7f9fbb400640 0 client.4584 ms_handle_reset on v2:172.21.15.60:6834/2677133122 2024-04-14T23:49:01.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.978+0000 7f9fdc174740 1 -- 172.21.15.60:0/1800974248 shutdown_connections 2024-04-14T23:49:01.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.978+0000 7f9fdc174740 1 -- 172.21.15.60:0/1800974248 wait complete. 2024-04-14T23:49:01.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.978+0000 7f9fdc174740 20 client.4584 trim_cache size 0 max 16384 2024-04-14T23:49:01.987 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-14T23:49:01.987 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_fsid": "5847db56-a76d-4486-b563-d83f1186af92", 2024-04-14T23:49:01.987 INFO:teuthology.orchestra.run.smithi060.stdout: "whoami": 0, 2024-04-14T23:49:01.987 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 4279, 2024-04-14T23:49:01.987 INFO:teuthology.orchestra.run.smithi060.stdout: "want_state": "up:active", 2024-04-14T23:49:01.987 INFO:teuthology.orchestra.run.smithi060.stdout: "state": "up:active", 2024-04-14T23:49:01.987 INFO:teuthology.orchestra.run.smithi060.stdout: "fs_name": "cephfs", 2024-04-14T23:49:01.987 INFO:teuthology.orchestra.run.smithi060.stdout: "rank_uptime": 18.770750837000001, 2024-04-14T23:49:01.987 INFO:teuthology.orchestra.run.smithi060.stdout: "mdsmap_epoch": 8, 2024-04-14T23:49:01.987 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch": 21, 2024-04-14T23:49:01.987 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch_barrier": 21, 2024-04-14T23:49:01.987 INFO:teuthology.orchestra.run.smithi060.stdout: "uptime": 26.798722518000002 2024-04-14T23:49:01.987 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-14T23:49:01.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.978+0000 7f9fd3200640 1 -- 172.21.15.60:0/2052790840 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f9fb006b9e0 msgr2=0x7f9fb006dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:01.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.978+0000 7f9fd3200640 1 --2- 172.21.15.60:0/2052790840 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f9fb006b9e0 0x7f9fb006dea0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9fc402b980 tx=0x7f9fc403b040 comp rx=0 tx=0).stop 2024-04-14T23:49:01.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.978+0000 7f9fd3200640 1 -- 172.21.15.60:0/2052790840 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9fd4155120 msgr2=0x7f9fd40c9a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:01.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.978+0000 7f9fd3200640 1 --2- 172.21.15.60:0/2052790840 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9fd4155120 0x7f9fd40c9a70 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc009db0 tx=0x7f9fbc00c490 comp rx=0 tx=0).stop 2024-04-14T23:49:01.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.978+0000 7f9fd3200640 1 -- 172.21.15.60:0/2052790840 shutdown_connections 2024-04-14T23:49:01.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.978+0000 7f9fd3200640 1 --2- 172.21.15.60:0/2052790840 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f9fb006b9e0 0x7f9fb006dea0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:01.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.978+0000 7f9fd3200640 1 --2- 172.21.15.60:0/2052790840 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9fd415b830 0x7f9fd41f1890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:01.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.978+0000 7f9fd3200640 1 --2- 172.21.15.60:0/2052790840 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9fd4155120 0x7f9fd40c9a70 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:01.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.978+0000 7f9fd3200640 1 --2- 172.21.15.60:0/2052790840 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9fd4154d40 0x7f9fd40c9530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:01.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.978+0000 7f9fd3200640 1 -- 172.21.15.60:0/2052790840 >> 172.21.15.60:0/2052790840 conn(0x7f9fd4066f10 msgr2=0x7f9fd40ce1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:49:01.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.979+0000 7f9fd3200640 1 -- 172.21.15.60:0/2052790840 shutdown_connections 2024-04-14T23:49:01.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:01.979+0000 7f9fd3200640 1 -- 172.21.15.60:0/2052790840 wait complete. 2024-04-14T23:49:02.002 INFO:teuthology.run_tasks:Running task sequential... 2024-04-14T23:49:02.012 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-14T23:49:02.012 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-14T23:49:02.012 INFO:teuthology.task.print:Enabling mgr modules 2024-04-14T23:49:02.012 INFO:teuthology.run_tasks:Running task kclient... 2024-04-14T23:49:02.053 INFO:tasks.kclient:Mounting kernel clients... 2024-04-14T23:49:02.053 INFO:tasks.kclient:config is {'client.0': {}} 2024-04-14T23:49:02.053 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-14T23:49:02.053 DEBUG:teuthology.orchestra.run.smithi060:> ip netns list 2024-04-14T23:49:02.115 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-14T23:49:02.115 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link delete ceph-brx 2024-04-14T23:49:02.221 INFO:teuthology.orchestra.run.smithi060.stderr:Cannot find device "ceph-brx" 2024-04-14T23:49:02.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T23:49:02.226 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2024-04-14T23:49:02.226 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-14T23:49:02.226 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-14T23:49:02.226 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-14T23:49:02.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.443+0000 7f5565000640 1 Processor -- start 2024-04-14T23:49:02.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.443+0000 7f5565000640 1 -- start start 2024-04-14T23:49:02.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.445+0000 7f5565000640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f55600c5160 0x7f556006c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:02.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.445+0000 7f5565000640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f556015b820 con 0x7f5560154d60 2024-04-14T23:49:02.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.445+0000 7f5565000640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f556015b260 con 0x7f556015a8d0 2024-04-14T23:49:02.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.445+0000 7f5565000640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f556006c160 con 0x7f55600c5160 2024-04-14T23:49:02.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.445+0000 7f555f400640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f5560154d60 0x7f55600cb370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33888/0 (socket says 172.21.15.60:33888) 2024-04-14T23:49:02.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.445+0000 7f555f400640 1 -- 172.21.15.60:0/2481945025 learned_addr learned my addr 172.21.15.60:0/2481945025 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:49:02.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.446+0000 7f555ea00640 1 --2- 172.21.15.60:0/2481945025 >> v2:172.21.15.60:3300/0 conn(0x7f55600c5160 0x7f556006c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:02.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.446+0000 7f555e000640 1 -- 172.21.15.60:0/2481945025 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4166944156 0 0) 0x7f556015b820 con 0x7f5560154d60 2024-04-14T23:49:02.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.446+0000 7f555e000640 1 -- 172.21.15.60:0/2481945025 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5544003520 con 0x7f5560154d60 2024-04-14T23:49:02.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.446+0000 7f555e000640 1 -- 172.21.15.60:0/2481945025 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2689256963 0 0) 0x7f556015b260 con 0x7f556015a8d0 2024-04-14T23:49:02.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.446+0000 7f555e000640 1 -- 172.21.15.60:0/2481945025 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f556015b820 con 0x7f556015a8d0 2024-04-14T23:49:02.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.446+0000 7f555ea00640 1 -- 172.21.15.60:0/2481945025 >> v1:172.21.15.186:6789/0 conn(0x7f556015a8d0 legacy=0x7f55600cc140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:02.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.446+0000 7f555ea00640 1 -- 172.21.15.60:0/2481945025 >> v1:172.21.15.60:6789/0 conn(0x7f5560154d60 legacy=0x7f55600cb370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:02.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.446+0000 7f555ea00640 1 -- 172.21.15.60:0/2481945025 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f556015b540 con 0x7f55600c5160 2024-04-14T23:49:02.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.447+0000 7f555ea00640 1 --2- 172.21.15.60:0/2481945025 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55600c5160 0x7f556006c5f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f554c0097a0 tx=0x7f554c02b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70c65cc49ad11a39 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:02.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.448+0000 7f555e000640 1 -- 172.21.15.60:0/2481945025 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f554c02f810 con 0x7f55600c5160 2024-04-14T23:49:02.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.448+0000 7f555e000640 1 -- 172.21.15.60:0/2481945025 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f554c02f9b0 con 0x7f55600c5160 2024-04-14T23:49:02.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.448+0000 7f555e000640 1 -- 172.21.15.60:0/2481945025 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f554c02fd80 con 0x7f55600c5160 2024-04-14T23:49:02.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.448+0000 7f5565000640 1 -- 172.21.15.60:0/2481945025 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55600c5160 msgr2=0x7f556006c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:02.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.448+0000 7f5565000640 1 --2- 172.21.15.60:0/2481945025 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55600c5160 0x7f556006c5f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f554c0097a0 tx=0x7f554c02b9f0 comp rx=0 tx=0).stop 2024-04-14T23:49:02.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.448+0000 7f5565000640 1 -- 172.21.15.60:0/2481945025 shutdown_connections 2024-04-14T23:49:02.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.448+0000 7f5565000640 1 --2- 172.21.15.60:0/2481945025 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55600c5160 0x7f556006c5f0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:02.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.448+0000 7f5565000640 1 -- 172.21.15.60:0/2481945025 >> 172.21.15.60:0/2481945025 conn(0x7f55600670f0 msgr2=0x7f5560067500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:49:02.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.449+0000 7f5565000640 1 -- 172.21.15.60:0/2481945025 shutdown_connections 2024-04-14T23:49:02.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.449+0000 7f5565000640 1 -- 172.21.15.60:0/2481945025 wait complete. 2024-04-14T23:49:02.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.450+0000 7f5565000640 1 Processor -- start 2024-04-14T23:49:02.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.450+0000 7f5565000640 1 -- start start 2024-04-14T23:49:02.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.451+0000 7f5565000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55600c5160 0x7f55601458c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:02.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.451+0000 7f5565000640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5560154d60 0x7f55601e6500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:02.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.451+0000 7f5565000640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f556015a8d0 0x7f55601e42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:02.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.451+0000 7f5565000640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f55600cc850 con 0x7f556015a8d0 2024-04-14T23:49:02.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.451+0000 7f5565000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f556006bdf0 con 0x7f55600c5160 2024-04-14T23:49:02.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.451+0000 7f5565000640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f55600cc9d0 con 0x7f5560154d60 2024-04-14T23:49:02.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.452+0000 7f555ea00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5560154d60 0x7f55601e6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:02.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.452+0000 7f555ea00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5560154d60 0x7f55601e6500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.60:48400/0 (socket says 172.21.15.60:48400) 2024-04-14T23:49:02.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.452+0000 7f555ea00640 1 -- 172.21.15.60:0/2043126615 learned_addr learned my addr 172.21.15.60:0/2043126615 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:49:02.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.452+0000 7f555fe00640 1 --2- 172.21.15.60:0/2043126615 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f556015a8d0 0x7f55601e42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:02.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.452+0000 7f555f400640 1 --2- 172.21.15.60:0/2043126615 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55600c5160 0x7f55601458c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:02.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.452+0000 7f555ea00640 1 -- 172.21.15.60:0/2043126615 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55600c5160 msgr2=0x7f55601458c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:02.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.452+0000 7f555ea00640 1 --2- 172.21.15.60:0/2043126615 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55600c5160 0x7f55601458c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:02.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.452+0000 7f555ea00640 1 -- 172.21.15.60:0/2043126615 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f556015a8d0 msgr2=0x7f55601e42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:02.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.452+0000 7f555ea00640 1 --2- 172.21.15.60:0/2043126615 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f556015a8d0 0x7f55601e42d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:02.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.452+0000 7f555ea00640 1 -- 172.21.15.60:0/2043126615 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55601451c0 con 0x7f5560154d60 2024-04-14T23:49:02.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.452+0000 7f555fe00640 1 --2- 172.21.15.60:0/2043126615 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f556015a8d0 0x7f55601e42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:49:02.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.452+0000 7f555ea00640 1 --2- 172.21.15.60:0/2043126615 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5560154d60 0x7f55601e6500 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f554c009890 tx=0x7f554c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:02.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.453+0000 7f5557400640 1 -- 172.21.15.60:0/2043126615 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f554c046070 con 0x7f5560154d60 2024-04-14T23:49:02.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.453+0000 7f5557400640 1 -- 172.21.15.60:0/2043126615 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f554c0378f0 con 0x7f5560154d60 2024-04-14T23:49:02.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.453+0000 7f5557400640 1 -- 172.21.15.60:0/2043126615 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f554c037ce0 con 0x7f5560154d60 2024-04-14T23:49:02.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.453+0000 7f5565000640 1 -- 172.21.15.60:0/2043126615 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55601454a0 con 0x7f5560154d60 2024-04-14T23:49:02.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.453+0000 7f5565000640 1 -- 172.21.15.60:0/2043126615 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f55601f1af0 con 0x7f5560154d60 2024-04-14T23:49:02.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.454+0000 7f5565000640 1 -- 172.21.15.60:0/2043126615 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f556004f540 con 0x7f5560154d60 2024-04-14T23:49:02.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.455+0000 7f5557400640 1 -- 172.21.15.60:0/2043126615 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f554c049070 con 0x7f5560154d60 2024-04-14T23:49:02.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.456+0000 7f5557400640 1 --2- 172.21.15.60:0/2043126615 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f554006b990 0x7f554006de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:02.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.456+0000 7f5557400640 1 -- 172.21.15.60:0/2043126615 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f554c0ac260 con 0x7f5560154d60 2024-04-14T23:49:02.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.456+0000 7f555f400640 1 --2- 172.21.15.60:0/2043126615 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f554006b990 0x7f554006de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:02.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.457+0000 7f555f400640 1 --2- 172.21.15.60:0/2043126615 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f554006b990 0x7f554006de50 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f5550000f60 tx=0x7f5550007d50 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:02.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.460+0000 7f5557400640 1 -- 172.21.15.60:0/2043126615 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f554c033030 con 0x7f5560154d60 2024-04-14T23:49:02.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.698+0000 7f5565000640 1 -- 172.21.15.60:0/2043126615 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f55600cbb60 con 0x7f5560154d60 2024-04-14T23:49:02.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.700+0000 7f5557400640 1 -- 172.21.15.60:0/2043126615 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+73 (secure 0 0 0) 0x7f554c07a780 con 0x7f5560154d60 2024-04-14T23:49:02.707 INFO:teuthology.orchestra.run.smithi060.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-14T23:49:02.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.701+0000 7f5565000640 1 -- 172.21.15.60:0/2043126615 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f554006b990 msgr2=0x7f554006de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:02.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.701+0000 7f5565000640 1 --2- 172.21.15.60:0/2043126615 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f554006b990 0x7f554006de50 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f5550000f60 tx=0x7f5550007d50 comp rx=0 tx=0).stop 2024-04-14T23:49:02.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.701+0000 7f5565000640 1 -- 172.21.15.60:0/2043126615 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5560154d60 msgr2=0x7f55601e6500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:02.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.701+0000 7f5565000640 1 --2- 172.21.15.60:0/2043126615 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5560154d60 0x7f55601e6500 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f554c009890 tx=0x7f554c004730 comp rx=0 tx=0).stop 2024-04-14T23:49:02.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.702+0000 7f5565000640 1 -- 172.21.15.60:0/2043126615 shutdown_connections 2024-04-14T23:49:02.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.702+0000 7f5565000640 1 --2- 172.21.15.60:0/2043126615 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f554006b990 0x7f554006de50 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:02.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.702+0000 7f5565000640 1 --2- 172.21.15.60:0/2043126615 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f556015a8d0 0x7f55601e42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:02.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.702+0000 7f5565000640 1 --2- 172.21.15.60:0/2043126615 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5560154d60 0x7f55601e6500 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:02.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.702+0000 7f5565000640 1 --2- 172.21.15.60:0/2043126615 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55600c5160 0x7f55601458c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:02.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.702+0000 7f5565000640 1 -- 172.21.15.60:0/2043126615 >> 172.21.15.60:0/2043126615 conn(0x7f55600670f0 msgr2=0x7f55600c5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:49:02.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.702+0000 7f5565000640 1 -- 172.21.15.60:0/2043126615 shutdown_connections 2024-04-14T23:49:02.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02.702+0000 7f5565000640 1 -- 172.21.15.60:0/2043126615 wait complete. 2024-04-14T23:49:02.726 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-14T23:49:02.727 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-14T23:49:02.727 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi060.front.sepia.ceph.com 2024-04-14T23:49:02.727 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-14T23:49:02.727 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-14T23:49:02.727 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-14T23:49:02.727 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-14T23:49:02.727 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-14T23:49:02.727 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-14T23:49:02.727 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-14T23:49:02.727 DEBUG:teuthology.orchestra.run.smithi060:> ip addr 2024-04-14T23:49:02.781 INFO:teuthology.orchestra.run.smithi060.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-14T23:49:02.781 INFO:teuthology.orchestra.run.smithi060.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-14T23:49:02.781 INFO:teuthology.orchestra.run.smithi060.stdout: inet 127.0.0.1/8 scope host lo 2024-04-14T23:49:02.781 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2024-04-14T23:49:02.781 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 ::1/128 scope host 2024-04-14T23:49:02.781 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2024-04-14T23:49:02.781 INFO:teuthology.orchestra.run.smithi060.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-14T23:49:02.782 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:6b:fe:cc brd ff:ff:ff:ff:ff:ff 2024-04-14T23:49:02.782 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp7s0f0 2024-04-14T23:49:02.782 INFO:teuthology.orchestra.run.smithi060.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-14T23:49:02.782 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:6b:fe:cd brd ff:ff:ff:ff:ff:ff 2024-04-14T23:49:02.782 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp7s0f1 2024-04-14T23:49:02.782 INFO:teuthology.orchestra.run.smithi060.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-14T23:49:02.782 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:bd:15:60 brd ff:ff:ff:ff:ff:ff 2024-04-14T23:49:02.782 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp1s0f0 2024-04-14T23:49:02.782 INFO:teuthology.orchestra.run.smithi060.stdout: inet 172.21.15.60/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-04-14T23:49:02.782 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft 42567sec preferred_lft 42567sec 2024-04-14T23:49:02.782 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::ec4:7aff:febd:1560/64 scope link noprefixroute 2024-04-14T23:49:02.782 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2024-04-14T23:49:02.782 INFO:teuthology.orchestra.run.smithi060.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-14T23:49:02.782 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:bd:15:61 brd ff:ff:ff:ff:ff:ff 2024-04-14T23:49:02.783 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp1s0f1 2024-04-14T23:49:02.783 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-14T23:49:02.783 DEBUG:teuthology.orchestra.run.smithi060:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-14T23:49:02.783 DEBUG:teuthology.orchestra.run.smithi060:> set -e 2024-04-14T23:49:02.783 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link add name ceph-brx type bridge 2024-04-14T23:49:02.783 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip addr flush dev ceph-brx 2024-04-14T23:49:02.783 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link set ceph-brx up 2024-04-14T23:49:02.783 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-14T23:49:02.784 DEBUG:teuthology.orchestra.run.smithi060:> ') 2024-04-14T23:49:02.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:02 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-14T23:49:03.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-14T23:49:03.352 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-14T23:49:03.352 DEBUG:teuthology.orchestra.run.smithi060:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-14T23:49:03.430 INFO:teuthology.orchestra.run.smithi060.stdout:1 2024-04-14T23:49:03.434 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-14T23:49:03.434 DEBUG:teuthology.orchestra.run.smithi060:> route 2024-04-14T23:49:03.491 INFO:teuthology.orchestra.run.smithi060.stdout:Kernel IP routing table 2024-04-14T23:49:03.492 INFO:teuthology.orchestra.run.smithi060.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-14T23:49:03.492 INFO:teuthology.orchestra.run.smithi060.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-14T23:49:03.492 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-14T23:49:03.492 INFO:teuthology.orchestra.run.smithi060.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-14T23:49:03.493 DEBUG:teuthology.orchestra.run.smithi060:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-14T23:49:03.493 DEBUG:teuthology.orchestra.run.smithi060:> set -e 2024-04-14T23:49:03.493 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-14T23:49:03.494 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-14T23:49:03.494 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-14T23:49:03.494 DEBUG:teuthology.orchestra.run.smithi060:> ') 2024-04-14T23:49:03.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:03 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-14T23:49:04.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-14T23:49:04.295 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-14T23:49:04.295 DEBUG:teuthology.orchestra.run.smithi060:> ip netns list 2024-04-14T23:49:04.354 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-14T23:49:04.354 DEBUG:teuthology.orchestra.run.smithi060:> ip netns list-id 2024-04-14T23:49:04.412 DEBUG:teuthology.orchestra.run.smithi060:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-14T23:49:04.412 DEBUG:teuthology.orchestra.run.smithi060:> set -e 2024-04-14T23:49:04.412 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-14T23:49:04.412 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-14T23:49:04.412 DEBUG:teuthology.orchestra.run.smithi060:> ') 2024-04-14T23:49:04.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:04 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-14T23:49:04.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-14T23:49:04.625 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-14T23:49:04.625 DEBUG:teuthology.orchestra.run.smithi060:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-14T23:49:04.625 DEBUG:teuthology.orchestra.run.smithi060:> set -e 2024-04-14T23:49:04.625 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-14T23:49:04.625 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-14T23:49:04.625 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-14T23:49:04.625 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-14T23:49:04.625 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-14T23:49:04.625 DEBUG:teuthology.orchestra.run.smithi060:> ') 2024-04-14T23:49:04.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:04 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-14T23:49:04.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-14T23:49:04.968 DEBUG:teuthology.orchestra.run.smithi060:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-14T23:49:04.968 DEBUG:teuthology.orchestra.run.smithi060:> set -e 2024-04-14T23:49:04.968 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link set brx.0 up 2024-04-14T23:49:04.968 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link set dev brx.0 master ceph-brx 2024-04-14T23:49:04.968 DEBUG:teuthology.orchestra.run.smithi060:> ') 2024-04-14T23:49:05.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:05 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-14T23:49:05.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-14T23:49:05.213 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-14T23:49:05.214 DEBUG:teuthology.orchestra.run.smithi060:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-14T23:49:05.261 INFO:teuthology.orchestra.run.smithi060.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-14T23:49:05.261 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-14T23:49:05.261 DEBUG:teuthology.orchestra.run.smithi060:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-14T23:49:05.352 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-04-14T23:49:05.352 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-14T23:49:05.352 DEBUG:teuthology.orchestra.run.smithi060:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-14T23:49:05.492 INFO:teuthology.orchestra.run.smithi060.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-14T23:49:05.492 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-14T23:49:05.492 INFO:teuthology.orchestra.run.smithi060.stdout:invalid new device string format 2024-04-14T23:49:06.167 INFO:teuthology.orchestra.run.smithi060.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-14T23:49:06.167 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-14T23:49:06.167 INFO:teuthology.orchestra.run.smithi060.stdout:invalid new device string format 2024-04-14T23:49:06.167 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: resolved to: "172.21.15.60:3300,172.21.15.186:3300,172.21.15.186:3301" 2024-04-14T23:49:06.167 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-14T23:49:06.204 INFO:teuthology.orchestra.run.smithi060.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-14T23:49:06.204 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-14T23:49:06.204 INFO:teuthology.orchestra.run.smithi060.stdout:invalid new device string format 2024-04-14T23:49:06.205 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: resolved to: "172.21.15.60:3300,172.21.15.186:3300,172.21.15.186:3301" 2024-04-14T23:49:06.205 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-14T23:49:06.205 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: resolved to: "172.21.15.60:6789,172.21.15.186:6789,172.21.15.186:6790" 2024-04-14T23:49:06.205 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.60:6789,172.21.15.186:6789,172.21.15.186:6790:/ 2024-04-14T23:49:06.205 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=5847db56-a76d-4486-b563-d83f1186af92" will pass to kernel 2024-04-14T23:49:06.209 INFO:teuthology.orchestra.run.smithi060.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-14T23:49:06.209 INFO:teuthology.orchestra.run.smithi060.stdout: You just mounted a file system that supports labels which does not 2024-04-14T23:49:06.209 INFO:teuthology.orchestra.run.smithi060.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-14T23:49:06.209 INFO:teuthology.orchestra.run.smithi060.stdout: applications will generate AVC messages and not be allowed access to 2024-04-14T23:49:06.209 INFO:teuthology.orchestra.run.smithi060.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-14T23:49:06.215 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-14T23:49:06.215 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-14T23:49:06.215 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-14T23:49:06.314 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-14T23:49:06.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:06 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-14T23:49:06.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:06 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-14T23:49:06.459 INFO:teuthology.orchestra.run.smithi060.stdout:client4596 2024-04-14T23:49:06.459 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-04-14T23:49:06.459 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-14T23:49:06.514 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-14T23:49:06.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:06 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-14T23:49:06.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:06 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-14T23:49:06.653 INFO:teuthology.orchestra.run.smithi060.stdout:client4596 2024-04-14T23:49:06.654 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-14T23:49:07.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:07 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-14T23:49:07.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:07 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-14T23:49:07.377 INFO:teuthology.orchestra.run.smithi060.stdout:client4596 2024-04-14T23:49:07.378 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/5847db56-a76d-4486-b563-d83f1186af92.client4596/status') 2024-04-14T23:49:07.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:07 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-14T23:49:07.547 INFO:teuthology.orchestra.run.smithi060.stdout:instance: client.4596 (3)192.168.0.1:0/1412319481 2024-04-14T23:49:07.547 INFO:teuthology.orchestra.run.smithi060.stdout:blocklisted: false 2024-04-14T23:49:07.548 INFO:teuthology.orchestra.run.smithi060.stderr:0+1 records in 2024-04-14T23:49:07.548 INFO:teuthology.orchestra.run.smithi060.stderr:0+1 records out 2024-04-14T23:49:07.548 INFO:teuthology.orchestra.run.smithi060.stderr:69 bytes copied, 8.5798e-05 s, 804 kB/s 2024-04-14T23:49:07.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:07 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-14T23:49:07.562 INFO:teuthology.run_tasks:Running task mds_thrash... 2024-04-14T23:49:07.574 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-14T23:49:07.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:07.795+0000 7fbb3a400640 1 Processor -- start 2024-04-14T23:49:07.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:07.796+0000 7fbb3a400640 1 -- start start 2024-04-14T23:49:07.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:07.797+0000 7fbb3a400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbb34154cf0 0x7fbb3415a940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:07.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:07.797+0000 7fbb3a400640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fbb3406c310 0x7fbb3406c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:07.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:07.797+0000 7fbb3a400640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb340c5520 con 0x7fbb340cb120 2024-04-14T23:49:07.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:07.797+0000 7fbb3a400640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fbb341550d0 con 0x7fbb3406c310 2024-04-14T23:49:07.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:07.797+0000 7fbb3a400640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fbb3415b040 con 0x7fbb34154cf0 2024-04-14T23:49:07.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:07.798+0000 7fbb39000640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbb34154cf0 0x7fbb3415a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:07.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:07.798+0000 7fbb39a00640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fbb340cb120 0x7fbb340cbd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33898/0 (socket says 172.21.15.60:33898) 2024-04-14T23:49:07.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:07.798+0000 7fbb39a00640 1 -- 172.21.15.60:0/371155533 learned_addr learned my addr 172.21.15.60:0/371155533 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:49:07.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:07.798+0000 7fbb33e00640 1 --2- 172.21.15.60:0/371155533 >> v2:172.21.15.186:3300/0 conn(0x7fbb3406c310 0x7fbb3406c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:07.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:07.798+0000 7fbb39000640 1 -- 172.21.15.60:0/371155533 >> v2:172.21.15.186:3300/0 conn(0x7fbb3406c310 msgr2=0x7fbb3406c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:49:07.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:07.798+0000 7fbb39000640 1 --2- 172.21.15.60:0/371155533 >> v2:172.21.15.186:3300/0 conn(0x7fbb3406c310 0x7fbb3406c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:07.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:07.798+0000 7fbb39000640 1 -- 172.21.15.60:0/371155533 >> v1:172.21.15.60:6789/0 conn(0x7fbb340cb120 legacy=0x7fbb340cbd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:07.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:07.798+0000 7fbb39000640 1 -- 172.21.15.60:0/371155533 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb3415b300 con 0x7fbb34154cf0 2024-04-14T23:49:07.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:07.798+0000 7fbb33400640 1 -- 172.21.15.60:0/371155533 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2492086563 0 0) 0x7fbb340c5520 con 0x7fbb340cb120 2024-04-14T23:49:07.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:07.800+0000 7fbb39000640 1 --2- 172.21.15.60:0/371155533 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbb34154cf0 0x7fbb3415a940 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fbb280097a0 tx=0x7fbb2802b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=adc7be3af217ed80 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:07.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:07.800+0000 7fbb33400640 1 -- 172.21.15.60:0/371155533 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb28034070 con 0x7fbb34154cf0 2024-04-14T23:49:07.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:07.800+0000 7fbb33400640 1 -- 172.21.15.60:0/371155533 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb28030800 con 0x7fbb34154cf0 2024-04-14T23:49:07.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:07.800+0000 7fbb33400640 1 -- 172.21.15.60:0/371155533 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb28030bd0 con 0x7fbb34154cf0 2024-04-14T23:49:07.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:07.801+0000 7fbb3a400640 1 -- 172.21.15.60:0/371155533 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbb34154cf0 msgr2=0x7fbb3415a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:07.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:07.801+0000 7fbb3a400640 1 --2- 172.21.15.60:0/371155533 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbb34154cf0 0x7fbb3415a940 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fbb280097a0 tx=0x7fbb2802b940 comp rx=0 tx=0).stop 2024-04-14T23:49:07.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:07.801+0000 7fbb3a400640 1 -- 172.21.15.60:0/371155533 shutdown_connections 2024-04-14T23:49:07.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:07.801+0000 7fbb3a400640 1 --2- 172.21.15.60:0/371155533 >> v2:172.21.15.186:3300/0 conn(0x7fbb3406c310 0x7fbb3406c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:07.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:07.801+0000 7fbb3a400640 1 --2- 172.21.15.60:0/371155533 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbb34154cf0 0x7fbb3415a940 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:07.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:07.801+0000 7fbb3a400640 1 -- 172.21.15.60:0/371155533 >> 172.21.15.60:0/371155533 conn(0x7fbb340670f0 msgr2=0x7fbb34067500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:49:07.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:07.801+0000 7fbb3a400640 1 -- 172.21.15.60:0/371155533 shutdown_connections 2024-04-14T23:49:07.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:07.801+0000 7fbb3a400640 1 -- 172.21.15.60:0/371155533 wait complete. 2024-04-14T23:49:07.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:07.803+0000 7fbb3a400640 1 Processor -- start 2024-04-14T23:49:07.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:07.803+0000 7fbb3a400640 1 -- start start 2024-04-14T23:49:07.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:07.804+0000 7fbb3a400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb3406c310 0x7fbb3415bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:07.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:07.804+0000 7fbb3a400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb340cb120 0x7fbb341f3c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:07.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:07.804+0000 7fbb3a400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbb34154cf0 0x7fbb3414aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:07.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:07.804+0000 7fbb3a400640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fbb3406bb10 con 0x7fbb340cb120 2024-04-14T23:49:07.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:07.804+0000 7fbb3a400640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbb3415b8c0 con 0x7fbb3406c310 2024-04-14T23:49:07.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:07.804+0000 7fbb3a400640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fbb3406bdf0 con 0x7fbb34154cf0 2024-04-14T23:49:07.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:07.804+0000 7fbb39000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb3406c310 0x7fbb3415bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:07.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:07.804+0000 7fbb39a00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbb34154cf0 0x7fbb3414aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:07.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:07.804+0000 7fbb39000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb3406c310 0x7fbb3415bd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:57642/0 (socket says 172.21.15.60:57642) 2024-04-14T23:49:07.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:07.804+0000 7fbb39000640 1 -- 172.21.15.60:0/732998384 learned_addr learned my addr 172.21.15.60:0/732998384 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:49:07.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:07.804+0000 7fbb33e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb340cb120 0x7fbb341f3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:07.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:07.805+0000 7fbb39000640 1 -- 172.21.15.60:0/732998384 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbb34154cf0 msgr2=0x7fbb3414aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:07.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:07.805+0000 7fbb39000640 1 --2- 172.21.15.60:0/732998384 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbb34154cf0 0x7fbb3414aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:07.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:07.805+0000 7fbb39000640 1 -- 172.21.15.60:0/732998384 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb340cb120 msgr2=0x7fbb341f3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:07.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:07.805+0000 7fbb39000640 1 --2- 172.21.15.60:0/732998384 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb340cb120 0x7fbb341f3c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:07.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:07.805+0000 7fbb39000640 1 -- 172.21.15.60:0/732998384 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb34156cc0 con 0x7fbb3406c310 2024-04-14T23:49:07.812 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:07.805+0000 7fbb39000640 1 --2- 172.21.15.60:0/732998384 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb3406c310 0x7fbb3415bd70 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fbb280098c0 tx=0x7fbb280046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:07.812 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:07.805+0000 7fbb30c00640 1 -- 172.21.15.60:0/732998384 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb28034040 con 0x7fbb3406c310 2024-04-14T23:49:07.812 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:07.805+0000 7fbb30c00640 1 -- 172.21.15.60:0/732998384 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb2803ada0 con 0x7fbb3406c310 2024-04-14T23:49:07.812 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:07.805+0000 7fbb3a400640 1 -- 172.21.15.60:0/732998384 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb34157c80 con 0x7fbb3406c310 2024-04-14T23:49:07.812 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:07.805+0000 7fbb30c00640 1 -- 172.21.15.60:0/732998384 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb28006e10 con 0x7fbb3406c310 2024-04-14T23:49:07.813 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:07.806+0000 7fbb3a400640 1 -- 172.21.15.60:0/732998384 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb34156f60 con 0x7fbb3406c310 2024-04-14T23:49:07.813 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:07.807+0000 7fbb3a400640 1 -- 172.21.15.60:0/732998384 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb3404f540 con 0x7fbb3406c310 2024-04-14T23:49:07.813 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:07.808+0000 7fbb30c00640 1 -- 172.21.15.60:0/732998384 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7fbb28037070 con 0x7fbb3406c310 2024-04-14T23:49:07.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:07.808+0000 7fbb30c00640 1 --2- 172.21.15.60:0/732998384 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fbb1406b990 0x7fbb1406de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:07.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:07.809+0000 7fbb30c00640 1 -- 172.21.15.60:0/732998384 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fbb280ac8b0 con 0x7fbb3406c310 2024-04-14T23:49:07.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:07.809+0000 7fbb33e00640 1 --2- 172.21.15.60:0/732998384 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fbb1406b990 0x7fbb1406de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:07.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:07.809+0000 7fbb33e00640 1 --2- 172.21.15.60:0/732998384 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fbb1406b990 0x7fbb1406de50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fbb3414f580 tx=0x7fbb20009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:07.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:07.813+0000 7fbb30c00640 1 -- 172.21.15.60:0/732998384 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fbb2803f070 con 0x7fbb3406c310 2024-04-14T23:49:08.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.017+0000 7fbb3a400640 1 -- 172.21.15.60:0/732998384 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbb34154160 con 0x7fbb3406c310 2024-04-14T23:49:08.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.019+0000 7fbb30c00640 1 -- 172.21.15.60:0/732998384 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16605 (secure 0 0 0) 0x7fbb2807ad40 con 0x7fbb3406c310 2024-04-14T23:49:08.025 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:49:08.025 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":21,"fsid":"5847db56-a76d-4486-b563-d83f1186af92","created":"2024-04-14T23:48:23.900003+0000","modified":"2024-04-14T23:48:44.219352+0000","last_up_change":"2024-04-14T23:48:34.203573+0000","last_in_change":"2024-04-14T23:48:26.345499+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-14T23:48:36.247372+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-14T23:48:40.274195+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-14T23:48:41.722183+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b2b2b755-7f42-4f9b-af9a-a178f7cc5a52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6817","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6818","nonce":2536528895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6822","nonce":2536528895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6827","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6828","nonce":2536528895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6825","nonce":2536528895}]},"public_addr":"172.21.15.60:6818/2536528895","cluster_addr":"172.21.15.60:6822/2536528895","heartbeat_back_addr":"172.21.15.60:6828/2536528895","heartbeat_front_addr":"172.21.15.60:6825/2536528895","state":["exists","up"]},{"osd":1,"uuid":"321418cd-5ab7-41a5-b9c8-b3fc12f68ebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6801","nonce":3115695676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6803","nonce":3115695676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6807","nonce":3115695676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6805","nonce":3115695676}]},"public_addr":"172.21.15.60:6801/3115695676","cluster_addr":"172.21.15.60:6803/3115695676","heartbeat_back_addr":"172.21.15.60:6807/3115695676","heartbeat_front_addr":"172.21.15.60:6805/3115695676","state":["exists","up"]},{"osd":2,"uuid":"07874e9f-3a38-41dc-be9f-2b75e84e53d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6819","nonce":3475168698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6824","nonce":3475168698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6831","nonce":3475168698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6829","nonce":3475168698}]},"public_addr":"172.21.15.60:6819/3475168698","cluster_addr":"172.21.15.60:6824/3475168698","heartbeat_back_addr":"172.21.15.60:6831/3475168698","heartbeat_front_addr":"172.21.15.60:6829/3475168698","state":["exists","up"]},{"osd":3,"uuid":"7e4ef6c9-b17a-47c6-a320-f5d16a6fe05e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6809","nonce":1730813674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6811","nonce":1730813674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6815","nonce":1730813674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6813","nonce":1730813674}]},"public_addr":"172.21.15.60:6809/1730813674","cluster_addr":"172.21.15.60:6811/1730813674","heartbeat_back_addr":"172.21.15.60:6815/1730813674","heartbeat_front_addr":"172.21.15.60:6813/1730813674","state":["exists","up"]},{"osd":4,"uuid":"759a3efc-dade-44c5-b57a-1623499aba50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6805","nonce":1464587038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6810","nonce":1464587038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6819","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6822","nonce":1464587038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6813","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6816","nonce":1464587038}]},"public_addr":"172.21.15.186:6805/1464587038","cluster_addr":"172.21.15.186:6810/1464587038","heartbeat_back_addr":"172.21.15.186:6822/1464587038","heartbeat_front_addr":"172.21.15.186:6816/1464587038","state":["exists","up"]},{"osd":5,"uuid":"24cd58c1-0cf1-4077-ab33-84c39c6907ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6801","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6803","nonce":1697098067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6809","nonce":1697098067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6820","nonce":1697098067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6815","nonce":1697098067}]},"public_addr":"172.21.15.186:6803/1697098067","cluster_addr":"172.21.15.186:6809/1697098067","heartbeat_back_addr":"172.21.15.186:6820/1697098067","heartbeat_front_addr":"172.21.15.186:6815/1697098067","state":["exists","up"]},{"osd":6,"uuid":"6add38f1-65e9-4b0d-927a-4ab77f5e2323","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6804","nonce":1837108559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6807","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6811","nonce":1837108559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6823","nonce":1837108559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6817","nonce":1837108559}]},"public_addr":"172.21.15.186:6804/1837108559","cluster_addr":"172.21.15.186:6811/1837108559","heartbeat_back_addr":"172.21.15.186:6823/1837108559","heartbeat_front_addr":"172.21.15.186:6817/1837108559","state":["exists","up"]},{"osd":7,"uuid":"a2326b6f-7601-4dda-a456-67c443ea5929","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6825","nonce":842537875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6827","nonce":842537875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6831","nonce":842537875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6829","nonce":842537875}]},"public_addr":"172.21.15.186:6825/842537875","cluster_addr":"172.21.15.186:6827/842537875","heartbeat_back_addr":"172.21.15.186:6831/842537875","heartbeat_front_addr":"172.21.15.186:6829/842537875","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.975583+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.976806+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.722725+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.763705+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.502140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.528972+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.517961+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.513350+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-14T23:49:08.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.022+0000 7fbb3a400640 1 -- 172.21.15.60:0/732998384 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fbb1406b990 msgr2=0x7fbb1406de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:08.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.022+0000 7fbb3a400640 1 --2- 172.21.15.60:0/732998384 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fbb1406b990 0x7fbb1406de50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fbb3414f580 tx=0x7fbb20009190 comp rx=0 tx=0).stop 2024-04-14T23:49:08.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.022+0000 7fbb3a400640 1 -- 172.21.15.60:0/732998384 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb3406c310 msgr2=0x7fbb3415bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:08.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.022+0000 7fbb3a400640 1 --2- 172.21.15.60:0/732998384 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb3406c310 0x7fbb3415bd70 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fbb280098c0 tx=0x7fbb280046c0 comp rx=0 tx=0).stop 2024-04-14T23:49:08.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.022+0000 7fbb3a400640 1 -- 172.21.15.60:0/732998384 shutdown_connections 2024-04-14T23:49:08.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.022+0000 7fbb3a400640 1 --2- 172.21.15.60:0/732998384 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fbb1406b990 0x7fbb1406de50 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:08.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.022+0000 7fbb3a400640 1 --2- 172.21.15.60:0/732998384 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbb34154cf0 0x7fbb3414aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:08.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.022+0000 7fbb3a400640 1 --2- 172.21.15.60:0/732998384 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb340cb120 0x7fbb341f3c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:08.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.022+0000 7fbb3a400640 1 --2- 172.21.15.60:0/732998384 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb3406c310 0x7fbb3415bd70 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:08.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.022+0000 7fbb3a400640 1 -- 172.21.15.60:0/732998384 >> 172.21.15.60:0/732998384 conn(0x7fbb340670f0 msgr2=0x7fbb340c5740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:49:08.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.022+0000 7fbb3a400640 1 -- 172.21.15.60:0/732998384 shutdown_connections 2024-04-14T23:49:08.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.022+0000 7fbb3a400640 1 -- 172.21.15.60:0/732998384 wait complete. 2024-04-14T23:49:08.049 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-14T23:48:36.247372+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-14T23:48:40.274195+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-14T23:48:41.722183+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-14T23:49:08.049 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-14T23:49:08.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.276+0000 7fcd7cc00640 1 Processor -- start 2024-04-14T23:49:08.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.276+0000 7fcd7cc00640 1 -- start start 2024-04-14T23:49:08.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.278+0000 7fcd7cc00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fcd780c63c0 0x7fcd780c6b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:08.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.278+0000 7fcd7cc00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd781530d0 con 0x7fcd78152cf0 2024-04-14T23:49:08.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.278+0000 7fcd7cc00640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd780751f0 con 0x7fcd7815c9b0 2024-04-14T23:49:08.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.278+0000 7fcd7cc00640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fcd780c8370 con 0x7fcd780c63c0 2024-04-14T23:49:08.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.278+0000 7fcd75e00640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fcd78152cf0 0x7fcd7815c2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33906/0 (socket says 172.21.15.60:33906) 2024-04-14T23:49:08.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.278+0000 7fcd75e00640 1 -- 172.21.15.60:0/3043718305 learned_addr learned my addr 172.21.15.60:0/3043718305 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:49:08.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.278+0000 7fcd75400640 1 --2- 172.21.15.60:0/3043718305 >> v2:172.21.15.60:3300/0 conn(0x7fcd780c63c0 0x7fcd780c6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:08.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.279+0000 7fcd74a00640 1 -- 172.21.15.60:0/3043718305 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2581671797 0 0) 0x7fcd780751f0 con 0x7fcd7815c9b0 2024-04-14T23:49:08.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.279+0000 7fcd75400640 1 -- 172.21.15.60:0/3043718305 >> v1:172.21.15.186:6789/0 conn(0x7fcd7815c9b0 legacy=0x7fcd780c70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:08.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.279+0000 7fcd75400640 1 -- 172.21.15.60:0/3043718305 >> v1:172.21.15.60:6789/0 conn(0x7fcd78152cf0 legacy=0x7fcd7815c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:08.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.279+0000 7fcd75400640 1 -- 172.21.15.60:0/3043718305 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd780c3810 con 0x7fcd780c63c0 2024-04-14T23:49:08.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.280+0000 7fcd75400640 1 --2- 172.21.15.60:0/3043718305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd780c63c0 0x7fcd780c6b70 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fcd600097a0 tx=0x7fcd6002b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c3dbe2d022bc7c server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:08.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.280+0000 7fcd74a00640 1 -- 172.21.15.60:0/3043718305 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd60038070 con 0x7fcd780c63c0 2024-04-14T23:49:08.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.280+0000 7fcd74a00640 1 -- 172.21.15.60:0/3043718305 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd60031700 con 0x7fcd780c63c0 2024-04-14T23:49:08.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.280+0000 7fcd74a00640 1 -- 172.21.15.60:0/3043718305 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd60031ad0 con 0x7fcd780c63c0 2024-04-14T23:49:08.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.281+0000 7fcd7cc00640 1 -- 172.21.15.60:0/3043718305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd780c63c0 msgr2=0x7fcd780c6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:08.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.281+0000 7fcd7cc00640 1 --2- 172.21.15.60:0/3043718305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd780c63c0 0x7fcd780c6b70 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fcd600097a0 tx=0x7fcd6002b820 comp rx=0 tx=0).stop 2024-04-14T23:49:08.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.281+0000 7fcd7cc00640 1 -- 172.21.15.60:0/3043718305 shutdown_connections 2024-04-14T23:49:08.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.281+0000 7fcd7cc00640 1 --2- 172.21.15.60:0/3043718305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd780c63c0 0x7fcd780c6b70 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:08.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.281+0000 7fcd7cc00640 1 -- 172.21.15.60:0/3043718305 >> 172.21.15.60:0/3043718305 conn(0x7fcd780cc170 msgr2=0x7fcd780cc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:49:08.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.281+0000 7fcd7cc00640 1 -- 172.21.15.60:0/3043718305 shutdown_connections 2024-04-14T23:49:08.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.281+0000 7fcd7cc00640 1 -- 172.21.15.60:0/3043718305 wait complete. 2024-04-14T23:49:08.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.283+0000 7fcd7cc00640 1 Processor -- start 2024-04-14T23:49:08.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.283+0000 7fcd7cc00640 1 -- start start 2024-04-14T23:49:08.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.284+0000 7fcd7cc00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd780c63c0 0x7fcd780c6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:08.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.284+0000 7fcd7cc00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcd78152cf0 0x7fcd780cb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:08.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.284+0000 7fcd7cc00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcd7815c9b0 0x7fcd7813f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:08.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.284+0000 7fcd7cc00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fcd780c88f0 con 0x7fcd780c63c0 2024-04-14T23:49:08.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.284+0000 7fcd7cc00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcd780c8610 con 0x7fcd78152cf0 2024-04-14T23:49:08.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.284+0000 7fcd7cc00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fcd780c77c0 con 0x7fcd7815c9b0 2024-04-14T23:49:08.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.284+0000 7fcd75e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd780c63c0 0x7fcd780c6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:08.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.284+0000 7fcd75400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcd78152cf0 0x7fcd780cb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:08.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.284+0000 7fcd75e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd780c63c0 0x7fcd780c6ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38482/0 (socket says 172.21.15.60:38482) 2024-04-14T23:49:08.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.284+0000 7fcd75e00640 1 -- 172.21.15.60:0/2119165814 learned_addr learned my addr 172.21.15.60:0/2119165814 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:49:08.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.284+0000 7fcd76800640 1 --2- 172.21.15.60:0/2119165814 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcd7815c9b0 0x7fcd7813f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:08.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.285+0000 7fcd75400640 1 -- 172.21.15.60:0/2119165814 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcd7815c9b0 msgr2=0x7fcd7813f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:08.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.285+0000 7fcd75400640 1 --2- 172.21.15.60:0/2119165814 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcd7815c9b0 0x7fcd7813f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:08.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.285+0000 7fcd75400640 1 -- 172.21.15.60:0/2119165814 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd780c63c0 msgr2=0x7fcd780c6ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:08.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.285+0000 7fcd75400640 1 --2- 172.21.15.60:0/2119165814 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd780c63c0 0x7fcd780c6ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:08.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.285+0000 7fcd75400640 1 -- 172.21.15.60:0/2119165814 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd78139880 con 0x7fcd78152cf0 2024-04-14T23:49:08.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.285+0000 7fcd75400640 1 --2- 172.21.15.60:0/2119165814 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcd78152cf0 0x7fcd780cb7c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fcd60033000 tx=0x7fcd600046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:08.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.286+0000 7fcd6e000640 1 -- 172.21.15.60:0/2119165814 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd60038040 con 0x7fcd78152cf0 2024-04-14T23:49:08.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.286+0000 7fcd6e000640 1 -- 172.21.15.60:0/2119165814 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd60004230 con 0x7fcd78152cf0 2024-04-14T23:49:08.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.286+0000 7fcd6e000640 1 -- 172.21.15.60:0/2119165814 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd600337a0 con 0x7fcd78152cf0 2024-04-14T23:49:08.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.286+0000 7fcd7cc00640 1 -- 172.21.15.60:0/2119165814 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd7813e700 con 0x7fcd78152cf0 2024-04-14T23:49:08.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.286+0000 7fcd7cc00640 1 -- 172.21.15.60:0/2119165814 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd781512a0 con 0x7fcd78152cf0 2024-04-14T23:49:08.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.287+0000 7fcd7cc00640 1 -- 172.21.15.60:0/2119165814 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd7813e230 con 0x7fcd78152cf0 2024-04-14T23:49:08.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.288+0000 7fcd6e000640 1 -- 172.21.15.60:0/2119165814 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7fcd60033940 con 0x7fcd78152cf0 2024-04-14T23:49:08.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.289+0000 7fcd6e000640 1 --2- 172.21.15.60:0/2119165814 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fcd5406b9c0 0x7fcd5406de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:08.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.289+0000 7fcd6e000640 1 -- 172.21.15.60:0/2119165814 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fcd600aca80 con 0x7fcd78152cf0 2024-04-14T23:49:08.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.289+0000 7fcd75e00640 1 --2- 172.21.15.60:0/2119165814 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fcd5406b9c0 0x7fcd5406de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:08.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.290+0000 7fcd75e00640 1 --2- 172.21.15.60:0/2119165814 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fcd5406b9c0 0x7fcd5406de80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fcd68000f60 tx=0x7fcd68007ec0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:08.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.294+0000 7fcd6e000640 1 -- 172.21.15.60:0/2119165814 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fcd600aee20 con 0x7fcd78152cf0 2024-04-14T23:49:08.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.499+0000 7fcd7cc00640 1 -- 172.21.15.60:0/2119165814 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fcd78076970 con 0x7fcd78152cf0 2024-04-14T23:49:08.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.500+0000 7fcd6e000640 1 -- 172.21.15.60:0/2119165814 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7fcd60030070 con 0x7fcd78152cf0 2024-04-14T23:49:08.505 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2024-04-14T23:49:08.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.502+0000 7fcd7cc00640 1 -- 172.21.15.60:0/2119165814 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fcd5406b9c0 msgr2=0x7fcd5406de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:08.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.502+0000 7fcd7cc00640 1 --2- 172.21.15.60:0/2119165814 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fcd5406b9c0 0x7fcd5406de80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fcd68000f60 tx=0x7fcd68007ec0 comp rx=0 tx=0).stop 2024-04-14T23:49:08.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.503+0000 7fcd7cc00640 1 -- 172.21.15.60:0/2119165814 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcd78152cf0 msgr2=0x7fcd780cb7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:08.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.503+0000 7fcd7cc00640 1 --2- 172.21.15.60:0/2119165814 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcd78152cf0 0x7fcd780cb7c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fcd60033000 tx=0x7fcd600046c0 comp rx=0 tx=0).stop 2024-04-14T23:49:08.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.503+0000 7fcd7cc00640 1 -- 172.21.15.60:0/2119165814 shutdown_connections 2024-04-14T23:49:08.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.503+0000 7fcd7cc00640 1 --2- 172.21.15.60:0/2119165814 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fcd5406b9c0 0x7fcd5406de80 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:08.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.503+0000 7fcd7cc00640 1 --2- 172.21.15.60:0/2119165814 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcd7815c9b0 0x7fcd7813f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:08.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.503+0000 7fcd7cc00640 1 --2- 172.21.15.60:0/2119165814 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcd78152cf0 0x7fcd780cb7c0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:08.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.503+0000 7fcd7cc00640 1 --2- 172.21.15.60:0/2119165814 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd780c63c0 0x7fcd780c6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:08.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.503+0000 7fcd7cc00640 1 -- 172.21.15.60:0/2119165814 >> 172.21.15.60:0/2119165814 conn(0x7fcd780cc170 msgr2=0x7fcd78074760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:49:08.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.503+0000 7fcd7cc00640 1 -- 172.21.15.60:0/2119165814 shutdown_connections 2024-04-14T23:49:08.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.503+0000 7fcd7cc00640 1 -- 172.21.15.60:0/2119165814 wait complete. 2024-04-14T23:49:08.528 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-14T23:49:08.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.754+0000 7f44b4800640 1 Processor -- start 2024-04-14T23:49:08.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.754+0000 7f44b4800640 1 -- start start 2024-04-14T23:49:08.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.755+0000 7f44b4800640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f44ac152cf0 0x7f44ac15c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:08.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.755+0000 7f44b4800640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f44ac0c63c0 0x7f44ac15c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:08.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.755+0000 7f44b4800640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44ac1530d0 con 0x7f44ac159450 2024-04-14T23:49:08.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.755+0000 7f44b4800640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f44ac0751f0 con 0x7f44ac0c63c0 2024-04-14T23:49:08.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.755+0000 7f44b4800640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f44ac0c8370 con 0x7f44ac152cf0 2024-04-14T23:49:08.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.756+0000 7f44b1400640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f44ac0c63c0 0x7f44ac15c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:08.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.756+0000 7f44b1e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f44ac152cf0 0x7f44ac15c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:08.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.756+0000 7f44b1e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f44ac152cf0 0x7f44ac15c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.60:48448/0 (socket says 172.21.15.60:48448) 2024-04-14T23:49:08.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.756+0000 7f44b1400640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f44ac0c63c0 0x7f44ac15c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38494/0 (socket says 172.21.15.60:38494) 2024-04-14T23:49:08.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.756+0000 7f44b1e00640 1 -- 172.21.15.60:0/2067606507 learned_addr learned my addr 172.21.15.60:0/2067606507 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:49:08.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.756+0000 7f44b2800640 1 --1- 172.21.15.60:0/2067606507 >> v1:172.21.15.186:6789/0 conn(0x7f44ac159450 0x7f44ac0c3f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.60:44652/0 (socket says 172.21.15.60:44652) 2024-04-14T23:49:08.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.756+0000 7f44b1e00640 1 -- 172.21.15.60:0/2067606507 >> v2:172.21.15.60:3300/0 conn(0x7f44ac0c63c0 msgr2=0x7f44ac15c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:49:08.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.756+0000 7f44b1e00640 1 --2- 172.21.15.60:0/2067606507 >> v2:172.21.15.60:3300/0 conn(0x7f44ac0c63c0 0x7f44ac15c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:08.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.756+0000 7f44b1e00640 1 -- 172.21.15.60:0/2067606507 >> v1:172.21.15.186:6789/0 conn(0x7f44ac159450 legacy=0x7f44ac0c3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:08.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.756+0000 7f44b1e00640 1 -- 172.21.15.60:0/2067606507 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44ac0c3810 con 0x7f44ac152cf0 2024-04-14T23:49:08.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.756+0000 7f44b0a00640 1 -- 172.21.15.60:0/2067606507 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1543949668 0 0) 0x7f44ac1530d0 con 0x7f44ac159450 2024-04-14T23:49:08.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.757+0000 7f44b1e00640 1 --2- 172.21.15.60:0/2067606507 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f44ac152cf0 0x7f44ac15c2a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f44a8009e50 tx=0x7f44a802bff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26929fc2ae236096 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:08.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.758+0000 7f44b0a00640 1 -- 172.21.15.60:0/2067606507 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a802cb90 con 0x7f44ac152cf0 2024-04-14T23:49:08.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.758+0000 7f44b0a00640 1 -- 172.21.15.60:0/2067606507 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44a802cd30 con 0x7f44ac152cf0 2024-04-14T23:49:08.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.758+0000 7f44b0a00640 1 -- 172.21.15.60:0/2067606507 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a8034610 con 0x7f44ac152cf0 2024-04-14T23:49:08.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.758+0000 7f44b4800640 1 -- 172.21.15.60:0/2067606507 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f44ac152cf0 msgr2=0x7f44ac15c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:08.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.758+0000 7f44b4800640 1 --2- 172.21.15.60:0/2067606507 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f44ac152cf0 0x7f44ac15c2a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f44a8009e50 tx=0x7f44a802bff0 comp rx=0 tx=0).stop 2024-04-14T23:49:08.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.758+0000 7f44b4800640 1 -- 172.21.15.60:0/2067606507 shutdown_connections 2024-04-14T23:49:08.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.758+0000 7f44b4800640 1 --2- 172.21.15.60:0/2067606507 >> v2:172.21.15.60:3300/0 conn(0x7f44ac0c63c0 0x7f44ac15c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:08.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.758+0000 7f44b4800640 1 --2- 172.21.15.60:0/2067606507 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f44ac152cf0 0x7f44ac15c2a0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:08.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.758+0000 7f44b4800640 1 -- 172.21.15.60:0/2067606507 >> 172.21.15.60:0/2067606507 conn(0x7f44ac0cc170 msgr2=0x7f44ac0cc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:49:08.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.758+0000 7f44b4800640 1 -- 172.21.15.60:0/2067606507 shutdown_connections 2024-04-14T23:49:08.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.759+0000 7f44b4800640 1 -- 172.21.15.60:0/2067606507 wait complete. 2024-04-14T23:49:08.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.760+0000 7f44b4800640 1 Processor -- start 2024-04-14T23:49:08.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.760+0000 7f44b4800640 1 -- start start 2024-04-14T23:49:08.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.761+0000 7f44b4800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f44ac0c63c0 0x7f44ac072cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:08.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.761+0000 7f44b4800640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f44ac152cf0 0x7f44ac071bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:08.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.761+0000 7f44b4800640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44ac159450 0x7f44ac1450b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:08.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.761+0000 7f44b4800640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f44ac0c88f0 con 0x7f44ac159450 2024-04-14T23:49:08.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.761+0000 7f44b4800640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f44ac0c8610 con 0x7f44ac0c63c0 2024-04-14T23:49:08.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.761+0000 7f44b4800640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f44ac0c4690 con 0x7f44ac152cf0 2024-04-14T23:49:08.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.762+0000 7f44b2800640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44ac159450 0x7f44ac1450b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:08.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.762+0000 7f44b1e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f44ac0c63c0 0x7f44ac072cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:08.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.762+0000 7f44b1e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f44ac0c63c0 0x7f44ac072cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:57668/0 (socket says 172.21.15.60:57668) 2024-04-14T23:49:08.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.762+0000 7f44b1e00640 1 -- 172.21.15.60:0/3350509134 learned_addr learned my addr 172.21.15.60:0/3350509134 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:49:08.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.762+0000 7f44b1400640 1 --2- 172.21.15.60:0/3350509134 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f44ac152cf0 0x7f44ac071bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:08.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.762+0000 7f44b1e00640 1 -- 172.21.15.60:0/3350509134 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f44ac152cf0 msgr2=0x7f44ac071bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:08.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.762+0000 7f44b1e00640 1 --2- 172.21.15.60:0/3350509134 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f44ac152cf0 0x7f44ac071bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:08.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.762+0000 7f44b1e00640 1 -- 172.21.15.60:0/3350509134 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44ac159450 msgr2=0x7f44ac1450b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:08.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.762+0000 7f44b1e00640 1 --2- 172.21.15.60:0/3350509134 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44ac159450 0x7f44ac1450b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:08.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.762+0000 7f44b1e00640 1 -- 172.21.15.60:0/3350509134 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44ac072600 con 0x7f44ac0c63c0 2024-04-14T23:49:08.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.762+0000 7f44b1e00640 1 --2- 172.21.15.60:0/3350509134 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f44ac0c63c0 0x7f44ac072cc0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f44a8009f70 tx=0x7f44a8004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:08.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.763+0000 7f44a2000640 1 -- 172.21.15.60:0/3350509134 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a8043020 con 0x7f44ac0c63c0 2024-04-14T23:49:08.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.763+0000 7f44a2000640 1 -- 172.21.15.60:0/3350509134 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44a803e450 con 0x7f44ac0c63c0 2024-04-14T23:49:08.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.763+0000 7f44b4800640 1 -- 172.21.15.60:0/3350509134 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44ac15b0a0 con 0x7f44ac0c63c0 2024-04-14T23:49:08.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.763+0000 7f44a2000640 1 -- 172.21.15.60:0/3350509134 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a8009510 con 0x7f44ac0c63c0 2024-04-14T23:49:08.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.763+0000 7f44b4800640 1 -- 172.21.15.60:0/3350509134 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44ac15a2c0 con 0x7f44ac0c63c0 2024-04-14T23:49:08.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.764+0000 7f44b4800640 1 -- 172.21.15.60:0/3350509134 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44ac068dd0 con 0x7f44ac0c63c0 2024-04-14T23:49:08.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.765+0000 7f44a2000640 1 -- 172.21.15.60:0/3350509134 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f44a8005d10 con 0x7f44ac0c63c0 2024-04-14T23:49:08.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.766+0000 7f44a2000640 1 --2- 172.21.15.60:0/3350509134 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f449006b990 0x7f449006de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:08.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.766+0000 7f44a2000640 1 -- 172.21.15.60:0/3350509134 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f44a80acda0 con 0x7f44ac0c63c0 2024-04-14T23:49:08.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.766+0000 7f44b1400640 1 --2- 172.21.15.60:0/3350509134 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f449006b990 0x7f449006de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:08.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.767+0000 7f44b1400640 1 --2- 172.21.15.60:0/3350509134 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f449006b990 0x7f449006de50 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f44ac06ce40 tx=0x7f449c00f040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:08.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.771+0000 7f44a2000640 1 -- 172.21.15.60:0/3350509134 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f44ac068dd0 con 0x7f44ac0c63c0 2024-04-14T23:49:08.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.970+0000 7f44b4800640 1 -- 172.21.15.60:0/3350509134 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f44ac14cd70 con 0x7f44ac0c63c0 2024-04-14T23:49:08.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.971+0000 7f44a2000640 1 -- 172.21.15.60:0/3350509134 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7f44ac14cd70 con 0x7f44ac0c63c0 2024-04-14T23:49:08.976 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2024-04-14T23:49:08.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.973+0000 7f44b4800640 1 -- 172.21.15.60:0/3350509134 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f449006b990 msgr2=0x7f449006de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:08.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.973+0000 7f44b4800640 1 --2- 172.21.15.60:0/3350509134 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f449006b990 0x7f449006de50 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f44ac06ce40 tx=0x7f449c00f040 comp rx=0 tx=0).stop 2024-04-14T23:49:08.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.973+0000 7f44b4800640 1 -- 172.21.15.60:0/3350509134 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f44ac0c63c0 msgr2=0x7f44ac072cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:08.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.973+0000 7f44b4800640 1 --2- 172.21.15.60:0/3350509134 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f44ac0c63c0 0x7f44ac072cc0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f44a8009f70 tx=0x7f44a8004a40 comp rx=0 tx=0).stop 2024-04-14T23:49:08.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.973+0000 7f44b4800640 1 -- 172.21.15.60:0/3350509134 shutdown_connections 2024-04-14T23:49:08.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.973+0000 7f44b4800640 1 --2- 172.21.15.60:0/3350509134 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f449006b990 0x7f449006de50 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:08.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.973+0000 7f44b4800640 1 --2- 172.21.15.60:0/3350509134 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44ac159450 0x7f44ac1450b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:08.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.973+0000 7f44b4800640 1 --2- 172.21.15.60:0/3350509134 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f44ac152cf0 0x7f44ac071bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:08.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.973+0000 7f44b4800640 1 --2- 172.21.15.60:0/3350509134 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f44ac0c63c0 0x7f44ac072cc0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:08.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.973+0000 7f44b4800640 1 -- 172.21.15.60:0/3350509134 >> 172.21.15.60:0/3350509134 conn(0x7f44ac0cc170 msgr2=0x7f44ac0cc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:49:08.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.973+0000 7f44b4800640 1 -- 172.21.15.60:0/3350509134 shutdown_connections 2024-04-14T23:49:08.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:08.973+0000 7f44b4800640 1 -- 172.21.15.60:0/3350509134 wait complete. 2024-04-14T23:49:08.996 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-14T23:49:09.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.213+0000 7f1ef2000640 1 Processor -- start 2024-04-14T23:49:09.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.213+0000 7f1ef2000640 1 -- start start 2024-04-14T23:49:09.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.215+0000 7f1ef2000640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1eec158610 0x7f1eec0b1740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:09.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.215+0000 7f1ef2000640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1eec0c3400 0x7f1eec06eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:09.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.215+0000 7f1ef2000640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1eec0b1fd0 con 0x7f1eec06f0a0 2024-04-14T23:49:09.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.215+0000 7f1ef2000640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f1eec15b070 con 0x7f1eec158610 2024-04-14T23:49:09.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.215+0000 7f1ef2000640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f1eec15b350 con 0x7f1eec0c3400 2024-04-14T23:49:09.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.215+0000 7f1ef0c00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1eec158610 0x7f1eec0b1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:09.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.215+0000 7f1ef0c00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1eec158610 0x7f1eec0b1740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38512/0 (socket says 172.21.15.60:38512) 2024-04-14T23:49:09.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.215+0000 7f1ef0c00640 1 -- 172.21.15.60:0/2629949028 learned_addr learned my addr 172.21.15.60:0/2629949028 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:49:09.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.215+0000 7f1eebe00640 1 --2- 172.21.15.60:0/2629949028 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1eec0c3400 0x7f1eec06eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:09.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.216+0000 7f1ef0c00640 1 -- 172.21.15.60:0/2629949028 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1eec0c3400 msgr2=0x7f1eec06eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:49:09.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.216+0000 7f1ef0c00640 1 --2- 172.21.15.60:0/2629949028 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1eec0c3400 0x7f1eec06eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:09.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.216+0000 7f1ef0c00640 1 -- 172.21.15.60:0/2629949028 >> v1:172.21.15.60:6789/0 conn(0x7f1eec06f0a0 legacy=0x7f1eec0c6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:09.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.216+0000 7f1ef0c00640 1 -- 172.21.15.60:0/2629949028 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1eec15b630 con 0x7f1eec158610 2024-04-14T23:49:09.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.217+0000 7f1ef0c00640 1 --2- 172.21.15.60:0/2629949028 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1eec158610 0x7f1eec0b1740 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1ee0009870 tx=0x7f1ee002ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=588470df5c0c6080 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:09.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.217+0000 7f1eeb400640 1 -- 172.21.15.60:0/2629949028 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ee002f830 con 0x7f1eec158610 2024-04-14T23:49:09.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.217+0000 7f1eeb400640 1 -- 172.21.15.60:0/2629949028 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ee002f9d0 con 0x7f1eec158610 2024-04-14T23:49:09.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.217+0000 7f1eeb400640 1 -- 172.21.15.60:0/2629949028 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ee002fda0 con 0x7f1eec158610 2024-04-14T23:49:09.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.218+0000 7f1ef2000640 1 -- 172.21.15.60:0/2629949028 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1eec158610 msgr2=0x7f1eec0b1740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:09.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.218+0000 7f1ef2000640 1 --2- 172.21.15.60:0/2629949028 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1eec158610 0x7f1eec0b1740 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1ee0009870 tx=0x7f1ee002ba10 comp rx=0 tx=0).stop 2024-04-14T23:49:09.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.218+0000 7f1ef2000640 1 -- 172.21.15.60:0/2629949028 shutdown_connections 2024-04-14T23:49:09.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.218+0000 7f1ef2000640 1 --2- 172.21.15.60:0/2629949028 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1eec0c3400 0x7f1eec06eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:09.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.218+0000 7f1ef2000640 1 --2- 172.21.15.60:0/2629949028 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1eec158610 0x7f1eec0b1740 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:09.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.218+0000 7f1ef2000640 1 -- 172.21.15.60:0/2629949028 >> 172.21.15.60:0/2629949028 conn(0x7f1eec0cc2f0 msgr2=0x7f1eec0cc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:49:09.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.218+0000 7f1ef2000640 1 -- 172.21.15.60:0/2629949028 shutdown_connections 2024-04-14T23:49:09.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.218+0000 7f1ef2000640 1 -- 172.21.15.60:0/2629949028 wait complete. 2024-04-14T23:49:09.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.220+0000 7f1ef2000640 1 Processor -- start 2024-04-14T23:49:09.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.220+0000 7f1ef2000640 1 -- start start 2024-04-14T23:49:09.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.221+0000 7f1ef2000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1eec06f0a0 0x7f1eec072a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:09.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.221+0000 7f1ef2000640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1eec0c3400 0x7f1eec0c0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:09.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.221+0000 7f1ef2000640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1eec158610 0x7f1eec14acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:09.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.221+0000 7f1ef2000640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f1eec0b1c80 con 0x7f1eec158610 2024-04-14T23:49:09.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.221+0000 7f1ef2000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1eec15bbf0 con 0x7f1eec06f0a0 2024-04-14T23:49:09.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.221+0000 7f1ef2000640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f1eec071730 con 0x7f1eec0c3400 2024-04-14T23:49:09.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.221+0000 7f1eebe00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1eec0c3400 0x7f1eec0c0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:09.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.221+0000 7f1ef1600640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1eec158610 0x7f1eec14acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:09.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.221+0000 7f1eebe00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1eec0c3400 0x7f1eec0c0e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.60:48474/0 (socket says 172.21.15.60:48474) 2024-04-14T23:49:09.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.221+0000 7f1ef0c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1eec06f0a0 0x7f1eec072a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:09.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.221+0000 7f1eebe00640 1 -- 172.21.15.60:0/2988520919 learned_addr learned my addr 172.21.15.60:0/2988520919 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:49:09.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.221+0000 7f1eebe00640 1 -- 172.21.15.60:0/2988520919 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1eec06f0a0 msgr2=0x7f1eec072a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:09.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.221+0000 7f1eebe00640 1 --2- 172.21.15.60:0/2988520919 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1eec06f0a0 0x7f1eec072a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:09.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.221+0000 7f1eebe00640 1 -- 172.21.15.60:0/2988520919 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1eec158610 msgr2=0x7f1eec14acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:09.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.221+0000 7f1eebe00640 1 --2- 172.21.15.60:0/2988520919 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1eec158610 0x7f1eec14acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:09.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.221+0000 7f1eebe00640 1 -- 172.21.15.60:0/2988520919 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1eec1646a0 con 0x7f1eec0c3400 2024-04-14T23:49:09.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.222+0000 7f1eebe00640 1 --2- 172.21.15.60:0/2988520919 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1eec0c3400 0x7f1eec0c0e50 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1ed800e9e0 tx=0x7f1ed800eea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:09.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.222+0000 7f1ee8c00640 1 -- 172.21.15.60:0/2988520919 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ed8017070 con 0x7f1eec0c3400 2024-04-14T23:49:09.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.222+0000 7f1ee8c00640 1 -- 172.21.15.60:0/2988520919 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ed800c9d0 con 0x7f1eec0c3400 2024-04-14T23:49:09.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.222+0000 7f1ef2000640 1 -- 172.21.15.60:0/2988520919 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1eec1f4630 con 0x7f1eec0c3400 2024-04-14T23:49:09.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.222+0000 7f1ee8c00640 1 -- 172.21.15.60:0/2988520919 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ed800cdc0 con 0x7f1eec0c3400 2024-04-14T23:49:09.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.222+0000 7f1ef2000640 1 -- 172.21.15.60:0/2988520919 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1eec0c4de0 con 0x7f1eec0c3400 2024-04-14T23:49:09.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.223+0000 7f1ef2000640 1 -- 172.21.15.60:0/2988520919 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1eec04f540 con 0x7f1eec0c3400 2024-04-14T23:49:09.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.224+0000 7f1ee8c00640 1 -- 172.21.15.60:0/2988520919 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f1ed801a070 con 0x7f1eec0c3400 2024-04-14T23:49:09.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.225+0000 7f1ee8c00640 1 --2- 172.21.15.60:0/2988520919 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f1ecc06b990 0x7f1ecc06de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:09.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.225+0000 7f1ee8c00640 1 -- 172.21.15.60:0/2988520919 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f1ed808abd0 con 0x7f1eec0c3400 2024-04-14T23:49:09.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.226+0000 7f1ef0c00640 1 --2- 172.21.15.60:0/2988520919 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f1ecc06b990 0x7f1ecc06de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:09.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.226+0000 7f1ef0c00640 1 --2- 172.21.15.60:0/2988520919 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f1ecc06b990 0x7f1ecc06de50 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1ee0036000 tx=0x7f1ee0008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:09.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.230+0000 7f1ee8c00640 1 -- 172.21.15.60:0/2988520919 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1ed80590f0 con 0x7f1eec0c3400 2024-04-14T23:49:09.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.425+0000 7f1ef2000640 1 -- 172.21.15.60:0/2988520919 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f1eec070d10 con 0x7f1eec0c3400 2024-04-14T23:49:09.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.425+0000 7f1ee8c00640 1 -- 172.21.15.60:0/2988520919 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7f1ed805d190 con 0x7f1eec0c3400 2024-04-14T23:49:09.431 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2024-04-14T23:49:09.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.428+0000 7f1ef2000640 1 -- 172.21.15.60:0/2988520919 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f1ecc06b990 msgr2=0x7f1ecc06de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:09.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.428+0000 7f1ef2000640 1 --2- 172.21.15.60:0/2988520919 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f1ecc06b990 0x7f1ecc06de50 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1ee0036000 tx=0x7f1ee0008030 comp rx=0 tx=0).stop 2024-04-14T23:49:09.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.428+0000 7f1ef2000640 1 -- 172.21.15.60:0/2988520919 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1eec0c3400 msgr2=0x7f1eec0c0e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:09.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.428+0000 7f1ef2000640 1 --2- 172.21.15.60:0/2988520919 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1eec0c3400 0x7f1eec0c0e50 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1ed800e9e0 tx=0x7f1ed800eea0 comp rx=0 tx=0).stop 2024-04-14T23:49:09.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.428+0000 7f1ef2000640 1 -- 172.21.15.60:0/2988520919 shutdown_connections 2024-04-14T23:49:09.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.428+0000 7f1ef2000640 1 --2- 172.21.15.60:0/2988520919 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f1ecc06b990 0x7f1ecc06de50 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:09.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.428+0000 7f1ef2000640 1 --2- 172.21.15.60:0/2988520919 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1eec158610 0x7f1eec14acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:09.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.428+0000 7f1ef2000640 1 --2- 172.21.15.60:0/2988520919 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1eec0c3400 0x7f1eec0c0e50 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:09.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.428+0000 7f1ef2000640 1 --2- 172.21.15.60:0/2988520919 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1eec06f0a0 0x7f1eec072a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:09.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.428+0000 7f1ef2000640 1 -- 172.21.15.60:0/2988520919 >> 172.21.15.60:0/2988520919 conn(0x7f1eec0cc2f0 msgr2=0x7f1eec14e240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:49:09.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.428+0000 7f1ef2000640 1 -- 172.21.15.60:0/2988520919 shutdown_connections 2024-04-14T23:49:09.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.428+0000 7f1ef2000640 1 -- 172.21.15.60:0/2988520919 wait complete. 2024-04-14T23:49:09.454 INFO:tasks.mds_thrash:mds thrasher using random seed: 1713138549 2024-04-14T23:49:09.454 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-14T23:49:09.673 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.667+0000 7f16d7e00640 1 Processor -- start 2024-04-14T23:49:09.673 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.668+0000 7f16d7e00640 1 -- start start 2024-04-14T23:49:09.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.669+0000 7f16d7e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f16d80ca180 0x7f16d80c6f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:09.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.669+0000 7f16d7e00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f16d8072db0 0x7f16d8073230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:09.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.669+0000 7f16d7e00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16d8073770 con 0x7f16d80ca560 2024-04-14T23:49:09.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.669+0000 7f16d7e00640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f16d80c78f0 con 0x7f16d8072db0 2024-04-14T23:49:09.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.669+0000 7f16d7e00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f16d80c7480 con 0x7f16d80ca180 2024-04-14T23:49:09.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.670+0000 7f16cfe00640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f16d80ca560 0x7f16d806b1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33928/0 (socket says 172.21.15.60:33928) 2024-04-14T23:49:09.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.670+0000 7f16cfe00640 1 -- 172.21.15.60:0/825739888 learned_addr learned my addr 172.21.15.60:0/825739888 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:49:09.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.670+0000 7f16d6a00640 1 --2- 172.21.15.60:0/825739888 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f16d80ca180 0x7f16d80c6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:09.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.670+0000 7f16d7400640 1 --2- 172.21.15.60:0/825739888 >> v2:172.21.15.60:3300/0 conn(0x7f16d8072db0 0x7f16d8073230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:09.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.670+0000 7f16d6000640 1 -- 172.21.15.60:0/825739888 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3624777950 0 0) 0x7f16c0001110 con 0x7f16d80ca560 2024-04-14T23:49:09.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.670+0000 7f16d6a00640 1 -- 172.21.15.60:0/825739888 >> v2:172.21.15.60:3300/0 conn(0x7f16d8072db0 msgr2=0x7f16d8073230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:49:09.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.670+0000 7f16d6a00640 1 --2- 172.21.15.60:0/825739888 >> v2:172.21.15.60:3300/0 conn(0x7f16d8072db0 0x7f16d8073230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:09.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.670+0000 7f16d6a00640 1 -- 172.21.15.60:0/825739888 >> v1:172.21.15.60:6789/0 conn(0x7f16d80ca560 legacy=0x7f16d806b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:09.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.670+0000 7f16d6a00640 1 -- 172.21.15.60:0/825739888 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16d80c7b40 con 0x7f16d80ca180 2024-04-14T23:49:09.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.671+0000 7f16d6a00640 1 --2- 172.21.15.60:0/825739888 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f16d80ca180 0x7f16d80c6f40 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f16c80097a0 tx=0x7f16c802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a027c3f2a4d79b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:09.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.672+0000 7f16d6000640 1 -- 172.21.15.60:0/825739888 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c8035020 con 0x7f16d80ca180 2024-04-14T23:49:09.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.672+0000 7f16d6000640 1 -- 172.21.15.60:0/825739888 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16c8031830 con 0x7f16d80ca180 2024-04-14T23:49:09.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.672+0000 7f16d6000640 1 -- 172.21.15.60:0/825739888 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c8031c00 con 0x7f16d80ca180 2024-04-14T23:49:09.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.672+0000 7f16d7e00640 1 -- 172.21.15.60:0/825739888 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f16d80ca180 msgr2=0x7f16d80c6f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:09.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.672+0000 7f16d7e00640 1 --2- 172.21.15.60:0/825739888 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f16d80ca180 0x7f16d80c6f40 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f16c80097a0 tx=0x7f16c802b8d0 comp rx=0 tx=0).stop 2024-04-14T23:49:09.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.673+0000 7f16d7e00640 1 -- 172.21.15.60:0/825739888 shutdown_connections 2024-04-14T23:49:09.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.673+0000 7f16d7e00640 1 --2- 172.21.15.60:0/825739888 >> v2:172.21.15.60:3300/0 conn(0x7f16d8072db0 0x7f16d8073230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:09.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.673+0000 7f16d7e00640 1 --2- 172.21.15.60:0/825739888 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f16d80ca180 0x7f16d80c6f40 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:09.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.673+0000 7f16d7e00640 1 -- 172.21.15.60:0/825739888 >> 172.21.15.60:0/825739888 conn(0x7f16d806fdc0 msgr2=0x7f16d80701d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:49:09.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.673+0000 7f16d7e00640 1 -- 172.21.15.60:0/825739888 shutdown_connections 2024-04-14T23:49:09.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.673+0000 7f16d7e00640 1 -- 172.21.15.60:0/825739888 wait complete. 2024-04-14T23:49:09.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.675+0000 7f16d7e00640 1 Processor -- start 2024-04-14T23:49:09.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.675+0000 7f16d7e00640 1 -- start start 2024-04-14T23:49:09.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.676+0000 7f16d7e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16d8072db0 0x7f16d81f0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:09.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.676+0000 7f16d7e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16d80ca180 0x7f16d80c6500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:09.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.676+0000 7f16d7e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f16d80ca560 0x7f16d80c6a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:09.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.676+0000 7f16d7e00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f16d80c83e0 con 0x7f16d8072db0 2024-04-14T23:49:09.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.676+0000 7f16d7e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f16d80c8100 con 0x7f16d80ca180 2024-04-14T23:49:09.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.676+0000 7f16d7e00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f16d815b180 con 0x7f16d80ca560 2024-04-14T23:49:09.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.676+0000 7f16d6a00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16d8072db0 0x7f16d81f0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:09.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.677+0000 7f16d6a00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16d8072db0 0x7f16d81f0de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38544/0 (socket says 172.21.15.60:38544) 2024-04-14T23:49:09.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.677+0000 7f16d6a00640 1 -- 172.21.15.60:0/1310652236 learned_addr learned my addr 172.21.15.60:0/1310652236 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:49:09.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.677+0000 7f16cfe00640 1 --2- 172.21.15.60:0/1310652236 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16d80ca180 0x7f16d80c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:09.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.677+0000 7f16d7400640 1 --2- 172.21.15.60:0/1310652236 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f16d80ca560 0x7f16d80c6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:09.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.677+0000 7f16d6a00640 1 -- 172.21.15.60:0/1310652236 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f16d80ca560 msgr2=0x7f16d80c6a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:09.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.677+0000 7f16d6a00640 1 --2- 172.21.15.60:0/1310652236 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f16d80ca560 0x7f16d80c6a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:09.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.677+0000 7f16d6a00640 1 -- 172.21.15.60:0/1310652236 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16d80ca180 msgr2=0x7f16d80c6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:09.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.677+0000 7f16d6a00640 1 --2- 172.21.15.60:0/1310652236 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16d80ca180 0x7f16d80c6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:09.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.677+0000 7f16d6a00640 1 -- 172.21.15.60:0/1310652236 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16d81f1320 con 0x7f16d8072db0 2024-04-14T23:49:09.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.677+0000 7f16d6a00640 1 --2- 172.21.15.60:0/1310652236 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16d8072db0 0x7f16d81f0de0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f16c802bee0 tx=0x7f16c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:09.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.678+0000 7f16cea00640 1 -- 172.21.15.60:0/1310652236 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c8035040 con 0x7f16d8072db0 2024-04-14T23:49:09.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.678+0000 7f16cea00640 1 -- 172.21.15.60:0/1310652236 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16c803b920 con 0x7f16d8072db0 2024-04-14T23:49:09.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.678+0000 7f16cea00640 1 -- 172.21.15.60:0/1310652236 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c803bd10 con 0x7f16d8072db0 2024-04-14T23:49:09.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.678+0000 7f16d7e00640 1 -- 172.21.15.60:0/1310652236 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16d80c19f0 con 0x7f16d8072db0 2024-04-14T23:49:09.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.678+0000 7f16d7e00640 1 -- 172.21.15.60:0/1310652236 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16d80c1fb0 con 0x7f16d8072db0 2024-04-14T23:49:09.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.679+0000 7f16d7e00640 1 -- 172.21.15.60:0/1310652236 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16d804f540 con 0x7f16d8072db0 2024-04-14T23:49:09.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.680+0000 7f16cea00640 1 -- 172.21.15.60:0/1310652236 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f16c8038020 con 0x7f16d8072db0 2024-04-14T23:49:09.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.681+0000 7f16cea00640 1 --2- 172.21.15.60:0/1310652236 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f16b406b990 0x7f16b406de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:09.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.681+0000 7f16cea00640 1 -- 172.21.15.60:0/1310652236 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f16c80ac9c0 con 0x7f16d8072db0 2024-04-14T23:49:09.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.682+0000 7f16cfe00640 1 --2- 172.21.15.60:0/1310652236 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f16b406b990 0x7f16b406de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:09.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.682+0000 7f16cfe00640 1 --2- 172.21.15.60:0/1310652236 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f16b406b990 0x7f16b406de50 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f16d81f49e0 tx=0x7f16c0007de0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:09.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.687+0000 7f16cea00640 1 -- 172.21.15.60:0/1310652236 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f16c8031800 con 0x7f16d8072db0 2024-04-14T23:49:09.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.890+0000 7f16d7e00640 1 -- 172.21.15.60:0/1310652236 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f16d806db90 con 0x7f16d8072db0 2024-04-14T23:49:09.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.892+0000 7f16cea00640 1 -- 172.21.15.60:0/1310652236 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16605 (secure 0 0 0) 0x7f16c807ae60 con 0x7f16d8072db0 2024-04-14T23:49:09.898 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:49:09.898 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":21,"fsid":"5847db56-a76d-4486-b563-d83f1186af92","created":"2024-04-14T23:48:23.900003+0000","modified":"2024-04-14T23:48:44.219352+0000","last_up_change":"2024-04-14T23:48:34.203573+0000","last_in_change":"2024-04-14T23:48:26.345499+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-14T23:48:36.247372+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-14T23:48:40.274195+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-14T23:48:41.722183+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b2b2b755-7f42-4f9b-af9a-a178f7cc5a52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6817","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6818","nonce":2536528895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6822","nonce":2536528895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6827","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6828","nonce":2536528895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6825","nonce":2536528895}]},"public_addr":"172.21.15.60:6818/2536528895","cluster_addr":"172.21.15.60:6822/2536528895","heartbeat_back_addr":"172.21.15.60:6828/2536528895","heartbeat_front_addr":"172.21.15.60:6825/2536528895","state":["exists","up"]},{"osd":1,"uuid":"321418cd-5ab7-41a5-b9c8-b3fc12f68ebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6801","nonce":3115695676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6803","nonce":3115695676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6807","nonce":3115695676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6805","nonce":3115695676}]},"public_addr":"172.21.15.60:6801/3115695676","cluster_addr":"172.21.15.60:6803/3115695676","heartbeat_back_addr":"172.21.15.60:6807/3115695676","heartbeat_front_addr":"172.21.15.60:6805/3115695676","state":["exists","up"]},{"osd":2,"uuid":"07874e9f-3a38-41dc-be9f-2b75e84e53d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6819","nonce":3475168698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6824","nonce":3475168698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6831","nonce":3475168698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6829","nonce":3475168698}]},"public_addr":"172.21.15.60:6819/3475168698","cluster_addr":"172.21.15.60:6824/3475168698","heartbeat_back_addr":"172.21.15.60:6831/3475168698","heartbeat_front_addr":"172.21.15.60:6829/3475168698","state":["exists","up"]},{"osd":3,"uuid":"7e4ef6c9-b17a-47c6-a320-f5d16a6fe05e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6809","nonce":1730813674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6811","nonce":1730813674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6815","nonce":1730813674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6813","nonce":1730813674}]},"public_addr":"172.21.15.60:6809/1730813674","cluster_addr":"172.21.15.60:6811/1730813674","heartbeat_back_addr":"172.21.15.60:6815/1730813674","heartbeat_front_addr":"172.21.15.60:6813/1730813674","state":["exists","up"]},{"osd":4,"uuid":"759a3efc-dade-44c5-b57a-1623499aba50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6805","nonce":1464587038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6810","nonce":1464587038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6819","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6822","nonce":1464587038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6813","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6816","nonce":1464587038}]},"public_addr":"172.21.15.186:6805/1464587038","cluster_addr":"172.21.15.186:6810/1464587038","heartbeat_back_addr":"172.21.15.186:6822/1464587038","heartbeat_front_addr":"172.21.15.186:6816/1464587038","state":["exists","up"]},{"osd":5,"uuid":"24cd58c1-0cf1-4077-ab33-84c39c6907ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6801","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6803","nonce":1697098067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6809","nonce":1697098067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6820","nonce":1697098067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6815","nonce":1697098067}]},"public_addr":"172.21.15.186:6803/1697098067","cluster_addr":"172.21.15.186:6809/1697098067","heartbeat_back_addr":"172.21.15.186:6820/1697098067","heartbeat_front_addr":"172.21.15.186:6815/1697098067","state":["exists","up"]},{"osd":6,"uuid":"6add38f1-65e9-4b0d-927a-4ab77f5e2323","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6804","nonce":1837108559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6807","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6811","nonce":1837108559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6823","nonce":1837108559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6817","nonce":1837108559}]},"public_addr":"172.21.15.186:6804/1837108559","cluster_addr":"172.21.15.186:6811/1837108559","heartbeat_back_addr":"172.21.15.186:6823/1837108559","heartbeat_front_addr":"172.21.15.186:6817/1837108559","state":["exists","up"]},{"osd":7,"uuid":"a2326b6f-7601-4dda-a456-67c443ea5929","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6825","nonce":842537875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6827","nonce":842537875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6831","nonce":842537875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6829","nonce":842537875}]},"public_addr":"172.21.15.186:6825/842537875","cluster_addr":"172.21.15.186:6827/842537875","heartbeat_back_addr":"172.21.15.186:6831/842537875","heartbeat_front_addr":"172.21.15.186:6829/842537875","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.975583+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.976806+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.722725+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.763705+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.502140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.528972+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.517961+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.513350+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-14T23:49:09.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.894+0000 7f16d7e00640 1 -- 172.21.15.60:0/1310652236 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f16b406b990 msgr2=0x7f16b406de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:09.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.894+0000 7f16d7e00640 1 --2- 172.21.15.60:0/1310652236 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f16b406b990 0x7f16b406de50 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f16d81f49e0 tx=0x7f16c0007de0 comp rx=0 tx=0).stop 2024-04-14T23:49:09.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.894+0000 7f16d7e00640 1 -- 172.21.15.60:0/1310652236 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16d8072db0 msgr2=0x7f16d81f0de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:09.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.894+0000 7f16d7e00640 1 --2- 172.21.15.60:0/1310652236 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16d8072db0 0x7f16d81f0de0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f16c802bee0 tx=0x7f16c80046c0 comp rx=0 tx=0).stop 2024-04-14T23:49:09.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.895+0000 7f16d7e00640 1 -- 172.21.15.60:0/1310652236 shutdown_connections 2024-04-14T23:49:09.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.895+0000 7f16d7e00640 1 --2- 172.21.15.60:0/1310652236 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f16b406b990 0x7f16b406de50 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:09.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.895+0000 7f16d7e00640 1 --2- 172.21.15.60:0/1310652236 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f16d80ca560 0x7f16d80c6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:09.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.895+0000 7f16d7e00640 1 --2- 172.21.15.60:0/1310652236 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16d80ca180 0x7f16d80c6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:09.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.895+0000 7f16d7e00640 1 --2- 172.21.15.60:0/1310652236 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16d8072db0 0x7f16d81f0de0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:09.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.895+0000 7f16d7e00640 1 -- 172.21.15.60:0/1310652236 >> 172.21.15.60:0/1310652236 conn(0x7f16d806fdc0 msgr2=0x7f16d815b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:49:09.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.895+0000 7f16d7e00640 1 -- 172.21.15.60:0/1310652236 shutdown_connections 2024-04-14T23:49:09.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:09.895+0000 7f16d7e00640 1 -- 172.21.15.60:0/1310652236 wait complete. 2024-04-14T23:49:09.921 INFO:tasks.mds_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-14T23:48:36.247372+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-14T23:48:40.274195+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-14T23:48:41.722183+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-14T23:49:09.921 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-14T23:49:10.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.143+0000 7f7c4f000640 1 Processor -- start 2024-04-14T23:49:10.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.144+0000 7f7c4f000640 1 -- start start 2024-04-14T23:49:10.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.145+0000 7f7c4f000640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7c480c6d70 0x7f7c480c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:10.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.145+0000 7f7c4f000640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f7c480711a0 0x7f7c48071610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:10.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.145+0000 7f7c4f000640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c48071b50 con 0x7f7c4815b740 2024-04-14T23:49:10.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.145+0000 7f7c4f000640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f7c48070d10 con 0x7f7c480c6d70 2024-04-14T23:49:10.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.145+0000 7f7c4f000640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f7c480c76f0 con 0x7f7c480711a0 2024-04-14T23:49:10.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.146+0000 7f7c4d200640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7c480c6d70 0x7f7c480c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:10.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.146+0000 7f7c4d200640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7c480c6d70 0x7f7c480c71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38554/0 (socket says 172.21.15.60:38554) 2024-04-14T23:49:10.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.146+0000 7f7c4e600640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f7c480711a0 0x7f7c48071610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:10.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.146+0000 7f7c4d200640 1 -- 172.21.15.60:0/3942707515 learned_addr learned my addr 172.21.15.60:0/3942707515 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:49:10.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.146+0000 7f7c4e600640 1 -- 172.21.15.60:0/3942707515 >> v2:172.21.15.60:3300/0 conn(0x7f7c480c6d70 msgr2=0x7f7c480c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:49:10.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.146+0000 7f7c4e600640 1 --2- 172.21.15.60:0/3942707515 >> v2:172.21.15.60:3300/0 conn(0x7f7c480c6d70 0x7f7c480c71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:10.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.146+0000 7f7c4e600640 1 -- 172.21.15.60:0/3942707515 >> v1:172.21.15.60:6789/0 conn(0x7f7c4815b740 legacy=0x7f7c4806eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:10.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.146+0000 7f7c4e600640 1 -- 172.21.15.60:0/3942707515 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c4815bb20 con 0x7f7c480711a0 2024-04-14T23:49:10.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.146+0000 7f7c4d200640 1 --2- 172.21.15.60:0/3942707515 >> v2:172.21.15.60:3300/0 conn(0x7f7c480c6d70 0x7f7c480c71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:49:10.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.147+0000 7f7c4e600640 1 --2- 172.21.15.60:0/3942707515 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c480711a0 0x7f7c48071610 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7c4806e6d0 tx=0x7f7c4002bec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2140d5eeca3787d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:10.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.147+0000 7f7c47e00640 1 -- 172.21.15.60:0/3942707515 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c40039070 con 0x7f7c480711a0 2024-04-14T23:49:10.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.148+0000 7f7c47e00640 1 -- 172.21.15.60:0/3942707515 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c4002cc70 con 0x7f7c480711a0 2024-04-14T23:49:10.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.148+0000 7f7c47e00640 1 -- 172.21.15.60:0/3942707515 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c40034660 con 0x7f7c480711a0 2024-04-14T23:49:10.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.148+0000 7f7c4f000640 1 -- 172.21.15.60:0/3942707515 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c480711a0 msgr2=0x7f7c48071610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:10.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.148+0000 7f7c4f000640 1 --2- 172.21.15.60:0/3942707515 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c480711a0 0x7f7c48071610 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7c4806e6d0 tx=0x7f7c4002bec0 comp rx=0 tx=0).stop 2024-04-14T23:49:10.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.148+0000 7f7c4f000640 1 -- 172.21.15.60:0/3942707515 shutdown_connections 2024-04-14T23:49:10.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.148+0000 7f7c4f000640 1 --2- 172.21.15.60:0/3942707515 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c480711a0 0x7f7c48071610 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:10.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.148+0000 7f7c4f000640 1 --2- 172.21.15.60:0/3942707515 >> v2:172.21.15.60:3300/0 conn(0x7f7c480c6d70 0x7f7c480c71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:10.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.148+0000 7f7c4f000640 1 -- 172.21.15.60:0/3942707515 >> 172.21.15.60:0/3942707515 conn(0x7f7c480cc2f0 msgr2=0x7f7c480cc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:49:10.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.148+0000 7f7c4f000640 1 -- 172.21.15.60:0/3942707515 shutdown_connections 2024-04-14T23:49:10.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.149+0000 7f7c4f000640 1 -- 172.21.15.60:0/3942707515 wait complete. 2024-04-14T23:49:10.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.151+0000 7f7c4f000640 1 Processor -- start 2024-04-14T23:49:10.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.151+0000 7f7c4f000640 1 -- start start 2024-04-14T23:49:10.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.152+0000 7f7c4f000640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c480711a0 0x7f7c4815bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:10.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.152+0000 7f7c4f000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c480c6d70 0x7f7c481f3d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:10.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.152+0000 7f7c4f000640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7c4815b740 0x7f7c481f4250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:10.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.152+0000 7f7c4f000640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f7c4815e8d0 con 0x7f7c480711a0 2024-04-14T23:49:10.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.152+0000 7f7c4f000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7c4815e5f0 con 0x7f7c480c6d70 2024-04-14T23:49:10.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.152+0000 7f7c4f000640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f7c4815ebb0 con 0x7f7c4815b740 2024-04-14T23:49:10.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.152+0000 7f7c4dc00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c480711a0 0x7f7c4815bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:10.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.153+0000 7f7c4d200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c480c6d70 0x7f7c481f3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:10.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.153+0000 7f7c4dc00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c480711a0 0x7f7c4815bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38562/0 (socket says 172.21.15.60:38562) 2024-04-14T23:49:10.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.153+0000 7f7c4dc00640 1 -- 172.21.15.60:0/1891278663 learned_addr learned my addr 172.21.15.60:0/1891278663 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:49:10.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.153+0000 7f7c4e600640 1 --2- 172.21.15.60:0/1891278663 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7c4815b740 0x7f7c481f4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:10.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.153+0000 7f7c4d200640 1 -- 172.21.15.60:0/1891278663 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7c4815b740 msgr2=0x7f7c481f4250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:10.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.153+0000 7f7c4d200640 1 --2- 172.21.15.60:0/1891278663 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7c4815b740 0x7f7c481f4250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:10.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.153+0000 7f7c4d200640 1 -- 172.21.15.60:0/1891278663 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c480711a0 msgr2=0x7f7c4815bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:10.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.153+0000 7f7c4d200640 1 --2- 172.21.15.60:0/1891278663 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c480711a0 0x7f7c4815bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:10.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.153+0000 7f7c4d200640 1 -- 172.21.15.60:0/1891278663 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c481f4790 con 0x7f7c480c6d70 2024-04-14T23:49:10.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.153+0000 7f7c4d200640 1 --2- 172.21.15.60:0/1891278663 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c480c6d70 0x7f7c481f3d10 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7c3400e960 tx=0x7f7c3400ee20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:10.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.154+0000 7f7c45600640 1 -- 172.21.15.60:0/1891278663 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c3400ca00 con 0x7f7c480c6d70 2024-04-14T23:49:10.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.154+0000 7f7c45600640 1 -- 172.21.15.60:0/1891278663 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c3400cba0 con 0x7f7c480c6d70 2024-04-14T23:49:10.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.154+0000 7f7c45600640 1 -- 172.21.15.60:0/1891278663 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c34004030 con 0x7f7c480c6d70 2024-04-14T23:49:10.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.154+0000 7f7c4f000640 1 -- 172.21.15.60:0/1891278663 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c4815de20 con 0x7f7c480c6d70 2024-04-14T23:49:10.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.154+0000 7f7c4f000640 1 -- 172.21.15.60:0/1891278663 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c481578d0 con 0x7f7c480c6d70 2024-04-14T23:49:10.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.155+0000 7f7c4f000640 1 -- 172.21.15.60:0/1891278663 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c4804f540 con 0x7f7c480c6d70 2024-04-14T23:49:10.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.156+0000 7f7c45600640 1 -- 172.21.15.60:0/1891278663 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f7c340041d0 con 0x7f7c480c6d70 2024-04-14T23:49:10.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.157+0000 7f7c45600640 1 --2- 172.21.15.60:0/1891278663 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f7c2806b990 0x7f7c2806de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:10.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.157+0000 7f7c45600640 1 -- 172.21.15.60:0/1891278663 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f7c3408a880 con 0x7f7c480c6d70 2024-04-14T23:49:10.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.157+0000 7f7c4dc00640 1 --2- 172.21.15.60:0/1891278663 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f7c2806b990 0x7f7c2806de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:10.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.158+0000 7f7c4dc00640 1 --2- 172.21.15.60:0/1891278663 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f7c2806b990 0x7f7c2806de50 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c0017a0 tx=0x7f7c3c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:10.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.162+0000 7f7c45600640 1 -- 172.21.15.60:0/1891278663 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7c34017020 con 0x7f7c480c6d70 2024-04-14T23:49:10.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.366+0000 7f7c4f000640 1 -- 172.21.15.60:0/1891278663 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7c480c4c80 con 0x7f7c480c6d70 2024-04-14T23:49:10.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.367+0000 7f7c45600640 1 -- 172.21.15.60:0/1891278663 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7f7c34058d10 con 0x7f7c480c6d70 2024-04-14T23:49:10.372 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2024-04-14T23:49:10.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.369+0000 7f7c4f000640 1 -- 172.21.15.60:0/1891278663 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f7c2806b990 msgr2=0x7f7c2806de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:10.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.369+0000 7f7c4f000640 1 --2- 172.21.15.60:0/1891278663 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f7c2806b990 0x7f7c2806de50 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c0017a0 tx=0x7f7c3c009040 comp rx=0 tx=0).stop 2024-04-14T23:49:10.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.369+0000 7f7c4f000640 1 -- 172.21.15.60:0/1891278663 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c480c6d70 msgr2=0x7f7c481f3d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:10.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.369+0000 7f7c4f000640 1 --2- 172.21.15.60:0/1891278663 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c480c6d70 0x7f7c481f3d10 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7c3400e960 tx=0x7f7c3400ee20 comp rx=0 tx=0).stop 2024-04-14T23:49:10.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.369+0000 7f7c4f000640 1 -- 172.21.15.60:0/1891278663 shutdown_connections 2024-04-14T23:49:10.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.369+0000 7f7c4f000640 1 --2- 172.21.15.60:0/1891278663 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f7c2806b990 0x7f7c2806de50 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:10.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.369+0000 7f7c4f000640 1 --2- 172.21.15.60:0/1891278663 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7c4815b740 0x7f7c481f4250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:10.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.369+0000 7f7c4f000640 1 --2- 172.21.15.60:0/1891278663 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c480c6d70 0x7f7c481f3d10 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:10.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.369+0000 7f7c4f000640 1 --2- 172.21.15.60:0/1891278663 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c480711a0 0x7f7c4815bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:10.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.369+0000 7f7c4f000640 1 -- 172.21.15.60:0/1891278663 >> 172.21.15.60:0/1891278663 conn(0x7f7c480cc2f0 msgr2=0x7f7c48159ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:49:10.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.370+0000 7f7c4f000640 1 -- 172.21.15.60:0/1891278663 shutdown_connections 2024-04-14T23:49:10.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.370+0000 7f7c4f000640 1 -- 172.21.15.60:0/1891278663 wait complete. 2024-04-14T23:49:10.398 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-14T23:49:10.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.623+0000 7fc336c00640 1 Processor -- start 2024-04-14T23:49:10.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.623+0000 7fc336c00640 1 -- start start 2024-04-14T23:49:10.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.625+0000 7fc336c00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3300712e0 0x7fc330162580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:10.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.625+0000 7fc336c00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc3300c8560 0x7fc33006eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:10.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.625+0000 7fc336c00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc330162c30 con 0x7fc33015c870 2024-04-14T23:49:10.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.625+0000 7fc336c00640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fc3301634d0 con 0x7fc3300c8560 2024-04-14T23:49:10.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.625+0000 7fc336c00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fc330162f10 con 0x7fc3300712e0 2024-04-14T23:49:10.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.625+0000 7fc335800640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fc33015c870 0x7fc33015fa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33940/0 (socket says 172.21.15.60:33940) 2024-04-14T23:49:10.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.625+0000 7fc335800640 1 -- 172.21.15.60:0/311714992 learned_addr learned my addr 172.21.15.60:0/311714992 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:49:10.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.625+0000 7fc336200640 1 --2- 172.21.15.60:0/311714992 >> v2:172.21.15.60:3300/0 conn(0x7fc3300c8560 0x7fc33006eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:10.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.626+0000 7fc334e00640 1 --2- 172.21.15.60:0/311714992 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3300712e0 0x7fc330162580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:10.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.626+0000 7fc336200640 1 -- 172.21.15.60:0/311714992 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3300712e0 msgr2=0x7fc330162580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:49:10.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.626+0000 7fc336200640 1 --2- 172.21.15.60:0/311714992 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3300712e0 0x7fc330162580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:10.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.626+0000 7fc336200640 1 -- 172.21.15.60:0/311714992 >> v1:172.21.15.60:6789/0 conn(0x7fc33015c870 legacy=0x7fc33015fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:10.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.626+0000 7fc336200640 1 -- 172.21.15.60:0/311714992 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3301631f0 con 0x7fc3300c8560 2024-04-14T23:49:10.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.626+0000 7fc334e00640 1 --2- 172.21.15.60:0/311714992 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3300712e0 0x7fc330162580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:49:10.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.626+0000 7fc32fe00640 1 -- 172.21.15.60:0/311714992 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2137544251 0 0) 0x7fc330162c30 con 0x7fc33015c870 2024-04-14T23:49:10.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.627+0000 7fc336200640 1 --2- 172.21.15.60:0/311714992 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3300c8560 0x7fc33006eb60 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc328009d90 tx=0x7fc32802bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d55e4063e8353442 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:10.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.627+0000 7fc32fe00640 1 -- 172.21.15.60:0/311714992 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc328039070 con 0x7fc3300c8560 2024-04-14T23:49:10.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.628+0000 7fc32fe00640 1 -- 172.21.15.60:0/311714992 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc32802ccf0 con 0x7fc3300c8560 2024-04-14T23:49:10.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.628+0000 7fc32fe00640 1 -- 172.21.15.60:0/311714992 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc328034660 con 0x7fc3300c8560 2024-04-14T23:49:10.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.628+0000 7fc336c00640 1 -- 172.21.15.60:0/311714992 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3300c8560 msgr2=0x7fc33006eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:10.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.628+0000 7fc336c00640 1 --2- 172.21.15.60:0/311714992 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3300c8560 0x7fc33006eb60 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc328009d90 tx=0x7fc32802bf40 comp rx=0 tx=0).stop 2024-04-14T23:49:10.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.628+0000 7fc336c00640 1 -- 172.21.15.60:0/311714992 shutdown_connections 2024-04-14T23:49:10.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.628+0000 7fc336c00640 1 --2- 172.21.15.60:0/311714992 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3300c8560 0x7fc33006eb60 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:10.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.628+0000 7fc336c00640 1 --2- 172.21.15.60:0/311714992 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3300712e0 0x7fc330162580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:10.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.628+0000 7fc336c00640 1 -- 172.21.15.60:0/311714992 >> 172.21.15.60:0/311714992 conn(0x7fc3300cc2f0 msgr2=0x7fc3300cc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:49:10.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.628+0000 7fc336c00640 1 -- 172.21.15.60:0/311714992 shutdown_connections 2024-04-14T23:49:10.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.628+0000 7fc336c00640 1 -- 172.21.15.60:0/311714992 wait complete. 2024-04-14T23:49:10.636 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.630+0000 7fc336c00640 1 Processor -- start 2024-04-14T23:49:10.636 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.630+0000 7fc336c00640 1 -- start start 2024-04-14T23:49:10.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.631+0000 7fc336c00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3300712e0 0x7fc3301f50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:10.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.631+0000 7fc336c00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3300c8560 0x7fc330157b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:10.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.631+0000 7fc336c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc33015c870 0x7fc33015e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:10.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.631+0000 7fc336c00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fc3300c7b10 con 0x7fc3300c8560 2024-04-14T23:49:10.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.631+0000 7fc336c00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc33015f500 con 0x7fc33015c870 2024-04-14T23:49:10.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.631+0000 7fc336c00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fc3300c7c90 con 0x7fc3300712e0 2024-04-14T23:49:10.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.631+0000 7fc334e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3300c8560 0x7fc330157b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:10.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.632+0000 7fc334e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3300c8560 0x7fc330157b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38572/0 (socket says 172.21.15.60:38572) 2024-04-14T23:49:10.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.632+0000 7fc334e00640 1 -- 172.21.15.60:0/3021786483 learned_addr learned my addr 172.21.15.60:0/3021786483 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:49:10.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.632+0000 7fc335800640 1 --2- 172.21.15.60:0/3021786483 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3300712e0 0x7fc3301f50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:10.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.632+0000 7fc336200640 1 --2- 172.21.15.60:0/3021786483 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc33015c870 0x7fc33015e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:10.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.632+0000 7fc334e00640 1 -- 172.21.15.60:0/3021786483 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3300712e0 msgr2=0x7fc3301f50d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:10.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.632+0000 7fc334e00640 1 --2- 172.21.15.60:0/3021786483 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3300712e0 0x7fc3301f50d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:10.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.632+0000 7fc334e00640 1 -- 172.21.15.60:0/3021786483 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc33015c870 msgr2=0x7fc33015e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:10.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.632+0000 7fc334e00640 1 --2- 172.21.15.60:0/3021786483 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc33015c870 0x7fc33015e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:10.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.632+0000 7fc334e00640 1 -- 172.21.15.60:0/3021786483 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3301f91d0 con 0x7fc3300c8560 2024-04-14T23:49:10.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.632+0000 7fc334e00640 1 --2- 172.21.15.60:0/3021786483 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3300c8560 0x7fc330157b50 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc31c00e930 tx=0x7fc31c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:10.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.633+0000 7fc32d600640 1 -- 172.21.15.60:0/3021786483 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc31c017070 con 0x7fc3300c8560 2024-04-14T23:49:10.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.633+0000 7fc32d600640 1 -- 172.21.15.60:0/3021786483 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc31c00c9a0 con 0x7fc3300c8560 2024-04-14T23:49:10.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.633+0000 7fc32d600640 1 -- 172.21.15.60:0/3021786483 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc31c00cd90 con 0x7fc3300c8560 2024-04-14T23:49:10.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.633+0000 7fc336c00640 1 -- 172.21.15.60:0/3021786483 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3301f94b0 con 0x7fc3300c8560 2024-04-14T23:49:10.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.633+0000 7fc336c00640 1 -- 172.21.15.60:0/3021786483 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc33015da80 con 0x7fc3300c8560 2024-04-14T23:49:10.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.634+0000 7fc336c00640 1 -- 172.21.15.60:0/3021786483 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc33004f540 con 0x7fc3300c8560 2024-04-14T23:49:10.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.635+0000 7fc32d600640 1 -- 172.21.15.60:0/3021786483 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7fc31c01a070 con 0x7fc3300c8560 2024-04-14T23:49:10.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.636+0000 7fc32d600640 1 --2- 172.21.15.60:0/3021786483 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fc31006b990 0x7fc31006de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:10.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.636+0000 7fc32d600640 1 -- 172.21.15.60:0/3021786483 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fc31c08a780 con 0x7fc3300c8560 2024-04-14T23:49:10.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.636+0000 7fc335800640 1 --2- 172.21.15.60:0/3021786483 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fc31006b990 0x7fc31006de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:10.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.637+0000 7fc335800640 1 --2- 172.21.15.60:0/3021786483 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fc31006b990 0x7fc31006de50 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc324000f60 tx=0x7fc324007d50 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:10.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.641+0000 7fc32d600640 1 -- 172.21.15.60:0/3021786483 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc31c058c10 con 0x7fc3300c8560 2024-04-14T23:49:10.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.842+0000 7fc336c00640 1 -- 172.21.15.60:0/3021786483 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc3300c78f0 con 0x7fc3300c8560 2024-04-14T23:49:10.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.843+0000 7fc32d600640 1 -- 172.21.15.60:0/3021786483 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7fc31c05ccb0 con 0x7fc3300c8560 2024-04-14T23:49:10.850 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2024-04-14T23:49:10.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.846+0000 7fc336c00640 1 -- 172.21.15.60:0/3021786483 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fc31006b990 msgr2=0x7fc31006de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:10.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.846+0000 7fc336c00640 1 --2- 172.21.15.60:0/3021786483 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fc31006b990 0x7fc31006de50 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc324000f60 tx=0x7fc324007d50 comp rx=0 tx=0).stop 2024-04-14T23:49:10.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.846+0000 7fc336c00640 1 -- 172.21.15.60:0/3021786483 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3300c8560 msgr2=0x7fc330157b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:10.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.846+0000 7fc336c00640 1 --2- 172.21.15.60:0/3021786483 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3300c8560 0x7fc330157b50 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc31c00e930 tx=0x7fc31c00edf0 comp rx=0 tx=0).stop 2024-04-14T23:49:10.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.846+0000 7fc336c00640 1 -- 172.21.15.60:0/3021786483 shutdown_connections 2024-04-14T23:49:10.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.846+0000 7fc336c00640 1 --2- 172.21.15.60:0/3021786483 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fc31006b990 0x7fc31006de50 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:10.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.846+0000 7fc336c00640 1 --2- 172.21.15.60:0/3021786483 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc33015c870 0x7fc33015e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:10.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.846+0000 7fc336c00640 1 --2- 172.21.15.60:0/3021786483 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3300c8560 0x7fc330157b50 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:10.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.846+0000 7fc336c00640 1 --2- 172.21.15.60:0/3021786483 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3300712e0 0x7fc3301f50d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:10.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.846+0000 7fc336c00640 1 -- 172.21.15.60:0/3021786483 >> 172.21.15.60:0/3021786483 conn(0x7fc3300cc2f0 msgr2=0x7fc330158c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:49:10.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.846+0000 7fc336c00640 1 -- 172.21.15.60:0/3021786483 shutdown_connections 2024-04-14T23:49:10.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:10.846+0000 7fc336c00640 1 -- 172.21.15.60:0/3021786483 wait complete. 2024-04-14T23:49:10.870 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-14T23:49:11.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.086+0000 7fe001400640 1 Processor -- start 2024-04-14T23:49:11.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.086+0000 7fe001400640 1 -- start start 2024-04-14T23:49:11.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.088+0000 7fe001400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdffc158610 0x7fdffc0b1740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:11.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.088+0000 7fe001400640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fdffc0c3400 0x7fdffc06eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:11.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.088+0000 7fe001400640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdffc06f0a0 0x7fdffc0c6800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:11.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.088+0000 7fe001400640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fdffc0b1fd0 con 0x7fdffc06f0a0 2024-04-14T23:49:11.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.088+0000 7fe001400640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fdffc15b070 con 0x7fdffc0c3400 2024-04-14T23:49:11.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.088+0000 7fe001400640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fdffc15b350 con 0x7fdffc158610 2024-04-14T23:49:11.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.088+0000 7fe000a00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdffc06f0a0 0x7fdffc0c6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:11.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.088+0000 7fdffbe00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdffc158610 0x7fdffc0b1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:11.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.088+0000 7fe000a00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdffc06f0a0 0x7fdffc0c6800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38588/0 (socket says 172.21.15.60:38588) 2024-04-14T23:49:11.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.088+0000 7fe000a00640 1 -- 172.21.15.60:0/3908600926 learned_addr learned my addr 172.21.15.60:0/3908600926 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:49:11.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.088+0000 7fdff3400640 1 --2- 172.21.15.60:0/3908600926 >> v2:172.21.15.186:3300/0 conn(0x7fdffc0c3400 0x7fdffc06eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:11.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.089+0000 7fe000a00640 1 -- 172.21.15.60:0/3908600926 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdffc158610 msgr2=0x7fdffc0b1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:49:11.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.089+0000 7fe000a00640 1 --2- 172.21.15.60:0/3908600926 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdffc158610 0x7fdffc0b1740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:11.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.089+0000 7fe000a00640 1 -- 172.21.15.60:0/3908600926 >> v2:172.21.15.186:3300/0 conn(0x7fdffc0c3400 msgr2=0x7fdffc06eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:49:11.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.089+0000 7fe000a00640 1 --2- 172.21.15.60:0/3908600926 >> v2:172.21.15.186:3300/0 conn(0x7fdffc0c3400 0x7fdffc06eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:11.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.089+0000 7fe000a00640 1 -- 172.21.15.60:0/3908600926 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdffc15b630 con 0x7fdffc06f0a0 2024-04-14T23:49:11.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.089+0000 7fdffbe00640 1 --2- 172.21.15.60:0/3908600926 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdffc158610 0x7fdffc0b1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:49:11.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.090+0000 7fe000a00640 1 --2- 172.21.15.60:0/3908600926 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdffc06f0a0 0x7fdffc0c6800 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fdffc0c8960 tx=0x7fdff402e170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82d2917891510811 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:11.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.090+0000 7fdffb400640 1 -- 172.21.15.60:0/3908600926 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff402ed10 con 0x7fdffc06f0a0 2024-04-14T23:49:11.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.090+0000 7fdffb400640 1 -- 172.21.15.60:0/3908600926 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdff40363f0 con 0x7fdffc06f0a0 2024-04-14T23:49:11.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.090+0000 7fdffb400640 1 -- 172.21.15.60:0/3908600926 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff40367c0 con 0x7fdffc06f0a0 2024-04-14T23:49:11.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.091+0000 7fe001400640 1 -- 172.21.15.60:0/3908600926 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdffc06f0a0 msgr2=0x7fdffc0c6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:11.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.091+0000 7fe001400640 1 --2- 172.21.15.60:0/3908600926 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdffc06f0a0 0x7fdffc0c6800 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fdffc0c8960 tx=0x7fdff402e170 comp rx=0 tx=0).stop 2024-04-14T23:49:11.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.091+0000 7fe001400640 1 -- 172.21.15.60:0/3908600926 shutdown_connections 2024-04-14T23:49:11.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.091+0000 7fe001400640 1 --2- 172.21.15.60:0/3908600926 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdffc06f0a0 0x7fdffc0c6800 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:11.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.091+0000 7fe001400640 1 --2- 172.21.15.60:0/3908600926 >> v2:172.21.15.186:3300/0 conn(0x7fdffc0c3400 0x7fdffc06eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:11.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.091+0000 7fe001400640 1 --2- 172.21.15.60:0/3908600926 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdffc158610 0x7fdffc0b1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:11.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.091+0000 7fe001400640 1 -- 172.21.15.60:0/3908600926 >> 172.21.15.60:0/3908600926 conn(0x7fdffc0cc2f0 msgr2=0x7fdffc0cc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:49:11.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.091+0000 7fe001400640 1 -- 172.21.15.60:0/3908600926 shutdown_connections 2024-04-14T23:49:11.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.091+0000 7fe001400640 1 -- 172.21.15.60:0/3908600926 wait complete. 2024-04-14T23:49:11.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.093+0000 7fe001400640 1 Processor -- start 2024-04-14T23:49:11.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.094+0000 7fe001400640 1 -- start start 2024-04-14T23:49:11.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.095+0000 7fe001400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdffc06f0a0 0x7fdffc150690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:11.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.095+0000 7fe001400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdffc0c3400 0x7fdffc0c2520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:11.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.095+0000 7fe001400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdffc158610 0x7fdffc0cb180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:11.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.095+0000 7fe001400640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fdffc0b1c80 con 0x7fdffc06f0a0 2024-04-14T23:49:11.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.095+0000 7fe001400640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdffc15bbf0 con 0x7fdffc0c3400 2024-04-14T23:49:11.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.095+0000 7fe001400640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fdffc071730 con 0x7fdffc158610 2024-04-14T23:49:11.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.095+0000 7fdffbe00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdffc06f0a0 0x7fdffc150690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:11.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.095+0000 7fdff3400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdffc0c3400 0x7fdffc0c2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:11.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.095+0000 7fdffbe00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdffc06f0a0 0x7fdffc150690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38590/0 (socket says 172.21.15.60:38590) 2024-04-14T23:49:11.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.095+0000 7fdffbe00640 1 -- 172.21.15.60:0/1637017769 learned_addr learned my addr 172.21.15.60:0/1637017769 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:49:11.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.095+0000 7fe000a00640 1 --2- 172.21.15.60:0/1637017769 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdffc158610 0x7fdffc0cb180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:11.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.096+0000 7fe000a00640 1 -- 172.21.15.60:0/1637017769 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdffc0c3400 msgr2=0x7fdffc0c2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:11.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.096+0000 7fe000a00640 1 --2- 172.21.15.60:0/1637017769 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdffc0c3400 0x7fdffc0c2520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:11.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.096+0000 7fe000a00640 1 -- 172.21.15.60:0/1637017769 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdffc06f0a0 msgr2=0x7fdffc150690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:11.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.096+0000 7fe000a00640 1 --2- 172.21.15.60:0/1637017769 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdffc06f0a0 0x7fdffc150690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:11.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.096+0000 7fe000a00640 1 -- 172.21.15.60:0/1637017769 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdffc0c77d0 con 0x7fdffc158610 2024-04-14T23:49:11.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.096+0000 7fe000a00640 1 --2- 172.21.15.60:0/1637017769 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdffc158610 0x7fdffc0cb180 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fdffc06c440 tx=0x7fdff4004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:11.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.096+0000 7fdff8c00640 1 -- 172.21.15.60:0/1637017769 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff4009510 con 0x7fdffc158610 2024-04-14T23:49:11.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.097+0000 7fdff8c00640 1 -- 172.21.15.60:0/1637017769 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdff4043070 con 0x7fdffc158610 2024-04-14T23:49:11.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.097+0000 7fdff8c00640 1 -- 172.21.15.60:0/1637017769 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff403e650 con 0x7fdffc158610 2024-04-14T23:49:11.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.097+0000 7fe001400640 1 -- 172.21.15.60:0/1637017769 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdffc1f38f0 con 0x7fdffc158610 2024-04-14T23:49:11.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.097+0000 7fe001400640 1 -- 172.21.15.60:0/1637017769 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdffc1f3eb0 con 0x7fdffc158610 2024-04-14T23:49:11.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.098+0000 7fe001400640 1 -- 172.21.15.60:0/1637017769 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdffc04f540 con 0x7fdffc158610 2024-04-14T23:49:11.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.099+0000 7fdff8c00640 1 -- 172.21.15.60:0/1637017769 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7fdff4005d10 con 0x7fdffc158610 2024-04-14T23:49:11.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.100+0000 7fdff8c00640 1 --2- 172.21.15.60:0/1637017769 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fdfd806b990 0x7fdfd806de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:11.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.100+0000 7fdff8c00640 1 -- 172.21.15.60:0/1637017769 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fdff40ac2f0 con 0x7fdffc158610 2024-04-14T23:49:11.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.100+0000 7fdffbe00640 1 --2- 172.21.15.60:0/1637017769 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fdfd806b990 0x7fdfd806de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:11.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.101+0000 7fdffbe00640 1 --2- 172.21.15.60:0/1637017769 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fdfd806b990 0x7fdfd806de50 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fdfec009560 tx=0x7fdfec009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:11.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.105+0000 7fdff8c00640 1 -- 172.21.15.60:0/1637017769 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fdff404d020 con 0x7fdffc158610 2024-04-14T23:49:11.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.310+0000 7fe001400640 1 -- 172.21.15.60:0/1637017769 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fdffc157140 con 0x7fdffc158610 2024-04-14T23:49:11.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.310+0000 7fdff8c00640 1 -- 172.21.15.60:0/1637017769 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7fdffc0c77d0 con 0x7fdffc158610 2024-04-14T23:49:11.316 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2024-04-14T23:49:11.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.313+0000 7fe001400640 1 -- 172.21.15.60:0/1637017769 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fdfd806b990 msgr2=0x7fdfd806de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:11.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.313+0000 7fe001400640 1 --2- 172.21.15.60:0/1637017769 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fdfd806b990 0x7fdfd806de50 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fdfec009560 tx=0x7fdfec009190 comp rx=0 tx=0).stop 2024-04-14T23:49:11.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.313+0000 7fe001400640 1 -- 172.21.15.60:0/1637017769 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdffc158610 msgr2=0x7fdffc0cb180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:11.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.313+0000 7fe001400640 1 --2- 172.21.15.60:0/1637017769 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdffc158610 0x7fdffc0cb180 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fdffc06c440 tx=0x7fdff4004a40 comp rx=0 tx=0).stop 2024-04-14T23:49:11.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.314+0000 7fe001400640 1 -- 172.21.15.60:0/1637017769 shutdown_connections 2024-04-14T23:49:11.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.314+0000 7fe001400640 1 --2- 172.21.15.60:0/1637017769 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fdfd806b990 0x7fdfd806de50 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:11.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.314+0000 7fe001400640 1 --2- 172.21.15.60:0/1637017769 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdffc158610 0x7fdffc0cb180 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:11.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.314+0000 7fe001400640 1 --2- 172.21.15.60:0/1637017769 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdffc0c3400 0x7fdffc0c2520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:11.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.314+0000 7fe001400640 1 --2- 172.21.15.60:0/1637017769 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdffc06f0a0 0x7fdffc150690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:11.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.314+0000 7fe001400640 1 -- 172.21.15.60:0/1637017769 >> 172.21.15.60:0/1637017769 conn(0x7fdffc0cc2f0 msgr2=0x7fdffc0cc6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:49:11.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.314+0000 7fe001400640 1 -- 172.21.15.60:0/1637017769 shutdown_connections 2024-04-14T23:49:11.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.314+0000 7fe001400640 1 -- 172.21.15.60:0/1637017769 wait complete. 2024-04-14T23:49:11.339 INFO:tasks.mds_thrash:Wait for all MDSs to reach steady state... 2024-04-14T23:49:11.340 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:49:11.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.552+0000 7f00e1e00640 1 Processor -- start 2024-04-14T23:49:11.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.553+0000 7f00e1e00640 1 -- start start 2024-04-14T23:49:11.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.554+0000 7f00e1e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f00dc154cf0 0x7f00dc15a940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:11.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.554+0000 7f00e1e00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f00dc06c310 0x7f00dc06c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:11.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.554+0000 7f00e1e00640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00dc0c5520 con 0x7f00dc0cb120 2024-04-14T23:49:11.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.554+0000 7f00e1e00640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f00dc1550d0 con 0x7f00dc06c310 2024-04-14T23:49:11.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.554+0000 7f00e1e00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f00dc15b040 con 0x7f00dc154cf0 2024-04-14T23:49:11.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.555+0000 7f00e0a00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f00dc154cf0 0x7f00dc15a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:11.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.555+0000 7f00e0a00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f00dc154cf0 0x7f00dc15a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.60:48520/0 (socket says 172.21.15.60:48520) 2024-04-14T23:49:11.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.555+0000 7f00dbe00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f00dc06c310 0x7f00dc06c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:11.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.555+0000 7f00e0a00640 1 -- 172.21.15.60:0/2028621113 learned_addr learned my addr 172.21.15.60:0/2028621113 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:49:11.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.555+0000 7f00e1400640 1 --1- 172.21.15.60:0/2028621113 >> v1:172.21.15.186:6789/0 conn(0x7f00dc0cb120 0x7f00dc0cbd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.60:44654/0 (socket says 172.21.15.60:44654) 2024-04-14T23:49:11.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.555+0000 7f00e0a00640 1 -- 172.21.15.60:0/2028621113 >> v2:172.21.15.186:3300/0 conn(0x7f00dc06c310 msgr2=0x7f00dc06c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:49:11.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.555+0000 7f00e0a00640 1 --2- 172.21.15.60:0/2028621113 >> v2:172.21.15.186:3300/0 conn(0x7f00dc06c310 0x7f00dc06c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:11.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.555+0000 7f00e0a00640 1 -- 172.21.15.60:0/2028621113 >> v1:172.21.15.186:6789/0 conn(0x7f00dc0cb120 legacy=0x7f00dc0cbd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:11.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.555+0000 7f00e0a00640 1 -- 172.21.15.60:0/2028621113 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00dc15b300 con 0x7f00dc154cf0 2024-04-14T23:49:11.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.555+0000 7f00dbe00640 1 --2- 172.21.15.60:0/2028621113 >> v2:172.21.15.186:3300/0 conn(0x7f00dc06c310 0x7f00dc06c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:49:11.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.557+0000 7f00e0a00640 1 --2- 172.21.15.60:0/2028621113 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f00dc154cf0 0x7f00dc15a940 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f00cc009870 tx=0x7f00cc02b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6fb2da48cf6a7dd server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:11.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.557+0000 7f00db400640 1 -- 172.21.15.60:0/2028621113 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00cc038070 con 0x7f00dc154cf0 2024-04-14T23:49:11.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.557+0000 7f00db400640 1 -- 172.21.15.60:0/2028621113 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00cc031740 con 0x7f00dc154cf0 2024-04-14T23:49:11.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.557+0000 7f00db400640 1 -- 172.21.15.60:0/2028621113 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00cc02f050 con 0x7f00dc154cf0 2024-04-14T23:49:11.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.557+0000 7f00e1e00640 1 -- 172.21.15.60:0/2028621113 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f00dc154cf0 msgr2=0x7f00dc15a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:11.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.557+0000 7f00e1e00640 1 --2- 172.21.15.60:0/2028621113 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f00dc154cf0 0x7f00dc15a940 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f00cc009870 tx=0x7f00cc02b860 comp rx=0 tx=0).stop 2024-04-14T23:49:11.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.558+0000 7f00e1e00640 1 -- 172.21.15.60:0/2028621113 shutdown_connections 2024-04-14T23:49:11.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.558+0000 7f00e1e00640 1 --2- 172.21.15.60:0/2028621113 >> v2:172.21.15.186:3300/0 conn(0x7f00dc06c310 0x7f00dc06c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:11.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.558+0000 7f00e1e00640 1 --2- 172.21.15.60:0/2028621113 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f00dc154cf0 0x7f00dc15a940 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:11.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.558+0000 7f00e1e00640 1 -- 172.21.15.60:0/2028621113 >> 172.21.15.60:0/2028621113 conn(0x7f00dc0670f0 msgr2=0x7f00dc067500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:49:11.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.558+0000 7f00e1e00640 1 -- 172.21.15.60:0/2028621113 shutdown_connections 2024-04-14T23:49:11.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.558+0000 7f00e1e00640 1 -- 172.21.15.60:0/2028621113 wait complete. 2024-04-14T23:49:11.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.559+0000 7f00e1e00640 1 Processor -- start 2024-04-14T23:49:11.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.560+0000 7f00e1e00640 1 -- start start 2024-04-14T23:49:11.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.561+0000 7f00e1e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00dc06c310 0x7f00dc1f16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:11.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.561+0000 7f00e1e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f00dc0cb120 0x7f00dc0b4280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:11.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.561+0000 7f00e1e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00dc154cf0 0x7f00dc0cfea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:11.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.561+0000 7f00e1e00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f00dc06bb10 con 0x7f00dc06c310 2024-04-14T23:49:11.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.561+0000 7f00e1e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f00dc15b8c0 con 0x7f00dc154cf0 2024-04-14T23:49:11.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.561+0000 7f00e1e00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f00dc06bdf0 con 0x7f00dc0cb120 2024-04-14T23:49:11.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.561+0000 7f00e0a00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00dc06c310 0x7f00dc1f16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:11.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.561+0000 7f00e0a00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00dc06c310 0x7f00dc1f16e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38594/0 (socket says 172.21.15.60:38594) 2024-04-14T23:49:11.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.561+0000 7f00e0a00640 1 -- 172.21.15.60:0/2504781938 learned_addr learned my addr 172.21.15.60:0/2504781938 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:49:11.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.561+0000 7f00dbe00640 1 --2- 172.21.15.60:0/2504781938 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f00dc0cb120 0x7f00dc0b4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:11.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.561+0000 7f00e1400640 1 --2- 172.21.15.60:0/2504781938 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00dc154cf0 0x7f00dc0cfea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:11.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.561+0000 7f00e0a00640 1 -- 172.21.15.60:0/2504781938 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f00dc0cb120 msgr2=0x7f00dc0b4280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:11.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.561+0000 7f00e0a00640 1 --2- 172.21.15.60:0/2504781938 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f00dc0cb120 0x7f00dc0b4280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:11.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.562+0000 7f00e0a00640 1 -- 172.21.15.60:0/2504781938 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00dc154cf0 msgr2=0x7f00dc0cfea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:11.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.562+0000 7f00e0a00640 1 --2- 172.21.15.60:0/2504781938 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00dc154cf0 0x7f00dc0cfea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:11.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.562+0000 7f00e0a00640 1 -- 172.21.15.60:0/2504781938 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00dc1f0c30 con 0x7f00dc06c310 2024-04-14T23:49:11.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.562+0000 7f00e0a00640 1 --2- 172.21.15.60:0/2504781938 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00dc06c310 0x7f00dc1f16e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f00cc009870 tx=0x7f00cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:11.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.562+0000 7f00d8c00640 1 -- 172.21.15.60:0/2504781938 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00cc043070 con 0x7f00dc06c310 2024-04-14T23:49:11.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.562+0000 7f00d8c00640 1 -- 172.21.15.60:0/2504781938 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00cc03f870 con 0x7f00dc06c310 2024-04-14T23:49:11.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.562+0000 7f00d8c00640 1 -- 172.21.15.60:0/2504781938 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00cc038040 con 0x7f00dc06c310 2024-04-14T23:49:11.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.562+0000 7f00e1e00640 1 -- 172.21.15.60:0/2504781938 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00dc0d0da0 con 0x7f00dc06c310 2024-04-14T23:49:11.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.562+0000 7f00e1e00640 1 -- 172.21.15.60:0/2504781938 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00dc1f0ee0 con 0x7f00dc06c310 2024-04-14T23:49:11.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.564+0000 7f00e1e00640 1 -- 172.21.15.60:0/2504781938 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00dc0ca910 con 0x7f00dc06c310 2024-04-14T23:49:11.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.565+0000 7f00d8c00640 1 -- 172.21.15.60:0/2504781938 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f00cc046070 con 0x7f00dc06c310 2024-04-14T23:49:11.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.565+0000 7f00d8c00640 1 --2- 172.21.15.60:0/2504781938 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f00b806b9c0 0x7f00b806de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:11.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.566+0000 7f00d8c00640 1 -- 172.21.15.60:0/2504781938 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f00cc0ac2d0 con 0x7f00dc06c310 2024-04-14T23:49:11.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.566+0000 7f00dbe00640 1 --2- 172.21.15.60:0/2504781938 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f00b806b9c0 0x7f00b806de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:11.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.567+0000 7f00dbe00640 1 --2- 172.21.15.60:0/2504781938 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f00b806b9c0 0x7f00b806de80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f00dc0c64a0 tx=0x7f00c400b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:11.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.570+0000 7f00d8c00640 1 -- 172.21.15.60:0/2504781938 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f00dc0ca910 con 0x7f00dc06c310 2024-04-14T23:49:11.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.809+0000 7f00e1e00640 1 -- 172.21.15.60:0/2504781938 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f00dc04f540 con 0x7f00dc06c310 2024-04-14T23:49:11.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.810+0000 7f00d8c00640 1 -- 172.21.15.60:0/2504781938 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6827 (secure 0 0 0) 0x7f00cc04d030 con 0x7f00dc06c310 2024-04-14T23:49:11.816 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:49:11.816 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/78303888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":78303888},{"type":"v1","addr":"172.21.15.60:6833","nonce":78303888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:48:49.256965+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.60:6835/2677133122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2677133122},{"type":"v1","addr":"172.21.15.60:6835","nonce":2677133122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4279,"qdb_cluster":[4279]},"id":1}]} 2024-04-14T23:49:11.817 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 8 2024-04-14T23:49:11.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.813+0000 7f00e1e00640 1 -- 172.21.15.60:0/2504781938 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f00b806b9c0 msgr2=0x7f00b806de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:11.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.813+0000 7f00e1e00640 1 --2- 172.21.15.60:0/2504781938 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f00b806b9c0 0x7f00b806de80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f00dc0c64a0 tx=0x7f00c400b040 comp rx=0 tx=0).stop 2024-04-14T23:49:11.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.813+0000 7f00e1e00640 1 -- 172.21.15.60:0/2504781938 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00dc06c310 msgr2=0x7f00dc1f16e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:11.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.813+0000 7f00e1e00640 1 --2- 172.21.15.60:0/2504781938 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00dc06c310 0x7f00dc1f16e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f00cc009870 tx=0x7f00cc0046c0 comp rx=0 tx=0).stop 2024-04-14T23:49:11.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.813+0000 7f00e1e00640 1 -- 172.21.15.60:0/2504781938 shutdown_connections 2024-04-14T23:49:11.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.813+0000 7f00e1e00640 1 --2- 172.21.15.60:0/2504781938 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f00b806b9c0 0x7f00b806de80 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:11.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.813+0000 7f00e1e00640 1 --2- 172.21.15.60:0/2504781938 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00dc154cf0 0x7f00dc0cfea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:11.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.813+0000 7f00e1e00640 1 --2- 172.21.15.60:0/2504781938 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f00dc0cb120 0x7f00dc0b4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:11.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.813+0000 7f00e1e00640 1 --2- 172.21.15.60:0/2504781938 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00dc06c310 0x7f00dc1f16e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:11.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.813+0000 7f00e1e00640 1 -- 172.21.15.60:0/2504781938 >> 172.21.15.60:0/2504781938 conn(0x7f00dc0670f0 msgr2=0x7f00dc0c6710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:49:11.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.813+0000 7f00e1e00640 1 -- 172.21.15.60:0/2504781938 shutdown_connections 2024-04-14T23:49:11.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:11.813+0000 7f00e1e00640 1 -- 172.21.15.60:0/2504781938 wait complete. 2024-04-14T23:49:11.838 INFO:tasks.mds_thrash:Ready to start thrashing 2024-04-14T23:49:11.838 INFO:tasks.mds_thrash.ceph_manager:waiting for clean 2024-04-14T23:49:11.839 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-14T23:49:12.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.057+0000 7f0e74000640 1 Processor -- start 2024-04-14T23:49:12.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.057+0000 7f0e74000640 1 -- start start 2024-04-14T23:49:12.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.059+0000 7f0e74000640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f0e6c0c63c0 0x7f0e6c0c6b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:12.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.059+0000 7f0e74000640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0e6c15c9b0 0x7f0e6c0c70b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:12.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.059+0000 7f0e74000640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e6c1530d0 con 0x7f0e6c152cf0 2024-04-14T23:49:12.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.059+0000 7f0e74000640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f0e6c0751f0 con 0x7f0e6c15c9b0 2024-04-14T23:49:12.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.059+0000 7f0e74000640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f0e6c0c8370 con 0x7f0e6c0c63c0 2024-04-14T23:49:12.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.059+0000 7f0e72000640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0e6c15c9b0 0x7f0e6c0c70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:12.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.059+0000 7f0e70c00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f0e6c0c63c0 0x7f0e6c0c6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:12.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.059+0000 7f0e72000640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0e6c15c9b0 0x7f0e6c0c70b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37278/0 (socket says 172.21.15.60:37278) 2024-04-14T23:49:12.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.059+0000 7f0e72000640 1 -- 172.21.15.60:0/3363033035 learned_addr learned my addr 172.21.15.60:0/3363033035 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:49:12.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.060+0000 7f0e70c00640 1 -- 172.21.15.60:0/3363033035 >> v2:172.21.15.60:3300/0 conn(0x7f0e6c15c9b0 msgr2=0x7f0e6c0c70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:49:12.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.060+0000 7f0e70c00640 1 --2- 172.21.15.60:0/3363033035 >> v2:172.21.15.60:3300/0 conn(0x7f0e6c15c9b0 0x7f0e6c0c70b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:12.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.060+0000 7f0e70c00640 1 -- 172.21.15.60:0/3363033035 >> v1:172.21.15.186:6789/0 conn(0x7f0e6c152cf0 legacy=0x7f0e6c15c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:12.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.060+0000 7f0e70c00640 1 -- 172.21.15.60:0/3363033035 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e6c0c3810 con 0x7f0e6c0c63c0 2024-04-14T23:49:12.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.061+0000 7f0e70c00640 1 --2- 172.21.15.60:0/3363033035 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e6c0c63c0 0x7f0e6c0c6b70 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c070e80 tx=0x7f0e5802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9fd99b3e8f94c2f server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:12.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.061+0000 7f0e6be00640 1 -- 172.21.15.60:0/3363033035 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e58034070 con 0x7f0e6c0c63c0 2024-04-14T23:49:12.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.061+0000 7f0e6be00640 1 -- 172.21.15.60:0/3363033035 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e58030610 con 0x7f0e6c0c63c0 2024-04-14T23:49:12.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.061+0000 7f0e6be00640 1 -- 172.21.15.60:0/3363033035 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e580309e0 con 0x7f0e6c0c63c0 2024-04-14T23:49:12.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.062+0000 7f0e74000640 1 -- 172.21.15.60:0/3363033035 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e6c0c63c0 msgr2=0x7f0e6c0c6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:12.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.062+0000 7f0e74000640 1 --2- 172.21.15.60:0/3363033035 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e6c0c63c0 0x7f0e6c0c6b70 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c070e80 tx=0x7f0e5802b960 comp rx=0 tx=0).stop 2024-04-14T23:49:12.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.062+0000 7f0e74000640 1 -- 172.21.15.60:0/3363033035 shutdown_connections 2024-04-14T23:49:12.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.062+0000 7f0e74000640 1 --2- 172.21.15.60:0/3363033035 >> v2:172.21.15.60:3300/0 conn(0x7f0e6c15c9b0 0x7f0e6c0c70b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:12.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.062+0000 7f0e74000640 1 --2- 172.21.15.60:0/3363033035 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e6c0c63c0 0x7f0e6c0c6b70 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:12.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.062+0000 7f0e74000640 1 -- 172.21.15.60:0/3363033035 >> 172.21.15.60:0/3363033035 conn(0x7f0e6c0cc170 msgr2=0x7f0e6c0cc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:49:12.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.062+0000 7f0e74000640 1 -- 172.21.15.60:0/3363033035 shutdown_connections 2024-04-14T23:49:12.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.062+0000 7f0e74000640 1 -- 172.21.15.60:0/3363033035 wait complete. 2024-04-14T23:49:12.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.063+0000 7f0e74000640 1 Processor -- start 2024-04-14T23:49:12.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.064+0000 7f0e74000640 1 -- start start 2024-04-14T23:49:12.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.064+0000 7f0e74000640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e6c0c63c0 0x7f0e6c0c6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:12.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.065+0000 7f0e74000640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0e6c152cf0 0x7f0e6c13c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:12.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.065+0000 7f0e74000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e6c15c9b0 0x7f0e6c13c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:12.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.065+0000 7f0e74000640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f0e6c0c88f0 con 0x7f0e6c0c63c0 2024-04-14T23:49:12.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.065+0000 7f0e74000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0e6c0c8610 con 0x7f0e6c15c9b0 2024-04-14T23:49:12.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.065+0000 7f0e74000640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f0e6c159450 con 0x7f0e6c152cf0 2024-04-14T23:49:12.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.065+0000 7f0e71600640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e6c0c63c0 0x7f0e6c0c6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:12.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.065+0000 7f0e71600640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e6c0c63c0 0x7f0e6c0c6ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37284/0 (socket says 172.21.15.60:37284) 2024-04-14T23:49:12.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.065+0000 7f0e71600640 1 -- 172.21.15.60:0/1400281565 learned_addr learned my addr 172.21.15.60:0/1400281565 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:49:12.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.065+0000 7f0e70c00640 1 --2- 172.21.15.60:0/1400281565 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0e6c152cf0 0x7f0e6c13c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:12.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.065+0000 7f0e72000640 1 --2- 172.21.15.60:0/1400281565 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e6c15c9b0 0x7f0e6c13c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:12.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.065+0000 7f0e71600640 1 -- 172.21.15.60:0/1400281565 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0e6c152cf0 msgr2=0x7f0e6c13c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:12.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.065+0000 7f0e71600640 1 --2- 172.21.15.60:0/1400281565 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0e6c152cf0 0x7f0e6c13c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:12.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.065+0000 7f0e71600640 1 -- 172.21.15.60:0/1400281565 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e6c15c9b0 msgr2=0x7f0e6c13c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:12.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.065+0000 7f0e71600640 1 --2- 172.21.15.60:0/1400281565 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e6c15c9b0 0x7f0e6c13c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:12.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.065+0000 7f0e71600640 1 -- 172.21.15.60:0/1400281565 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e6c159fb0 con 0x7f0e6c0c63c0 2024-04-14T23:49:12.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.065+0000 7f0e70c00640 1 --2- 172.21.15.60:0/1400281565 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0e6c152cf0 0x7f0e6c13c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-14T23:49:12.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.066+0000 7f0e71600640 1 --2- 172.21.15.60:0/1400281565 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e6c0c63c0 0x7f0e6c0c6ad0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f0e6000c8a0 tx=0x7f0e6000cd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:12.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.066+0000 7f0e69600640 1 -- 172.21.15.60:0/1400281565 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e60003b70 con 0x7f0e6c0c63c0 2024-04-14T23:49:12.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.066+0000 7f0e69600640 1 -- 172.21.15.60:0/1400281565 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e60003d30 con 0x7f0e6c0c63c0 2024-04-14T23:49:12.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.066+0000 7f0e69600640 1 -- 172.21.15.60:0/1400281565 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e6000aa90 con 0x7f0e6c0c63c0 2024-04-14T23:49:12.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.066+0000 7f0e74000640 1 -- 172.21.15.60:0/1400281565 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e6c153360 con 0x7f0e6c0c63c0 2024-04-14T23:49:12.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.066+0000 7f0e74000640 1 -- 172.21.15.60:0/1400281565 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e6c153900 con 0x7f0e6c0c63c0 2024-04-14T23:49:12.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.067+0000 7f0e74000640 1 -- 172.21.15.60:0/1400281565 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e6c076970 con 0x7f0e6c0c63c0 2024-04-14T23:49:12.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.068+0000 7f0e69600640 1 -- 172.21.15.60:0/1400281565 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f0e6000ac30 con 0x7f0e6c0c63c0 2024-04-14T23:49:12.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.069+0000 7f0e69600640 1 --2- 172.21.15.60:0/1400281565 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f0e4c06b9c0 0x7f0e4c06de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:12.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.069+0000 7f0e69600640 1 -- 172.21.15.60:0/1400281565 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f0e6008a7b0 con 0x7f0e6c0c63c0 2024-04-14T23:49:12.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.069+0000 7f0e70c00640 1 --2- 172.21.15.60:0/1400281565 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f0e4c06b9c0 0x7f0e4c06de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:12.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.070+0000 7f0e70c00640 1 --2- 172.21.15.60:0/1400281565 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f0e4c06b9c0 0x7f0e4c06de80 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0e58036000 tx=0x7f0e5803b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:12.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.073+0000 7f0e69600640 1 -- 172.21.15.60:0/1400281565 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0e60059330 con 0x7f0e6c0c63c0 2024-04-14T23:49:12.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.270+0000 7f0e74000640 1 -- 172.21.15.60:0/1400281565 --> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0e6c06ee80 con 0x7f0e4c06b9c0 2024-04-14T23:49:12.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.277+0000 7f0e69600640 1 -- 172.21.15.60:0/1400281565 <== mgr.4100 v2:172.21.15.186:6832/714104835 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251932 (secure 0 0 0) 0x7f0e6c06ee80 con 0x7f0e4c06b9c0 2024-04-14T23:49:12.286 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:49:12.288 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-04-14T23:49:12.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.282+0000 7f0e74000640 1 -- 172.21.15.60:0/1400281565 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f0e4c06b9c0 msgr2=0x7f0e4c06de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:12.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.282+0000 7f0e74000640 1 --2- 172.21.15.60:0/1400281565 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f0e4c06b9c0 0x7f0e4c06de80 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0e58036000 tx=0x7f0e5803b040 comp rx=0 tx=0).stop 2024-04-14T23:49:12.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.283+0000 7f0e74000640 1 -- 172.21.15.60:0/1400281565 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e6c0c63c0 msgr2=0x7f0e6c0c6ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:12.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.283+0000 7f0e74000640 1 --2- 172.21.15.60:0/1400281565 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e6c0c63c0 0x7f0e6c0c6ad0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f0e6000c8a0 tx=0x7f0e6000cd60 comp rx=0 tx=0).stop 2024-04-14T23:49:12.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.283+0000 7f0e74000640 1 -- 172.21.15.60:0/1400281565 shutdown_connections 2024-04-14T23:49:12.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.283+0000 7f0e74000640 1 --2- 172.21.15.60:0/1400281565 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f0e4c06b9c0 0x7f0e4c06de80 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:12.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.283+0000 7f0e74000640 1 --2- 172.21.15.60:0/1400281565 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e6c15c9b0 0x7f0e6c13c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:12.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.283+0000 7f0e74000640 1 --2- 172.21.15.60:0/1400281565 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0e6c152cf0 0x7f0e6c13c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:12.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.283+0000 7f0e74000640 1 --2- 172.21.15.60:0/1400281565 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e6c0c63c0 0x7f0e6c0c6ad0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:12.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.283+0000 7f0e74000640 1 -- 172.21.15.60:0/1400281565 >> 172.21.15.60:0/1400281565 conn(0x7f0e6c0cc170 msgr2=0x7f0e6c0c77b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:49:12.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.283+0000 7f0e74000640 1 -- 172.21.15.60:0/1400281565 shutdown_connections 2024-04-14T23:49:12.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.283+0000 7f0e74000640 1 -- 172.21.15.60:0/1400281565 wait complete. 2024-04-14T23:49:12.308 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2024-04-14T23:49:12.173811+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594023,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218016,"kb_used_data":1888,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756704,"statfs":{"total":773094113280,"available":772870864896,"internally_reserved":0,"allocated":1933312,"data_stored":1792886,"data_compressed":65128,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1245,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003322"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.549104+0000","last_change":"2024-04-14T23:48:43.222542+0000","last_active":"2024-04-14T23:48:45.549104+0000","last_peered":"2024-04-14T23:48:45.549104+0000","last_clean":"2024-04-14T23:48:45.549104+0000","last_became_active":"2024-04-14T23:48:43.222278+0000","last_became_peered":"2024-04-14T23:48:43.222278+0000","last_unstale":"2024-04-14T23:48:45.549104+0000","last_undegraded":"2024-04-14T23:48:45.549104+0000","last_fullsized":"2024-04-14T23:48:45.549104+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:13:06.877485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.049059+0000","last_change":"2024-04-14T23:48:43.226781+0000","last_active":"2024-04-14T23:48:45.049059+0000","last_peered":"2024-04-14T23:48:45.049059+0000","last_clean":"2024-04-14T23:48:45.049059+0000","last_became_active":"2024-04-14T23:48:43.226426+0000","last_became_peered":"2024-04-14T23:48:43.226426+0000","last_unstale":"2024-04-14T23:48:45.049059+0000","last_undegraded":"2024-04-14T23:48:45.049059+0000","last_fullsized":"2024-04-14T23:48:45.049059+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:35:24.780859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.235142+0000","last_change":"2024-04-14T23:48:43.236939+0000","last_active":"2024-04-14T23:48:44.235142+0000","last_peered":"2024-04-14T23:48:44.235142+0000","last_clean":"2024-04-14T23:48:44.235142+0000","last_became_active":"2024-04-14T23:48:43.236616+0000","last_became_peered":"2024-04-14T23:48:43.236616+0000","last_unstale":"2024-04-14T23:48:44.235142+0000","last_undegraded":"2024-04-14T23:48:44.235142+0000","last_fullsized":"2024-04-14T23:48:44.235142+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:35:42.619139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230420+0000","last_change":"2024-04-14T23:48:43.242349+0000","last_active":"2024-04-14T23:48:44.230420+0000","last_peered":"2024-04-14T23:48:44.230420+0000","last_clean":"2024-04-14T23:48:44.230420+0000","last_became_active":"2024-04-14T23:48:43.242000+0000","last_became_peered":"2024-04-14T23:48:43.242000+0000","last_unstale":"2024-04-14T23:48:44.230420+0000","last_undegraded":"2024-04-14T23:48:44.230420+0000","last_fullsized":"2024-04-14T23:48:44.230420+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:33:52.458016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548549+0000","last_change":"2024-04-14T23:48:43.233937+0000","last_active":"2024-04-14T23:48:45.548549+0000","last_peered":"2024-04-14T23:48:45.548549+0000","last_clean":"2024-04-14T23:48:45.548549+0000","last_became_active":"2024-04-14T23:48:43.233674+0000","last_became_peered":"2024-04-14T23:48:43.233674+0000","last_unstale":"2024-04-14T23:48:45.548549+0000","last_undegraded":"2024-04-14T23:48:45.548549+0000","last_fullsized":"2024-04-14T23:48:45.548549+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:00:07.838600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230260+0000","last_change":"2024-04-14T23:48:43.230925+0000","last_active":"2024-04-14T23:48:44.230260+0000","last_peered":"2024-04-14T23:48:44.230260+0000","last_clean":"2024-04-14T23:48:44.230260+0000","last_became_active":"2024-04-14T23:48:43.230405+0000","last_became_peered":"2024-04-14T23:48:43.230405+0000","last_unstale":"2024-04-14T23:48:44.230260+0000","last_undegraded":"2024-04-14T23:48:44.230260+0000","last_fullsized":"2024-04-14T23:48:44.230260+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:15:12.452964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548354+0000","last_change":"2024-04-14T23:48:43.231088+0000","last_active":"2024-04-14T23:48:45.548354+0000","last_peered":"2024-04-14T23:48:45.548354+0000","last_clean":"2024-04-14T23:48:45.548354+0000","last_became_active":"2024-04-14T23:48:43.230784+0000","last_became_peered":"2024-04-14T23:48:43.230784+0000","last_unstale":"2024-04-14T23:48:45.548354+0000","last_undegraded":"2024-04-14T23:48:45.548354+0000","last_fullsized":"2024-04-14T23:48:45.548354+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:23:10.461088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.231230+0000","last_change":"2024-04-14T23:48:43.236019+0000","last_active":"2024-04-14T23:48:44.231230+0000","last_peered":"2024-04-14T23:48:44.231230+0000","last_clean":"2024-04-14T23:48:44.231230+0000","last_became_active":"2024-04-14T23:48:43.235719+0000","last_became_peered":"2024-04-14T23:48:43.235719+0000","last_unstale":"2024-04-14T23:48:44.231230+0000","last_undegraded":"2024-04-14T23:48:44.231230+0000","last_fullsized":"2024-04-14T23:48:44.231230+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T04:41:16.130730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.549156+0000","last_change":"2024-04-14T23:48:43.232166+0000","last_active":"2024-04-14T23:48:45.549156+0000","last_peered":"2024-04-14T23:48:45.549156+0000","last_clean":"2024-04-14T23:48:45.549156+0000","last_became_active":"2024-04-14T23:48:43.231751+0000","last_became_peered":"2024-04-14T23:48:43.231751+0000","last_unstale":"2024-04-14T23:48:45.549156+0000","last_undegraded":"2024-04-14T23:48:45.549156+0000","last_fullsized":"2024-04-14T23:48:45.549156+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:05:47.289092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548848+0000","last_change":"2024-04-14T23:48:43.224938+0000","last_active":"2024-04-14T23:48:45.548848+0000","last_peered":"2024-04-14T23:48:45.548848+0000","last_clean":"2024-04-14T23:48:45.548848+0000","last_became_active":"2024-04-14T23:48:43.224592+0000","last_became_peered":"2024-04-14T23:48:43.224592+0000","last_unstale":"2024-04-14T23:48:45.548848+0000","last_undegraded":"2024-04-14T23:48:45.548848+0000","last_fullsized":"2024-04-14T23:48:45.548848+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:59:12.923596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548591+0000","last_change":"2024-04-14T23:48:43.225290+0000","last_active":"2024-04-14T23:48:45.548591+0000","last_peered":"2024-04-14T23:48:45.548591+0000","last_clean":"2024-04-14T23:48:45.548591+0000","last_became_active":"2024-04-14T23:48:43.224989+0000","last_became_peered":"2024-04-14T23:48:43.224989+0000","last_unstale":"2024-04-14T23:48:45.548591+0000","last_undegraded":"2024-04-14T23:48:45.548591+0000","last_fullsized":"2024-04-14T23:48:45.548591+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:44:49.116229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548742+0000","last_change":"2024-04-14T23:48:43.226194+0000","last_active":"2024-04-14T23:48:45.548742+0000","last_peered":"2024-04-14T23:48:45.548742+0000","last_clean":"2024-04-14T23:48:45.548742+0000","last_became_active":"2024-04-14T23:48:43.225892+0000","last_became_peered":"2024-04-14T23:48:43.225892+0000","last_unstale":"2024-04-14T23:48:45.548742+0000","last_undegraded":"2024-04-14T23:48:45.548742+0000","last_fullsized":"2024-04-14T23:48:45.548742+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:04:12.377770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.233039+0000","last_change":"2024-04-14T23:48:43.220571+0000","last_active":"2024-04-14T23:48:44.233039+0000","last_peered":"2024-04-14T23:48:44.233039+0000","last_clean":"2024-04-14T23:48:44.233039+0000","last_became_active":"2024-04-14T23:48:43.219614+0000","last_became_peered":"2024-04-14T23:48:43.219614+0000","last_unstale":"2024-04-14T23:48:44.233039+0000","last_undegraded":"2024-04-14T23:48:44.233039+0000","last_fullsized":"2024-04-14T23:48:44.233039+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:14:41.854957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.234686+0000","last_change":"2024-04-14T23:48:43.239403+0000","last_active":"2024-04-14T23:48:44.234686+0000","last_peered":"2024-04-14T23:48:44.234686+0000","last_clean":"2024-04-14T23:48:44.234686+0000","last_became_active":"2024-04-14T23:48:43.239136+0000","last_became_peered":"2024-04-14T23:48:43.239136+0000","last_unstale":"2024-04-14T23:48:44.234686+0000","last_undegraded":"2024-04-14T23:48:44.234686+0000","last_fullsized":"2024-04-14T23:48:44.234686+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:27:30.512861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.048083+0000","last_change":"2024-04-14T23:48:43.228157+0000","last_active":"2024-04-14T23:48:45.048083+0000","last_peered":"2024-04-14T23:48:45.048083+0000","last_clean":"2024-04-14T23:48:45.048083+0000","last_became_active":"2024-04-14T23:48:43.227898+0000","last_became_peered":"2024-04-14T23:48:43.227898+0000","last_unstale":"2024-04-14T23:48:45.048083+0000","last_undegraded":"2024-04-14T23:48:45.048083+0000","last_fullsized":"2024-04-14T23:48:45.048083+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:06:37.602935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230387+0000","last_change":"2024-04-14T23:48:43.238986+0000","last_active":"2024-04-14T23:48:44.230387+0000","last_peered":"2024-04-14T23:48:44.230387+0000","last_clean":"2024-04-14T23:48:44.230387+0000","last_became_active":"2024-04-14T23:48:43.238714+0000","last_became_peered":"2024-04-14T23:48:43.238714+0000","last_unstale":"2024-04-14T23:48:44.230387+0000","last_undegraded":"2024-04-14T23:48:44.230387+0000","last_fullsized":"2024-04-14T23:48:44.230387+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:05:30.655048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.231004+0000","last_change":"2024-04-14T23:48:43.229325+0000","last_active":"2024-04-14T23:48:44.231004+0000","last_peered":"2024-04-14T23:48:44.231004+0000","last_clean":"2024-04-14T23:48:44.231004+0000","last_became_active":"2024-04-14T23:48:43.228618+0000","last_became_peered":"2024-04-14T23:48:43.228618+0000","last_unstale":"2024-04-14T23:48:44.231004+0000","last_undegraded":"2024-04-14T23:48:44.231004+0000","last_fullsized":"2024-04-14T23:48:44.231004+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:06:43.745698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.649355+0000","last_change":"2024-04-14T23:48:43.233288+0000","last_active":"2024-04-14T23:48:44.649355+0000","last_peered":"2024-04-14T23:48:44.649355+0000","last_clean":"2024-04-14T23:48:44.649355+0000","last_became_active":"2024-04-14T23:48:43.232971+0000","last_became_peered":"2024-04-14T23:48:43.232971+0000","last_unstale":"2024-04-14T23:48:44.649355+0000","last_undegraded":"2024-04-14T23:48:44.649355+0000","last_fullsized":"2024-04-14T23:48:44.649355+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:06:23.588161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.549346+0000","last_change":"2024-04-14T23:48:43.233169+0000","last_active":"2024-04-14T23:48:45.549346+0000","last_peered":"2024-04-14T23:48:45.549346+0000","last_clean":"2024-04-14T23:48:45.549346+0000","last_became_active":"2024-04-14T23:48:43.232135+0000","last_became_peered":"2024-04-14T23:48:43.232135+0000","last_unstale":"2024-04-14T23:48:45.549346+0000","last_undegraded":"2024-04-14T23:48:45.549346+0000","last_fullsized":"2024-04-14T23:48:45.549346+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:37:06.166596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.547820+0000","last_change":"2024-04-14T23:48:43.228728+0000","last_active":"2024-04-14T23:48:45.547820+0000","last_peered":"2024-04-14T23:48:45.547820+0000","last_clean":"2024-04-14T23:48:45.547820+0000","last_became_active":"2024-04-14T23:48:43.226225+0000","last_became_peered":"2024-04-14T23:48:43.226225+0000","last_unstale":"2024-04-14T23:48:45.547820+0000","last_undegraded":"2024-04-14T23:48:45.547820+0000","last_fullsized":"2024-04-14T23:48:45.547820+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:06:03.453599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.234949+0000","last_change":"2024-04-14T23:48:43.237234+0000","last_active":"2024-04-14T23:48:44.234949+0000","last_peered":"2024-04-14T23:48:44.234949+0000","last_clean":"2024-04-14T23:48:44.234949+0000","last_became_active":"2024-04-14T23:48:43.230056+0000","last_became_peered":"2024-04-14T23:48:43.230056+0000","last_unstale":"2024-04-14T23:48:44.234949+0000","last_undegraded":"2024-04-14T23:48:44.234949+0000","last_fullsized":"2024-04-14T23:48:44.234949+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:43:44.966384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230697+0000","last_change":"2024-04-14T23:48:43.229387+0000","last_active":"2024-04-14T23:48:44.230697+0000","last_peered":"2024-04-14T23:48:44.230697+0000","last_clean":"2024-04-14T23:48:44.230697+0000","last_became_active":"2024-04-14T23:48:43.229110+0000","last_became_peered":"2024-04-14T23:48:43.229110+0000","last_unstale":"2024-04-14T23:48:44.230697+0000","last_undegraded":"2024-04-14T23:48:44.230697+0000","last_fullsized":"2024-04-14T23:48:44.230697+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T09:18:46.939940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.232750+0000","last_change":"2024-04-14T23:48:43.234154+0000","last_active":"2024-04-14T23:48:44.232750+0000","last_peered":"2024-04-14T23:48:44.232750+0000","last_clean":"2024-04-14T23:48:44.232750+0000","last_became_active":"2024-04-14T23:48:43.233853+0000","last_became_peered":"2024-04-14T23:48:43.233853+0000","last_unstale":"2024-04-14T23:48:44.232750+0000","last_undegraded":"2024-04-14T23:48:44.232750+0000","last_fullsized":"2024-04-14T23:48:44.232750+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T23:54:02.200200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230330+0000","last_change":"2024-04-14T23:48:43.242154+0000","last_active":"2024-04-14T23:48:44.230330+0000","last_peered":"2024-04-14T23:48:44.230330+0000","last_clean":"2024-04-14T23:48:44.230330+0000","last_became_active":"2024-04-14T23:48:43.241808+0000","last_became_peered":"2024-04-14T23:48:43.241808+0000","last_unstale":"2024-04-14T23:48:44.230330+0000","last_undegraded":"2024-04-14T23:48:44.230330+0000","last_fullsized":"2024-04-14T23:48:44.230330+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:48:42.680808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.232996+0000","last_change":"2024-04-14T23:48:43.233530+0000","last_active":"2024-04-14T23:48:44.232996+0000","last_peered":"2024-04-14T23:48:44.232996+0000","last_clean":"2024-04-14T23:48:44.232996+0000","last_became_active":"2024-04-14T23:48:43.233238+0000","last_became_peered":"2024-04-14T23:48:43.233238+0000","last_unstale":"2024-04-14T23:48:44.232996+0000","last_undegraded":"2024-04-14T23:48:44.232996+0000","last_fullsized":"2024-04-14T23:48:44.232996+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:10:58.525655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.549193+0000","last_change":"2024-04-14T23:48:43.222945+0000","last_active":"2024-04-14T23:48:45.549193+0000","last_peered":"2024-04-14T23:48:45.549193+0000","last_clean":"2024-04-14T23:48:45.549193+0000","last_became_active":"2024-04-14T23:48:43.222624+0000","last_became_peered":"2024-04-14T23:48:43.222624+0000","last_unstale":"2024-04-14T23:48:45.549193+0000","last_undegraded":"2024-04-14T23:48:45.549193+0000","last_fullsized":"2024-04-14T23:48:45.549193+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:22:08.289006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.048719+0000","last_change":"2024-04-14T23:48:43.232567+0000","last_active":"2024-04-14T23:48:45.048719+0000","last_peered":"2024-04-14T23:48:45.048719+0000","last_clean":"2024-04-14T23:48:45.048719+0000","last_became_active":"2024-04-14T23:48:43.232277+0000","last_became_peered":"2024-04-14T23:48:43.232277+0000","last_unstale":"2024-04-14T23:48:45.048719+0000","last_undegraded":"2024-04-14T23:48:45.048719+0000","last_fullsized":"2024-04-14T23:48:45.048719+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T04:59:39.714566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.232426+0000","last_change":"2024-04-14T23:48:43.229443+0000","last_active":"2024-04-14T23:48:44.232426+0000","last_peered":"2024-04-14T23:48:44.232426+0000","last_clean":"2024-04-14T23:48:44.232426+0000","last_became_active":"2024-04-14T23:48:43.229172+0000","last_became_peered":"2024-04-14T23:48:43.229172+0000","last_unstale":"2024-04-14T23:48:44.232426+0000","last_undegraded":"2024-04-14T23:48:44.232426+0000","last_fullsized":"2024-04-14T23:48:44.232426+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:41:31.405287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.234812+0000","last_change":"2024-04-14T23:48:43.242202+0000","last_active":"2024-04-14T23:48:44.234812+0000","last_peered":"2024-04-14T23:48:44.234812+0000","last_clean":"2024-04-14T23:48:44.234812+0000","last_became_active":"2024-04-14T23:48:43.241879+0000","last_became_peered":"2024-04-14T23:48:43.241879+0000","last_unstale":"2024-04-14T23:48:44.234812+0000","last_undegraded":"2024-04-14T23:48:44.234812+0000","last_fullsized":"2024-04-14T23:48:44.234812+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:11:27.552454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230871+0000","last_change":"2024-04-14T23:48:43.242068+0000","last_active":"2024-04-14T23:48:44.230871+0000","last_peered":"2024-04-14T23:48:44.230871+0000","last_clean":"2024-04-14T23:48:44.230871+0000","last_became_active":"2024-04-14T23:48:43.241734+0000","last_became_peered":"2024-04-14T23:48:43.241734+0000","last_unstale":"2024-04-14T23:48:44.230871+0000","last_undegraded":"2024-04-14T23:48:44.230871+0000","last_fullsized":"2024-04-14T23:48:44.230871+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:32:04.579906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.648997+0000","last_change":"2024-04-14T23:48:43.236034+0000","last_active":"2024-04-14T23:48:44.648997+0000","last_peered":"2024-04-14T23:48:44.648997+0000","last_clean":"2024-04-14T23:48:44.648997+0000","last_became_active":"2024-04-14T23:48:43.233567+0000","last_became_peered":"2024-04-14T23:48:43.233567+0000","last_unstale":"2024-04-14T23:48:44.648997+0000","last_undegraded":"2024-04-14T23:48:44.648997+0000","last_fullsized":"2024-04-14T23:48:44.648997+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:07:15.904075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.547675+0000","last_change":"2024-04-14T23:48:43.231045+0000","last_active":"2024-04-14T23:48:45.547675+0000","last_peered":"2024-04-14T23:48:45.547675+0000","last_clean":"2024-04-14T23:48:45.547675+0000","last_became_active":"2024-04-14T23:48:43.230626+0000","last_became_peered":"2024-04-14T23:48:43.230626+0000","last_unstale":"2024-04-14T23:48:45.547675+0000","last_undegraded":"2024-04-14T23:48:45.547675+0000","last_fullsized":"2024-04-14T23:48:45.547675+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:24:03.654061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548613+0000","last_change":"2024-04-14T23:48:43.227543+0000","last_active":"2024-04-14T23:48:45.548613+0000","last_peered":"2024-04-14T23:48:45.548613+0000","last_clean":"2024-04-14T23:48:45.548613+0000","last_became_active":"2024-04-14T23:48:43.226773+0000","last_became_peered":"2024-04-14T23:48:43.226773+0000","last_unstale":"2024-04-14T23:48:45.548613+0000","last_undegraded":"2024-04-14T23:48:45.548613+0000","last_fullsized":"2024-04-14T23:48:45.548613+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:44:56.691708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.211656+0000","last_change":"2024-04-14T23:48:42.239254+0000","last_active":"2024-04-14T23:48:45.211656+0000","last_peered":"2024-04-14T23:48:45.211656+0000","last_clean":"2024-04-14T23:48:45.211656+0000","last_became_active":"2024-04-14T23:48:42.238847+0000","last_became_peered":"2024-04-14T23:48:42.238847+0000","last_unstale":"2024-04-14T23:48:45.211656+0000","last_undegraded":"2024-04-14T23:48:45.211656+0000","last_fullsized":"2024-04-14T23:48:45.211656+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:46:16.301900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.235039+0000","last_change":"2024-04-14T23:48:43.237720+0000","last_active":"2024-04-14T23:48:44.235039+0000","last_peered":"2024-04-14T23:48:44.235039+0000","last_clean":"2024-04-14T23:48:44.235039+0000","last_became_active":"2024-04-14T23:48:43.237391+0000","last_became_peered":"2024-04-14T23:48:43.237391+0000","last_unstale":"2024-04-14T23:48:44.235039+0000","last_undegraded":"2024-04-14T23:48:44.235039+0000","last_fullsized":"2024-04-14T23:48:44.235039+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:38:26.879906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548161+0000","last_change":"2024-04-14T23:48:42.219277+0000","last_active":"2024-04-14T23:48:45.548161+0000","last_peered":"2024-04-14T23:48:45.548161+0000","last_clean":"2024-04-14T23:48:45.548161+0000","last_became_active":"2024-04-14T23:48:42.218995+0000","last_became_peered":"2024-04-14T23:48:42.218995+0000","last_unstale":"2024-04-14T23:48:45.548161+0000","last_undegraded":"2024-04-14T23:48:45.548161+0000","last_fullsized":"2024-04-14T23:48:45.548161+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T23:51:33.370857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.232590+0000","last_change":"2024-04-14T23:48:42.236803+0000","last_active":"2024-04-14T23:48:44.232590+0000","last_peered":"2024-04-14T23:48:44.232590+0000","last_clean":"2024-04-14T23:48:44.232590+0000","last_became_active":"2024-04-14T23:48:42.225937+0000","last_became_peered":"2024-04-14T23:48:42.225937+0000","last_unstale":"2024-04-14T23:48:44.232590+0000","last_undegraded":"2024-04-14T23:48:44.232590+0000","last_fullsized":"2024-04-14T23:48:44.232590+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:52:32.609126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.231732+0000","last_change":"2024-04-14T23:48:43.229506+0000","last_active":"2024-04-14T23:48:44.231732+0000","last_peered":"2024-04-14T23:48:44.231732+0000","last_clean":"2024-04-14T23:48:44.231732+0000","last_became_active":"2024-04-14T23:48:43.227356+0000","last_became_peered":"2024-04-14T23:48:43.227356+0000","last_unstale":"2024-04-14T23:48:44.231732+0000","last_undegraded":"2024-04-14T23:48:44.231732+0000","last_fullsized":"2024-04-14T23:48:44.231732+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:13:04.840655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230660+0000","last_change":"2024-04-14T23:48:42.210883+0000","last_active":"2024-04-14T23:48:44.230660+0000","last_peered":"2024-04-14T23:48:44.230660+0000","last_clean":"2024-04-14T23:48:44.230660+0000","last_became_active":"2024-04-14T23:48:42.210607+0000","last_became_peered":"2024-04-14T23:48:42.210607+0000","last_unstale":"2024-04-14T23:48:44.230660+0000","last_undegraded":"2024-04-14T23:48:44.230660+0000","last_fullsized":"2024-04-14T23:48:44.230660+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:33:32.526223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.048141+0000","last_change":"2024-04-14T23:48:43.218259+0000","last_active":"2024-04-14T23:48:45.048141+0000","last_peered":"2024-04-14T23:48:45.048141+0000","last_clean":"2024-04-14T23:48:45.048141+0000","last_became_active":"2024-04-14T23:48:43.217801+0000","last_became_peered":"2024-04-14T23:48:43.217801+0000","last_unstale":"2024-04-14T23:48:45.048141+0000","last_undegraded":"2024-04-14T23:48:45.048141+0000","last_fullsized":"2024-04-14T23:48:45.048141+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T04:13:42.425034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.547845+0000","last_change":"2024-04-14T23:48:42.218652+0000","last_active":"2024-04-14T23:48:45.547845+0000","last_peered":"2024-04-14T23:48:45.547845+0000","last_clean":"2024-04-14T23:48:45.547845+0000","last_became_active":"2024-04-14T23:48:42.218321+0000","last_became_peered":"2024-04-14T23:48:42.218321+0000","last_unstale":"2024-04-14T23:48:45.547845+0000","last_undegraded":"2024-04-14T23:48:45.547845+0000","last_fullsized":"2024-04-14T23:48:45.547845+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T04:22:06.957747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230879+0000","last_change":"2024-04-14T23:48:43.229117+0000","last_active":"2024-04-14T23:48:44.230879+0000","last_peered":"2024-04-14T23:48:44.230879+0000","last_clean":"2024-04-14T23:48:44.230879+0000","last_became_active":"2024-04-14T23:48:43.228805+0000","last_became_peered":"2024-04-14T23:48:43.228805+0000","last_unstale":"2024-04-14T23:48:44.230879+0000","last_undegraded":"2024-04-14T23:48:44.230879+0000","last_fullsized":"2024-04-14T23:48:44.230879+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:10:39.043818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230930+0000","last_change":"2024-04-14T23:48:42.221104+0000","last_active":"2024-04-14T23:48:44.230930+0000","last_peered":"2024-04-14T23:48:44.230930+0000","last_clean":"2024-04-14T23:48:44.230930+0000","last_became_active":"2024-04-14T23:48:42.220767+0000","last_became_peered":"2024-04-14T23:48:42.220767+0000","last_unstale":"2024-04-14T23:48:44.230930+0000","last_undegraded":"2024-04-14T23:48:44.230930+0000","last_fullsized":"2024-04-14T23:48:44.230930+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:23:57.213764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.235822+0000","last_change":"2024-04-14T23:48:43.236916+0000","last_active":"2024-04-14T23:48:44.235822+0000","last_peered":"2024-04-14T23:48:44.235822+0000","last_clean":"2024-04-14T23:48:44.235822+0000","last_became_active":"2024-04-14T23:48:43.236168+0000","last_became_peered":"2024-04-14T23:48:43.236168+0000","last_unstale":"2024-04-14T23:48:44.235822+0000","last_undegraded":"2024-04-14T23:48:44.235822+0000","last_fullsized":"2024-04-14T23:48:44.235822+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:45:52.057070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:48.220199+0000","last_change":"2024-04-14T23:48:42.231329+0000","last_active":"2024-04-14T23:48:48.220199+0000","last_peered":"2024-04-14T23:48:48.220199+0000","last_clean":"2024-04-14T23:48:48.220199+0000","last_became_active":"2024-04-14T23:48:42.230969+0000","last_became_peered":"2024-04-14T23:48:42.230969+0000","last_unstale":"2024-04-14T23:48:48.220199+0000","last_undegraded":"2024-04-14T23:48:48.220199+0000","last_fullsized":"2024-04-14T23:48:48.220199+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:06:16.557589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548833+0000","last_change":"2024-04-14T23:48:43.230531+0000","last_active":"2024-04-14T23:48:45.548833+0000","last_peered":"2024-04-14T23:48:45.548833+0000","last_clean":"2024-04-14T23:48:45.548833+0000","last_became_active":"2024-04-14T23:48:43.230253+0000","last_became_peered":"2024-04-14T23:48:43.230253+0000","last_unstale":"2024-04-14T23:48:45.548833+0000","last_undegraded":"2024-04-14T23:48:45.548833+0000","last_fullsized":"2024-04-14T23:48:45.548833+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:34:37.449824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'9","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:49:06.196149+0000","last_change":"2024-04-14T23:48:42.237069+0000","last_active":"2024-04-14T23:49:06.196149+0000","last_peered":"2024-04-14T23:49:06.196149+0000","last_clean":"2024-04-14T23:49:06.196149+0000","last_became_active":"2024-04-14T23:48:42.236804+0000","last_became_peered":"2024-04-14T23:48:42.236804+0000","last_unstale":"2024-04-14T23:49:06.196149+0000","last_undegraded":"2024-04-14T23:49:06.196149+0000","last_fullsized":"2024-04-14T23:49:06.196149+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:25:37.318379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2857,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548904+0000","last_change":"2024-04-14T23:48:43.221771+0000","last_active":"2024-04-14T23:48:45.548904+0000","last_peered":"2024-04-14T23:48:45.548904+0000","last_clean":"2024-04-14T23:48:45.548904+0000","last_became_active":"2024-04-14T23:48:43.221462+0000","last_became_peered":"2024-04-14T23:48:43.221462+0000","last_unstale":"2024-04-14T23:48:45.548904+0000","last_undegraded":"2024-04-14T23:48:45.548904+0000","last_fullsized":"2024-04-14T23:48:45.548904+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:29:52.737196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:48.227204+0000","last_change":"2024-04-14T23:48:42.230518+0000","last_active":"2024-04-14T23:48:48.227204+0000","last_peered":"2024-04-14T23:48:48.227204+0000","last_clean":"2024-04-14T23:48:48.227204+0000","last_became_active":"2024-04-14T23:48:42.229758+0000","last_became_peered":"2024-04-14T23:48:42.229758+0000","last_unstale":"2024-04-14T23:48:48.227204+0000","last_undegraded":"2024-04-14T23:48:48.227204+0000","last_fullsized":"2024-04-14T23:48:48.227204+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:01:41.332553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.234816+0000","last_change":"2024-04-14T23:48:43.229873+0000","last_active":"2024-04-14T23:48:44.234816+0000","last_peered":"2024-04-14T23:48:44.234816+0000","last_clean":"2024-04-14T23:48:44.234816+0000","last_became_active":"2024-04-14T23:48:43.229476+0000","last_became_peered":"2024-04-14T23:48:43.229476+0000","last_unstale":"2024-04-14T23:48:44.234816+0000","last_undegraded":"2024-04-14T23:48:44.234816+0000","last_fullsized":"2024-04-14T23:48:44.234816+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:54:22.874344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230858+0000","last_change":"2024-04-14T23:48:42.225404+0000","last_active":"2024-04-14T23:48:44.230858+0000","last_peered":"2024-04-14T23:48:44.230858+0000","last_clean":"2024-04-14T23:48:44.230858+0000","last_became_active":"2024-04-14T23:48:42.225211+0000","last_became_peered":"2024-04-14T23:48:42.225211+0000","last_unstale":"2024-04-14T23:48:44.230858+0000","last_undegraded":"2024-04-14T23:48:44.230858+0000","last_fullsized":"2024-04-14T23:48:44.230858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:49:45.282259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548573+0000","last_change":"2024-04-14T23:48:43.229543+0000","last_active":"2024-04-14T23:48:45.548573+0000","last_peered":"2024-04-14T23:48:45.548573+0000","last_clean":"2024-04-14T23:48:45.548573+0000","last_became_active":"2024-04-14T23:48:43.229297+0000","last_became_peered":"2024-04-14T23:48:43.229297+0000","last_unstale":"2024-04-14T23:48:45.548573+0000","last_undegraded":"2024-04-14T23:48:45.548573+0000","last_fullsized":"2024-04-14T23:48:45.548573+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:00:17.923013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.547633+0000","last_change":"2024-04-14T23:48:42.220492+0000","last_active":"2024-04-14T23:48:45.547633+0000","last_peered":"2024-04-14T23:48:45.547633+0000","last_clean":"2024-04-14T23:48:45.547633+0000","last_became_active":"2024-04-14T23:48:42.220145+0000","last_became_peered":"2024-04-14T23:48:42.220145+0000","last_unstale":"2024-04-14T23:48:45.547633+0000","last_undegraded":"2024-04-14T23:48:45.547633+0000","last_fullsized":"2024-04-14T23:48:45.547633+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:17:45.278993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.547774+0000","last_change":"2024-04-14T23:48:43.235676+0000","last_active":"2024-04-14T23:48:45.547774+0000","last_peered":"2024-04-14T23:48:45.547774+0000","last_clean":"2024-04-14T23:48:45.547774+0000","last_became_active":"2024-04-14T23:48:43.235319+0000","last_became_peered":"2024-04-14T23:48:43.235319+0000","last_unstale":"2024-04-14T23:48:45.547774+0000","last_undegraded":"2024-04-14T23:48:45.547774+0000","last_fullsized":"2024-04-14T23:48:45.547774+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:32:06.283229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:48.230880+0000","last_change":"2024-04-14T23:48:42.231178+0000","last_active":"2024-04-14T23:48:48.230880+0000","last_peered":"2024-04-14T23:48:48.230880+0000","last_clean":"2024-04-14T23:48:48.230880+0000","last_became_active":"2024-04-14T23:48:42.230792+0000","last_became_peered":"2024-04-14T23:48:42.230792+0000","last_unstale":"2024-04-14T23:48:48.230880+0000","last_undegraded":"2024-04-14T23:48:48.230880+0000","last_fullsized":"2024-04-14T23:48:48.230880+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:22:40.744314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548527+0000","last_change":"2024-04-14T23:48:43.228290+0000","last_active":"2024-04-14T23:48:45.548527+0000","last_peered":"2024-04-14T23:48:45.548527+0000","last_clean":"2024-04-14T23:48:45.548527+0000","last_became_active":"2024-04-14T23:48:43.228004+0000","last_became_peered":"2024-04-14T23:48:43.228004+0000","last_unstale":"2024-04-14T23:48:45.548527+0000","last_undegraded":"2024-04-14T23:48:45.548527+0000","last_fullsized":"2024-04-14T23:48:45.548527+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:20:06.572625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548424+0000","last_change":"2024-04-14T23:48:42.231936+0000","last_active":"2024-04-14T23:48:45.548424+0000","last_peered":"2024-04-14T23:48:45.548424+0000","last_clean":"2024-04-14T23:48:45.548424+0000","last_became_active":"2024-04-14T23:48:42.231612+0000","last_became_peered":"2024-04-14T23:48:42.231612+0000","last_unstale":"2024-04-14T23:48:45.548424+0000","last_undegraded":"2024-04-14T23:48:45.548424+0000","last_fullsized":"2024-04-14T23:48:45.548424+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:45:03.098024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.549200+0000","last_change":"2024-04-14T23:48:43.228933+0000","last_active":"2024-04-14T23:48:45.549200+0000","last_peered":"2024-04-14T23:48:45.549200+0000","last_clean":"2024-04-14T23:48:45.549200+0000","last_became_active":"2024-04-14T23:48:43.228665+0000","last_became_peered":"2024-04-14T23:48:43.228665+0000","last_unstale":"2024-04-14T23:48:45.549200+0000","last_undegraded":"2024-04-14T23:48:45.549200+0000","last_fullsized":"2024-04-14T23:48:45.549200+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:43:04.747711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:48.230335+0000","last_change":"2024-04-14T23:48:42.239794+0000","last_active":"2024-04-14T23:48:48.230335+0000","last_peered":"2024-04-14T23:48:48.230335+0000","last_clean":"2024-04-14T23:48:48.230335+0000","last_became_active":"2024-04-14T23:48:42.239505+0000","last_became_peered":"2024-04-14T23:48:42.239505+0000","last_unstale":"2024-04-14T23:48:48.230335+0000","last_undegraded":"2024-04-14T23:48:48.230335+0000","last_fullsized":"2024-04-14T23:48:48.230335+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T09:43:16.582708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.549265+0000","last_change":"2024-04-14T23:48:43.230251+0000","last_active":"2024-04-14T23:48:45.549265+0000","last_peered":"2024-04-14T23:48:45.549265+0000","last_clean":"2024-04-14T23:48:45.549265+0000","last_became_active":"2024-04-14T23:48:43.229949+0000","last_became_peered":"2024-04-14T23:48:43.229949+0000","last_unstale":"2024-04-14T23:48:45.549265+0000","last_undegraded":"2024-04-14T23:48:45.549265+0000","last_fullsized":"2024-04-14T23:48:45.549265+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T09:54:42.581135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.048901+0000","last_change":"2024-04-14T23:48:42.227708+0000","last_active":"2024-04-14T23:48:45.048901+0000","last_peered":"2024-04-14T23:48:45.048901+0000","last_clean":"2024-04-14T23:48:45.048901+0000","last_became_active":"2024-04-14T23:48:42.226845+0000","last_became_peered":"2024-04-14T23:48:42.226845+0000","last_unstale":"2024-04-14T23:48:45.048901+0000","last_undegraded":"2024-04-14T23:48:45.048901+0000","last_fullsized":"2024-04-14T23:48:45.048901+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:02:54.468796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230624+0000","last_change":"2024-04-14T23:48:43.227439+0000","last_active":"2024-04-14T23:48:44.230624+0000","last_peered":"2024-04-14T23:48:44.230624+0000","last_clean":"2024-04-14T23:48:44.230624+0000","last_became_active":"2024-04-14T23:48:43.226717+0000","last_became_peered":"2024-04-14T23:48:43.226717+0000","last_unstale":"2024-04-14T23:48:44.230624+0000","last_undegraded":"2024-04-14T23:48:44.230624+0000","last_fullsized":"2024-04-14T23:48:44.230624+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:01:11.249999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.649211+0000","last_change":"2024-04-14T23:48:42.232164+0000","last_active":"2024-04-14T23:48:44.649211+0000","last_peered":"2024-04-14T23:48:44.649211+0000","last_clean":"2024-04-14T23:48:44.649211+0000","last_became_active":"2024-04-14T23:48:42.229785+0000","last_became_peered":"2024-04-14T23:48:42.229785+0000","last_unstale":"2024-04-14T23:48:44.649211+0000","last_undegraded":"2024-04-14T23:48:44.649211+0000","last_fullsized":"2024-04-14T23:48:44.649211+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T04:21:37.538735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.549097+0000","last_change":"2024-04-14T23:48:43.232949+0000","last_active":"2024-04-14T23:48:45.549097+0000","last_peered":"2024-04-14T23:48:45.549097+0000","last_clean":"2024-04-14T23:48:45.549097+0000","last_became_active":"2024-04-14T23:48:43.232612+0000","last_became_peered":"2024-04-14T23:48:43.232612+0000","last_unstale":"2024-04-14T23:48:45.549097+0000","last_undegraded":"2024-04-14T23:48:45.549097+0000","last_fullsized":"2024-04-14T23:48:45.549097+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:01:00.524663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548069+0000","last_change":"2024-04-14T23:48:38.150200+0000","last_active":"2024-04-14T23:48:45.548069+0000","last_peered":"2024-04-14T23:48:45.548069+0000","last_clean":"2024-04-14T23:48:45.548069+0000","last_became_active":"2024-04-14T23:48:38.149301+0000","last_became_peered":"2024-04-14T23:48:38.149301+0000","last_unstale":"2024-04-14T23:48:45.548069+0000","last_undegraded":"2024-04-14T23:48:45.548069+0000","last_fullsized":"2024-04-14T23:48:45.548069+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:37.129663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:37.129663+0000","last_clean_scrub_stamp":"2024-04-14T23:48:37.129663+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T23:58:01.303851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230439+0000","last_change":"2024-04-14T23:48:42.229238+0000","last_active":"2024-04-14T23:48:44.230439+0000","last_peered":"2024-04-14T23:48:44.230439+0000","last_clean":"2024-04-14T23:48:44.230439+0000","last_became_active":"2024-04-14T23:48:42.228994+0000","last_became_peered":"2024-04-14T23:48:42.228994+0000","last_unstale":"2024-04-14T23:48:44.230439+0000","last_undegraded":"2024-04-14T23:48:44.230439+0000","last_fullsized":"2024-04-14T23:48:44.230439+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:47:15.406228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.232256+0000","last_change":"2024-04-14T23:48:43.223973+0000","last_active":"2024-04-14T23:48:44.232256+0000","last_peered":"2024-04-14T23:48:44.232256+0000","last_clean":"2024-04-14T23:48:44.232256+0000","last_became_active":"2024-04-14T23:48:43.223665+0000","last_became_peered":"2024-04-14T23:48:43.223665+0000","last_unstale":"2024-04-14T23:48:44.232256+0000","last_undegraded":"2024-04-14T23:48:44.232256+0000","last_fullsized":"2024-04-14T23:48:44.232256+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:23:28.495821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.547518+0000","last_change":"2024-04-14T23:48:42.230404+0000","last_active":"2024-04-14T23:48:45.547518+0000","last_peered":"2024-04-14T23:48:45.547518+0000","last_clean":"2024-04-14T23:48:45.547518+0000","last_became_active":"2024-04-14T23:48:42.230173+0000","last_became_peered":"2024-04-14T23:48:42.230173+0000","last_unstale":"2024-04-14T23:48:45.547518+0000","last_undegraded":"2024-04-14T23:48:45.547518+0000","last_fullsized":"2024-04-14T23:48:45.547518+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:23:45.666680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.234372+0000","last_change":"2024-04-14T23:48:43.231958+0000","last_active":"2024-04-14T23:48:44.234372+0000","last_peered":"2024-04-14T23:48:44.234372+0000","last_clean":"2024-04-14T23:48:44.234372+0000","last_became_active":"2024-04-14T23:48:43.231025+0000","last_became_peered":"2024-04-14T23:48:43.231025+0000","last_unstale":"2024-04-14T23:48:44.234372+0000","last_undegraded":"2024-04-14T23:48:44.234372+0000","last_fullsized":"2024-04-14T23:48:44.234372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T23:51:47.947116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.649047+0000","last_change":"2024-04-14T23:48:42.240651+0000","last_active":"2024-04-14T23:48:44.649047+0000","last_peered":"2024-04-14T23:48:44.649047+0000","last_clean":"2024-04-14T23:48:44.649047+0000","last_became_active":"2024-04-14T23:48:42.240351+0000","last_became_peered":"2024-04-14T23:48:42.240351+0000","last_unstale":"2024-04-14T23:48:44.649047+0000","last_undegraded":"2024-04-14T23:48:44.649047+0000","last_fullsized":"2024-04-14T23:48:44.649047+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:03:02.804444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.048231+0000","last_change":"2024-04-14T23:48:43.232415+0000","last_active":"2024-04-14T23:48:45.048231+0000","last_peered":"2024-04-14T23:48:45.048231+0000","last_clean":"2024-04-14T23:48:45.048231+0000","last_became_active":"2024-04-14T23:48:43.232120+0000","last_became_peered":"2024-04-14T23:48:43.232120+0000","last_unstale":"2024-04-14T23:48:45.048231+0000","last_undegraded":"2024-04-14T23:48:45.048231+0000","last_fullsized":"2024-04-14T23:48:45.048231+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:05:27.562738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230988+0000","last_change":"2024-04-14T23:48:42.233743+0000","last_active":"2024-04-14T23:48:44.230988+0000","last_peered":"2024-04-14T23:48:44.230988+0000","last_clean":"2024-04-14T23:48:44.230988+0000","last_became_active":"2024-04-14T23:48:42.233354+0000","last_became_peered":"2024-04-14T23:48:42.233354+0000","last_unstale":"2024-04-14T23:48:44.230988+0000","last_undegraded":"2024-04-14T23:48:44.230988+0000","last_fullsized":"2024-04-14T23:48:44.230988+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:49:23.045968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548229+0000","last_change":"2024-04-14T23:48:43.232468+0000","last_active":"2024-04-14T23:48:45.548229+0000","last_peered":"2024-04-14T23:48:45.548229+0000","last_clean":"2024-04-14T23:48:45.548229+0000","last_became_active":"2024-04-14T23:48:43.232119+0000","last_became_peered":"2024-04-14T23:48:43.232119+0000","last_unstale":"2024-04-14T23:48:45.548229+0000","last_undegraded":"2024-04-14T23:48:45.548229+0000","last_fullsized":"2024-04-14T23:48:45.548229+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:24:12.553325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.048630+0000","last_change":"2024-04-14T23:48:42.242249+0000","last_active":"2024-04-14T23:48:45.048630+0000","last_peered":"2024-04-14T23:48:45.048630+0000","last_clean":"2024-04-14T23:48:45.048630+0000","last_became_active":"2024-04-14T23:48:42.242039+0000","last_became_peered":"2024-04-14T23:48:42.242039+0000","last_unstale":"2024-04-14T23:48:45.048630+0000","last_undegraded":"2024-04-14T23:48:45.048630+0000","last_fullsized":"2024-04-14T23:48:45.048630+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:46:16.301900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230960+0000","last_change":"2024-04-14T23:48:43.242228+0000","last_active":"2024-04-14T23:48:44.230960+0000","last_peered":"2024-04-14T23:48:44.230960+0000","last_clean":"2024-04-14T23:48:44.230960+0000","last_became_active":"2024-04-14T23:48:43.241776+0000","last_became_peered":"2024-04-14T23:48:43.241776+0000","last_unstale":"2024-04-14T23:48:44.230960+0000","last_undegraded":"2024-04-14T23:48:44.230960+0000","last_fullsized":"2024-04-14T23:48:44.230960+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:48:23.653866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.549314+0000","last_change":"2024-04-14T23:48:42.232537+0000","last_active":"2024-04-14T23:48:45.549314+0000","last_peered":"2024-04-14T23:48:45.549314+0000","last_clean":"2024-04-14T23:48:45.549314+0000","last_became_active":"2024-04-14T23:48:42.232325+0000","last_became_peered":"2024-04-14T23:48:42.232325+0000","last_unstale":"2024-04-14T23:48:45.549314+0000","last_undegraded":"2024-04-14T23:48:45.549314+0000","last_fullsized":"2024-04-14T23:48:45.549314+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:34:37.448301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.048333+0000","last_change":"2024-04-14T23:48:43.234445+0000","last_active":"2024-04-14T23:48:45.048333+0000","last_peered":"2024-04-14T23:48:45.048333+0000","last_clean":"2024-04-14T23:48:45.048333+0000","last_became_active":"2024-04-14T23:48:43.234150+0000","last_became_peered":"2024-04-14T23:48:43.234150+0000","last_unstale":"2024-04-14T23:48:45.048333+0000","last_undegraded":"2024-04-14T23:48:45.048333+0000","last_fullsized":"2024-04-14T23:48:45.048333+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:50:59.708087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.049158+0000","last_change":"2024-04-14T23:48:42.230014+0000","last_active":"2024-04-14T23:48:45.049158+0000","last_peered":"2024-04-14T23:48:45.049158+0000","last_clean":"2024-04-14T23:48:45.049158+0000","last_became_active":"2024-04-14T23:48:42.229712+0000","last_became_peered":"2024-04-14T23:48:42.229712+0000","last_unstale":"2024-04-14T23:48:45.049158+0000","last_undegraded":"2024-04-14T23:48:45.049158+0000","last_fullsized":"2024-04-14T23:48:45.049158+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:54:20.595391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.233392+0000","last_change":"2024-04-14T23:48:43.229128+0000","last_active":"2024-04-14T23:48:44.233392+0000","last_peered":"2024-04-14T23:48:44.233392+0000","last_clean":"2024-04-14T23:48:44.233392+0000","last_became_active":"2024-04-14T23:48:43.228739+0000","last_became_peered":"2024-04-14T23:48:43.228739+0000","last_unstale":"2024-04-14T23:48:44.233392+0000","last_undegraded":"2024-04-14T23:48:44.233392+0000","last_fullsized":"2024-04-14T23:48:44.233392+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:08:08.148345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:48.223491+0000","last_change":"2024-04-14T23:48:42.220035+0000","last_active":"2024-04-14T23:48:48.223491+0000","last_peered":"2024-04-14T23:48:48.223491+0000","last_clean":"2024-04-14T23:48:48.223491+0000","last_became_active":"2024-04-14T23:48:42.219750+0000","last_became_peered":"2024-04-14T23:48:42.219750+0000","last_unstale":"2024-04-14T23:48:48.223491+0000","last_undegraded":"2024-04-14T23:48:48.223491+0000","last_fullsized":"2024-04-14T23:48:48.223491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:00:58.928166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.235426+0000","last_change":"2024-04-14T23:48:43.232997+0000","last_active":"2024-04-14T23:48:44.235426+0000","last_peered":"2024-04-14T23:48:44.235426+0000","last_clean":"2024-04-14T23:48:44.235426+0000","last_became_active":"2024-04-14T23:48:43.232730+0000","last_became_peered":"2024-04-14T23:48:43.232730+0000","last_unstale":"2024-04-14T23:48:44.235426+0000","last_undegraded":"2024-04-14T23:48:44.235426+0000","last_fullsized":"2024-04-14T23:48:44.235426+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:38:44.068804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.547928+0000","last_change":"2024-04-14T23:48:42.232087+0000","last_active":"2024-04-14T23:48:45.547928+0000","last_peered":"2024-04-14T23:48:45.547928+0000","last_clean":"2024-04-14T23:48:45.547928+0000","last_became_active":"2024-04-14T23:48:42.231796+0000","last_became_peered":"2024-04-14T23:48:42.231796+0000","last_unstale":"2024-04-14T23:48:45.547928+0000","last_undegraded":"2024-04-14T23:48:45.547928+0000","last_fullsized":"2024-04-14T23:48:45.547928+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:14:35.050093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548007+0000","last_change":"2024-04-14T23:48:43.232871+0000","last_active":"2024-04-14T23:48:45.548007+0000","last_peered":"2024-04-14T23:48:45.548007+0000","last_clean":"2024-04-14T23:48:45.548007+0000","last_became_active":"2024-04-14T23:48:43.232594+0000","last_became_peered":"2024-04-14T23:48:43.232594+0000","last_unstale":"2024-04-14T23:48:45.548007+0000","last_undegraded":"2024-04-14T23:48:45.548007+0000","last_fullsized":"2024-04-14T23:48:45.548007+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:27:37.320994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.048370+0000","last_change":"2024-04-14T23:48:42.231237+0000","last_active":"2024-04-14T23:48:45.048370+0000","last_peered":"2024-04-14T23:48:45.048370+0000","last_clean":"2024-04-14T23:48:45.048370+0000","last_became_active":"2024-04-14T23:48:42.230902+0000","last_became_peered":"2024-04-14T23:48:42.230902+0000","last_unstale":"2024-04-14T23:48:45.048370+0000","last_undegraded":"2024-04-14T23:48:45.048370+0000","last_fullsized":"2024-04-14T23:48:45.048370+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:28:37.228039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.232080+0000","last_change":"2024-04-14T23:48:43.233035+0000","last_active":"2024-04-14T23:48:44.232080+0000","last_peered":"2024-04-14T23:48:44.232080+0000","last_clean":"2024-04-14T23:48:44.232080+0000","last_became_active":"2024-04-14T23:48:43.232747+0000","last_became_peered":"2024-04-14T23:48:43.232747+0000","last_unstale":"2024-04-14T23:48:44.232080+0000","last_undegraded":"2024-04-14T23:48:44.232080+0000","last_fullsized":"2024-04-14T23:48:44.232080+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:38:57.324408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.048301+0000","last_change":"2024-04-14T23:48:42.240496+0000","last_active":"2024-04-14T23:48:45.048301+0000","last_peered":"2024-04-14T23:48:45.048301+0000","last_clean":"2024-04-14T23:48:45.048301+0000","last_became_active":"2024-04-14T23:48:42.238524+0000","last_became_peered":"2024-04-14T23:48:42.238524+0000","last_unstale":"2024-04-14T23:48:45.048301+0000","last_undegraded":"2024-04-14T23:48:45.048301+0000","last_fullsized":"2024-04-14T23:48:45.048301+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:29:35.218759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.233225+0000","last_change":"2024-04-14T23:48:43.234025+0000","last_active":"2024-04-14T23:48:44.233225+0000","last_peered":"2024-04-14T23:48:44.233225+0000","last_clean":"2024-04-14T23:48:44.233225+0000","last_became_active":"2024-04-14T23:48:43.233651+0000","last_became_peered":"2024-04-14T23:48:43.233651+0000","last_unstale":"2024-04-14T23:48:44.233225+0000","last_undegraded":"2024-04-14T23:48:44.233225+0000","last_fullsized":"2024-04-14T23:48:44.233225+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:26:58.094545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:48.217913+0000","last_change":"2024-04-14T23:48:42.218897+0000","last_active":"2024-04-14T23:48:48.217913+0000","last_peered":"2024-04-14T23:48:48.217913+0000","last_clean":"2024-04-14T23:48:48.217913+0000","last_became_active":"2024-04-14T23:48:42.218580+0000","last_became_peered":"2024-04-14T23:48:42.218580+0000","last_unstale":"2024-04-14T23:48:48.217913+0000","last_undegraded":"2024-04-14T23:48:48.217913+0000","last_fullsized":"2024-04-14T23:48:48.217913+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:26:44.477638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548008+0000","last_change":"2024-04-14T23:48:43.219871+0000","last_active":"2024-04-14T23:48:45.548008+0000","last_peered":"2024-04-14T23:48:45.548008+0000","last_clean":"2024-04-14T23:48:45.548008+0000","last_became_active":"2024-04-14T23:48:43.219601+0000","last_became_peered":"2024-04-14T23:48:43.219601+0000","last_unstale":"2024-04-14T23:48:45.548008+0000","last_undegraded":"2024-04-14T23:48:45.548008+0000","last_fullsized":"2024-04-14T23:48:45.548008+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:36:20.379172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230971+0000","last_change":"2024-04-14T23:48:43.222259+0000","last_active":"2024-04-14T23:48:44.230971+0000","last_peered":"2024-04-14T23:48:44.230971+0000","last_clean":"2024-04-14T23:48:44.230971+0000","last_became_active":"2024-04-14T23:48:43.221842+0000","last_became_peered":"2024-04-14T23:48:43.221842+0000","last_unstale":"2024-04-14T23:48:44.230971+0000","last_undegraded":"2024-04-14T23:48:44.230971+0000","last_fullsized":"2024-04-14T23:48:44.230971+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:12:20.038987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:48.217880+0000","last_change":"2024-04-14T23:48:42.221314+0000","last_active":"2024-04-14T23:48:48.217880+0000","last_peered":"2024-04-14T23:48:48.217880+0000","last_clean":"2024-04-14T23:48:48.217880+0000","last_became_active":"2024-04-14T23:48:42.221028+0000","last_became_peered":"2024-04-14T23:48:42.221028+0000","last_unstale":"2024-04-14T23:48:48.217880+0000","last_undegraded":"2024-04-14T23:48:48.217880+0000","last_fullsized":"2024-04-14T23:48:48.217880+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T09:58:18.195400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230606+0000","last_change":"2024-04-14T23:48:43.224577+0000","last_active":"2024-04-14T23:48:44.230606+0000","last_peered":"2024-04-14T23:48:44.230606+0000","last_clean":"2024-04-14T23:48:44.230606+0000","last_became_active":"2024-04-14T23:48:43.224312+0000","last_became_peered":"2024-04-14T23:48:43.224312+0000","last_unstale":"2024-04-14T23:48:44.230606+0000","last_undegraded":"2024-04-14T23:48:44.230606+0000","last_fullsized":"2024-04-14T23:48:44.230606+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:52:44.955773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:48.217170+0000","last_change":"2024-04-14T23:48:42.221000+0000","last_active":"2024-04-14T23:48:48.217170+0000","last_peered":"2024-04-14T23:48:48.217170+0000","last_clean":"2024-04-14T23:48:48.217170+0000","last_became_active":"2024-04-14T23:48:42.219813+0000","last_became_peered":"2024-04-14T23:48:42.219813+0000","last_unstale":"2024-04-14T23:48:48.217170+0000","last_undegraded":"2024-04-14T23:48:48.217170+0000","last_fullsized":"2024-04-14T23:48:48.217170+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:00:38.107953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.235379+0000","last_change":"2024-04-14T23:48:43.236946+0000","last_active":"2024-04-14T23:48:44.235379+0000","last_peered":"2024-04-14T23:48:44.235379+0000","last_clean":"2024-04-14T23:48:44.235379+0000","last_became_active":"2024-04-14T23:48:43.236595+0000","last_became_peered":"2024-04-14T23:48:43.236595+0000","last_unstale":"2024-04-14T23:48:44.235379+0000","last_undegraded":"2024-04-14T23:48:44.235379+0000","last_fullsized":"2024-04-14T23:48:44.235379+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:56:46.037478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.211743+0000","last_change":"2024-04-14T23:48:42.239255+0000","last_active":"2024-04-14T23:48:45.211743+0000","last_peered":"2024-04-14T23:48:45.211743+0000","last_clean":"2024-04-14T23:48:45.211743+0000","last_became_active":"2024-04-14T23:48:42.238892+0000","last_became_peered":"2024-04-14T23:48:42.238892+0000","last_unstale":"2024-04-14T23:48:45.211743+0000","last_undegraded":"2024-04-14T23:48:45.211743+0000","last_fullsized":"2024-04-14T23:48:45.211743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:44:39.632155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.233893+0000","last_change":"2024-04-14T23:48:43.227240+0000","last_active":"2024-04-14T23:48:44.233893+0000","last_peered":"2024-04-14T23:48:44.233893+0000","last_clean":"2024-04-14T23:48:44.233893+0000","last_became_active":"2024-04-14T23:48:43.222707+0000","last_became_peered":"2024-04-14T23:48:43.222707+0000","last_unstale":"2024-04-14T23:48:44.233893+0000","last_undegraded":"2024-04-14T23:48:44.233893+0000","last_fullsized":"2024-04-14T23:48:44.233893+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:02:17.231763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:48.228047+0000","last_change":"2024-04-14T23:48:42.226563+0000","last_active":"2024-04-14T23:48:48.228047+0000","last_peered":"2024-04-14T23:48:48.228047+0000","last_clean":"2024-04-14T23:48:48.228047+0000","last_became_active":"2024-04-14T23:48:42.226238+0000","last_became_peered":"2024-04-14T23:48:42.226238+0000","last_unstale":"2024-04-14T23:48:48.228047+0000","last_undegraded":"2024-04-14T23:48:48.228047+0000","last_fullsized":"2024-04-14T23:48:48.228047+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:01:36.516898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":3655,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65128,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574858,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":286720,"data_stored":666555,"data_compressed":32564,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574859,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":78462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574858,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":221184,"data_stored":75695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574859,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":75731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":78462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574858,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574858,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":225280,"data_stored":75765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":294912,"data_stored":666611,"data_compressed":32564,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32564,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32564,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-14T23:49:12.313 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-14T23:49:12.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.568+0000 7f8086400640 1 Processor -- start 2024-04-14T23:49:12.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.568+0000 7f8086400640 1 -- start start 2024-04-14T23:49:12.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.569+0000 7f8086400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f80800c63c0 0x7f80800c6b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:12.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.570+0000 7f8086400640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f808015c9b0 0x7f80800c70b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:12.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.570+0000 7f8086400640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80801530d0 con 0x7f8080152cf0 2024-04-14T23:49:12.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.570+0000 7f8086400640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f80800751f0 con 0x7f808015c9b0 2024-04-14T23:49:12.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.570+0000 7f8086400640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f80800c8370 con 0x7f80800c63c0 2024-04-14T23:49:12.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.570+0000 7f807ea00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f80800c63c0 0x7f80800c6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:12.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.570+0000 7f807fe00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f808015c9b0 0x7f80800c70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:12.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.570+0000 7f807f400640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f8080152cf0 0x7f808015c2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.60:36236/0 (socket says 172.21.15.60:36236) 2024-04-14T23:49:12.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.570+0000 7f807f400640 1 -- 172.21.15.60:0/2478512536 learned_addr learned my addr 172.21.15.60:0/2478512536 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:49:12.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.570+0000 7f807e000640 1 -- 172.21.15.60:0/2478512536 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 230406570 0 0) 0x7f80801530d0 con 0x7f8080152cf0 2024-04-14T23:49:12.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.570+0000 7f807e000640 1 -- 172.21.15.60:0/2478512536 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8064003520 con 0x7f8080152cf0 2024-04-14T23:49:12.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.571+0000 7f807ea00640 1 -- 172.21.15.60:0/2478512536 >> v2:172.21.15.186:3300/0 conn(0x7f808015c9b0 msgr2=0x7f80800c70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:49:12.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.571+0000 7f807ea00640 1 --2- 172.21.15.60:0/2478512536 >> v2:172.21.15.186:3300/0 conn(0x7f808015c9b0 0x7f80800c70b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:12.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.571+0000 7f807ea00640 1 -- 172.21.15.60:0/2478512536 >> v1:172.21.15.186:6789/0 conn(0x7f8080152cf0 legacy=0x7f808015c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:12.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.571+0000 7f807ea00640 1 -- 172.21.15.60:0/2478512536 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80800c3810 con 0x7f80800c63c0 2024-04-14T23:49:12.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.571+0000 7f807fe00640 1 --2- 172.21.15.60:0/2478512536 >> v2:172.21.15.186:3300/0 conn(0x7f808015c9b0 0x7f80800c70b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:49:12.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.572+0000 7f807ea00640 1 --2- 172.21.15.60:0/2478512536 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f80800c63c0 0x7f80800c6b70 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f80680097a0 tx=0x7f806802b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0512714de777b0f server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:12.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.572+0000 7f807e000640 1 -- 172.21.15.60:0/2478512536 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8068034070 con 0x7f80800c63c0 2024-04-14T23:49:12.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.572+0000 7f807e000640 1 -- 172.21.15.60:0/2478512536 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8068030610 con 0x7f80800c63c0 2024-04-14T23:49:12.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.572+0000 7f807e000640 1 -- 172.21.15.60:0/2478512536 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80680309c0 con 0x7f80800c63c0 2024-04-14T23:49:12.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.573+0000 7f8086400640 1 -- 172.21.15.60:0/2478512536 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f80800c63c0 msgr2=0x7f80800c6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:12.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.573+0000 7f8086400640 1 --2- 172.21.15.60:0/2478512536 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f80800c63c0 0x7f80800c6b70 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f80680097a0 tx=0x7f806802b960 comp rx=0 tx=0).stop 2024-04-14T23:49:12.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.573+0000 7f8086400640 1 -- 172.21.15.60:0/2478512536 shutdown_connections 2024-04-14T23:49:12.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.573+0000 7f8086400640 1 --2- 172.21.15.60:0/2478512536 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f80800c63c0 0x7f80800c6b70 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:12.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.573+0000 7f8086400640 1 --2- 172.21.15.60:0/2478512536 >> v2:172.21.15.186:3300/0 conn(0x7f808015c9b0 0x7f80800c70b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:12.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.573+0000 7f8086400640 1 -- 172.21.15.60:0/2478512536 >> 172.21.15.60:0/2478512536 conn(0x7f80800cc170 msgr2=0x7f80800cc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:49:12.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.573+0000 7f8086400640 1 -- 172.21.15.60:0/2478512536 shutdown_connections 2024-04-14T23:49:12.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.573+0000 7f8086400640 1 -- 172.21.15.60:0/2478512536 wait complete. 2024-04-14T23:49:12.581 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.575+0000 7f8086400640 1 Processor -- start 2024-04-14T23:49:12.581 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.576+0000 7f8086400640 1 -- start start 2024-04-14T23:49:12.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.577+0000 7f8086400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f80800c63c0 0x7f80800cb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:12.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.577+0000 7f8086400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8080152cf0 0x7f80800c4520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:12.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.577+0000 7f8086400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f808015c9b0 0x7f808013ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:12.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.577+0000 7f8086400640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f80800c88f0 con 0x7f80800c63c0 2024-04-14T23:49:12.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.577+0000 7f8086400640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f80800c8610 con 0x7f808015c9b0 2024-04-14T23:49:12.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.577+0000 7f8086400640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f8080159450 con 0x7f8080152cf0 2024-04-14T23:49:12.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.577+0000 7f807f400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f80800c63c0 0x7f80800cb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:12.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.577+0000 7f807ea00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8080152cf0 0x7f80800c4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:12.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.577+0000 7f807fe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f808015c9b0 0x7f808013ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:12.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.577+0000 7f807fe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f808015c9b0 0x7f808013ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:39746/0 (socket says 172.21.15.60:39746) 2024-04-14T23:49:12.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.577+0000 7f807ea00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8080152cf0 0x7f80800c4520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.60:32846/0 (socket says 172.21.15.60:32846) 2024-04-14T23:49:12.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.577+0000 7f807fe00640 1 -- 172.21.15.60:0/1755564129 learned_addr learned my addr 172.21.15.60:0/1755564129 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:49:12.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.578+0000 7f807ea00640 1 -- 172.21.15.60:0/1755564129 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f808015c9b0 msgr2=0x7f808013ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:12.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.578+0000 7f807ea00640 1 --2- 172.21.15.60:0/1755564129 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f808015c9b0 0x7f808013ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:12.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.578+0000 7f807ea00640 1 -- 172.21.15.60:0/1755564129 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f80800c63c0 msgr2=0x7f80800cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:12.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.578+0000 7f807ea00640 1 --2- 172.21.15.60:0/1755564129 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f80800c63c0 0x7f80800cb9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:12.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.578+0000 7f807ea00640 1 -- 172.21.15.60:0/1755564129 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80800cb5a0 con 0x7f8080152cf0 2024-04-14T23:49:12.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.578+0000 7f807fe00640 1 --2- 172.21.15.60:0/1755564129 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f808015c9b0 0x7f808013ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:49:12.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.578+0000 7f807ea00640 1 --2- 172.21.15.60:0/1755564129 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8080152cf0 0x7f80800c4520 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8068033000 tx=0x7f80680046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:12.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.579+0000 7f8077400640 1 -- 172.21.15.60:0/1755564129 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8068048070 con 0x7f8080152cf0 2024-04-14T23:49:12.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.579+0000 7f8077400640 1 -- 172.21.15.60:0/1755564129 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8068030410 con 0x7f8080152cf0 2024-04-14T23:49:12.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.579+0000 7f8077400640 1 -- 172.21.15.60:0/1755564129 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8068034050 con 0x7f8080152cf0 2024-04-14T23:49:12.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.579+0000 7f8086400640 1 -- 172.21.15.60:0/1755564129 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f808014ed10 con 0x7f8080152cf0 2024-04-14T23:49:12.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.579+0000 7f8086400640 1 -- 172.21.15.60:0/1755564129 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f808014d130 con 0x7f8080152cf0 2024-04-14T23:49:12.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.580+0000 7f8086400640 1 -- 172.21.15.60:0/1755564129 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80800c7d30 con 0x7f8080152cf0 2024-04-14T23:49:12.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.581+0000 7f8077400640 1 -- 172.21.15.60:0/1755564129 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f806804b070 con 0x7f8080152cf0 2024-04-14T23:49:12.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.582+0000 7f8077400640 1 --2- 172.21.15.60:0/1755564129 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f805c06b9e0 0x7f805c06dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:12.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.582+0000 7f8077400640 1 -- 172.21.15.60:0/1755564129 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f80680ac420 con 0x7f8080152cf0 2024-04-14T23:49:12.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.582+0000 7f807f400640 1 --2- 172.21.15.60:0/1755564129 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f805c06b9e0 0x7f805c06dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:12.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.583+0000 7f807f400640 1 --2- 172.21.15.60:0/1755564129 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f805c06b9e0 0x7f805c06dea0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8070005880 tx=0x7f8070009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:12.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.587+0000 7f8077400640 1 -- 172.21.15.60:0/1755564129 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f80800c7d30 con 0x7f8080152cf0 2024-04-14T23:49:12.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.781+0000 7f8086400640 1 -- 172.21.15.60:0/1755564129 --> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f808014cd70 con 0x7f805c06b9e0 2024-04-14T23:49:12.793 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.788+0000 7f8077400640 1 -- 172.21.15.60:0/1755564129 <== mgr.4100 v2:172.21.15.186:6832/714104835 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251932 (secure 0 0 0) 0x7f808014cd70 con 0x7f805c06b9e0 2024-04-14T23:49:12.797 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:49:12.798 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-04-14T23:49:12.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.792+0000 7f8086400640 1 -- 172.21.15.60:0/1755564129 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f805c06b9e0 msgr2=0x7f805c06dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:12.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.793+0000 7f8086400640 1 --2- 172.21.15.60:0/1755564129 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f805c06b9e0 0x7f805c06dea0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8070005880 tx=0x7f8070009040 comp rx=0 tx=0).stop 2024-04-14T23:49:12.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.793+0000 7f8086400640 1 -- 172.21.15.60:0/1755564129 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8080152cf0 msgr2=0x7f80800c4520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:12.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.793+0000 7f8086400640 1 --2- 172.21.15.60:0/1755564129 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8080152cf0 0x7f80800c4520 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8068033000 tx=0x7f80680046c0 comp rx=0 tx=0).stop 2024-04-14T23:49:12.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.793+0000 7f8086400640 1 -- 172.21.15.60:0/1755564129 shutdown_connections 2024-04-14T23:49:12.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.793+0000 7f8086400640 1 --2- 172.21.15.60:0/1755564129 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f805c06b9e0 0x7f805c06dea0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:12.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.793+0000 7f8086400640 1 --2- 172.21.15.60:0/1755564129 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f808015c9b0 0x7f808013ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:12.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.793+0000 7f8086400640 1 --2- 172.21.15.60:0/1755564129 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8080152cf0 0x7f80800c4520 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:12.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.793+0000 7f8086400640 1 --2- 172.21.15.60:0/1755564129 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f80800c63c0 0x7f80800cb9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:12.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.793+0000 7f8086400640 1 -- 172.21.15.60:0/1755564129 >> 172.21.15.60:0/1755564129 conn(0x7f80800cc170 msgr2=0x7f8080148cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:49:12.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.793+0000 7f8086400640 1 -- 172.21.15.60:0/1755564129 shutdown_connections 2024-04-14T23:49:12.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:12.793+0000 7f8086400640 1 -- 172.21.15.60:0/1755564129 wait complete. 2024-04-14T23:49:12.817 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2024-04-14T23:49:12.173811+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594023,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218016,"kb_used_data":1888,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756704,"statfs":{"total":773094113280,"available":772870864896,"internally_reserved":0,"allocated":1933312,"data_stored":1792886,"data_compressed":65128,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1245,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003322"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.549104+0000","last_change":"2024-04-14T23:48:43.222542+0000","last_active":"2024-04-14T23:48:45.549104+0000","last_peered":"2024-04-14T23:48:45.549104+0000","last_clean":"2024-04-14T23:48:45.549104+0000","last_became_active":"2024-04-14T23:48:43.222278+0000","last_became_peered":"2024-04-14T23:48:43.222278+0000","last_unstale":"2024-04-14T23:48:45.549104+0000","last_undegraded":"2024-04-14T23:48:45.549104+0000","last_fullsized":"2024-04-14T23:48:45.549104+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:13:06.877485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.049059+0000","last_change":"2024-04-14T23:48:43.226781+0000","last_active":"2024-04-14T23:48:45.049059+0000","last_peered":"2024-04-14T23:48:45.049059+0000","last_clean":"2024-04-14T23:48:45.049059+0000","last_became_active":"2024-04-14T23:48:43.226426+0000","last_became_peered":"2024-04-14T23:48:43.226426+0000","last_unstale":"2024-04-14T23:48:45.049059+0000","last_undegraded":"2024-04-14T23:48:45.049059+0000","last_fullsized":"2024-04-14T23:48:45.049059+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:35:24.780859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.235142+0000","last_change":"2024-04-14T23:48:43.236939+0000","last_active":"2024-04-14T23:48:44.235142+0000","last_peered":"2024-04-14T23:48:44.235142+0000","last_clean":"2024-04-14T23:48:44.235142+0000","last_became_active":"2024-04-14T23:48:43.236616+0000","last_became_peered":"2024-04-14T23:48:43.236616+0000","last_unstale":"2024-04-14T23:48:44.235142+0000","last_undegraded":"2024-04-14T23:48:44.235142+0000","last_fullsized":"2024-04-14T23:48:44.235142+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:35:42.619139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230420+0000","last_change":"2024-04-14T23:48:43.242349+0000","last_active":"2024-04-14T23:48:44.230420+0000","last_peered":"2024-04-14T23:48:44.230420+0000","last_clean":"2024-04-14T23:48:44.230420+0000","last_became_active":"2024-04-14T23:48:43.242000+0000","last_became_peered":"2024-04-14T23:48:43.242000+0000","last_unstale":"2024-04-14T23:48:44.230420+0000","last_undegraded":"2024-04-14T23:48:44.230420+0000","last_fullsized":"2024-04-14T23:48:44.230420+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:33:52.458016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548549+0000","last_change":"2024-04-14T23:48:43.233937+0000","last_active":"2024-04-14T23:48:45.548549+0000","last_peered":"2024-04-14T23:48:45.548549+0000","last_clean":"2024-04-14T23:48:45.548549+0000","last_became_active":"2024-04-14T23:48:43.233674+0000","last_became_peered":"2024-04-14T23:48:43.233674+0000","last_unstale":"2024-04-14T23:48:45.548549+0000","last_undegraded":"2024-04-14T23:48:45.548549+0000","last_fullsized":"2024-04-14T23:48:45.548549+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:00:07.838600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230260+0000","last_change":"2024-04-14T23:48:43.230925+0000","last_active":"2024-04-14T23:48:44.230260+0000","last_peered":"2024-04-14T23:48:44.230260+0000","last_clean":"2024-04-14T23:48:44.230260+0000","last_became_active":"2024-04-14T23:48:43.230405+0000","last_became_peered":"2024-04-14T23:48:43.230405+0000","last_unstale":"2024-04-14T23:48:44.230260+0000","last_undegraded":"2024-04-14T23:48:44.230260+0000","last_fullsized":"2024-04-14T23:48:44.230260+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:15:12.452964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548354+0000","last_change":"2024-04-14T23:48:43.231088+0000","last_active":"2024-04-14T23:48:45.548354+0000","last_peered":"2024-04-14T23:48:45.548354+0000","last_clean":"2024-04-14T23:48:45.548354+0000","last_became_active":"2024-04-14T23:48:43.230784+0000","last_became_peered":"2024-04-14T23:48:43.230784+0000","last_unstale":"2024-04-14T23:48:45.548354+0000","last_undegraded":"2024-04-14T23:48:45.548354+0000","last_fullsized":"2024-04-14T23:48:45.548354+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:23:10.461088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.231230+0000","last_change":"2024-04-14T23:48:43.236019+0000","last_active":"2024-04-14T23:48:44.231230+0000","last_peered":"2024-04-14T23:48:44.231230+0000","last_clean":"2024-04-14T23:48:44.231230+0000","last_became_active":"2024-04-14T23:48:43.235719+0000","last_became_peered":"2024-04-14T23:48:43.235719+0000","last_unstale":"2024-04-14T23:48:44.231230+0000","last_undegraded":"2024-04-14T23:48:44.231230+0000","last_fullsized":"2024-04-14T23:48:44.231230+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T04:41:16.130730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.549156+0000","last_change":"2024-04-14T23:48:43.232166+0000","last_active":"2024-04-14T23:48:45.549156+0000","last_peered":"2024-04-14T23:48:45.549156+0000","last_clean":"2024-04-14T23:48:45.549156+0000","last_became_active":"2024-04-14T23:48:43.231751+0000","last_became_peered":"2024-04-14T23:48:43.231751+0000","last_unstale":"2024-04-14T23:48:45.549156+0000","last_undegraded":"2024-04-14T23:48:45.549156+0000","last_fullsized":"2024-04-14T23:48:45.549156+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:05:47.289092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548848+0000","last_change":"2024-04-14T23:48:43.224938+0000","last_active":"2024-04-14T23:48:45.548848+0000","last_peered":"2024-04-14T23:48:45.548848+0000","last_clean":"2024-04-14T23:48:45.548848+0000","last_became_active":"2024-04-14T23:48:43.224592+0000","last_became_peered":"2024-04-14T23:48:43.224592+0000","last_unstale":"2024-04-14T23:48:45.548848+0000","last_undegraded":"2024-04-14T23:48:45.548848+0000","last_fullsized":"2024-04-14T23:48:45.548848+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:59:12.923596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548591+0000","last_change":"2024-04-14T23:48:43.225290+0000","last_active":"2024-04-14T23:48:45.548591+0000","last_peered":"2024-04-14T23:48:45.548591+0000","last_clean":"2024-04-14T23:48:45.548591+0000","last_became_active":"2024-04-14T23:48:43.224989+0000","last_became_peered":"2024-04-14T23:48:43.224989+0000","last_unstale":"2024-04-14T23:48:45.548591+0000","last_undegraded":"2024-04-14T23:48:45.548591+0000","last_fullsized":"2024-04-14T23:48:45.548591+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:44:49.116229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548742+0000","last_change":"2024-04-14T23:48:43.226194+0000","last_active":"2024-04-14T23:48:45.548742+0000","last_peered":"2024-04-14T23:48:45.548742+0000","last_clean":"2024-04-14T23:48:45.548742+0000","last_became_active":"2024-04-14T23:48:43.225892+0000","last_became_peered":"2024-04-14T23:48:43.225892+0000","last_unstale":"2024-04-14T23:48:45.548742+0000","last_undegraded":"2024-04-14T23:48:45.548742+0000","last_fullsized":"2024-04-14T23:48:45.548742+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:04:12.377770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.233039+0000","last_change":"2024-04-14T23:48:43.220571+0000","last_active":"2024-04-14T23:48:44.233039+0000","last_peered":"2024-04-14T23:48:44.233039+0000","last_clean":"2024-04-14T23:48:44.233039+0000","last_became_active":"2024-04-14T23:48:43.219614+0000","last_became_peered":"2024-04-14T23:48:43.219614+0000","last_unstale":"2024-04-14T23:48:44.233039+0000","last_undegraded":"2024-04-14T23:48:44.233039+0000","last_fullsized":"2024-04-14T23:48:44.233039+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:14:41.854957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.234686+0000","last_change":"2024-04-14T23:48:43.239403+0000","last_active":"2024-04-14T23:48:44.234686+0000","last_peered":"2024-04-14T23:48:44.234686+0000","last_clean":"2024-04-14T23:48:44.234686+0000","last_became_active":"2024-04-14T23:48:43.239136+0000","last_became_peered":"2024-04-14T23:48:43.239136+0000","last_unstale":"2024-04-14T23:48:44.234686+0000","last_undegraded":"2024-04-14T23:48:44.234686+0000","last_fullsized":"2024-04-14T23:48:44.234686+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:27:30.512861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.048083+0000","last_change":"2024-04-14T23:48:43.228157+0000","last_active":"2024-04-14T23:48:45.048083+0000","last_peered":"2024-04-14T23:48:45.048083+0000","last_clean":"2024-04-14T23:48:45.048083+0000","last_became_active":"2024-04-14T23:48:43.227898+0000","last_became_peered":"2024-04-14T23:48:43.227898+0000","last_unstale":"2024-04-14T23:48:45.048083+0000","last_undegraded":"2024-04-14T23:48:45.048083+0000","last_fullsized":"2024-04-14T23:48:45.048083+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:06:37.602935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230387+0000","last_change":"2024-04-14T23:48:43.238986+0000","last_active":"2024-04-14T23:48:44.230387+0000","last_peered":"2024-04-14T23:48:44.230387+0000","last_clean":"2024-04-14T23:48:44.230387+0000","last_became_active":"2024-04-14T23:48:43.238714+0000","last_became_peered":"2024-04-14T23:48:43.238714+0000","last_unstale":"2024-04-14T23:48:44.230387+0000","last_undegraded":"2024-04-14T23:48:44.230387+0000","last_fullsized":"2024-04-14T23:48:44.230387+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:05:30.655048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.231004+0000","last_change":"2024-04-14T23:48:43.229325+0000","last_active":"2024-04-14T23:48:44.231004+0000","last_peered":"2024-04-14T23:48:44.231004+0000","last_clean":"2024-04-14T23:48:44.231004+0000","last_became_active":"2024-04-14T23:48:43.228618+0000","last_became_peered":"2024-04-14T23:48:43.228618+0000","last_unstale":"2024-04-14T23:48:44.231004+0000","last_undegraded":"2024-04-14T23:48:44.231004+0000","last_fullsized":"2024-04-14T23:48:44.231004+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:06:43.745698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.649355+0000","last_change":"2024-04-14T23:48:43.233288+0000","last_active":"2024-04-14T23:48:44.649355+0000","last_peered":"2024-04-14T23:48:44.649355+0000","last_clean":"2024-04-14T23:48:44.649355+0000","last_became_active":"2024-04-14T23:48:43.232971+0000","last_became_peered":"2024-04-14T23:48:43.232971+0000","last_unstale":"2024-04-14T23:48:44.649355+0000","last_undegraded":"2024-04-14T23:48:44.649355+0000","last_fullsized":"2024-04-14T23:48:44.649355+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:06:23.588161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.549346+0000","last_change":"2024-04-14T23:48:43.233169+0000","last_active":"2024-04-14T23:48:45.549346+0000","last_peered":"2024-04-14T23:48:45.549346+0000","last_clean":"2024-04-14T23:48:45.549346+0000","last_became_active":"2024-04-14T23:48:43.232135+0000","last_became_peered":"2024-04-14T23:48:43.232135+0000","last_unstale":"2024-04-14T23:48:45.549346+0000","last_undegraded":"2024-04-14T23:48:45.549346+0000","last_fullsized":"2024-04-14T23:48:45.549346+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:37:06.166596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.547820+0000","last_change":"2024-04-14T23:48:43.228728+0000","last_active":"2024-04-14T23:48:45.547820+0000","last_peered":"2024-04-14T23:48:45.547820+0000","last_clean":"2024-04-14T23:48:45.547820+0000","last_became_active":"2024-04-14T23:48:43.226225+0000","last_became_peered":"2024-04-14T23:48:43.226225+0000","last_unstale":"2024-04-14T23:48:45.547820+0000","last_undegraded":"2024-04-14T23:48:45.547820+0000","last_fullsized":"2024-04-14T23:48:45.547820+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:06:03.453599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.234949+0000","last_change":"2024-04-14T23:48:43.237234+0000","last_active":"2024-04-14T23:48:44.234949+0000","last_peered":"2024-04-14T23:48:44.234949+0000","last_clean":"2024-04-14T23:48:44.234949+0000","last_became_active":"2024-04-14T23:48:43.230056+0000","last_became_peered":"2024-04-14T23:48:43.230056+0000","last_unstale":"2024-04-14T23:48:44.234949+0000","last_undegraded":"2024-04-14T23:48:44.234949+0000","last_fullsized":"2024-04-14T23:48:44.234949+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:43:44.966384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230697+0000","last_change":"2024-04-14T23:48:43.229387+0000","last_active":"2024-04-14T23:48:44.230697+0000","last_peered":"2024-04-14T23:48:44.230697+0000","last_clean":"2024-04-14T23:48:44.230697+0000","last_became_active":"2024-04-14T23:48:43.229110+0000","last_became_peered":"2024-04-14T23:48:43.229110+0000","last_unstale":"2024-04-14T23:48:44.230697+0000","last_undegraded":"2024-04-14T23:48:44.230697+0000","last_fullsized":"2024-04-14T23:48:44.230697+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T09:18:46.939940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.232750+0000","last_change":"2024-04-14T23:48:43.234154+0000","last_active":"2024-04-14T23:48:44.232750+0000","last_peered":"2024-04-14T23:48:44.232750+0000","last_clean":"2024-04-14T23:48:44.232750+0000","last_became_active":"2024-04-14T23:48:43.233853+0000","last_became_peered":"2024-04-14T23:48:43.233853+0000","last_unstale":"2024-04-14T23:48:44.232750+0000","last_undegraded":"2024-04-14T23:48:44.232750+0000","last_fullsized":"2024-04-14T23:48:44.232750+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T23:54:02.200200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230330+0000","last_change":"2024-04-14T23:48:43.242154+0000","last_active":"2024-04-14T23:48:44.230330+0000","last_peered":"2024-04-14T23:48:44.230330+0000","last_clean":"2024-04-14T23:48:44.230330+0000","last_became_active":"2024-04-14T23:48:43.241808+0000","last_became_peered":"2024-04-14T23:48:43.241808+0000","last_unstale":"2024-04-14T23:48:44.230330+0000","last_undegraded":"2024-04-14T23:48:44.230330+0000","last_fullsized":"2024-04-14T23:48:44.230330+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:48:42.680808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.232996+0000","last_change":"2024-04-14T23:48:43.233530+0000","last_active":"2024-04-14T23:48:44.232996+0000","last_peered":"2024-04-14T23:48:44.232996+0000","last_clean":"2024-04-14T23:48:44.232996+0000","last_became_active":"2024-04-14T23:48:43.233238+0000","last_became_peered":"2024-04-14T23:48:43.233238+0000","last_unstale":"2024-04-14T23:48:44.232996+0000","last_undegraded":"2024-04-14T23:48:44.232996+0000","last_fullsized":"2024-04-14T23:48:44.232996+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:10:58.525655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.549193+0000","last_change":"2024-04-14T23:48:43.222945+0000","last_active":"2024-04-14T23:48:45.549193+0000","last_peered":"2024-04-14T23:48:45.549193+0000","last_clean":"2024-04-14T23:48:45.549193+0000","last_became_active":"2024-04-14T23:48:43.222624+0000","last_became_peered":"2024-04-14T23:48:43.222624+0000","last_unstale":"2024-04-14T23:48:45.549193+0000","last_undegraded":"2024-04-14T23:48:45.549193+0000","last_fullsized":"2024-04-14T23:48:45.549193+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:22:08.289006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.048719+0000","last_change":"2024-04-14T23:48:43.232567+0000","last_active":"2024-04-14T23:48:45.048719+0000","last_peered":"2024-04-14T23:48:45.048719+0000","last_clean":"2024-04-14T23:48:45.048719+0000","last_became_active":"2024-04-14T23:48:43.232277+0000","last_became_peered":"2024-04-14T23:48:43.232277+0000","last_unstale":"2024-04-14T23:48:45.048719+0000","last_undegraded":"2024-04-14T23:48:45.048719+0000","last_fullsized":"2024-04-14T23:48:45.048719+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T04:59:39.714566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.232426+0000","last_change":"2024-04-14T23:48:43.229443+0000","last_active":"2024-04-14T23:48:44.232426+0000","last_peered":"2024-04-14T23:48:44.232426+0000","last_clean":"2024-04-14T23:48:44.232426+0000","last_became_active":"2024-04-14T23:48:43.229172+0000","last_became_peered":"2024-04-14T23:48:43.229172+0000","last_unstale":"2024-04-14T23:48:44.232426+0000","last_undegraded":"2024-04-14T23:48:44.232426+0000","last_fullsized":"2024-04-14T23:48:44.232426+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:41:31.405287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.234812+0000","last_change":"2024-04-14T23:48:43.242202+0000","last_active":"2024-04-14T23:48:44.234812+0000","last_peered":"2024-04-14T23:48:44.234812+0000","last_clean":"2024-04-14T23:48:44.234812+0000","last_became_active":"2024-04-14T23:48:43.241879+0000","last_became_peered":"2024-04-14T23:48:43.241879+0000","last_unstale":"2024-04-14T23:48:44.234812+0000","last_undegraded":"2024-04-14T23:48:44.234812+0000","last_fullsized":"2024-04-14T23:48:44.234812+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:11:27.552454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230871+0000","last_change":"2024-04-14T23:48:43.242068+0000","last_active":"2024-04-14T23:48:44.230871+0000","last_peered":"2024-04-14T23:48:44.230871+0000","last_clean":"2024-04-14T23:48:44.230871+0000","last_became_active":"2024-04-14T23:48:43.241734+0000","last_became_peered":"2024-04-14T23:48:43.241734+0000","last_unstale":"2024-04-14T23:48:44.230871+0000","last_undegraded":"2024-04-14T23:48:44.230871+0000","last_fullsized":"2024-04-14T23:48:44.230871+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:32:04.579906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.648997+0000","last_change":"2024-04-14T23:48:43.236034+0000","last_active":"2024-04-14T23:48:44.648997+0000","last_peered":"2024-04-14T23:48:44.648997+0000","last_clean":"2024-04-14T23:48:44.648997+0000","last_became_active":"2024-04-14T23:48:43.233567+0000","last_became_peered":"2024-04-14T23:48:43.233567+0000","last_unstale":"2024-04-14T23:48:44.648997+0000","last_undegraded":"2024-04-14T23:48:44.648997+0000","last_fullsized":"2024-04-14T23:48:44.648997+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:07:15.904075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.547675+0000","last_change":"2024-04-14T23:48:43.231045+0000","last_active":"2024-04-14T23:48:45.547675+0000","last_peered":"2024-04-14T23:48:45.547675+0000","last_clean":"2024-04-14T23:48:45.547675+0000","last_became_active":"2024-04-14T23:48:43.230626+0000","last_became_peered":"2024-04-14T23:48:43.230626+0000","last_unstale":"2024-04-14T23:48:45.547675+0000","last_undegraded":"2024-04-14T23:48:45.547675+0000","last_fullsized":"2024-04-14T23:48:45.547675+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:24:03.654061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548613+0000","last_change":"2024-04-14T23:48:43.227543+0000","last_active":"2024-04-14T23:48:45.548613+0000","last_peered":"2024-04-14T23:48:45.548613+0000","last_clean":"2024-04-14T23:48:45.548613+0000","last_became_active":"2024-04-14T23:48:43.226773+0000","last_became_peered":"2024-04-14T23:48:43.226773+0000","last_unstale":"2024-04-14T23:48:45.548613+0000","last_undegraded":"2024-04-14T23:48:45.548613+0000","last_fullsized":"2024-04-14T23:48:45.548613+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:44:56.691708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.211656+0000","last_change":"2024-04-14T23:48:42.239254+0000","last_active":"2024-04-14T23:48:45.211656+0000","last_peered":"2024-04-14T23:48:45.211656+0000","last_clean":"2024-04-14T23:48:45.211656+0000","last_became_active":"2024-04-14T23:48:42.238847+0000","last_became_peered":"2024-04-14T23:48:42.238847+0000","last_unstale":"2024-04-14T23:48:45.211656+0000","last_undegraded":"2024-04-14T23:48:45.211656+0000","last_fullsized":"2024-04-14T23:48:45.211656+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:46:16.301900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.235039+0000","last_change":"2024-04-14T23:48:43.237720+0000","last_active":"2024-04-14T23:48:44.235039+0000","last_peered":"2024-04-14T23:48:44.235039+0000","last_clean":"2024-04-14T23:48:44.235039+0000","last_became_active":"2024-04-14T23:48:43.237391+0000","last_became_peered":"2024-04-14T23:48:43.237391+0000","last_unstale":"2024-04-14T23:48:44.235039+0000","last_undegraded":"2024-04-14T23:48:44.235039+0000","last_fullsized":"2024-04-14T23:48:44.235039+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:38:26.879906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548161+0000","last_change":"2024-04-14T23:48:42.219277+0000","last_active":"2024-04-14T23:48:45.548161+0000","last_peered":"2024-04-14T23:48:45.548161+0000","last_clean":"2024-04-14T23:48:45.548161+0000","last_became_active":"2024-04-14T23:48:42.218995+0000","last_became_peered":"2024-04-14T23:48:42.218995+0000","last_unstale":"2024-04-14T23:48:45.548161+0000","last_undegraded":"2024-04-14T23:48:45.548161+0000","last_fullsized":"2024-04-14T23:48:45.548161+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T23:51:33.370857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.232590+0000","last_change":"2024-04-14T23:48:42.236803+0000","last_active":"2024-04-14T23:48:44.232590+0000","last_peered":"2024-04-14T23:48:44.232590+0000","last_clean":"2024-04-14T23:48:44.232590+0000","last_became_active":"2024-04-14T23:48:42.225937+0000","last_became_peered":"2024-04-14T23:48:42.225937+0000","last_unstale":"2024-04-14T23:48:44.232590+0000","last_undegraded":"2024-04-14T23:48:44.232590+0000","last_fullsized":"2024-04-14T23:48:44.232590+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:52:32.609126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.231732+0000","last_change":"2024-04-14T23:48:43.229506+0000","last_active":"2024-04-14T23:48:44.231732+0000","last_peered":"2024-04-14T23:48:44.231732+0000","last_clean":"2024-04-14T23:48:44.231732+0000","last_became_active":"2024-04-14T23:48:43.227356+0000","last_became_peered":"2024-04-14T23:48:43.227356+0000","last_unstale":"2024-04-14T23:48:44.231732+0000","last_undegraded":"2024-04-14T23:48:44.231732+0000","last_fullsized":"2024-04-14T23:48:44.231732+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:13:04.840655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230660+0000","last_change":"2024-04-14T23:48:42.210883+0000","last_active":"2024-04-14T23:48:44.230660+0000","last_peered":"2024-04-14T23:48:44.230660+0000","last_clean":"2024-04-14T23:48:44.230660+0000","last_became_active":"2024-04-14T23:48:42.210607+0000","last_became_peered":"2024-04-14T23:48:42.210607+0000","last_unstale":"2024-04-14T23:48:44.230660+0000","last_undegraded":"2024-04-14T23:48:44.230660+0000","last_fullsized":"2024-04-14T23:48:44.230660+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:33:32.526223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.048141+0000","last_change":"2024-04-14T23:48:43.218259+0000","last_active":"2024-04-14T23:48:45.048141+0000","last_peered":"2024-04-14T23:48:45.048141+0000","last_clean":"2024-04-14T23:48:45.048141+0000","last_became_active":"2024-04-14T23:48:43.217801+0000","last_became_peered":"2024-04-14T23:48:43.217801+0000","last_unstale":"2024-04-14T23:48:45.048141+0000","last_undegraded":"2024-04-14T23:48:45.048141+0000","last_fullsized":"2024-04-14T23:48:45.048141+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T04:13:42.425034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.547845+0000","last_change":"2024-04-14T23:48:42.218652+0000","last_active":"2024-04-14T23:48:45.547845+0000","last_peered":"2024-04-14T23:48:45.547845+0000","last_clean":"2024-04-14T23:48:45.547845+0000","last_became_active":"2024-04-14T23:48:42.218321+0000","last_became_peered":"2024-04-14T23:48:42.218321+0000","last_unstale":"2024-04-14T23:48:45.547845+0000","last_undegraded":"2024-04-14T23:48:45.547845+0000","last_fullsized":"2024-04-14T23:48:45.547845+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T04:22:06.957747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230879+0000","last_change":"2024-04-14T23:48:43.229117+0000","last_active":"2024-04-14T23:48:44.230879+0000","last_peered":"2024-04-14T23:48:44.230879+0000","last_clean":"2024-04-14T23:48:44.230879+0000","last_became_active":"2024-04-14T23:48:43.228805+0000","last_became_peered":"2024-04-14T23:48:43.228805+0000","last_unstale":"2024-04-14T23:48:44.230879+0000","last_undegraded":"2024-04-14T23:48:44.230879+0000","last_fullsized":"2024-04-14T23:48:44.230879+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:10:39.043818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230930+0000","last_change":"2024-04-14T23:48:42.221104+0000","last_active":"2024-04-14T23:48:44.230930+0000","last_peered":"2024-04-14T23:48:44.230930+0000","last_clean":"2024-04-14T23:48:44.230930+0000","last_became_active":"2024-04-14T23:48:42.220767+0000","last_became_peered":"2024-04-14T23:48:42.220767+0000","last_unstale":"2024-04-14T23:48:44.230930+0000","last_undegraded":"2024-04-14T23:48:44.230930+0000","last_fullsized":"2024-04-14T23:48:44.230930+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:23:57.213764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.235822+0000","last_change":"2024-04-14T23:48:43.236916+0000","last_active":"2024-04-14T23:48:44.235822+0000","last_peered":"2024-04-14T23:48:44.235822+0000","last_clean":"2024-04-14T23:48:44.235822+0000","last_became_active":"2024-04-14T23:48:43.236168+0000","last_became_peered":"2024-04-14T23:48:43.236168+0000","last_unstale":"2024-04-14T23:48:44.235822+0000","last_undegraded":"2024-04-14T23:48:44.235822+0000","last_fullsized":"2024-04-14T23:48:44.235822+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:45:52.057070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:48.220199+0000","last_change":"2024-04-14T23:48:42.231329+0000","last_active":"2024-04-14T23:48:48.220199+0000","last_peered":"2024-04-14T23:48:48.220199+0000","last_clean":"2024-04-14T23:48:48.220199+0000","last_became_active":"2024-04-14T23:48:42.230969+0000","last_became_peered":"2024-04-14T23:48:42.230969+0000","last_unstale":"2024-04-14T23:48:48.220199+0000","last_undegraded":"2024-04-14T23:48:48.220199+0000","last_fullsized":"2024-04-14T23:48:48.220199+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:06:16.557589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548833+0000","last_change":"2024-04-14T23:48:43.230531+0000","last_active":"2024-04-14T23:48:45.548833+0000","last_peered":"2024-04-14T23:48:45.548833+0000","last_clean":"2024-04-14T23:48:45.548833+0000","last_became_active":"2024-04-14T23:48:43.230253+0000","last_became_peered":"2024-04-14T23:48:43.230253+0000","last_unstale":"2024-04-14T23:48:45.548833+0000","last_undegraded":"2024-04-14T23:48:45.548833+0000","last_fullsized":"2024-04-14T23:48:45.548833+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:34:37.449824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'9","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:49:06.196149+0000","last_change":"2024-04-14T23:48:42.237069+0000","last_active":"2024-04-14T23:49:06.196149+0000","last_peered":"2024-04-14T23:49:06.196149+0000","last_clean":"2024-04-14T23:49:06.196149+0000","last_became_active":"2024-04-14T23:48:42.236804+0000","last_became_peered":"2024-04-14T23:48:42.236804+0000","last_unstale":"2024-04-14T23:49:06.196149+0000","last_undegraded":"2024-04-14T23:49:06.196149+0000","last_fullsized":"2024-04-14T23:49:06.196149+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:25:37.318379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2857,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548904+0000","last_change":"2024-04-14T23:48:43.221771+0000","last_active":"2024-04-14T23:48:45.548904+0000","last_peered":"2024-04-14T23:48:45.548904+0000","last_clean":"2024-04-14T23:48:45.548904+0000","last_became_active":"2024-04-14T23:48:43.221462+0000","last_became_peered":"2024-04-14T23:48:43.221462+0000","last_unstale":"2024-04-14T23:48:45.548904+0000","last_undegraded":"2024-04-14T23:48:45.548904+0000","last_fullsized":"2024-04-14T23:48:45.548904+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:29:52.737196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:48.227204+0000","last_change":"2024-04-14T23:48:42.230518+0000","last_active":"2024-04-14T23:48:48.227204+0000","last_peered":"2024-04-14T23:48:48.227204+0000","last_clean":"2024-04-14T23:48:48.227204+0000","last_became_active":"2024-04-14T23:48:42.229758+0000","last_became_peered":"2024-04-14T23:48:42.229758+0000","last_unstale":"2024-04-14T23:48:48.227204+0000","last_undegraded":"2024-04-14T23:48:48.227204+0000","last_fullsized":"2024-04-14T23:48:48.227204+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:01:41.332553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.234816+0000","last_change":"2024-04-14T23:48:43.229873+0000","last_active":"2024-04-14T23:48:44.234816+0000","last_peered":"2024-04-14T23:48:44.234816+0000","last_clean":"2024-04-14T23:48:44.234816+0000","last_became_active":"2024-04-14T23:48:43.229476+0000","last_became_peered":"2024-04-14T23:48:43.229476+0000","last_unstale":"2024-04-14T23:48:44.234816+0000","last_undegraded":"2024-04-14T23:48:44.234816+0000","last_fullsized":"2024-04-14T23:48:44.234816+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:54:22.874344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230858+0000","last_change":"2024-04-14T23:48:42.225404+0000","last_active":"2024-04-14T23:48:44.230858+0000","last_peered":"2024-04-14T23:48:44.230858+0000","last_clean":"2024-04-14T23:48:44.230858+0000","last_became_active":"2024-04-14T23:48:42.225211+0000","last_became_peered":"2024-04-14T23:48:42.225211+0000","last_unstale":"2024-04-14T23:48:44.230858+0000","last_undegraded":"2024-04-14T23:48:44.230858+0000","last_fullsized":"2024-04-14T23:48:44.230858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:49:45.282259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548573+0000","last_change":"2024-04-14T23:48:43.229543+0000","last_active":"2024-04-14T23:48:45.548573+0000","last_peered":"2024-04-14T23:48:45.548573+0000","last_clean":"2024-04-14T23:48:45.548573+0000","last_became_active":"2024-04-14T23:48:43.229297+0000","last_became_peered":"2024-04-14T23:48:43.229297+0000","last_unstale":"2024-04-14T23:48:45.548573+0000","last_undegraded":"2024-04-14T23:48:45.548573+0000","last_fullsized":"2024-04-14T23:48:45.548573+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:00:17.923013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.547633+0000","last_change":"2024-04-14T23:48:42.220492+0000","last_active":"2024-04-14T23:48:45.547633+0000","last_peered":"2024-04-14T23:48:45.547633+0000","last_clean":"2024-04-14T23:48:45.547633+0000","last_became_active":"2024-04-14T23:48:42.220145+0000","last_became_peered":"2024-04-14T23:48:42.220145+0000","last_unstale":"2024-04-14T23:48:45.547633+0000","last_undegraded":"2024-04-14T23:48:45.547633+0000","last_fullsized":"2024-04-14T23:48:45.547633+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:17:45.278993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.547774+0000","last_change":"2024-04-14T23:48:43.235676+0000","last_active":"2024-04-14T23:48:45.547774+0000","last_peered":"2024-04-14T23:48:45.547774+0000","last_clean":"2024-04-14T23:48:45.547774+0000","last_became_active":"2024-04-14T23:48:43.235319+0000","last_became_peered":"2024-04-14T23:48:43.235319+0000","last_unstale":"2024-04-14T23:48:45.547774+0000","last_undegraded":"2024-04-14T23:48:45.547774+0000","last_fullsized":"2024-04-14T23:48:45.547774+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:32:06.283229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:48.230880+0000","last_change":"2024-04-14T23:48:42.231178+0000","last_active":"2024-04-14T23:48:48.230880+0000","last_peered":"2024-04-14T23:48:48.230880+0000","last_clean":"2024-04-14T23:48:48.230880+0000","last_became_active":"2024-04-14T23:48:42.230792+0000","last_became_peered":"2024-04-14T23:48:42.230792+0000","last_unstale":"2024-04-14T23:48:48.230880+0000","last_undegraded":"2024-04-14T23:48:48.230880+0000","last_fullsized":"2024-04-14T23:48:48.230880+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:22:40.744314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548527+0000","last_change":"2024-04-14T23:48:43.228290+0000","last_active":"2024-04-14T23:48:45.548527+0000","last_peered":"2024-04-14T23:48:45.548527+0000","last_clean":"2024-04-14T23:48:45.548527+0000","last_became_active":"2024-04-14T23:48:43.228004+0000","last_became_peered":"2024-04-14T23:48:43.228004+0000","last_unstale":"2024-04-14T23:48:45.548527+0000","last_undegraded":"2024-04-14T23:48:45.548527+0000","last_fullsized":"2024-04-14T23:48:45.548527+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:20:06.572625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548424+0000","last_change":"2024-04-14T23:48:42.231936+0000","last_active":"2024-04-14T23:48:45.548424+0000","last_peered":"2024-04-14T23:48:45.548424+0000","last_clean":"2024-04-14T23:48:45.548424+0000","last_became_active":"2024-04-14T23:48:42.231612+0000","last_became_peered":"2024-04-14T23:48:42.231612+0000","last_unstale":"2024-04-14T23:48:45.548424+0000","last_undegraded":"2024-04-14T23:48:45.548424+0000","last_fullsized":"2024-04-14T23:48:45.548424+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:45:03.098024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.549200+0000","last_change":"2024-04-14T23:48:43.228933+0000","last_active":"2024-04-14T23:48:45.549200+0000","last_peered":"2024-04-14T23:48:45.549200+0000","last_clean":"2024-04-14T23:48:45.549200+0000","last_became_active":"2024-04-14T23:48:43.228665+0000","last_became_peered":"2024-04-14T23:48:43.228665+0000","last_unstale":"2024-04-14T23:48:45.549200+0000","last_undegraded":"2024-04-14T23:48:45.549200+0000","last_fullsized":"2024-04-14T23:48:45.549200+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:43:04.747711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:48.230335+0000","last_change":"2024-04-14T23:48:42.239794+0000","last_active":"2024-04-14T23:48:48.230335+0000","last_peered":"2024-04-14T23:48:48.230335+0000","last_clean":"2024-04-14T23:48:48.230335+0000","last_became_active":"2024-04-14T23:48:42.239505+0000","last_became_peered":"2024-04-14T23:48:42.239505+0000","last_unstale":"2024-04-14T23:48:48.230335+0000","last_undegraded":"2024-04-14T23:48:48.230335+0000","last_fullsized":"2024-04-14T23:48:48.230335+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T09:43:16.582708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.549265+0000","last_change":"2024-04-14T23:48:43.230251+0000","last_active":"2024-04-14T23:48:45.549265+0000","last_peered":"2024-04-14T23:48:45.549265+0000","last_clean":"2024-04-14T23:48:45.549265+0000","last_became_active":"2024-04-14T23:48:43.229949+0000","last_became_peered":"2024-04-14T23:48:43.229949+0000","last_unstale":"2024-04-14T23:48:45.549265+0000","last_undegraded":"2024-04-14T23:48:45.549265+0000","last_fullsized":"2024-04-14T23:48:45.549265+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T09:54:42.581135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.048901+0000","last_change":"2024-04-14T23:48:42.227708+0000","last_active":"2024-04-14T23:48:45.048901+0000","last_peered":"2024-04-14T23:48:45.048901+0000","last_clean":"2024-04-14T23:48:45.048901+0000","last_became_active":"2024-04-14T23:48:42.226845+0000","last_became_peered":"2024-04-14T23:48:42.226845+0000","last_unstale":"2024-04-14T23:48:45.048901+0000","last_undegraded":"2024-04-14T23:48:45.048901+0000","last_fullsized":"2024-04-14T23:48:45.048901+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:02:54.468796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230624+0000","last_change":"2024-04-14T23:48:43.227439+0000","last_active":"2024-04-14T23:48:44.230624+0000","last_peered":"2024-04-14T23:48:44.230624+0000","last_clean":"2024-04-14T23:48:44.230624+0000","last_became_active":"2024-04-14T23:48:43.226717+0000","last_became_peered":"2024-04-14T23:48:43.226717+0000","last_unstale":"2024-04-14T23:48:44.230624+0000","last_undegraded":"2024-04-14T23:48:44.230624+0000","last_fullsized":"2024-04-14T23:48:44.230624+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:01:11.249999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.649211+0000","last_change":"2024-04-14T23:48:42.232164+0000","last_active":"2024-04-14T23:48:44.649211+0000","last_peered":"2024-04-14T23:48:44.649211+0000","last_clean":"2024-04-14T23:48:44.649211+0000","last_became_active":"2024-04-14T23:48:42.229785+0000","last_became_peered":"2024-04-14T23:48:42.229785+0000","last_unstale":"2024-04-14T23:48:44.649211+0000","last_undegraded":"2024-04-14T23:48:44.649211+0000","last_fullsized":"2024-04-14T23:48:44.649211+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T04:21:37.538735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.549097+0000","last_change":"2024-04-14T23:48:43.232949+0000","last_active":"2024-04-14T23:48:45.549097+0000","last_peered":"2024-04-14T23:48:45.549097+0000","last_clean":"2024-04-14T23:48:45.549097+0000","last_became_active":"2024-04-14T23:48:43.232612+0000","last_became_peered":"2024-04-14T23:48:43.232612+0000","last_unstale":"2024-04-14T23:48:45.549097+0000","last_undegraded":"2024-04-14T23:48:45.549097+0000","last_fullsized":"2024-04-14T23:48:45.549097+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:01:00.524663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548069+0000","last_change":"2024-04-14T23:48:38.150200+0000","last_active":"2024-04-14T23:48:45.548069+0000","last_peered":"2024-04-14T23:48:45.548069+0000","last_clean":"2024-04-14T23:48:45.548069+0000","last_became_active":"2024-04-14T23:48:38.149301+0000","last_became_peered":"2024-04-14T23:48:38.149301+0000","last_unstale":"2024-04-14T23:48:45.548069+0000","last_undegraded":"2024-04-14T23:48:45.548069+0000","last_fullsized":"2024-04-14T23:48:45.548069+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:37.129663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:37.129663+0000","last_clean_scrub_stamp":"2024-04-14T23:48:37.129663+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T23:58:01.303851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230439+0000","last_change":"2024-04-14T23:48:42.229238+0000","last_active":"2024-04-14T23:48:44.230439+0000","last_peered":"2024-04-14T23:48:44.230439+0000","last_clean":"2024-04-14T23:48:44.230439+0000","last_became_active":"2024-04-14T23:48:42.228994+0000","last_became_peered":"2024-04-14T23:48:42.228994+0000","last_unstale":"2024-04-14T23:48:44.230439+0000","last_undegraded":"2024-04-14T23:48:44.230439+0000","last_fullsized":"2024-04-14T23:48:44.230439+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:47:15.406228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.232256+0000","last_change":"2024-04-14T23:48:43.223973+0000","last_active":"2024-04-14T23:48:44.232256+0000","last_peered":"2024-04-14T23:48:44.232256+0000","last_clean":"2024-04-14T23:48:44.232256+0000","last_became_active":"2024-04-14T23:48:43.223665+0000","last_became_peered":"2024-04-14T23:48:43.223665+0000","last_unstale":"2024-04-14T23:48:44.232256+0000","last_undegraded":"2024-04-14T23:48:44.232256+0000","last_fullsized":"2024-04-14T23:48:44.232256+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:23:28.495821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.547518+0000","last_change":"2024-04-14T23:48:42.230404+0000","last_active":"2024-04-14T23:48:45.547518+0000","last_peered":"2024-04-14T23:48:45.547518+0000","last_clean":"2024-04-14T23:48:45.547518+0000","last_became_active":"2024-04-14T23:48:42.230173+0000","last_became_peered":"2024-04-14T23:48:42.230173+0000","last_unstale":"2024-04-14T23:48:45.547518+0000","last_undegraded":"2024-04-14T23:48:45.547518+0000","last_fullsized":"2024-04-14T23:48:45.547518+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:23:45.666680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.234372+0000","last_change":"2024-04-14T23:48:43.231958+0000","last_active":"2024-04-14T23:48:44.234372+0000","last_peered":"2024-04-14T23:48:44.234372+0000","last_clean":"2024-04-14T23:48:44.234372+0000","last_became_active":"2024-04-14T23:48:43.231025+0000","last_became_peered":"2024-04-14T23:48:43.231025+0000","last_unstale":"2024-04-14T23:48:44.234372+0000","last_undegraded":"2024-04-14T23:48:44.234372+0000","last_fullsized":"2024-04-14T23:48:44.234372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T23:51:47.947116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.649047+0000","last_change":"2024-04-14T23:48:42.240651+0000","last_active":"2024-04-14T23:48:44.649047+0000","last_peered":"2024-04-14T23:48:44.649047+0000","last_clean":"2024-04-14T23:48:44.649047+0000","last_became_active":"2024-04-14T23:48:42.240351+0000","last_became_peered":"2024-04-14T23:48:42.240351+0000","last_unstale":"2024-04-14T23:48:44.649047+0000","last_undegraded":"2024-04-14T23:48:44.649047+0000","last_fullsized":"2024-04-14T23:48:44.649047+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:03:02.804444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.048231+0000","last_change":"2024-04-14T23:48:43.232415+0000","last_active":"2024-04-14T23:48:45.048231+0000","last_peered":"2024-04-14T23:48:45.048231+0000","last_clean":"2024-04-14T23:48:45.048231+0000","last_became_active":"2024-04-14T23:48:43.232120+0000","last_became_peered":"2024-04-14T23:48:43.232120+0000","last_unstale":"2024-04-14T23:48:45.048231+0000","last_undegraded":"2024-04-14T23:48:45.048231+0000","last_fullsized":"2024-04-14T23:48:45.048231+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:05:27.562738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230988+0000","last_change":"2024-04-14T23:48:42.233743+0000","last_active":"2024-04-14T23:48:44.230988+0000","last_peered":"2024-04-14T23:48:44.230988+0000","last_clean":"2024-04-14T23:48:44.230988+0000","last_became_active":"2024-04-14T23:48:42.233354+0000","last_became_peered":"2024-04-14T23:48:42.233354+0000","last_unstale":"2024-04-14T23:48:44.230988+0000","last_undegraded":"2024-04-14T23:48:44.230988+0000","last_fullsized":"2024-04-14T23:48:44.230988+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:49:23.045968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548229+0000","last_change":"2024-04-14T23:48:43.232468+0000","last_active":"2024-04-14T23:48:45.548229+0000","last_peered":"2024-04-14T23:48:45.548229+0000","last_clean":"2024-04-14T23:48:45.548229+0000","last_became_active":"2024-04-14T23:48:43.232119+0000","last_became_peered":"2024-04-14T23:48:43.232119+0000","last_unstale":"2024-04-14T23:48:45.548229+0000","last_undegraded":"2024-04-14T23:48:45.548229+0000","last_fullsized":"2024-04-14T23:48:45.548229+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:24:12.553325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.048630+0000","last_change":"2024-04-14T23:48:42.242249+0000","last_active":"2024-04-14T23:48:45.048630+0000","last_peered":"2024-04-14T23:48:45.048630+0000","last_clean":"2024-04-14T23:48:45.048630+0000","last_became_active":"2024-04-14T23:48:42.242039+0000","last_became_peered":"2024-04-14T23:48:42.242039+0000","last_unstale":"2024-04-14T23:48:45.048630+0000","last_undegraded":"2024-04-14T23:48:45.048630+0000","last_fullsized":"2024-04-14T23:48:45.048630+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:46:16.301900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230960+0000","last_change":"2024-04-14T23:48:43.242228+0000","last_active":"2024-04-14T23:48:44.230960+0000","last_peered":"2024-04-14T23:48:44.230960+0000","last_clean":"2024-04-14T23:48:44.230960+0000","last_became_active":"2024-04-14T23:48:43.241776+0000","last_became_peered":"2024-04-14T23:48:43.241776+0000","last_unstale":"2024-04-14T23:48:44.230960+0000","last_undegraded":"2024-04-14T23:48:44.230960+0000","last_fullsized":"2024-04-14T23:48:44.230960+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:48:23.653866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.549314+0000","last_change":"2024-04-14T23:48:42.232537+0000","last_active":"2024-04-14T23:48:45.549314+0000","last_peered":"2024-04-14T23:48:45.549314+0000","last_clean":"2024-04-14T23:48:45.549314+0000","last_became_active":"2024-04-14T23:48:42.232325+0000","last_became_peered":"2024-04-14T23:48:42.232325+0000","last_unstale":"2024-04-14T23:48:45.549314+0000","last_undegraded":"2024-04-14T23:48:45.549314+0000","last_fullsized":"2024-04-14T23:48:45.549314+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:34:37.448301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.048333+0000","last_change":"2024-04-14T23:48:43.234445+0000","last_active":"2024-04-14T23:48:45.048333+0000","last_peered":"2024-04-14T23:48:45.048333+0000","last_clean":"2024-04-14T23:48:45.048333+0000","last_became_active":"2024-04-14T23:48:43.234150+0000","last_became_peered":"2024-04-14T23:48:43.234150+0000","last_unstale":"2024-04-14T23:48:45.048333+0000","last_undegraded":"2024-04-14T23:48:45.048333+0000","last_fullsized":"2024-04-14T23:48:45.048333+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:50:59.708087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.049158+0000","last_change":"2024-04-14T23:48:42.230014+0000","last_active":"2024-04-14T23:48:45.049158+0000","last_peered":"2024-04-14T23:48:45.049158+0000","last_clean":"2024-04-14T23:48:45.049158+0000","last_became_active":"2024-04-14T23:48:42.229712+0000","last_became_peered":"2024-04-14T23:48:42.229712+0000","last_unstale":"2024-04-14T23:48:45.049158+0000","last_undegraded":"2024-04-14T23:48:45.049158+0000","last_fullsized":"2024-04-14T23:48:45.049158+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:54:20.595391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.233392+0000","last_change":"2024-04-14T23:48:43.229128+0000","last_active":"2024-04-14T23:48:44.233392+0000","last_peered":"2024-04-14T23:48:44.233392+0000","last_clean":"2024-04-14T23:48:44.233392+0000","last_became_active":"2024-04-14T23:48:43.228739+0000","last_became_peered":"2024-04-14T23:48:43.228739+0000","last_unstale":"2024-04-14T23:48:44.233392+0000","last_undegraded":"2024-04-14T23:48:44.233392+0000","last_fullsized":"2024-04-14T23:48:44.233392+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:08:08.148345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:48.223491+0000","last_change":"2024-04-14T23:48:42.220035+0000","last_active":"2024-04-14T23:48:48.223491+0000","last_peered":"2024-04-14T23:48:48.223491+0000","last_clean":"2024-04-14T23:48:48.223491+0000","last_became_active":"2024-04-14T23:48:42.219750+0000","last_became_peered":"2024-04-14T23:48:42.219750+0000","last_unstale":"2024-04-14T23:48:48.223491+0000","last_undegraded":"2024-04-14T23:48:48.223491+0000","last_fullsized":"2024-04-14T23:48:48.223491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:00:58.928166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.235426+0000","last_change":"2024-04-14T23:48:43.232997+0000","last_active":"2024-04-14T23:48:44.235426+0000","last_peered":"2024-04-14T23:48:44.235426+0000","last_clean":"2024-04-14T23:48:44.235426+0000","last_became_active":"2024-04-14T23:48:43.232730+0000","last_became_peered":"2024-04-14T23:48:43.232730+0000","last_unstale":"2024-04-14T23:48:44.235426+0000","last_undegraded":"2024-04-14T23:48:44.235426+0000","last_fullsized":"2024-04-14T23:48:44.235426+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:38:44.068804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.547928+0000","last_change":"2024-04-14T23:48:42.232087+0000","last_active":"2024-04-14T23:48:45.547928+0000","last_peered":"2024-04-14T23:48:45.547928+0000","last_clean":"2024-04-14T23:48:45.547928+0000","last_became_active":"2024-04-14T23:48:42.231796+0000","last_became_peered":"2024-04-14T23:48:42.231796+0000","last_unstale":"2024-04-14T23:48:45.547928+0000","last_undegraded":"2024-04-14T23:48:45.547928+0000","last_fullsized":"2024-04-14T23:48:45.547928+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:14:35.050093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548007+0000","last_change":"2024-04-14T23:48:43.232871+0000","last_active":"2024-04-14T23:48:45.548007+0000","last_peered":"2024-04-14T23:48:45.548007+0000","last_clean":"2024-04-14T23:48:45.548007+0000","last_became_active":"2024-04-14T23:48:43.232594+0000","last_became_peered":"2024-04-14T23:48:43.232594+0000","last_unstale":"2024-04-14T23:48:45.548007+0000","last_undegraded":"2024-04-14T23:48:45.548007+0000","last_fullsized":"2024-04-14T23:48:45.548007+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:27:37.320994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.048370+0000","last_change":"2024-04-14T23:48:42.231237+0000","last_active":"2024-04-14T23:48:45.048370+0000","last_peered":"2024-04-14T23:48:45.048370+0000","last_clean":"2024-04-14T23:48:45.048370+0000","last_became_active":"2024-04-14T23:48:42.230902+0000","last_became_peered":"2024-04-14T23:48:42.230902+0000","last_unstale":"2024-04-14T23:48:45.048370+0000","last_undegraded":"2024-04-14T23:48:45.048370+0000","last_fullsized":"2024-04-14T23:48:45.048370+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:28:37.228039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.232080+0000","last_change":"2024-04-14T23:48:43.233035+0000","last_active":"2024-04-14T23:48:44.232080+0000","last_peered":"2024-04-14T23:48:44.232080+0000","last_clean":"2024-04-14T23:48:44.232080+0000","last_became_active":"2024-04-14T23:48:43.232747+0000","last_became_peered":"2024-04-14T23:48:43.232747+0000","last_unstale":"2024-04-14T23:48:44.232080+0000","last_undegraded":"2024-04-14T23:48:44.232080+0000","last_fullsized":"2024-04-14T23:48:44.232080+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:38:57.324408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.048301+0000","last_change":"2024-04-14T23:48:42.240496+0000","last_active":"2024-04-14T23:48:45.048301+0000","last_peered":"2024-04-14T23:48:45.048301+0000","last_clean":"2024-04-14T23:48:45.048301+0000","last_became_active":"2024-04-14T23:48:42.238524+0000","last_became_peered":"2024-04-14T23:48:42.238524+0000","last_unstale":"2024-04-14T23:48:45.048301+0000","last_undegraded":"2024-04-14T23:48:45.048301+0000","last_fullsized":"2024-04-14T23:48:45.048301+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:29:35.218759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.233225+0000","last_change":"2024-04-14T23:48:43.234025+0000","last_active":"2024-04-14T23:48:44.233225+0000","last_peered":"2024-04-14T23:48:44.233225+0000","last_clean":"2024-04-14T23:48:44.233225+0000","last_became_active":"2024-04-14T23:48:43.233651+0000","last_became_peered":"2024-04-14T23:48:43.233651+0000","last_unstale":"2024-04-14T23:48:44.233225+0000","last_undegraded":"2024-04-14T23:48:44.233225+0000","last_fullsized":"2024-04-14T23:48:44.233225+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:26:58.094545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:48.217913+0000","last_change":"2024-04-14T23:48:42.218897+0000","last_active":"2024-04-14T23:48:48.217913+0000","last_peered":"2024-04-14T23:48:48.217913+0000","last_clean":"2024-04-14T23:48:48.217913+0000","last_became_active":"2024-04-14T23:48:42.218580+0000","last_became_peered":"2024-04-14T23:48:42.218580+0000","last_unstale":"2024-04-14T23:48:48.217913+0000","last_undegraded":"2024-04-14T23:48:48.217913+0000","last_fullsized":"2024-04-14T23:48:48.217913+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:26:44.477638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548008+0000","last_change":"2024-04-14T23:48:43.219871+0000","last_active":"2024-04-14T23:48:45.548008+0000","last_peered":"2024-04-14T23:48:45.548008+0000","last_clean":"2024-04-14T23:48:45.548008+0000","last_became_active":"2024-04-14T23:48:43.219601+0000","last_became_peered":"2024-04-14T23:48:43.219601+0000","last_unstale":"2024-04-14T23:48:45.548008+0000","last_undegraded":"2024-04-14T23:48:45.548008+0000","last_fullsized":"2024-04-14T23:48:45.548008+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:36:20.379172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230971+0000","last_change":"2024-04-14T23:48:43.222259+0000","last_active":"2024-04-14T23:48:44.230971+0000","last_peered":"2024-04-14T23:48:44.230971+0000","last_clean":"2024-04-14T23:48:44.230971+0000","last_became_active":"2024-04-14T23:48:43.221842+0000","last_became_peered":"2024-04-14T23:48:43.221842+0000","last_unstale":"2024-04-14T23:48:44.230971+0000","last_undegraded":"2024-04-14T23:48:44.230971+0000","last_fullsized":"2024-04-14T23:48:44.230971+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:12:20.038987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:48.217880+0000","last_change":"2024-04-14T23:48:42.221314+0000","last_active":"2024-04-14T23:48:48.217880+0000","last_peered":"2024-04-14T23:48:48.217880+0000","last_clean":"2024-04-14T23:48:48.217880+0000","last_became_active":"2024-04-14T23:48:42.221028+0000","last_became_peered":"2024-04-14T23:48:42.221028+0000","last_unstale":"2024-04-14T23:48:48.217880+0000","last_undegraded":"2024-04-14T23:48:48.217880+0000","last_fullsized":"2024-04-14T23:48:48.217880+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T09:58:18.195400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230606+0000","last_change":"2024-04-14T23:48:43.224577+0000","last_active":"2024-04-14T23:48:44.230606+0000","last_peered":"2024-04-14T23:48:44.230606+0000","last_clean":"2024-04-14T23:48:44.230606+0000","last_became_active":"2024-04-14T23:48:43.224312+0000","last_became_peered":"2024-04-14T23:48:43.224312+0000","last_unstale":"2024-04-14T23:48:44.230606+0000","last_undegraded":"2024-04-14T23:48:44.230606+0000","last_fullsized":"2024-04-14T23:48:44.230606+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:52:44.955773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:48.217170+0000","last_change":"2024-04-14T23:48:42.221000+0000","last_active":"2024-04-14T23:48:48.217170+0000","last_peered":"2024-04-14T23:48:48.217170+0000","last_clean":"2024-04-14T23:48:48.217170+0000","last_became_active":"2024-04-14T23:48:42.219813+0000","last_became_peered":"2024-04-14T23:48:42.219813+0000","last_unstale":"2024-04-14T23:48:48.217170+0000","last_undegraded":"2024-04-14T23:48:48.217170+0000","last_fullsized":"2024-04-14T23:48:48.217170+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:00:38.107953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.235379+0000","last_change":"2024-04-14T23:48:43.236946+0000","last_active":"2024-04-14T23:48:44.235379+0000","last_peered":"2024-04-14T23:48:44.235379+0000","last_clean":"2024-04-14T23:48:44.235379+0000","last_became_active":"2024-04-14T23:48:43.236595+0000","last_became_peered":"2024-04-14T23:48:43.236595+0000","last_unstale":"2024-04-14T23:48:44.235379+0000","last_undegraded":"2024-04-14T23:48:44.235379+0000","last_fullsized":"2024-04-14T23:48:44.235379+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:56:46.037478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.211743+0000","last_change":"2024-04-14T23:48:42.239255+0000","last_active":"2024-04-14T23:48:45.211743+0000","last_peered":"2024-04-14T23:48:45.211743+0000","last_clean":"2024-04-14T23:48:45.211743+0000","last_became_active":"2024-04-14T23:48:42.238892+0000","last_became_peered":"2024-04-14T23:48:42.238892+0000","last_unstale":"2024-04-14T23:48:45.211743+0000","last_undegraded":"2024-04-14T23:48:45.211743+0000","last_fullsized":"2024-04-14T23:48:45.211743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:44:39.632155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.233893+0000","last_change":"2024-04-14T23:48:43.227240+0000","last_active":"2024-04-14T23:48:44.233893+0000","last_peered":"2024-04-14T23:48:44.233893+0000","last_clean":"2024-04-14T23:48:44.233893+0000","last_became_active":"2024-04-14T23:48:43.222707+0000","last_became_peered":"2024-04-14T23:48:43.222707+0000","last_unstale":"2024-04-14T23:48:44.233893+0000","last_undegraded":"2024-04-14T23:48:44.233893+0000","last_fullsized":"2024-04-14T23:48:44.233893+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:02:17.231763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:48.228047+0000","last_change":"2024-04-14T23:48:42.226563+0000","last_active":"2024-04-14T23:48:48.228047+0000","last_peered":"2024-04-14T23:48:48.228047+0000","last_clean":"2024-04-14T23:48:48.228047+0000","last_became_active":"2024-04-14T23:48:42.226238+0000","last_became_peered":"2024-04-14T23:48:42.226238+0000","last_unstale":"2024-04-14T23:48:48.228047+0000","last_undegraded":"2024-04-14T23:48:48.228047+0000","last_fullsized":"2024-04-14T23:48:48.228047+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:01:36.516898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":3655,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65128,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574858,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":286720,"data_stored":666555,"data_compressed":32564,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574859,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":78462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574858,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":221184,"data_stored":75695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574859,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":75731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":78462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574858,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574858,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":225280,"data_stored":75765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":294912,"data_stored":666611,"data_compressed":32564,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32564,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32564,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-14T23:49:12.821 INFO:tasks.mds_thrash.ceph_manager:clean! 2024-04-14T23:49:12.821 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-14T23:49:13.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.077+0000 7fe938a00640 1 Processor -- start 2024-04-14T23:49:13.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.077+0000 7fe938a00640 1 -- start start 2024-04-14T23:49:13.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.078+0000 7fe938a00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe9340c7080 0x7fe93406b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:13.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.078+0000 7fe938a00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe93415ba70 0x7fe934070d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:13.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.078+0000 7fe938a00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe93406b850 con 0x7fe9340c6950 2024-04-14T23:49:13.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.078+0000 7fe938a00640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fe93406c740 con 0x7fe93415ba70 2024-04-14T23:49:13.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.078+0000 7fe938a00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fe93406bb30 con 0x7fe9340c7080 2024-04-14T23:49:13.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.078+0000 7fe933e00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe93415ba70 0x7fe934070d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:13.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.078+0000 7fe933e00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe93415ba70 0x7fe934070d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37302/0 (socket says 172.21.15.60:37302) 2024-04-14T23:49:13.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.078+0000 7fe933400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe9340c7080 0x7fe93406b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:13.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.078+0000 7fe933e00640 1 -- 172.21.15.60:0/2113511417 learned_addr learned my addr 172.21.15.60:0/2113511417 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:49:13.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.079+0000 7fe933400640 1 -- 172.21.15.60:0/2113511417 >> v2:172.21.15.60:3300/0 conn(0x7fe93415ba70 msgr2=0x7fe934070d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:49:13.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.079+0000 7fe933400640 1 --2- 172.21.15.60:0/2113511417 >> v2:172.21.15.60:3300/0 conn(0x7fe93415ba70 0x7fe934070d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:13.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.079+0000 7fe933400640 1 -- 172.21.15.60:0/2113511417 >> v1:172.21.15.60:6789/0 conn(0x7fe9340c6950 legacy=0x7fe93415b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:13.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.079+0000 7fe933400640 1 -- 172.21.15.60:0/2113511417 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe93406be10 con 0x7fe9340c7080 2024-04-14T23:49:13.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.079+0000 7fe933e00640 1 --2- 172.21.15.60:0/2113511417 >> v2:172.21.15.60:3300/0 conn(0x7fe93415ba70 0x7fe934070d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:49:13.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.080+0000 7fe933400640 1 --2- 172.21.15.60:0/2113511417 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe9340c7080 0x7fe93406b1c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe9240097a0 tx=0x7fe92402b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=503e5a4c1a3c56e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:13.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.080+0000 7fe932a00640 1 -- 172.21.15.60:0/2113511417 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe924034070 con 0x7fe9340c7080 2024-04-14T23:49:13.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.081+0000 7fe932a00640 1 -- 172.21.15.60:0/2113511417 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe924030800 con 0x7fe9340c7080 2024-04-14T23:49:13.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.081+0000 7fe932a00640 1 -- 172.21.15.60:0/2113511417 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe924030bd0 con 0x7fe9340c7080 2024-04-14T23:49:13.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.081+0000 7fe938a00640 1 -- 172.21.15.60:0/2113511417 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe9340c7080 msgr2=0x7fe93406b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:13.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.081+0000 7fe938a00640 1 --2- 172.21.15.60:0/2113511417 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe9340c7080 0x7fe93406b1c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe9240097a0 tx=0x7fe92402b940 comp rx=0 tx=0).stop 2024-04-14T23:49:13.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.081+0000 7fe938a00640 1 -- 172.21.15.60:0/2113511417 shutdown_connections 2024-04-14T23:49:13.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.081+0000 7fe938a00640 1 --2- 172.21.15.60:0/2113511417 >> v2:172.21.15.60:3300/0 conn(0x7fe93415ba70 0x7fe934070d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:13.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.081+0000 7fe938a00640 1 --2- 172.21.15.60:0/2113511417 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe9340c7080 0x7fe93406b1c0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:13.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.081+0000 7fe938a00640 1 -- 172.21.15.60:0/2113511417 >> 172.21.15.60:0/2113511417 conn(0x7fe9340c8b50 msgr2=0x7fe9340c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:49:13.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.081+0000 7fe938a00640 1 -- 172.21.15.60:0/2113511417 shutdown_connections 2024-04-14T23:49:13.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.082+0000 7fe938a00640 1 -- 172.21.15.60:0/2113511417 wait complete. 2024-04-14T23:49:13.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.083+0000 7fe938a00640 1 Processor -- start 2024-04-14T23:49:13.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.083+0000 7fe938a00640 1 -- start start 2024-04-14T23:49:13.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.084+0000 7fe938a00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe9340c6950 0x7fe9340cb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:13.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.084+0000 7fe938a00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9340c7080 0x7fe934151030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:13.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.084+0000 7fe938a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe93415ba70 0x7fe934151570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:13.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.084+0000 7fe938a00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fe9340c7460 con 0x7fe9340c7080 2024-04-14T23:49:13.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.084+0000 7fe938a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe93406c3d0 con 0x7fe93415ba70 2024-04-14T23:49:13.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.084+0000 7fe938a00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fe9340c75e0 con 0x7fe9340c6950 2024-04-14T23:49:13.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.085+0000 7fe933400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe9340c6950 0x7fe9340cb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:13.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.085+0000 7fe933400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe9340c6950 0x7fe9340cb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.60:32860/0 (socket says 172.21.15.60:32860) 2024-04-14T23:49:13.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.085+0000 7fe933400640 1 -- 172.21.15.60:0/3541443223 learned_addr learned my addr 172.21.15.60:0/3541443223 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:49:13.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.085+0000 7fe933e00640 1 --2- 172.21.15.60:0/3541443223 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe93415ba70 0x7fe934151570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:13.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.085+0000 7fe92be00640 1 --2- 172.21.15.60:0/3541443223 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9340c7080 0x7fe934151030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:13.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.085+0000 7fe933400640 1 -- 172.21.15.60:0/3541443223 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe93415ba70 msgr2=0x7fe934151570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:13.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.085+0000 7fe933400640 1 --2- 172.21.15.60:0/3541443223 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe93415ba70 0x7fe934151570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:13.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.085+0000 7fe933400640 1 -- 172.21.15.60:0/3541443223 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9340c7080 msgr2=0x7fe934151030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:13.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.085+0000 7fe933400640 1 --2- 172.21.15.60:0/3541443223 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9340c7080 0x7fe934151030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:13.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.085+0000 7fe933400640 1 -- 172.21.15.60:0/3541443223 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9341f0ed0 con 0x7fe9340c6950 2024-04-14T23:49:13.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.085+0000 7fe92be00640 1 --2- 172.21.15.60:0/3541443223 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9340c7080 0x7fe934151030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-14T23:49:13.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.085+0000 7fe933400640 1 --2- 172.21.15.60:0/3541443223 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe9340c6950 0x7fe9340cb1b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe924009890 tx=0x7fe9240046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:13.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.086+0000 7fe92b400640 1 -- 172.21.15.60:0/3541443223 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe924034040 con 0x7fe9340c6950 2024-04-14T23:49:13.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.086+0000 7fe92b400640 1 -- 172.21.15.60:0/3541443223 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe924030600 con 0x7fe9340c6950 2024-04-14T23:49:13.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.086+0000 7fe92b400640 1 -- 172.21.15.60:0/3541443223 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe924006e10 con 0x7fe9340c6950 2024-04-14T23:49:13.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.086+0000 7fe938a00640 1 -- 172.21.15.60:0/3541443223 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9341f11b0 con 0x7fe9340c6950 2024-04-14T23:49:13.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.086+0000 7fe938a00640 1 -- 172.21.15.60:0/3541443223 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9341f1770 con 0x7fe9340c6950 2024-04-14T23:49:13.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.087+0000 7fe938a00640 1 -- 172.21.15.60:0/3541443223 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9340672e0 con 0x7fe9340c6950 2024-04-14T23:49:13.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.088+0000 7fe92b400640 1 -- 172.21.15.60:0/3541443223 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7fe924037070 con 0x7fe9340c6950 2024-04-14T23:49:13.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.089+0000 7fe92b400640 1 --2- 172.21.15.60:0/3541443223 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fe91006b990 0x7fe91006de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:13.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.089+0000 7fe92b400640 1 -- 172.21.15.60:0/3541443223 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fe9240ac920 con 0x7fe9340c6950 2024-04-14T23:49:13.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.089+0000 7fe92be00640 1 --2- 172.21.15.60:0/3541443223 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fe91006b990 0x7fe91006de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:13.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.090+0000 7fe92be00640 1 --2- 172.21.15.60:0/3541443223 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fe91006b990 0x7fe91006de50 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe9341f4040 tx=0x7fe91c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:13.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.094+0000 7fe92b400640 1 -- 172.21.15.60:0/3541443223 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe9340672e0 con 0x7fe9340c6950 2024-04-14T23:49:13.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.289+0000 7fe938a00640 1 -- 172.21.15.60:0/3541443223 --> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe9341f0c80 con 0x7fe91006b990 2024-04-14T23:49:13.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.296+0000 7fe92b400640 1 -- 172.21.15.60:0/3541443223 <== mgr.4100 v2:172.21.15.186:6832/714104835 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251932 (secure 0 0 0) 0x7fe9341f0c80 con 0x7fe91006b990 2024-04-14T23:49:13.305 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:49:13.307 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-04-14T23:49:13.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.301+0000 7fe938a00640 1 -- 172.21.15.60:0/3541443223 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fe91006b990 msgr2=0x7fe91006de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:13.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.301+0000 7fe938a00640 1 --2- 172.21.15.60:0/3541443223 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fe91006b990 0x7fe91006de50 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe9341f4040 tx=0x7fe91c009040 comp rx=0 tx=0).stop 2024-04-14T23:49:13.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.301+0000 7fe938a00640 1 -- 172.21.15.60:0/3541443223 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe9340c6950 msgr2=0x7fe9340cb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:13.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.301+0000 7fe938a00640 1 --2- 172.21.15.60:0/3541443223 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe9340c6950 0x7fe9340cb1b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe924009890 tx=0x7fe9240046c0 comp rx=0 tx=0).stop 2024-04-14T23:49:13.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.301+0000 7fe938a00640 1 -- 172.21.15.60:0/3541443223 shutdown_connections 2024-04-14T23:49:13.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.301+0000 7fe938a00640 1 --2- 172.21.15.60:0/3541443223 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fe91006b990 0x7fe91006de50 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:13.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.301+0000 7fe938a00640 1 --2- 172.21.15.60:0/3541443223 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe93415ba70 0x7fe934151570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:13.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.301+0000 7fe938a00640 1 --2- 172.21.15.60:0/3541443223 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9340c7080 0x7fe934151030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:13.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.301+0000 7fe938a00640 1 --2- 172.21.15.60:0/3541443223 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe9340c6950 0x7fe9340cb1b0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:13.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.301+0000 7fe938a00640 1 -- 172.21.15.60:0/3541443223 >> 172.21.15.60:0/3541443223 conn(0x7fe9340c8b50 msgr2=0x7fe9340c82c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:49:13.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.301+0000 7fe938a00640 1 -- 172.21.15.60:0/3541443223 shutdown_connections 2024-04-14T23:49:13.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.301+0000 7fe938a00640 1 -- 172.21.15.60:0/3541443223 wait complete. 2024-04-14T23:49:13.325 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2024-04-14T23:49:12.173811+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594023,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218016,"kb_used_data":1888,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756704,"statfs":{"total":773094113280,"available":772870864896,"internally_reserved":0,"allocated":1933312,"data_stored":1792886,"data_compressed":65128,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1245,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003322"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.549104+0000","last_change":"2024-04-14T23:48:43.222542+0000","last_active":"2024-04-14T23:48:45.549104+0000","last_peered":"2024-04-14T23:48:45.549104+0000","last_clean":"2024-04-14T23:48:45.549104+0000","last_became_active":"2024-04-14T23:48:43.222278+0000","last_became_peered":"2024-04-14T23:48:43.222278+0000","last_unstale":"2024-04-14T23:48:45.549104+0000","last_undegraded":"2024-04-14T23:48:45.549104+0000","last_fullsized":"2024-04-14T23:48:45.549104+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:13:06.877485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.049059+0000","last_change":"2024-04-14T23:48:43.226781+0000","last_active":"2024-04-14T23:48:45.049059+0000","last_peered":"2024-04-14T23:48:45.049059+0000","last_clean":"2024-04-14T23:48:45.049059+0000","last_became_active":"2024-04-14T23:48:43.226426+0000","last_became_peered":"2024-04-14T23:48:43.226426+0000","last_unstale":"2024-04-14T23:48:45.049059+0000","last_undegraded":"2024-04-14T23:48:45.049059+0000","last_fullsized":"2024-04-14T23:48:45.049059+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:35:24.780859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.235142+0000","last_change":"2024-04-14T23:48:43.236939+0000","last_active":"2024-04-14T23:48:44.235142+0000","last_peered":"2024-04-14T23:48:44.235142+0000","last_clean":"2024-04-14T23:48:44.235142+0000","last_became_active":"2024-04-14T23:48:43.236616+0000","last_became_peered":"2024-04-14T23:48:43.236616+0000","last_unstale":"2024-04-14T23:48:44.235142+0000","last_undegraded":"2024-04-14T23:48:44.235142+0000","last_fullsized":"2024-04-14T23:48:44.235142+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:35:42.619139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230420+0000","last_change":"2024-04-14T23:48:43.242349+0000","last_active":"2024-04-14T23:48:44.230420+0000","last_peered":"2024-04-14T23:48:44.230420+0000","last_clean":"2024-04-14T23:48:44.230420+0000","last_became_active":"2024-04-14T23:48:43.242000+0000","last_became_peered":"2024-04-14T23:48:43.242000+0000","last_unstale":"2024-04-14T23:48:44.230420+0000","last_undegraded":"2024-04-14T23:48:44.230420+0000","last_fullsized":"2024-04-14T23:48:44.230420+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:33:52.458016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548549+0000","last_change":"2024-04-14T23:48:43.233937+0000","last_active":"2024-04-14T23:48:45.548549+0000","last_peered":"2024-04-14T23:48:45.548549+0000","last_clean":"2024-04-14T23:48:45.548549+0000","last_became_active":"2024-04-14T23:48:43.233674+0000","last_became_peered":"2024-04-14T23:48:43.233674+0000","last_unstale":"2024-04-14T23:48:45.548549+0000","last_undegraded":"2024-04-14T23:48:45.548549+0000","last_fullsized":"2024-04-14T23:48:45.548549+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:00:07.838600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230260+0000","last_change":"2024-04-14T23:48:43.230925+0000","last_active":"2024-04-14T23:48:44.230260+0000","last_peered":"2024-04-14T23:48:44.230260+0000","last_clean":"2024-04-14T23:48:44.230260+0000","last_became_active":"2024-04-14T23:48:43.230405+0000","last_became_peered":"2024-04-14T23:48:43.230405+0000","last_unstale":"2024-04-14T23:48:44.230260+0000","last_undegraded":"2024-04-14T23:48:44.230260+0000","last_fullsized":"2024-04-14T23:48:44.230260+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:15:12.452964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548354+0000","last_change":"2024-04-14T23:48:43.231088+0000","last_active":"2024-04-14T23:48:45.548354+0000","last_peered":"2024-04-14T23:48:45.548354+0000","last_clean":"2024-04-14T23:48:45.548354+0000","last_became_active":"2024-04-14T23:48:43.230784+0000","last_became_peered":"2024-04-14T23:48:43.230784+0000","last_unstale":"2024-04-14T23:48:45.548354+0000","last_undegraded":"2024-04-14T23:48:45.548354+0000","last_fullsized":"2024-04-14T23:48:45.548354+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:23:10.461088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.231230+0000","last_change":"2024-04-14T23:48:43.236019+0000","last_active":"2024-04-14T23:48:44.231230+0000","last_peered":"2024-04-14T23:48:44.231230+0000","last_clean":"2024-04-14T23:48:44.231230+0000","last_became_active":"2024-04-14T23:48:43.235719+0000","last_became_peered":"2024-04-14T23:48:43.235719+0000","last_unstale":"2024-04-14T23:48:44.231230+0000","last_undegraded":"2024-04-14T23:48:44.231230+0000","last_fullsized":"2024-04-14T23:48:44.231230+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T04:41:16.130730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.549156+0000","last_change":"2024-04-14T23:48:43.232166+0000","last_active":"2024-04-14T23:48:45.549156+0000","last_peered":"2024-04-14T23:48:45.549156+0000","last_clean":"2024-04-14T23:48:45.549156+0000","last_became_active":"2024-04-14T23:48:43.231751+0000","last_became_peered":"2024-04-14T23:48:43.231751+0000","last_unstale":"2024-04-14T23:48:45.549156+0000","last_undegraded":"2024-04-14T23:48:45.549156+0000","last_fullsized":"2024-04-14T23:48:45.549156+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:05:47.289092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548848+0000","last_change":"2024-04-14T23:48:43.224938+0000","last_active":"2024-04-14T23:48:45.548848+0000","last_peered":"2024-04-14T23:48:45.548848+0000","last_clean":"2024-04-14T23:48:45.548848+0000","last_became_active":"2024-04-14T23:48:43.224592+0000","last_became_peered":"2024-04-14T23:48:43.224592+0000","last_unstale":"2024-04-14T23:48:45.548848+0000","last_undegraded":"2024-04-14T23:48:45.548848+0000","last_fullsized":"2024-04-14T23:48:45.548848+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:59:12.923596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548591+0000","last_change":"2024-04-14T23:48:43.225290+0000","last_active":"2024-04-14T23:48:45.548591+0000","last_peered":"2024-04-14T23:48:45.548591+0000","last_clean":"2024-04-14T23:48:45.548591+0000","last_became_active":"2024-04-14T23:48:43.224989+0000","last_became_peered":"2024-04-14T23:48:43.224989+0000","last_unstale":"2024-04-14T23:48:45.548591+0000","last_undegraded":"2024-04-14T23:48:45.548591+0000","last_fullsized":"2024-04-14T23:48:45.548591+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:44:49.116229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548742+0000","last_change":"2024-04-14T23:48:43.226194+0000","last_active":"2024-04-14T23:48:45.548742+0000","last_peered":"2024-04-14T23:48:45.548742+0000","last_clean":"2024-04-14T23:48:45.548742+0000","last_became_active":"2024-04-14T23:48:43.225892+0000","last_became_peered":"2024-04-14T23:48:43.225892+0000","last_unstale":"2024-04-14T23:48:45.548742+0000","last_undegraded":"2024-04-14T23:48:45.548742+0000","last_fullsized":"2024-04-14T23:48:45.548742+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:04:12.377770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.233039+0000","last_change":"2024-04-14T23:48:43.220571+0000","last_active":"2024-04-14T23:48:44.233039+0000","last_peered":"2024-04-14T23:48:44.233039+0000","last_clean":"2024-04-14T23:48:44.233039+0000","last_became_active":"2024-04-14T23:48:43.219614+0000","last_became_peered":"2024-04-14T23:48:43.219614+0000","last_unstale":"2024-04-14T23:48:44.233039+0000","last_undegraded":"2024-04-14T23:48:44.233039+0000","last_fullsized":"2024-04-14T23:48:44.233039+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:14:41.854957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.234686+0000","last_change":"2024-04-14T23:48:43.239403+0000","last_active":"2024-04-14T23:48:44.234686+0000","last_peered":"2024-04-14T23:48:44.234686+0000","last_clean":"2024-04-14T23:48:44.234686+0000","last_became_active":"2024-04-14T23:48:43.239136+0000","last_became_peered":"2024-04-14T23:48:43.239136+0000","last_unstale":"2024-04-14T23:48:44.234686+0000","last_undegraded":"2024-04-14T23:48:44.234686+0000","last_fullsized":"2024-04-14T23:48:44.234686+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:27:30.512861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.048083+0000","last_change":"2024-04-14T23:48:43.228157+0000","last_active":"2024-04-14T23:48:45.048083+0000","last_peered":"2024-04-14T23:48:45.048083+0000","last_clean":"2024-04-14T23:48:45.048083+0000","last_became_active":"2024-04-14T23:48:43.227898+0000","last_became_peered":"2024-04-14T23:48:43.227898+0000","last_unstale":"2024-04-14T23:48:45.048083+0000","last_undegraded":"2024-04-14T23:48:45.048083+0000","last_fullsized":"2024-04-14T23:48:45.048083+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:06:37.602935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230387+0000","last_change":"2024-04-14T23:48:43.238986+0000","last_active":"2024-04-14T23:48:44.230387+0000","last_peered":"2024-04-14T23:48:44.230387+0000","last_clean":"2024-04-14T23:48:44.230387+0000","last_became_active":"2024-04-14T23:48:43.238714+0000","last_became_peered":"2024-04-14T23:48:43.238714+0000","last_unstale":"2024-04-14T23:48:44.230387+0000","last_undegraded":"2024-04-14T23:48:44.230387+0000","last_fullsized":"2024-04-14T23:48:44.230387+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:05:30.655048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.231004+0000","last_change":"2024-04-14T23:48:43.229325+0000","last_active":"2024-04-14T23:48:44.231004+0000","last_peered":"2024-04-14T23:48:44.231004+0000","last_clean":"2024-04-14T23:48:44.231004+0000","last_became_active":"2024-04-14T23:48:43.228618+0000","last_became_peered":"2024-04-14T23:48:43.228618+0000","last_unstale":"2024-04-14T23:48:44.231004+0000","last_undegraded":"2024-04-14T23:48:44.231004+0000","last_fullsized":"2024-04-14T23:48:44.231004+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:06:43.745698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.649355+0000","last_change":"2024-04-14T23:48:43.233288+0000","last_active":"2024-04-14T23:48:44.649355+0000","last_peered":"2024-04-14T23:48:44.649355+0000","last_clean":"2024-04-14T23:48:44.649355+0000","last_became_active":"2024-04-14T23:48:43.232971+0000","last_became_peered":"2024-04-14T23:48:43.232971+0000","last_unstale":"2024-04-14T23:48:44.649355+0000","last_undegraded":"2024-04-14T23:48:44.649355+0000","last_fullsized":"2024-04-14T23:48:44.649355+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:06:23.588161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.549346+0000","last_change":"2024-04-14T23:48:43.233169+0000","last_active":"2024-04-14T23:48:45.549346+0000","last_peered":"2024-04-14T23:48:45.549346+0000","last_clean":"2024-04-14T23:48:45.549346+0000","last_became_active":"2024-04-14T23:48:43.232135+0000","last_became_peered":"2024-04-14T23:48:43.232135+0000","last_unstale":"2024-04-14T23:48:45.549346+0000","last_undegraded":"2024-04-14T23:48:45.549346+0000","last_fullsized":"2024-04-14T23:48:45.549346+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:37:06.166596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.547820+0000","last_change":"2024-04-14T23:48:43.228728+0000","last_active":"2024-04-14T23:48:45.547820+0000","last_peered":"2024-04-14T23:48:45.547820+0000","last_clean":"2024-04-14T23:48:45.547820+0000","last_became_active":"2024-04-14T23:48:43.226225+0000","last_became_peered":"2024-04-14T23:48:43.226225+0000","last_unstale":"2024-04-14T23:48:45.547820+0000","last_undegraded":"2024-04-14T23:48:45.547820+0000","last_fullsized":"2024-04-14T23:48:45.547820+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:06:03.453599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.234949+0000","last_change":"2024-04-14T23:48:43.237234+0000","last_active":"2024-04-14T23:48:44.234949+0000","last_peered":"2024-04-14T23:48:44.234949+0000","last_clean":"2024-04-14T23:48:44.234949+0000","last_became_active":"2024-04-14T23:48:43.230056+0000","last_became_peered":"2024-04-14T23:48:43.230056+0000","last_unstale":"2024-04-14T23:48:44.234949+0000","last_undegraded":"2024-04-14T23:48:44.234949+0000","last_fullsized":"2024-04-14T23:48:44.234949+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:43:44.966384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230697+0000","last_change":"2024-04-14T23:48:43.229387+0000","last_active":"2024-04-14T23:48:44.230697+0000","last_peered":"2024-04-14T23:48:44.230697+0000","last_clean":"2024-04-14T23:48:44.230697+0000","last_became_active":"2024-04-14T23:48:43.229110+0000","last_became_peered":"2024-04-14T23:48:43.229110+0000","last_unstale":"2024-04-14T23:48:44.230697+0000","last_undegraded":"2024-04-14T23:48:44.230697+0000","last_fullsized":"2024-04-14T23:48:44.230697+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T09:18:46.939940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.232750+0000","last_change":"2024-04-14T23:48:43.234154+0000","last_active":"2024-04-14T23:48:44.232750+0000","last_peered":"2024-04-14T23:48:44.232750+0000","last_clean":"2024-04-14T23:48:44.232750+0000","last_became_active":"2024-04-14T23:48:43.233853+0000","last_became_peered":"2024-04-14T23:48:43.233853+0000","last_unstale":"2024-04-14T23:48:44.232750+0000","last_undegraded":"2024-04-14T23:48:44.232750+0000","last_fullsized":"2024-04-14T23:48:44.232750+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T23:54:02.200200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230330+0000","last_change":"2024-04-14T23:48:43.242154+0000","last_active":"2024-04-14T23:48:44.230330+0000","last_peered":"2024-04-14T23:48:44.230330+0000","last_clean":"2024-04-14T23:48:44.230330+0000","last_became_active":"2024-04-14T23:48:43.241808+0000","last_became_peered":"2024-04-14T23:48:43.241808+0000","last_unstale":"2024-04-14T23:48:44.230330+0000","last_undegraded":"2024-04-14T23:48:44.230330+0000","last_fullsized":"2024-04-14T23:48:44.230330+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:48:42.680808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.232996+0000","last_change":"2024-04-14T23:48:43.233530+0000","last_active":"2024-04-14T23:48:44.232996+0000","last_peered":"2024-04-14T23:48:44.232996+0000","last_clean":"2024-04-14T23:48:44.232996+0000","last_became_active":"2024-04-14T23:48:43.233238+0000","last_became_peered":"2024-04-14T23:48:43.233238+0000","last_unstale":"2024-04-14T23:48:44.232996+0000","last_undegraded":"2024-04-14T23:48:44.232996+0000","last_fullsized":"2024-04-14T23:48:44.232996+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:10:58.525655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.549193+0000","last_change":"2024-04-14T23:48:43.222945+0000","last_active":"2024-04-14T23:48:45.549193+0000","last_peered":"2024-04-14T23:48:45.549193+0000","last_clean":"2024-04-14T23:48:45.549193+0000","last_became_active":"2024-04-14T23:48:43.222624+0000","last_became_peered":"2024-04-14T23:48:43.222624+0000","last_unstale":"2024-04-14T23:48:45.549193+0000","last_undegraded":"2024-04-14T23:48:45.549193+0000","last_fullsized":"2024-04-14T23:48:45.549193+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:22:08.289006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.048719+0000","last_change":"2024-04-14T23:48:43.232567+0000","last_active":"2024-04-14T23:48:45.048719+0000","last_peered":"2024-04-14T23:48:45.048719+0000","last_clean":"2024-04-14T23:48:45.048719+0000","last_became_active":"2024-04-14T23:48:43.232277+0000","last_became_peered":"2024-04-14T23:48:43.232277+0000","last_unstale":"2024-04-14T23:48:45.048719+0000","last_undegraded":"2024-04-14T23:48:45.048719+0000","last_fullsized":"2024-04-14T23:48:45.048719+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T04:59:39.714566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.232426+0000","last_change":"2024-04-14T23:48:43.229443+0000","last_active":"2024-04-14T23:48:44.232426+0000","last_peered":"2024-04-14T23:48:44.232426+0000","last_clean":"2024-04-14T23:48:44.232426+0000","last_became_active":"2024-04-14T23:48:43.229172+0000","last_became_peered":"2024-04-14T23:48:43.229172+0000","last_unstale":"2024-04-14T23:48:44.232426+0000","last_undegraded":"2024-04-14T23:48:44.232426+0000","last_fullsized":"2024-04-14T23:48:44.232426+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:41:31.405287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.234812+0000","last_change":"2024-04-14T23:48:43.242202+0000","last_active":"2024-04-14T23:48:44.234812+0000","last_peered":"2024-04-14T23:48:44.234812+0000","last_clean":"2024-04-14T23:48:44.234812+0000","last_became_active":"2024-04-14T23:48:43.241879+0000","last_became_peered":"2024-04-14T23:48:43.241879+0000","last_unstale":"2024-04-14T23:48:44.234812+0000","last_undegraded":"2024-04-14T23:48:44.234812+0000","last_fullsized":"2024-04-14T23:48:44.234812+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:11:27.552454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230871+0000","last_change":"2024-04-14T23:48:43.242068+0000","last_active":"2024-04-14T23:48:44.230871+0000","last_peered":"2024-04-14T23:48:44.230871+0000","last_clean":"2024-04-14T23:48:44.230871+0000","last_became_active":"2024-04-14T23:48:43.241734+0000","last_became_peered":"2024-04-14T23:48:43.241734+0000","last_unstale":"2024-04-14T23:48:44.230871+0000","last_undegraded":"2024-04-14T23:48:44.230871+0000","last_fullsized":"2024-04-14T23:48:44.230871+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:32:04.579906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.648997+0000","last_change":"2024-04-14T23:48:43.236034+0000","last_active":"2024-04-14T23:48:44.648997+0000","last_peered":"2024-04-14T23:48:44.648997+0000","last_clean":"2024-04-14T23:48:44.648997+0000","last_became_active":"2024-04-14T23:48:43.233567+0000","last_became_peered":"2024-04-14T23:48:43.233567+0000","last_unstale":"2024-04-14T23:48:44.648997+0000","last_undegraded":"2024-04-14T23:48:44.648997+0000","last_fullsized":"2024-04-14T23:48:44.648997+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:07:15.904075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.547675+0000","last_change":"2024-04-14T23:48:43.231045+0000","last_active":"2024-04-14T23:48:45.547675+0000","last_peered":"2024-04-14T23:48:45.547675+0000","last_clean":"2024-04-14T23:48:45.547675+0000","last_became_active":"2024-04-14T23:48:43.230626+0000","last_became_peered":"2024-04-14T23:48:43.230626+0000","last_unstale":"2024-04-14T23:48:45.547675+0000","last_undegraded":"2024-04-14T23:48:45.547675+0000","last_fullsized":"2024-04-14T23:48:45.547675+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:24:03.654061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548613+0000","last_change":"2024-04-14T23:48:43.227543+0000","last_active":"2024-04-14T23:48:45.548613+0000","last_peered":"2024-04-14T23:48:45.548613+0000","last_clean":"2024-04-14T23:48:45.548613+0000","last_became_active":"2024-04-14T23:48:43.226773+0000","last_became_peered":"2024-04-14T23:48:43.226773+0000","last_unstale":"2024-04-14T23:48:45.548613+0000","last_undegraded":"2024-04-14T23:48:45.548613+0000","last_fullsized":"2024-04-14T23:48:45.548613+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:44:56.691708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.211656+0000","last_change":"2024-04-14T23:48:42.239254+0000","last_active":"2024-04-14T23:48:45.211656+0000","last_peered":"2024-04-14T23:48:45.211656+0000","last_clean":"2024-04-14T23:48:45.211656+0000","last_became_active":"2024-04-14T23:48:42.238847+0000","last_became_peered":"2024-04-14T23:48:42.238847+0000","last_unstale":"2024-04-14T23:48:45.211656+0000","last_undegraded":"2024-04-14T23:48:45.211656+0000","last_fullsized":"2024-04-14T23:48:45.211656+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:46:16.301900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.235039+0000","last_change":"2024-04-14T23:48:43.237720+0000","last_active":"2024-04-14T23:48:44.235039+0000","last_peered":"2024-04-14T23:48:44.235039+0000","last_clean":"2024-04-14T23:48:44.235039+0000","last_became_active":"2024-04-14T23:48:43.237391+0000","last_became_peered":"2024-04-14T23:48:43.237391+0000","last_unstale":"2024-04-14T23:48:44.235039+0000","last_undegraded":"2024-04-14T23:48:44.235039+0000","last_fullsized":"2024-04-14T23:48:44.235039+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:38:26.879906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548161+0000","last_change":"2024-04-14T23:48:42.219277+0000","last_active":"2024-04-14T23:48:45.548161+0000","last_peered":"2024-04-14T23:48:45.548161+0000","last_clean":"2024-04-14T23:48:45.548161+0000","last_became_active":"2024-04-14T23:48:42.218995+0000","last_became_peered":"2024-04-14T23:48:42.218995+0000","last_unstale":"2024-04-14T23:48:45.548161+0000","last_undegraded":"2024-04-14T23:48:45.548161+0000","last_fullsized":"2024-04-14T23:48:45.548161+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T23:51:33.370857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.232590+0000","last_change":"2024-04-14T23:48:42.236803+0000","last_active":"2024-04-14T23:48:44.232590+0000","last_peered":"2024-04-14T23:48:44.232590+0000","last_clean":"2024-04-14T23:48:44.232590+0000","last_became_active":"2024-04-14T23:48:42.225937+0000","last_became_peered":"2024-04-14T23:48:42.225937+0000","last_unstale":"2024-04-14T23:48:44.232590+0000","last_undegraded":"2024-04-14T23:48:44.232590+0000","last_fullsized":"2024-04-14T23:48:44.232590+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:52:32.609126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.231732+0000","last_change":"2024-04-14T23:48:43.229506+0000","last_active":"2024-04-14T23:48:44.231732+0000","last_peered":"2024-04-14T23:48:44.231732+0000","last_clean":"2024-04-14T23:48:44.231732+0000","last_became_active":"2024-04-14T23:48:43.227356+0000","last_became_peered":"2024-04-14T23:48:43.227356+0000","last_unstale":"2024-04-14T23:48:44.231732+0000","last_undegraded":"2024-04-14T23:48:44.231732+0000","last_fullsized":"2024-04-14T23:48:44.231732+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:13:04.840655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230660+0000","last_change":"2024-04-14T23:48:42.210883+0000","last_active":"2024-04-14T23:48:44.230660+0000","last_peered":"2024-04-14T23:48:44.230660+0000","last_clean":"2024-04-14T23:48:44.230660+0000","last_became_active":"2024-04-14T23:48:42.210607+0000","last_became_peered":"2024-04-14T23:48:42.210607+0000","last_unstale":"2024-04-14T23:48:44.230660+0000","last_undegraded":"2024-04-14T23:48:44.230660+0000","last_fullsized":"2024-04-14T23:48:44.230660+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:33:32.526223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.048141+0000","last_change":"2024-04-14T23:48:43.218259+0000","last_active":"2024-04-14T23:48:45.048141+0000","last_peered":"2024-04-14T23:48:45.048141+0000","last_clean":"2024-04-14T23:48:45.048141+0000","last_became_active":"2024-04-14T23:48:43.217801+0000","last_became_peered":"2024-04-14T23:48:43.217801+0000","last_unstale":"2024-04-14T23:48:45.048141+0000","last_undegraded":"2024-04-14T23:48:45.048141+0000","last_fullsized":"2024-04-14T23:48:45.048141+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T04:13:42.425034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.547845+0000","last_change":"2024-04-14T23:48:42.218652+0000","last_active":"2024-04-14T23:48:45.547845+0000","last_peered":"2024-04-14T23:48:45.547845+0000","last_clean":"2024-04-14T23:48:45.547845+0000","last_became_active":"2024-04-14T23:48:42.218321+0000","last_became_peered":"2024-04-14T23:48:42.218321+0000","last_unstale":"2024-04-14T23:48:45.547845+0000","last_undegraded":"2024-04-14T23:48:45.547845+0000","last_fullsized":"2024-04-14T23:48:45.547845+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T04:22:06.957747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230879+0000","last_change":"2024-04-14T23:48:43.229117+0000","last_active":"2024-04-14T23:48:44.230879+0000","last_peered":"2024-04-14T23:48:44.230879+0000","last_clean":"2024-04-14T23:48:44.230879+0000","last_became_active":"2024-04-14T23:48:43.228805+0000","last_became_peered":"2024-04-14T23:48:43.228805+0000","last_unstale":"2024-04-14T23:48:44.230879+0000","last_undegraded":"2024-04-14T23:48:44.230879+0000","last_fullsized":"2024-04-14T23:48:44.230879+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:10:39.043818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230930+0000","last_change":"2024-04-14T23:48:42.221104+0000","last_active":"2024-04-14T23:48:44.230930+0000","last_peered":"2024-04-14T23:48:44.230930+0000","last_clean":"2024-04-14T23:48:44.230930+0000","last_became_active":"2024-04-14T23:48:42.220767+0000","last_became_peered":"2024-04-14T23:48:42.220767+0000","last_unstale":"2024-04-14T23:48:44.230930+0000","last_undegraded":"2024-04-14T23:48:44.230930+0000","last_fullsized":"2024-04-14T23:48:44.230930+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:23:57.213764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.235822+0000","last_change":"2024-04-14T23:48:43.236916+0000","last_active":"2024-04-14T23:48:44.235822+0000","last_peered":"2024-04-14T23:48:44.235822+0000","last_clean":"2024-04-14T23:48:44.235822+0000","last_became_active":"2024-04-14T23:48:43.236168+0000","last_became_peered":"2024-04-14T23:48:43.236168+0000","last_unstale":"2024-04-14T23:48:44.235822+0000","last_undegraded":"2024-04-14T23:48:44.235822+0000","last_fullsized":"2024-04-14T23:48:44.235822+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:45:52.057070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:48.220199+0000","last_change":"2024-04-14T23:48:42.231329+0000","last_active":"2024-04-14T23:48:48.220199+0000","last_peered":"2024-04-14T23:48:48.220199+0000","last_clean":"2024-04-14T23:48:48.220199+0000","last_became_active":"2024-04-14T23:48:42.230969+0000","last_became_peered":"2024-04-14T23:48:42.230969+0000","last_unstale":"2024-04-14T23:48:48.220199+0000","last_undegraded":"2024-04-14T23:48:48.220199+0000","last_fullsized":"2024-04-14T23:48:48.220199+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:06:16.557589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548833+0000","last_change":"2024-04-14T23:48:43.230531+0000","last_active":"2024-04-14T23:48:45.548833+0000","last_peered":"2024-04-14T23:48:45.548833+0000","last_clean":"2024-04-14T23:48:45.548833+0000","last_became_active":"2024-04-14T23:48:43.230253+0000","last_became_peered":"2024-04-14T23:48:43.230253+0000","last_unstale":"2024-04-14T23:48:45.548833+0000","last_undegraded":"2024-04-14T23:48:45.548833+0000","last_fullsized":"2024-04-14T23:48:45.548833+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:34:37.449824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'9","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:49:06.196149+0000","last_change":"2024-04-14T23:48:42.237069+0000","last_active":"2024-04-14T23:49:06.196149+0000","last_peered":"2024-04-14T23:49:06.196149+0000","last_clean":"2024-04-14T23:49:06.196149+0000","last_became_active":"2024-04-14T23:48:42.236804+0000","last_became_peered":"2024-04-14T23:48:42.236804+0000","last_unstale":"2024-04-14T23:49:06.196149+0000","last_undegraded":"2024-04-14T23:49:06.196149+0000","last_fullsized":"2024-04-14T23:49:06.196149+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:25:37.318379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2857,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548904+0000","last_change":"2024-04-14T23:48:43.221771+0000","last_active":"2024-04-14T23:48:45.548904+0000","last_peered":"2024-04-14T23:48:45.548904+0000","last_clean":"2024-04-14T23:48:45.548904+0000","last_became_active":"2024-04-14T23:48:43.221462+0000","last_became_peered":"2024-04-14T23:48:43.221462+0000","last_unstale":"2024-04-14T23:48:45.548904+0000","last_undegraded":"2024-04-14T23:48:45.548904+0000","last_fullsized":"2024-04-14T23:48:45.548904+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:29:52.737196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:48.227204+0000","last_change":"2024-04-14T23:48:42.230518+0000","last_active":"2024-04-14T23:48:48.227204+0000","last_peered":"2024-04-14T23:48:48.227204+0000","last_clean":"2024-04-14T23:48:48.227204+0000","last_became_active":"2024-04-14T23:48:42.229758+0000","last_became_peered":"2024-04-14T23:48:42.229758+0000","last_unstale":"2024-04-14T23:48:48.227204+0000","last_undegraded":"2024-04-14T23:48:48.227204+0000","last_fullsized":"2024-04-14T23:48:48.227204+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:01:41.332553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.234816+0000","last_change":"2024-04-14T23:48:43.229873+0000","last_active":"2024-04-14T23:48:44.234816+0000","last_peered":"2024-04-14T23:48:44.234816+0000","last_clean":"2024-04-14T23:48:44.234816+0000","last_became_active":"2024-04-14T23:48:43.229476+0000","last_became_peered":"2024-04-14T23:48:43.229476+0000","last_unstale":"2024-04-14T23:48:44.234816+0000","last_undegraded":"2024-04-14T23:48:44.234816+0000","last_fullsized":"2024-04-14T23:48:44.234816+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:54:22.874344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230858+0000","last_change":"2024-04-14T23:48:42.225404+0000","last_active":"2024-04-14T23:48:44.230858+0000","last_peered":"2024-04-14T23:48:44.230858+0000","last_clean":"2024-04-14T23:48:44.230858+0000","last_became_active":"2024-04-14T23:48:42.225211+0000","last_became_peered":"2024-04-14T23:48:42.225211+0000","last_unstale":"2024-04-14T23:48:44.230858+0000","last_undegraded":"2024-04-14T23:48:44.230858+0000","last_fullsized":"2024-04-14T23:48:44.230858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:49:45.282259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548573+0000","last_change":"2024-04-14T23:48:43.229543+0000","last_active":"2024-04-14T23:48:45.548573+0000","last_peered":"2024-04-14T23:48:45.548573+0000","last_clean":"2024-04-14T23:48:45.548573+0000","last_became_active":"2024-04-14T23:48:43.229297+0000","last_became_peered":"2024-04-14T23:48:43.229297+0000","last_unstale":"2024-04-14T23:48:45.548573+0000","last_undegraded":"2024-04-14T23:48:45.548573+0000","last_fullsized":"2024-04-14T23:48:45.548573+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:00:17.923013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.547633+0000","last_change":"2024-04-14T23:48:42.220492+0000","last_active":"2024-04-14T23:48:45.547633+0000","last_peered":"2024-04-14T23:48:45.547633+0000","last_clean":"2024-04-14T23:48:45.547633+0000","last_became_active":"2024-04-14T23:48:42.220145+0000","last_became_peered":"2024-04-14T23:48:42.220145+0000","last_unstale":"2024-04-14T23:48:45.547633+0000","last_undegraded":"2024-04-14T23:48:45.547633+0000","last_fullsized":"2024-04-14T23:48:45.547633+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:17:45.278993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.547774+0000","last_change":"2024-04-14T23:48:43.235676+0000","last_active":"2024-04-14T23:48:45.547774+0000","last_peered":"2024-04-14T23:48:45.547774+0000","last_clean":"2024-04-14T23:48:45.547774+0000","last_became_active":"2024-04-14T23:48:43.235319+0000","last_became_peered":"2024-04-14T23:48:43.235319+0000","last_unstale":"2024-04-14T23:48:45.547774+0000","last_undegraded":"2024-04-14T23:48:45.547774+0000","last_fullsized":"2024-04-14T23:48:45.547774+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:32:06.283229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:48.230880+0000","last_change":"2024-04-14T23:48:42.231178+0000","last_active":"2024-04-14T23:48:48.230880+0000","last_peered":"2024-04-14T23:48:48.230880+0000","last_clean":"2024-04-14T23:48:48.230880+0000","last_became_active":"2024-04-14T23:48:42.230792+0000","last_became_peered":"2024-04-14T23:48:42.230792+0000","last_unstale":"2024-04-14T23:48:48.230880+0000","last_undegraded":"2024-04-14T23:48:48.230880+0000","last_fullsized":"2024-04-14T23:48:48.230880+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:22:40.744314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548527+0000","last_change":"2024-04-14T23:48:43.228290+0000","last_active":"2024-04-14T23:48:45.548527+0000","last_peered":"2024-04-14T23:48:45.548527+0000","last_clean":"2024-04-14T23:48:45.548527+0000","last_became_active":"2024-04-14T23:48:43.228004+0000","last_became_peered":"2024-04-14T23:48:43.228004+0000","last_unstale":"2024-04-14T23:48:45.548527+0000","last_undegraded":"2024-04-14T23:48:45.548527+0000","last_fullsized":"2024-04-14T23:48:45.548527+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:20:06.572625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548424+0000","last_change":"2024-04-14T23:48:42.231936+0000","last_active":"2024-04-14T23:48:45.548424+0000","last_peered":"2024-04-14T23:48:45.548424+0000","last_clean":"2024-04-14T23:48:45.548424+0000","last_became_active":"2024-04-14T23:48:42.231612+0000","last_became_peered":"2024-04-14T23:48:42.231612+0000","last_unstale":"2024-04-14T23:48:45.548424+0000","last_undegraded":"2024-04-14T23:48:45.548424+0000","last_fullsized":"2024-04-14T23:48:45.548424+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:45:03.098024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.549200+0000","last_change":"2024-04-14T23:48:43.228933+0000","last_active":"2024-04-14T23:48:45.549200+0000","last_peered":"2024-04-14T23:48:45.549200+0000","last_clean":"2024-04-14T23:48:45.549200+0000","last_became_active":"2024-04-14T23:48:43.228665+0000","last_became_peered":"2024-04-14T23:48:43.228665+0000","last_unstale":"2024-04-14T23:48:45.549200+0000","last_undegraded":"2024-04-14T23:48:45.549200+0000","last_fullsized":"2024-04-14T23:48:45.549200+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:43:04.747711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:48.230335+0000","last_change":"2024-04-14T23:48:42.239794+0000","last_active":"2024-04-14T23:48:48.230335+0000","last_peered":"2024-04-14T23:48:48.230335+0000","last_clean":"2024-04-14T23:48:48.230335+0000","last_became_active":"2024-04-14T23:48:42.239505+0000","last_became_peered":"2024-04-14T23:48:42.239505+0000","last_unstale":"2024-04-14T23:48:48.230335+0000","last_undegraded":"2024-04-14T23:48:48.230335+0000","last_fullsized":"2024-04-14T23:48:48.230335+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T09:43:16.582708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.549265+0000","last_change":"2024-04-14T23:48:43.230251+0000","last_active":"2024-04-14T23:48:45.549265+0000","last_peered":"2024-04-14T23:48:45.549265+0000","last_clean":"2024-04-14T23:48:45.549265+0000","last_became_active":"2024-04-14T23:48:43.229949+0000","last_became_peered":"2024-04-14T23:48:43.229949+0000","last_unstale":"2024-04-14T23:48:45.549265+0000","last_undegraded":"2024-04-14T23:48:45.549265+0000","last_fullsized":"2024-04-14T23:48:45.549265+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T09:54:42.581135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.048901+0000","last_change":"2024-04-14T23:48:42.227708+0000","last_active":"2024-04-14T23:48:45.048901+0000","last_peered":"2024-04-14T23:48:45.048901+0000","last_clean":"2024-04-14T23:48:45.048901+0000","last_became_active":"2024-04-14T23:48:42.226845+0000","last_became_peered":"2024-04-14T23:48:42.226845+0000","last_unstale":"2024-04-14T23:48:45.048901+0000","last_undegraded":"2024-04-14T23:48:45.048901+0000","last_fullsized":"2024-04-14T23:48:45.048901+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:02:54.468796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230624+0000","last_change":"2024-04-14T23:48:43.227439+0000","last_active":"2024-04-14T23:48:44.230624+0000","last_peered":"2024-04-14T23:48:44.230624+0000","last_clean":"2024-04-14T23:48:44.230624+0000","last_became_active":"2024-04-14T23:48:43.226717+0000","last_became_peered":"2024-04-14T23:48:43.226717+0000","last_unstale":"2024-04-14T23:48:44.230624+0000","last_undegraded":"2024-04-14T23:48:44.230624+0000","last_fullsized":"2024-04-14T23:48:44.230624+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:01:11.249999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.649211+0000","last_change":"2024-04-14T23:48:42.232164+0000","last_active":"2024-04-14T23:48:44.649211+0000","last_peered":"2024-04-14T23:48:44.649211+0000","last_clean":"2024-04-14T23:48:44.649211+0000","last_became_active":"2024-04-14T23:48:42.229785+0000","last_became_peered":"2024-04-14T23:48:42.229785+0000","last_unstale":"2024-04-14T23:48:44.649211+0000","last_undegraded":"2024-04-14T23:48:44.649211+0000","last_fullsized":"2024-04-14T23:48:44.649211+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T04:21:37.538735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.549097+0000","last_change":"2024-04-14T23:48:43.232949+0000","last_active":"2024-04-14T23:48:45.549097+0000","last_peered":"2024-04-14T23:48:45.549097+0000","last_clean":"2024-04-14T23:48:45.549097+0000","last_became_active":"2024-04-14T23:48:43.232612+0000","last_became_peered":"2024-04-14T23:48:43.232612+0000","last_unstale":"2024-04-14T23:48:45.549097+0000","last_undegraded":"2024-04-14T23:48:45.549097+0000","last_fullsized":"2024-04-14T23:48:45.549097+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:01:00.524663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548069+0000","last_change":"2024-04-14T23:48:38.150200+0000","last_active":"2024-04-14T23:48:45.548069+0000","last_peered":"2024-04-14T23:48:45.548069+0000","last_clean":"2024-04-14T23:48:45.548069+0000","last_became_active":"2024-04-14T23:48:38.149301+0000","last_became_peered":"2024-04-14T23:48:38.149301+0000","last_unstale":"2024-04-14T23:48:45.548069+0000","last_undegraded":"2024-04-14T23:48:45.548069+0000","last_fullsized":"2024-04-14T23:48:45.548069+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:37.129663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:37.129663+0000","last_clean_scrub_stamp":"2024-04-14T23:48:37.129663+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T23:58:01.303851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230439+0000","last_change":"2024-04-14T23:48:42.229238+0000","last_active":"2024-04-14T23:48:44.230439+0000","last_peered":"2024-04-14T23:48:44.230439+0000","last_clean":"2024-04-14T23:48:44.230439+0000","last_became_active":"2024-04-14T23:48:42.228994+0000","last_became_peered":"2024-04-14T23:48:42.228994+0000","last_unstale":"2024-04-14T23:48:44.230439+0000","last_undegraded":"2024-04-14T23:48:44.230439+0000","last_fullsized":"2024-04-14T23:48:44.230439+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:47:15.406228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.232256+0000","last_change":"2024-04-14T23:48:43.223973+0000","last_active":"2024-04-14T23:48:44.232256+0000","last_peered":"2024-04-14T23:48:44.232256+0000","last_clean":"2024-04-14T23:48:44.232256+0000","last_became_active":"2024-04-14T23:48:43.223665+0000","last_became_peered":"2024-04-14T23:48:43.223665+0000","last_unstale":"2024-04-14T23:48:44.232256+0000","last_undegraded":"2024-04-14T23:48:44.232256+0000","last_fullsized":"2024-04-14T23:48:44.232256+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:23:28.495821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.547518+0000","last_change":"2024-04-14T23:48:42.230404+0000","last_active":"2024-04-14T23:48:45.547518+0000","last_peered":"2024-04-14T23:48:45.547518+0000","last_clean":"2024-04-14T23:48:45.547518+0000","last_became_active":"2024-04-14T23:48:42.230173+0000","last_became_peered":"2024-04-14T23:48:42.230173+0000","last_unstale":"2024-04-14T23:48:45.547518+0000","last_undegraded":"2024-04-14T23:48:45.547518+0000","last_fullsized":"2024-04-14T23:48:45.547518+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:23:45.666680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.234372+0000","last_change":"2024-04-14T23:48:43.231958+0000","last_active":"2024-04-14T23:48:44.234372+0000","last_peered":"2024-04-14T23:48:44.234372+0000","last_clean":"2024-04-14T23:48:44.234372+0000","last_became_active":"2024-04-14T23:48:43.231025+0000","last_became_peered":"2024-04-14T23:48:43.231025+0000","last_unstale":"2024-04-14T23:48:44.234372+0000","last_undegraded":"2024-04-14T23:48:44.234372+0000","last_fullsized":"2024-04-14T23:48:44.234372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T23:51:47.947116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.649047+0000","last_change":"2024-04-14T23:48:42.240651+0000","last_active":"2024-04-14T23:48:44.649047+0000","last_peered":"2024-04-14T23:48:44.649047+0000","last_clean":"2024-04-14T23:48:44.649047+0000","last_became_active":"2024-04-14T23:48:42.240351+0000","last_became_peered":"2024-04-14T23:48:42.240351+0000","last_unstale":"2024-04-14T23:48:44.649047+0000","last_undegraded":"2024-04-14T23:48:44.649047+0000","last_fullsized":"2024-04-14T23:48:44.649047+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:03:02.804444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.048231+0000","last_change":"2024-04-14T23:48:43.232415+0000","last_active":"2024-04-14T23:48:45.048231+0000","last_peered":"2024-04-14T23:48:45.048231+0000","last_clean":"2024-04-14T23:48:45.048231+0000","last_became_active":"2024-04-14T23:48:43.232120+0000","last_became_peered":"2024-04-14T23:48:43.232120+0000","last_unstale":"2024-04-14T23:48:45.048231+0000","last_undegraded":"2024-04-14T23:48:45.048231+0000","last_fullsized":"2024-04-14T23:48:45.048231+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:05:27.562738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230988+0000","last_change":"2024-04-14T23:48:42.233743+0000","last_active":"2024-04-14T23:48:44.230988+0000","last_peered":"2024-04-14T23:48:44.230988+0000","last_clean":"2024-04-14T23:48:44.230988+0000","last_became_active":"2024-04-14T23:48:42.233354+0000","last_became_peered":"2024-04-14T23:48:42.233354+0000","last_unstale":"2024-04-14T23:48:44.230988+0000","last_undegraded":"2024-04-14T23:48:44.230988+0000","last_fullsized":"2024-04-14T23:48:44.230988+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:49:23.045968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548229+0000","last_change":"2024-04-14T23:48:43.232468+0000","last_active":"2024-04-14T23:48:45.548229+0000","last_peered":"2024-04-14T23:48:45.548229+0000","last_clean":"2024-04-14T23:48:45.548229+0000","last_became_active":"2024-04-14T23:48:43.232119+0000","last_became_peered":"2024-04-14T23:48:43.232119+0000","last_unstale":"2024-04-14T23:48:45.548229+0000","last_undegraded":"2024-04-14T23:48:45.548229+0000","last_fullsized":"2024-04-14T23:48:45.548229+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:24:12.553325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.048630+0000","last_change":"2024-04-14T23:48:42.242249+0000","last_active":"2024-04-14T23:48:45.048630+0000","last_peered":"2024-04-14T23:48:45.048630+0000","last_clean":"2024-04-14T23:48:45.048630+0000","last_became_active":"2024-04-14T23:48:42.242039+0000","last_became_peered":"2024-04-14T23:48:42.242039+0000","last_unstale":"2024-04-14T23:48:45.048630+0000","last_undegraded":"2024-04-14T23:48:45.048630+0000","last_fullsized":"2024-04-14T23:48:45.048630+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:46:16.301900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230960+0000","last_change":"2024-04-14T23:48:43.242228+0000","last_active":"2024-04-14T23:48:44.230960+0000","last_peered":"2024-04-14T23:48:44.230960+0000","last_clean":"2024-04-14T23:48:44.230960+0000","last_became_active":"2024-04-14T23:48:43.241776+0000","last_became_peered":"2024-04-14T23:48:43.241776+0000","last_unstale":"2024-04-14T23:48:44.230960+0000","last_undegraded":"2024-04-14T23:48:44.230960+0000","last_fullsized":"2024-04-14T23:48:44.230960+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:48:23.653866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.549314+0000","last_change":"2024-04-14T23:48:42.232537+0000","last_active":"2024-04-14T23:48:45.549314+0000","last_peered":"2024-04-14T23:48:45.549314+0000","last_clean":"2024-04-14T23:48:45.549314+0000","last_became_active":"2024-04-14T23:48:42.232325+0000","last_became_peered":"2024-04-14T23:48:42.232325+0000","last_unstale":"2024-04-14T23:48:45.549314+0000","last_undegraded":"2024-04-14T23:48:45.549314+0000","last_fullsized":"2024-04-14T23:48:45.549314+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:34:37.448301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.048333+0000","last_change":"2024-04-14T23:48:43.234445+0000","last_active":"2024-04-14T23:48:45.048333+0000","last_peered":"2024-04-14T23:48:45.048333+0000","last_clean":"2024-04-14T23:48:45.048333+0000","last_became_active":"2024-04-14T23:48:43.234150+0000","last_became_peered":"2024-04-14T23:48:43.234150+0000","last_unstale":"2024-04-14T23:48:45.048333+0000","last_undegraded":"2024-04-14T23:48:45.048333+0000","last_fullsized":"2024-04-14T23:48:45.048333+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:50:59.708087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.049158+0000","last_change":"2024-04-14T23:48:42.230014+0000","last_active":"2024-04-14T23:48:45.049158+0000","last_peered":"2024-04-14T23:48:45.049158+0000","last_clean":"2024-04-14T23:48:45.049158+0000","last_became_active":"2024-04-14T23:48:42.229712+0000","last_became_peered":"2024-04-14T23:48:42.229712+0000","last_unstale":"2024-04-14T23:48:45.049158+0000","last_undegraded":"2024-04-14T23:48:45.049158+0000","last_fullsized":"2024-04-14T23:48:45.049158+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:54:20.595391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.233392+0000","last_change":"2024-04-14T23:48:43.229128+0000","last_active":"2024-04-14T23:48:44.233392+0000","last_peered":"2024-04-14T23:48:44.233392+0000","last_clean":"2024-04-14T23:48:44.233392+0000","last_became_active":"2024-04-14T23:48:43.228739+0000","last_became_peered":"2024-04-14T23:48:43.228739+0000","last_unstale":"2024-04-14T23:48:44.233392+0000","last_undegraded":"2024-04-14T23:48:44.233392+0000","last_fullsized":"2024-04-14T23:48:44.233392+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:08:08.148345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:48.223491+0000","last_change":"2024-04-14T23:48:42.220035+0000","last_active":"2024-04-14T23:48:48.223491+0000","last_peered":"2024-04-14T23:48:48.223491+0000","last_clean":"2024-04-14T23:48:48.223491+0000","last_became_active":"2024-04-14T23:48:42.219750+0000","last_became_peered":"2024-04-14T23:48:42.219750+0000","last_unstale":"2024-04-14T23:48:48.223491+0000","last_undegraded":"2024-04-14T23:48:48.223491+0000","last_fullsized":"2024-04-14T23:48:48.223491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:00:58.928166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.235426+0000","last_change":"2024-04-14T23:48:43.232997+0000","last_active":"2024-04-14T23:48:44.235426+0000","last_peered":"2024-04-14T23:48:44.235426+0000","last_clean":"2024-04-14T23:48:44.235426+0000","last_became_active":"2024-04-14T23:48:43.232730+0000","last_became_peered":"2024-04-14T23:48:43.232730+0000","last_unstale":"2024-04-14T23:48:44.235426+0000","last_undegraded":"2024-04-14T23:48:44.235426+0000","last_fullsized":"2024-04-14T23:48:44.235426+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:38:44.068804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.547928+0000","last_change":"2024-04-14T23:48:42.232087+0000","last_active":"2024-04-14T23:48:45.547928+0000","last_peered":"2024-04-14T23:48:45.547928+0000","last_clean":"2024-04-14T23:48:45.547928+0000","last_became_active":"2024-04-14T23:48:42.231796+0000","last_became_peered":"2024-04-14T23:48:42.231796+0000","last_unstale":"2024-04-14T23:48:45.547928+0000","last_undegraded":"2024-04-14T23:48:45.547928+0000","last_fullsized":"2024-04-14T23:48:45.547928+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:14:35.050093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548007+0000","last_change":"2024-04-14T23:48:43.232871+0000","last_active":"2024-04-14T23:48:45.548007+0000","last_peered":"2024-04-14T23:48:45.548007+0000","last_clean":"2024-04-14T23:48:45.548007+0000","last_became_active":"2024-04-14T23:48:43.232594+0000","last_became_peered":"2024-04-14T23:48:43.232594+0000","last_unstale":"2024-04-14T23:48:45.548007+0000","last_undegraded":"2024-04-14T23:48:45.548007+0000","last_fullsized":"2024-04-14T23:48:45.548007+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:27:37.320994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.048370+0000","last_change":"2024-04-14T23:48:42.231237+0000","last_active":"2024-04-14T23:48:45.048370+0000","last_peered":"2024-04-14T23:48:45.048370+0000","last_clean":"2024-04-14T23:48:45.048370+0000","last_became_active":"2024-04-14T23:48:42.230902+0000","last_became_peered":"2024-04-14T23:48:42.230902+0000","last_unstale":"2024-04-14T23:48:45.048370+0000","last_undegraded":"2024-04-14T23:48:45.048370+0000","last_fullsized":"2024-04-14T23:48:45.048370+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:28:37.228039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.232080+0000","last_change":"2024-04-14T23:48:43.233035+0000","last_active":"2024-04-14T23:48:44.232080+0000","last_peered":"2024-04-14T23:48:44.232080+0000","last_clean":"2024-04-14T23:48:44.232080+0000","last_became_active":"2024-04-14T23:48:43.232747+0000","last_became_peered":"2024-04-14T23:48:43.232747+0000","last_unstale":"2024-04-14T23:48:44.232080+0000","last_undegraded":"2024-04-14T23:48:44.232080+0000","last_fullsized":"2024-04-14T23:48:44.232080+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:38:57.324408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.048301+0000","last_change":"2024-04-14T23:48:42.240496+0000","last_active":"2024-04-14T23:48:45.048301+0000","last_peered":"2024-04-14T23:48:45.048301+0000","last_clean":"2024-04-14T23:48:45.048301+0000","last_became_active":"2024-04-14T23:48:42.238524+0000","last_became_peered":"2024-04-14T23:48:42.238524+0000","last_unstale":"2024-04-14T23:48:45.048301+0000","last_undegraded":"2024-04-14T23:48:45.048301+0000","last_fullsized":"2024-04-14T23:48:45.048301+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:29:35.218759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.233225+0000","last_change":"2024-04-14T23:48:43.234025+0000","last_active":"2024-04-14T23:48:44.233225+0000","last_peered":"2024-04-14T23:48:44.233225+0000","last_clean":"2024-04-14T23:48:44.233225+0000","last_became_active":"2024-04-14T23:48:43.233651+0000","last_became_peered":"2024-04-14T23:48:43.233651+0000","last_unstale":"2024-04-14T23:48:44.233225+0000","last_undegraded":"2024-04-14T23:48:44.233225+0000","last_fullsized":"2024-04-14T23:48:44.233225+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:26:58.094545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:48.217913+0000","last_change":"2024-04-14T23:48:42.218897+0000","last_active":"2024-04-14T23:48:48.217913+0000","last_peered":"2024-04-14T23:48:48.217913+0000","last_clean":"2024-04-14T23:48:48.217913+0000","last_became_active":"2024-04-14T23:48:42.218580+0000","last_became_peered":"2024-04-14T23:48:42.218580+0000","last_unstale":"2024-04-14T23:48:48.217913+0000","last_undegraded":"2024-04-14T23:48:48.217913+0000","last_fullsized":"2024-04-14T23:48:48.217913+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:26:44.477638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.548008+0000","last_change":"2024-04-14T23:48:43.219871+0000","last_active":"2024-04-14T23:48:45.548008+0000","last_peered":"2024-04-14T23:48:45.548008+0000","last_clean":"2024-04-14T23:48:45.548008+0000","last_became_active":"2024-04-14T23:48:43.219601+0000","last_became_peered":"2024-04-14T23:48:43.219601+0000","last_unstale":"2024-04-14T23:48:45.548008+0000","last_undegraded":"2024-04-14T23:48:45.548008+0000","last_fullsized":"2024-04-14T23:48:45.548008+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:36:20.379172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230971+0000","last_change":"2024-04-14T23:48:43.222259+0000","last_active":"2024-04-14T23:48:44.230971+0000","last_peered":"2024-04-14T23:48:44.230971+0000","last_clean":"2024-04-14T23:48:44.230971+0000","last_became_active":"2024-04-14T23:48:43.221842+0000","last_became_peered":"2024-04-14T23:48:43.221842+0000","last_unstale":"2024-04-14T23:48:44.230971+0000","last_undegraded":"2024-04-14T23:48:44.230971+0000","last_fullsized":"2024-04-14T23:48:44.230971+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:12:20.038987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:48.217880+0000","last_change":"2024-04-14T23:48:42.221314+0000","last_active":"2024-04-14T23:48:48.217880+0000","last_peered":"2024-04-14T23:48:48.217880+0000","last_clean":"2024-04-14T23:48:48.217880+0000","last_became_active":"2024-04-14T23:48:42.221028+0000","last_became_peered":"2024-04-14T23:48:42.221028+0000","last_unstale":"2024-04-14T23:48:48.217880+0000","last_undegraded":"2024-04-14T23:48:48.217880+0000","last_fullsized":"2024-04-14T23:48:48.217880+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T09:58:18.195400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.230606+0000","last_change":"2024-04-14T23:48:43.224577+0000","last_active":"2024-04-14T23:48:44.230606+0000","last_peered":"2024-04-14T23:48:44.230606+0000","last_clean":"2024-04-14T23:48:44.230606+0000","last_became_active":"2024-04-14T23:48:43.224312+0000","last_became_peered":"2024-04-14T23:48:43.224312+0000","last_unstale":"2024-04-14T23:48:44.230606+0000","last_undegraded":"2024-04-14T23:48:44.230606+0000","last_fullsized":"2024-04-14T23:48:44.230606+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:52:44.955773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:48.217170+0000","last_change":"2024-04-14T23:48:42.221000+0000","last_active":"2024-04-14T23:48:48.217170+0000","last_peered":"2024-04-14T23:48:48.217170+0000","last_clean":"2024-04-14T23:48:48.217170+0000","last_became_active":"2024-04-14T23:48:42.219813+0000","last_became_peered":"2024-04-14T23:48:42.219813+0000","last_unstale":"2024-04-14T23:48:48.217170+0000","last_undegraded":"2024-04-14T23:48:48.217170+0000","last_fullsized":"2024-04-14T23:48:48.217170+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:00:38.107953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.235379+0000","last_change":"2024-04-14T23:48:43.236946+0000","last_active":"2024-04-14T23:48:44.235379+0000","last_peered":"2024-04-14T23:48:44.235379+0000","last_clean":"2024-04-14T23:48:44.235379+0000","last_became_active":"2024-04-14T23:48:43.236595+0000","last_became_peered":"2024-04-14T23:48:43.236595+0000","last_unstale":"2024-04-14T23:48:44.235379+0000","last_undegraded":"2024-04-14T23:48:44.235379+0000","last_fullsized":"2024-04-14T23:48:44.235379+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:56:46.037478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:45.211743+0000","last_change":"2024-04-14T23:48:42.239255+0000","last_active":"2024-04-14T23:48:45.211743+0000","last_peered":"2024-04-14T23:48:45.211743+0000","last_clean":"2024-04-14T23:48:45.211743+0000","last_became_active":"2024-04-14T23:48:42.238892+0000","last_became_peered":"2024-04-14T23:48:42.238892+0000","last_unstale":"2024-04-14T23:48:45.211743+0000","last_undegraded":"2024-04-14T23:48:45.211743+0000","last_fullsized":"2024-04-14T23:48:45.211743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:44:39.632155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:44.233893+0000","last_change":"2024-04-14T23:48:43.227240+0000","last_active":"2024-04-14T23:48:44.233893+0000","last_peered":"2024-04-14T23:48:44.233893+0000","last_clean":"2024-04-14T23:48:44.233893+0000","last_became_active":"2024-04-14T23:48:43.222707+0000","last_became_peered":"2024-04-14T23:48:43.222707+0000","last_unstale":"2024-04-14T23:48:44.233893+0000","last_undegraded":"2024-04-14T23:48:44.233893+0000","last_fullsized":"2024-04-14T23:48:44.233893+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:02:17.231763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-14T23:48:48.228047+0000","last_change":"2024-04-14T23:48:42.226563+0000","last_active":"2024-04-14T23:48:48.228047+0000","last_peered":"2024-04-14T23:48:48.228047+0000","last_clean":"2024-04-14T23:48:48.228047+0000","last_became_active":"2024-04-14T23:48:42.226238+0000","last_became_peered":"2024-04-14T23:48:42.226238+0000","last_unstale":"2024-04-14T23:48:48.228047+0000","last_undegraded":"2024-04-14T23:48:48.228047+0000","last_fullsized":"2024-04-14T23:48:48.228047+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:01:36.516898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":3655,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65128,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574858,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":286720,"data_stored":666555,"data_compressed":32564,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574859,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":78462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574858,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":221184,"data_stored":75695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574859,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":75731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":78462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574858,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574858,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":225280,"data_stored":75765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":294912,"data_stored":666611,"data_compressed":32564,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32564,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32564,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-14T23:49:13.330 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-14T23:49:13.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.580+0000 7f11fea00640 1 Processor -- start 2024-04-14T23:49:13.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.580+0000 7f11fea00640 1 -- start start 2024-04-14T23:49:13.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.583+0000 7f11fea00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f11f80c4fa0 0x7f11f80c77b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:13.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.583+0000 7f11fea00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f11f8070d50 0x7f11f80c7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:13.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.583+0000 7f11fea00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11f806b4b0 con 0x7f11f80c5380 2024-04-14T23:49:13.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.583+0000 7f11fea00640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f11f806b790 con 0x7f11f80c4fa0 2024-04-14T23:49:13.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.583+0000 7f11fea00640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f11f80c69f0 con 0x7f11f8070d50 2024-04-14T23:49:13.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.583+0000 7f11fd600640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f11f80c4fa0 0x7f11f80c77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:13.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.583+0000 7f11fd600640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f11f80c4fa0 0x7f11f80c77b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37320/0 (socket says 172.21.15.60:37320) 2024-04-14T23:49:13.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.583+0000 7f11fd600640 1 -- 172.21.15.60:0/173834738 learned_addr learned my addr 172.21.15.60:0/173834738 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:49:13.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.583+0000 7f11fcc00640 1 --2- 172.21.15.60:0/173834738 >> v2:172.21.15.186:3300/0 conn(0x7f11f8070d50 0x7f11f80c7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:13.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.583+0000 7f11fd600640 1 -- 172.21.15.60:0/173834738 >> v2:172.21.15.186:3300/0 conn(0x7f11f8070d50 msgr2=0x7f11f80c7cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:49:13.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.583+0000 7f11fd600640 1 --2- 172.21.15.60:0/173834738 >> v2:172.21.15.186:3300/0 conn(0x7f11f8070d50 0x7f11f80c7cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:13.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.583+0000 7f11fd600640 1 -- 172.21.15.60:0/173834738 >> v1:172.21.15.60:6789/0 conn(0x7f11f80c5380 legacy=0x7f11f815a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:13.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.583+0000 7f11fd600640 1 -- 172.21.15.60:0/173834738 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11f80c6cd0 con 0x7f11f80c4fa0 2024-04-14T23:49:13.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.583+0000 7f11f7e00640 1 -- 172.21.15.60:0/173834738 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1003184440 0 0) 0x7f11f806b4b0 con 0x7f11f80c5380 2024-04-14T23:49:13.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.584+0000 7f11fd600640 1 --2- 172.21.15.60:0/173834738 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f11f80c4fa0 0x7f11f80c77b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f11f80c8a80 tx=0x7f11ec02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fff06e0403f4e0e server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:13.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.584+0000 7f11f7e00640 1 -- 172.21.15.60:0/173834738 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11ec034070 con 0x7f11f80c4fa0 2024-04-14T23:49:13.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.584+0000 7f11f7e00640 1 -- 172.21.15.60:0/173834738 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11ec030860 con 0x7f11f80c4fa0 2024-04-14T23:49:13.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.584+0000 7f11f7e00640 1 -- 172.21.15.60:0/173834738 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11ec030c30 con 0x7f11f80c4fa0 2024-04-14T23:49:13.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.585+0000 7f11fea00640 1 -- 172.21.15.60:0/173834738 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f11f80c4fa0 msgr2=0x7f11f80c77b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:13.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.585+0000 7f11fea00640 1 --2- 172.21.15.60:0/173834738 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f11f80c4fa0 0x7f11f80c77b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f11f80c8a80 tx=0x7f11ec02b9a0 comp rx=0 tx=0).stop 2024-04-14T23:49:13.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.585+0000 7f11fea00640 1 -- 172.21.15.60:0/173834738 shutdown_connections 2024-04-14T23:49:13.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.585+0000 7f11fea00640 1 --2- 172.21.15.60:0/173834738 >> v2:172.21.15.186:3300/0 conn(0x7f11f8070d50 0x7f11f80c7cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:13.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.585+0000 7f11fea00640 1 --2- 172.21.15.60:0/173834738 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f11f80c4fa0 0x7f11f80c77b0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:13.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.585+0000 7f11fea00640 1 -- 172.21.15.60:0/173834738 >> 172.21.15.60:0/173834738 conn(0x7f11f80c8b50 msgr2=0x7f11f80c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:49:13.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.585+0000 7f11fea00640 1 -- 172.21.15.60:0/173834738 shutdown_connections 2024-04-14T23:49:13.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.585+0000 7f11fea00640 1 -- 172.21.15.60:0/173834738 wait complete. 2024-04-14T23:49:13.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.587+0000 7f11fea00640 1 Processor -- start 2024-04-14T23:49:13.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.587+0000 7f11fea00640 1 -- start start 2024-04-14T23:49:13.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.588+0000 7f11fea00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f11f8070d50 0x7f11f806bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:13.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.588+0000 7f11fea00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f11f80c4fa0 0x7f11f81e4290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:13.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.588+0000 7f11fea00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11f80c5380 0x7f11f8157450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:13.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.588+0000 7f11fea00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f11f806b1c0 con 0x7f11f80c4fa0 2024-04-14T23:49:13.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.588+0000 7f11fea00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f11f80c7290 con 0x7f11f80c5380 2024-04-14T23:49:13.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.588+0000 7f11fea00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f11f80c99d0 con 0x7f11f8070d50 2024-04-14T23:49:13.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.588+0000 7f11fcc00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f11f80c4fa0 0x7f11f81e4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:13.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.588+0000 7f11fd600640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f11f8070d50 0x7f11f806bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:13.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.588+0000 7f11fe000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11f80c5380 0x7f11f8157450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:13.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.588+0000 7f11fd600640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f11f8070d50 0x7f11f806bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.60:32864/0 (socket says 172.21.15.60:32864) 2024-04-14T23:49:13.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.588+0000 7f11fd600640 1 -- 172.21.15.60:0/986927367 learned_addr learned my addr 172.21.15.60:0/986927367 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:49:13.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.589+0000 7f11fd600640 1 -- 172.21.15.60:0/986927367 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11f80c5380 msgr2=0x7f11f8157450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:13.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.589+0000 7f11fd600640 1 --2- 172.21.15.60:0/986927367 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11f80c5380 0x7f11f8157450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:13.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.589+0000 7f11fd600640 1 -- 172.21.15.60:0/986927367 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f11f80c4fa0 msgr2=0x7f11f81e4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:13.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.589+0000 7f11fd600640 1 --2- 172.21.15.60:0/986927367 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f11f80c4fa0 0x7f11f81e4290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:13.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.589+0000 7f11fd600640 1 -- 172.21.15.60:0/986927367 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11f81e6480 con 0x7f11f8070d50 2024-04-14T23:49:13.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.589+0000 7f11fe000640 1 --2- 172.21.15.60:0/986927367 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11f80c5380 0x7f11f8157450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:49:13.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.589+0000 7f11fd600640 1 --2- 172.21.15.60:0/986927367 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f11f8070d50 0x7f11f806bcc0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f11ec009960 tx=0x7f11ec0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:13.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.589+0000 7f11f5600640 1 -- 172.21.15.60:0/986927367 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11ec034040 con 0x7f11f8070d50 2024-04-14T23:49:13.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.589+0000 7f11f5600640 1 -- 172.21.15.60:0/986927367 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11ec03ab90 con 0x7f11f8070d50 2024-04-14T23:49:13.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.589+0000 7f11fea00640 1 -- 172.21.15.60:0/986927367 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11f80aeb40 con 0x7f11f8070d50 2024-04-14T23:49:13.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.589+0000 7f11f5600640 1 -- 172.21.15.60:0/986927367 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11ec004030 con 0x7f11f8070d50 2024-04-14T23:49:13.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.590+0000 7f11fea00640 1 -- 172.21.15.60:0/986927367 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f11f80af100 con 0x7f11f8070d50 2024-04-14T23:49:13.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.591+0000 7f11fea00640 1 -- 172.21.15.60:0/986927367 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11f804f540 con 0x7f11f8070d50 2024-04-14T23:49:13.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.592+0000 7f11f5600640 1 -- 172.21.15.60:0/986927367 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f11ec037070 con 0x7f11f8070d50 2024-04-14T23:49:13.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.592+0000 7f11f5600640 1 --2- 172.21.15.60:0/986927367 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f11d806b990 0x7f11d806de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:13.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.593+0000 7f11f5600640 1 -- 172.21.15.60:0/986927367 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f11ec0ac7e0 con 0x7f11f8070d50 2024-04-14T23:49:13.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.593+0000 7f11fcc00640 1 --2- 172.21.15.60:0/986927367 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f11d806b990 0x7f11d806de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:13.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.594+0000 7f11fcc00640 1 --2- 172.21.15.60:0/986927367 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f11d806b990 0x7f11d806de50 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f11f81eadc0 tx=0x7f11e4009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:13.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.598+0000 7f11f5600640 1 -- 172.21.15.60:0/986927367 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f11ec03f070 con 0x7f11f8070d50 2024-04-14T23:49:13.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.800+0000 7f11fea00640 1 -- 172.21.15.60:0/986927367 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f11f80ca650 con 0x7f11f8070d50 2024-04-14T23:49:13.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.802+0000 7f11f5600640 1 -- 172.21.15.60:0/986927367 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16605 (secure 0 0 0) 0x7f11ec07ac70 con 0x7f11f8070d50 2024-04-14T23:49:13.807 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:49:13.807 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":21,"fsid":"5847db56-a76d-4486-b563-d83f1186af92","created":"2024-04-14T23:48:23.900003+0000","modified":"2024-04-14T23:48:44.219352+0000","last_up_change":"2024-04-14T23:48:34.203573+0000","last_in_change":"2024-04-14T23:48:26.345499+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-14T23:48:36.247372+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-14T23:48:40.274195+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-14T23:48:41.722183+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b2b2b755-7f42-4f9b-af9a-a178f7cc5a52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6817","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6818","nonce":2536528895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6822","nonce":2536528895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6827","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6828","nonce":2536528895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6825","nonce":2536528895}]},"public_addr":"172.21.15.60:6818/2536528895","cluster_addr":"172.21.15.60:6822/2536528895","heartbeat_back_addr":"172.21.15.60:6828/2536528895","heartbeat_front_addr":"172.21.15.60:6825/2536528895","state":["exists","up"]},{"osd":1,"uuid":"321418cd-5ab7-41a5-b9c8-b3fc12f68ebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6801","nonce":3115695676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6803","nonce":3115695676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6807","nonce":3115695676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6805","nonce":3115695676}]},"public_addr":"172.21.15.60:6801/3115695676","cluster_addr":"172.21.15.60:6803/3115695676","heartbeat_back_addr":"172.21.15.60:6807/3115695676","heartbeat_front_addr":"172.21.15.60:6805/3115695676","state":["exists","up"]},{"osd":2,"uuid":"07874e9f-3a38-41dc-be9f-2b75e84e53d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6819","nonce":3475168698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6824","nonce":3475168698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6831","nonce":3475168698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6829","nonce":3475168698}]},"public_addr":"172.21.15.60:6819/3475168698","cluster_addr":"172.21.15.60:6824/3475168698","heartbeat_back_addr":"172.21.15.60:6831/3475168698","heartbeat_front_addr":"172.21.15.60:6829/3475168698","state":["exists","up"]},{"osd":3,"uuid":"7e4ef6c9-b17a-47c6-a320-f5d16a6fe05e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6809","nonce":1730813674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6811","nonce":1730813674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6815","nonce":1730813674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6813","nonce":1730813674}]},"public_addr":"172.21.15.60:6809/1730813674","cluster_addr":"172.21.15.60:6811/1730813674","heartbeat_back_addr":"172.21.15.60:6815/1730813674","heartbeat_front_addr":"172.21.15.60:6813/1730813674","state":["exists","up"]},{"osd":4,"uuid":"759a3efc-dade-44c5-b57a-1623499aba50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6805","nonce":1464587038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6810","nonce":1464587038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6819","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6822","nonce":1464587038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6813","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6816","nonce":1464587038}]},"public_addr":"172.21.15.186:6805/1464587038","cluster_addr":"172.21.15.186:6810/1464587038","heartbeat_back_addr":"172.21.15.186:6822/1464587038","heartbeat_front_addr":"172.21.15.186:6816/1464587038","state":["exists","up"]},{"osd":5,"uuid":"24cd58c1-0cf1-4077-ab33-84c39c6907ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6801","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6803","nonce":1697098067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6809","nonce":1697098067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6820","nonce":1697098067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6815","nonce":1697098067}]},"public_addr":"172.21.15.186:6803/1697098067","cluster_addr":"172.21.15.186:6809/1697098067","heartbeat_back_addr":"172.21.15.186:6820/1697098067","heartbeat_front_addr":"172.21.15.186:6815/1697098067","state":["exists","up"]},{"osd":6,"uuid":"6add38f1-65e9-4b0d-927a-4ab77f5e2323","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6804","nonce":1837108559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6807","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6811","nonce":1837108559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6823","nonce":1837108559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6817","nonce":1837108559}]},"public_addr":"172.21.15.186:6804/1837108559","cluster_addr":"172.21.15.186:6811/1837108559","heartbeat_back_addr":"172.21.15.186:6823/1837108559","heartbeat_front_addr":"172.21.15.186:6817/1837108559","state":["exists","up"]},{"osd":7,"uuid":"a2326b6f-7601-4dda-a456-67c443ea5929","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6825","nonce":842537875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6827","nonce":842537875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6831","nonce":842537875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6829","nonce":842537875}]},"public_addr":"172.21.15.186:6825/842537875","cluster_addr":"172.21.15.186:6827/842537875","heartbeat_back_addr":"172.21.15.186:6831/842537875","heartbeat_front_addr":"172.21.15.186:6829/842537875","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.975583+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.976806+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.722725+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.763705+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.502140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.528972+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.517961+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.513350+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-14T23:49:13.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.805+0000 7f11fea00640 1 -- 172.21.15.60:0/986927367 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f11d806b990 msgr2=0x7f11d806de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:13.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.805+0000 7f11fea00640 1 --2- 172.21.15.60:0/986927367 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f11d806b990 0x7f11d806de50 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f11f81eadc0 tx=0x7f11e4009190 comp rx=0 tx=0).stop 2024-04-14T23:49:13.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.805+0000 7f11fea00640 1 -- 172.21.15.60:0/986927367 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f11f8070d50 msgr2=0x7f11f806bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:13.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.805+0000 7f11fea00640 1 --2- 172.21.15.60:0/986927367 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f11f8070d50 0x7f11f806bcc0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f11ec009960 tx=0x7f11ec0046c0 comp rx=0 tx=0).stop 2024-04-14T23:49:13.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.805+0000 7f11fea00640 1 -- 172.21.15.60:0/986927367 shutdown_connections 2024-04-14T23:49:13.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.805+0000 7f11fea00640 1 --2- 172.21.15.60:0/986927367 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f11d806b990 0x7f11d806de50 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:13.812 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.805+0000 7f11fea00640 1 --2- 172.21.15.60:0/986927367 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11f80c5380 0x7f11f8157450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:13.812 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.805+0000 7f11fea00640 1 --2- 172.21.15.60:0/986927367 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f11f80c4fa0 0x7f11f81e4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:13.812 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.805+0000 7f11fea00640 1 --2- 172.21.15.60:0/986927367 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f11f8070d50 0x7f11f806bcc0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:13.812 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.805+0000 7f11fea00640 1 -- 172.21.15.60:0/986927367 >> 172.21.15.60:0/986927367 conn(0x7f11f80c8b50 msgr2=0x7f11f80c9e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:49:13.812 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.805+0000 7f11fea00640 1 -- 172.21.15.60:0/986927367 shutdown_connections 2024-04-14T23:49:13.812 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:13.805+0000 7f11fea00640 1 -- 172.21.15.60:0/986927367 wait complete. 2024-04-14T23:49:13.831 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-14T23:48:36.247372+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-14T23:48:40.274195+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-14T23:48:41.722183+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-14T23:49:13.831 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-14T23:49:13.878 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-14T23:49:13.886 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-14T23:49:14.040 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.035+0000 7ffa9f200640 1 Processor -- start 2024-04-14T23:49:14.040 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.036+0000 7ffa9f200640 1 -- start start 2024-04-14T23:49:14.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.037+0000 7ffa9f200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffaa0152860 0x7ffaa00c31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:14.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.037+0000 7ffa9f200640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ffaa00c3720 0x7ffaa0163140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:14.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.037+0000 7ffa9f200640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffaa0159cb0 con 0x7ffaa00c8410 2024-04-14T23:49:14.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.037+0000 7ffa9f200640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7ffaa006eaf0 con 0x7ffaa00c3720 2024-04-14T23:49:14.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.037+0000 7ffa9f200640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7ffaa006fa30 con 0x7ffaa0152860 2024-04-14T23:49:14.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.037+0000 7ffa9de00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffaa0152860 0x7ffaa00c31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:14.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.037+0000 7ffa9d400640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7ffaa00c8410 0x7ffaa00ce150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.60:36246/0 (socket says 172.21.15.60:36246) 2024-04-14T23:49:14.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.037+0000 7ffa9d400640 1 -- 172.21.15.60:0/3132197310 learned_addr learned my addr 172.21.15.60:0/3132197310 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:49:14.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.037+0000 7ffa9e800640 1 --2- 172.21.15.60:0/3132197310 >> v2:172.21.15.60:3300/0 conn(0x7ffaa00c3720 0x7ffaa0163140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:14.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.037+0000 7ffa9ca00640 1 -- 172.21.15.60:0/3132197310 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2501296834 0 0) 0x7ffaa0159cb0 con 0x7ffaa00c8410 2024-04-14T23:49:14.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.037+0000 7ffa9de00640 1 -- 172.21.15.60:0/3132197310 >> v2:172.21.15.60:3300/0 conn(0x7ffaa00c3720 msgr2=0x7ffaa0163140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:49:14.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.037+0000 7ffa9de00640 1 --2- 172.21.15.60:0/3132197310 >> v2:172.21.15.60:3300/0 conn(0x7ffaa00c3720 0x7ffaa0163140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:14.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.037+0000 7ffa9de00640 1 -- 172.21.15.60:0/3132197310 >> v1:172.21.15.186:6789/0 conn(0x7ffaa00c8410 legacy=0x7ffaa00ce150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:14.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.037+0000 7ffa9de00640 1 -- 172.21.15.60:0/3132197310 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffaa00c50e0 con 0x7ffaa0152860 2024-04-14T23:49:14.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.038+0000 7ffa9de00640 1 --2- 172.21.15.60:0/3132197310 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffaa0152860 0x7ffaa00c31e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ffa900097a0 tx=0x7ffa9002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dbe98a831366c694 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:14.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.038+0000 7ffa9ca00640 1 -- 172.21.15.60:0/3132197310 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa90035020 con 0x7ffaa0152860 2024-04-14T23:49:14.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.038+0000 7ffa9ca00640 1 -- 172.21.15.60:0/3132197310 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa90031830 con 0x7ffaa0152860 2024-04-14T23:49:14.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.038+0000 7ffa9ca00640 1 -- 172.21.15.60:0/3132197310 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa90031c00 con 0x7ffaa0152860 2024-04-14T23:49:14.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.039+0000 7ffa9f200640 1 -- 172.21.15.60:0/3132197310 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffaa0152860 msgr2=0x7ffaa00c31e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:14.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.039+0000 7ffa9f200640 1 --2- 172.21.15.60:0/3132197310 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffaa0152860 0x7ffaa00c31e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ffa900097a0 tx=0x7ffa9002b8d0 comp rx=0 tx=0).stop 2024-04-14T23:49:14.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.039+0000 7ffa9f200640 1 -- 172.21.15.60:0/3132197310 shutdown_connections 2024-04-14T23:49:14.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.039+0000 7ffa9f200640 1 --2- 172.21.15.60:0/3132197310 >> v2:172.21.15.60:3300/0 conn(0x7ffaa00c3720 0x7ffaa0163140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:14.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.039+0000 7ffa9f200640 1 --2- 172.21.15.60:0/3132197310 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffaa0152860 0x7ffaa00c31e0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:14.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.039+0000 7ffa9f200640 1 -- 172.21.15.60:0/3132197310 >> 172.21.15.60:0/3132197310 conn(0x7ffaa0066f10 msgr2=0x7ffaa00672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:49:14.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.039+0000 7ffa9f200640 1 -- 172.21.15.60:0/3132197310 shutdown_connections 2024-04-14T23:49:14.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.039+0000 7ffa9f200640 1 -- 172.21.15.60:0/3132197310 wait complete. 2024-04-14T23:49:14.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.042+0000 7ffa9f200640 1 Processor -- start 2024-04-14T23:49:14.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.042+0000 7ffa9f200640 1 -- start start 2024-04-14T23:49:14.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.044+0000 7ffa9f200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffaa00c3720 0x7ffaa00cb080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:14.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.044+0000 7ffa9f200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffaa00c8410 0x7ffaa01f13f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:14.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.044+0000 7ffa9f200640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffaa0152860 0x7ffaa00c9410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:14.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.044+0000 7ffa9f200640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ffaa01592e0 con 0x7ffaa0152860 2024-04-14T23:49:14.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.044+0000 7ffa9f200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ffaa0152c40 con 0x7ffaa00c3720 2024-04-14T23:49:14.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.044+0000 7ffa9f200640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7ffaa01595c0 con 0x7ffaa00c8410 2024-04-14T23:49:14.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.044+0000 7ffa9d400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffaa00c8410 0x7ffaa01f13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:14.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.044+0000 7ffa9de00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffaa00c3720 0x7ffaa00cb080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:14.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.044+0000 7ffa9e800640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffaa0152860 0x7ffaa00c9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:14.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.044+0000 7ffa9de00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffaa00c3720 0x7ffaa00cb080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:39782/0 (socket says 172.21.15.60:39782) 2024-04-14T23:49:14.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.044+0000 7ffa9de00640 1 -- 172.21.15.60:0/48300908 learned_addr learned my addr 172.21.15.60:0/48300908 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:49:14.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.044+0000 7ffa9de00640 1 -- 172.21.15.60:0/48300908 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffaa00c8410 msgr2=0x7ffaa01f13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:14.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.044+0000 7ffa9de00640 1 --2- 172.21.15.60:0/48300908 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffaa00c8410 0x7ffaa01f13f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:14.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.044+0000 7ffa9de00640 1 -- 172.21.15.60:0/48300908 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffaa0152860 msgr2=0x7ffaa00c9410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:14.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.044+0000 7ffa9de00640 1 --2- 172.21.15.60:0/48300908 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffaa0152860 0x7ffaa00c9410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:14.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.044+0000 7ffa9de00640 1 -- 172.21.15.60:0/48300908 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffaa01e1ac0 con 0x7ffaa00c3720 2024-04-14T23:49:14.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.044+0000 7ffa9de00640 1 --2- 172.21.15.60:0/48300908 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffaa00c3720 0x7ffaa00cb080 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ffa90034040 tx=0x7ffa900046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:14.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.044+0000 7ffa96000640 1 -- 172.21.15.60:0/48300908 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa90040070 con 0x7ffaa00c3720 2024-04-14T23:49:14.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.044+0000 7ffa96000640 1 -- 172.21.15.60:0/48300908 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa9003a910 con 0x7ffaa00c3720 2024-04-14T23:49:14.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.044+0000 7ffa96000640 1 -- 172.21.15.60:0/48300908 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa90035040 con 0x7ffaa00c3720 2024-04-14T23:49:14.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.044+0000 7ffa9f200640 1 -- 172.21.15.60:0/48300908 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffaa01e1da0 con 0x7ffaa00c3720 2024-04-14T23:49:14.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.045+0000 7ffa9f200640 1 -- 172.21.15.60:0/48300908 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffaa01e2360 con 0x7ffaa00c3720 2024-04-14T23:49:14.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.046+0000 7ffa9f200640 1 -- 172.21.15.60:0/48300908 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffaa00cde40 con 0x7ffaa00c3720 2024-04-14T23:49:14.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.046+0000 7ffa96000640 1 -- 172.21.15.60:0/48300908 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7ffa90038020 con 0x7ffaa00c3720 2024-04-14T23:49:14.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.047+0000 7ffa96000640 1 --2- 172.21.15.60:0/48300908 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ffa7c06b9c0 0x7ffa7c06de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:14.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.047+0000 7ffa96000640 1 -- 172.21.15.60:0/48300908 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ffa900ac100 con 0x7ffaa00c3720 2024-04-14T23:49:14.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.048+0000 7ffa9d400640 1 --2- 172.21.15.60:0/48300908 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ffa7c06b9c0 0x7ffa7c06de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:14.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.048+0000 7ffa9d400640 1 --2- 172.21.15.60:0/48300908 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ffa7c06b9c0 0x7ffa7c06de80 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ffa88000f60 tx=0x7ffa88007d50 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:14.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.053+0000 7ffa96000640 1 -- 172.21.15.60:0/48300908 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ffaa00cde40 con 0x7ffaa00c3720 2024-04-14T23:49:14.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.257+0000 7ffa9f200640 1 -- 172.21.15.60:0/48300908 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ffaa004f540 con 0x7ffaa00c3720 2024-04-14T23:49:14.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.258+0000 7ffa96000640 1 -- 172.21.15.60:0/48300908 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7ffa90031800 con 0x7ffaa00c3720 2024-04-14T23:49:14.264 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2024-04-14T23:49:14.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.261+0000 7ffa9f200640 1 -- 172.21.15.60:0/48300908 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ffa7c06b9c0 msgr2=0x7ffa7c06de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:14.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.261+0000 7ffa9f200640 1 --2- 172.21.15.60:0/48300908 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ffa7c06b9c0 0x7ffa7c06de80 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ffa88000f60 tx=0x7ffa88007d50 comp rx=0 tx=0).stop 2024-04-14T23:49:14.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.261+0000 7ffa9f200640 1 -- 172.21.15.60:0/48300908 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffaa00c3720 msgr2=0x7ffaa00cb080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:14.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.261+0000 7ffa9f200640 1 --2- 172.21.15.60:0/48300908 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffaa00c3720 0x7ffaa00cb080 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ffa90034040 tx=0x7ffa900046c0 comp rx=0 tx=0).stop 2024-04-14T23:49:14.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.261+0000 7ffa9f200640 1 -- 172.21.15.60:0/48300908 shutdown_connections 2024-04-14T23:49:14.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.261+0000 7ffa9f200640 1 --2- 172.21.15.60:0/48300908 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ffa7c06b9c0 0x7ffa7c06de80 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:14.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.261+0000 7ffa9f200640 1 --2- 172.21.15.60:0/48300908 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffaa0152860 0x7ffaa00c9410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:14.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.261+0000 7ffa9f200640 1 --2- 172.21.15.60:0/48300908 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffaa00c8410 0x7ffaa01f13f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:14.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.261+0000 7ffa9f200640 1 --2- 172.21.15.60:0/48300908 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffaa00c3720 0x7ffaa00cb080 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:14.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.261+0000 7ffa9f200640 1 -- 172.21.15.60:0/48300908 >> 172.21.15.60:0/48300908 conn(0x7ffaa0066f10 msgr2=0x7ffaa006d580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:49:14.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.262+0000 7ffa9f200640 1 -- 172.21.15.60:0/48300908 shutdown_connections 2024-04-14T23:49:14.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.262+0000 7ffa9f200640 1 -- 172.21.15.60:0/48300908 wait complete. 2024-04-14T23:49:14.288 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-14T23:49:14.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.504+0000 7f9c6c400640 1 Processor -- start 2024-04-14T23:49:14.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.504+0000 7f9c6c400640 1 -- start start 2024-04-14T23:49:14.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.505+0000 7f9c6c400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9c640cae70 0x7f9c640cb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:14.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.505+0000 7f9c6c400640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c64155060 con 0x7f9c64154c80 2024-04-14T23:49:14.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.505+0000 7f9c6c400640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c640c68c0 con 0x7f9c640c5710 2024-04-14T23:49:14.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.505+0000 7f9c6c400640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f9c6415e880 con 0x7f9c640cae70 2024-04-14T23:49:14.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.506+0000 7f9c69000640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f9c640c5710 0x7f9c640c5ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.60:36258/0 (socket says 172.21.15.60:36258) 2024-04-14T23:49:14.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.506+0000 7f9c69000640 1 -- 172.21.15.60:0/2789781944 learned_addr learned my addr 172.21.15.60:0/2789781944 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:49:14.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.506+0000 7f9c6a400640 1 --2- 172.21.15.60:0/2789781944 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9c640cae70 0x7f9c640cb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:14.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.506+0000 7f9c63e00640 1 -- 172.21.15.60:0/2789781944 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3696699790 0 0) 0x7f9c640c68c0 con 0x7f9c640c5710 2024-04-14T23:49:14.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.506+0000 7f9c63e00640 1 -- 172.21.15.60:0/2789781944 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c48003520 con 0x7f9c640c5710 2024-04-14T23:49:14.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.506+0000 7f9c6a400640 1 -- 172.21.15.60:0/2789781944 >> v1:172.21.15.186:6789/0 conn(0x7f9c640c5710 legacy=0x7f9c640c5ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:14.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.506+0000 7f9c6a400640 1 -- 172.21.15.60:0/2789781944 >> v1:172.21.15.60:6789/0 conn(0x7f9c64154c80 legacy=0x7f9c6415e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:14.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.506+0000 7f9c6a400640 1 -- 172.21.15.60:0/2789781944 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c6415ac90 con 0x7f9c640cae70 2024-04-14T23:49:14.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.508+0000 7f9c6a400640 1 --2- 172.21.15.60:0/2789781944 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9c640cae70 0x7f9c640cb2e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f9c5c009dc0 tx=0x7f9c5c02bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a948ecdc404bc66d server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:14.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.508+0000 7f9c63e00640 1 -- 172.21.15.60:0/2789781944 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c5c039070 con 0x7f9c640cae70 2024-04-14T23:49:14.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.508+0000 7f9c63e00640 1 -- 172.21.15.60:0/2789781944 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c5c02ccc0 con 0x7f9c640cae70 2024-04-14T23:49:14.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.508+0000 7f9c63e00640 1 -- 172.21.15.60:0/2789781944 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c5c034660 con 0x7f9c640cae70 2024-04-14T23:49:14.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.509+0000 7f9c6c400640 1 -- 172.21.15.60:0/2789781944 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9c640cae70 msgr2=0x7f9c640cb2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:14.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.509+0000 7f9c6c400640 1 --2- 172.21.15.60:0/2789781944 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9c640cae70 0x7f9c640cb2e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f9c5c009dc0 tx=0x7f9c5c02bf10 comp rx=0 tx=0).stop 2024-04-14T23:49:14.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.509+0000 7f9c6c400640 1 -- 172.21.15.60:0/2789781944 shutdown_connections 2024-04-14T23:49:14.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.509+0000 7f9c6c400640 1 --2- 172.21.15.60:0/2789781944 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9c640cae70 0x7f9c640cb2e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:14.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.509+0000 7f9c6c400640 1 -- 172.21.15.60:0/2789781944 >> 172.21.15.60:0/2789781944 conn(0x7f9c6406c200 msgr2=0x7f9c6406cb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:49:14.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.509+0000 7f9c6c400640 1 -- 172.21.15.60:0/2789781944 shutdown_connections 2024-04-14T23:49:14.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.509+0000 7f9c6c400640 1 -- 172.21.15.60:0/2789781944 wait complete. 2024-04-14T23:49:14.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.512+0000 7f9c6c400640 1 Processor -- start 2024-04-14T23:49:14.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.512+0000 7f9c6c400640 1 -- start start 2024-04-14T23:49:14.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.513+0000 7f9c6c400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c640c5710 0x7f9c64153470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:14.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.513+0000 7f9c6c400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9c640cae70 0x7f9c641e54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:14.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.513+0000 7f9c6c400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9c64154c80 0x7f9c641e5a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:14.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.513+0000 7f9c6c400640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f9c6415b250 con 0x7f9c640c5710 2024-04-14T23:49:14.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.513+0000 7f9c6c400640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9c6415b5c0 con 0x7f9c64154c80 2024-04-14T23:49:14.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.513+0000 7f9c6c400640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f9c640c65b0 con 0x7f9c640cae70 2024-04-14T23:49:14.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.513+0000 7f9c69a00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c640c5710 0x7f9c64153470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:14.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.513+0000 7f9c69a00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c640c5710 0x7f9c64153470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37352/0 (socket says 172.21.15.60:37352) 2024-04-14T23:49:14.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.513+0000 7f9c69a00640 1 -- 172.21.15.60:0/822894640 learned_addr learned my addr 172.21.15.60:0/822894640 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:49:14.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.513+0000 7f9c69000640 1 --2- 172.21.15.60:0/822894640 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9c640cae70 0x7f9c641e54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:14.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.513+0000 7f9c6a400640 1 --2- 172.21.15.60:0/822894640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9c64154c80 0x7f9c641e5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:14.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.513+0000 7f9c69a00640 1 -- 172.21.15.60:0/822894640 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9c640cae70 msgr2=0x7f9c641e54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:14.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.513+0000 7f9c69a00640 1 --2- 172.21.15.60:0/822894640 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9c640cae70 0x7f9c641e54f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:14.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.513+0000 7f9c69a00640 1 -- 172.21.15.60:0/822894640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9c64154c80 msgr2=0x7f9c641e5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:14.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.513+0000 7f9c69a00640 1 --2- 172.21.15.60:0/822894640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9c64154c80 0x7f9c641e5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:14.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.513+0000 7f9c69a00640 1 -- 172.21.15.60:0/822894640 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c641f06c0 con 0x7f9c640c5710 2024-04-14T23:49:14.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.513+0000 7f9c6a400640 1 --2- 172.21.15.60:0/822894640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9c64154c80 0x7f9c641e5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:49:14.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.513+0000 7f9c69000640 1 --2- 172.21.15.60:0/822894640 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9c640cae70 0x7f9c641e54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:49:14.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.513+0000 7f9c69a00640 1 --2- 172.21.15.60:0/822894640 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c640c5710 0x7f9c64153470 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f9c5400c3c0 tx=0x7f9c5400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:14.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.513+0000 7f9c61600640 1 -- 172.21.15.60:0/822894640 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c540052d0 con 0x7f9c640c5710 2024-04-14T23:49:14.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.513+0000 7f9c61600640 1 -- 172.21.15.60:0/822894640 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c54005490 con 0x7f9c640c5710 2024-04-14T23:49:14.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.513+0000 7f9c61600640 1 -- 172.21.15.60:0/822894640 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c54007260 con 0x7f9c640c5710 2024-04-14T23:49:14.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.513+0000 7f9c6c400640 1 -- 172.21.15.60:0/822894640 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c641f09a0 con 0x7f9c640c5710 2024-04-14T23:49:14.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.514+0000 7f9c6c400640 1 -- 172.21.15.60:0/822894640 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c641f0f60 con 0x7f9c640c5710 2024-04-14T23:49:14.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.515+0000 7f9c6c400640 1 -- 172.21.15.60:0/822894640 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c64076970 con 0x7f9c640c5710 2024-04-14T23:49:14.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.516+0000 7f9c61600640 1 -- 172.21.15.60:0/822894640 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f9c54007400 con 0x7f9c640c5710 2024-04-14T23:49:14.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.517+0000 7f9c61600640 1 --2- 172.21.15.60:0/822894640 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f9c4406b990 0x7f9c4406de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:14.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.517+0000 7f9c61600640 1 -- 172.21.15.60:0/822894640 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f9c54089d60 con 0x7f9c640c5710 2024-04-14T23:49:14.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.517+0000 7f9c69000640 1 --2- 172.21.15.60:0/822894640 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f9c4406b990 0x7f9c4406de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:14.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.518+0000 7f9c69000640 1 --2- 172.21.15.60:0/822894640 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f9c4406b990 0x7f9c4406de50 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9c64154210 tx=0x7f9c50007d60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:14.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.522+0000 7f9c61600640 1 -- 172.21.15.60:0/822894640 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f9c64155060 con 0x7f9c640c5710 2024-04-14T23:49:14.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.724+0000 7f9c6c400640 1 -- 172.21.15.60:0/822894640 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f9c64074760 con 0x7f9c640c5710 2024-04-14T23:49:14.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.726+0000 7f9c61600640 1 -- 172.21.15.60:0/822894640 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7f9c54058280 con 0x7f9c640c5710 2024-04-14T23:49:14.731 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2024-04-14T23:49:14.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.729+0000 7f9c6c400640 1 -- 172.21.15.60:0/822894640 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f9c4406b990 msgr2=0x7f9c4406de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:14.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.729+0000 7f9c6c400640 1 --2- 172.21.15.60:0/822894640 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f9c4406b990 0x7f9c4406de50 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9c64154210 tx=0x7f9c50007d60 comp rx=0 tx=0).stop 2024-04-14T23:49:14.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.729+0000 7f9c6c400640 1 -- 172.21.15.60:0/822894640 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c640c5710 msgr2=0x7f9c64153470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:14.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.729+0000 7f9c6c400640 1 --2- 172.21.15.60:0/822894640 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c640c5710 0x7f9c64153470 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f9c5400c3c0 tx=0x7f9c5400c880 comp rx=0 tx=0).stop 2024-04-14T23:49:14.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.729+0000 7f9c6c400640 1 -- 172.21.15.60:0/822894640 shutdown_connections 2024-04-14T23:49:14.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.729+0000 7f9c6c400640 1 --2- 172.21.15.60:0/822894640 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f9c4406b990 0x7f9c4406de50 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:14.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.729+0000 7f9c6c400640 1 --2- 172.21.15.60:0/822894640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9c64154c80 0x7f9c641e5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:14.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.729+0000 7f9c6c400640 1 --2- 172.21.15.60:0/822894640 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9c640cae70 0x7f9c641e54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:14.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.729+0000 7f9c6c400640 1 --2- 172.21.15.60:0/822894640 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c640c5710 0x7f9c64153470 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:14.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.729+0000 7f9c6c400640 1 -- 172.21.15.60:0/822894640 >> 172.21.15.60:0/822894640 conn(0x7f9c6406c200 msgr2=0x7f9c6415a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:49:14.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.729+0000 7f9c6c400640 1 -- 172.21.15.60:0/822894640 shutdown_connections 2024-04-14T23:49:14.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.729+0000 7f9c6c400640 1 -- 172.21.15.60:0/822894640 wait complete. 2024-04-14T23:49:14.754 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-14T23:49:14.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.976+0000 7ff131e00640 1 Processor -- start 2024-04-14T23:49:14.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.976+0000 7ff131e00640 1 -- start start 2024-04-14T23:49:14.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.978+0000 7ff131e00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff12c152cf0 0x7ff12c15c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:14.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.978+0000 7ff131e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff12c0c63c0 0x7ff12c15c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:14.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.978+0000 7ff131e00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff12c1530d0 con 0x7ff12c159450 2024-04-14T23:49:14.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.978+0000 7ff131e00640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7ff12c0751f0 con 0x7ff12c152cf0 2024-04-14T23:49:14.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.978+0000 7ff131e00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7ff12c0c8370 con 0x7ff12c0c63c0 2024-04-14T23:49:14.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.978+0000 7ff12a000640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff12c0c63c0 0x7ff12c15c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:14.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.978+0000 7ff12aa00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff12c152cf0 0x7ff12c15c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:14.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.978+0000 7ff12a000640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff12c0c63c0 0x7ff12c15c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.60:32932/0 (socket says 172.21.15.60:32932) 2024-04-14T23:49:14.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.978+0000 7ff12a000640 1 -- 172.21.15.60:0/2756545367 learned_addr learned my addr 172.21.15.60:0/2756545367 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:49:14.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.979+0000 7ff12a000640 1 -- 172.21.15.60:0/2756545367 >> v2:172.21.15.186:3300/0 conn(0x7ff12c152cf0 msgr2=0x7ff12c15c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:49:14.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.979+0000 7ff12a000640 1 --2- 172.21.15.60:0/2756545367 >> v2:172.21.15.186:3300/0 conn(0x7ff12c152cf0 0x7ff12c15c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:14.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.979+0000 7ff12a000640 1 -- 172.21.15.60:0/2756545367 >> v1:172.21.15.60:6789/0 conn(0x7ff12c159450 legacy=0x7ff12c0c3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:14.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.979+0000 7ff12a000640 1 -- 172.21.15.60:0/2756545367 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff12c0c3810 con 0x7ff12c0c63c0 2024-04-14T23:49:14.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.979+0000 7ff12aa00640 1 --2- 172.21.15.60:0/2756545367 >> v2:172.21.15.186:3300/0 conn(0x7ff12c152cf0 0x7ff12c15c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:49:14.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.980+0000 7ff12a000640 1 --2- 172.21.15.60:0/2756545367 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff12c0c63c0 0x7ff12c15c7e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff118009870 tx=0x7ff11802ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c50e0a57257fbd0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:14.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.980+0000 7ff129600640 1 -- 172.21.15.60:0/2756545367 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff11802f830 con 0x7ff12c0c63c0 2024-04-14T23:49:14.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.981+0000 7ff129600640 1 -- 172.21.15.60:0/2756545367 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff11802f9d0 con 0x7ff12c0c63c0 2024-04-14T23:49:14.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.981+0000 7ff129600640 1 -- 172.21.15.60:0/2756545367 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff11802fda0 con 0x7ff12c0c63c0 2024-04-14T23:49:14.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.981+0000 7ff131e00640 1 -- 172.21.15.60:0/2756545367 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff12c0c63c0 msgr2=0x7ff12c15c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:14.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.981+0000 7ff131e00640 1 --2- 172.21.15.60:0/2756545367 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff12c0c63c0 0x7ff12c15c7e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff118009870 tx=0x7ff11802ba10 comp rx=0 tx=0).stop 2024-04-14T23:49:14.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.981+0000 7ff131e00640 1 -- 172.21.15.60:0/2756545367 shutdown_connections 2024-04-14T23:49:14.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.981+0000 7ff131e00640 1 --2- 172.21.15.60:0/2756545367 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff12c0c63c0 0x7ff12c15c7e0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:14.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.981+0000 7ff131e00640 1 --2- 172.21.15.60:0/2756545367 >> v2:172.21.15.186:3300/0 conn(0x7ff12c152cf0 0x7ff12c15c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:14.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.981+0000 7ff131e00640 1 -- 172.21.15.60:0/2756545367 >> 172.21.15.60:0/2756545367 conn(0x7ff12c0cc170 msgr2=0x7ff12c0cc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:49:14.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.981+0000 7ff131e00640 1 -- 172.21.15.60:0/2756545367 shutdown_connections 2024-04-14T23:49:14.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.981+0000 7ff131e00640 1 -- 172.21.15.60:0/2756545367 wait complete. 2024-04-14T23:49:14.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.983+0000 7ff131e00640 1 Processor -- start 2024-04-14T23:49:14.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.983+0000 7ff131e00640 1 -- start start 2024-04-14T23:49:14.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.984+0000 7ff131e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff12c0c63c0 0x7ff12c151d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:14.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.984+0000 7ff131e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff12c152cf0 0x7ff12c1f28d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:14.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.984+0000 7ff131e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff12c159450 0x7ff12c1f2e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:14.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.984+0000 7ff131e00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff12c0c88f0 con 0x7ff12c0c63c0 2024-04-14T23:49:14.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.984+0000 7ff131e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff12c0c8610 con 0x7ff12c152cf0 2024-04-14T23:49:14.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.984+0000 7ff131e00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7ff12c0c4690 con 0x7ff12c159450 2024-04-14T23:49:14.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.984+0000 7ff12aa00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff12c0c63c0 0x7ff12c151d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:14.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.984+0000 7ff12aa00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff12c0c63c0 0x7ff12c151d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37358/0 (socket says 172.21.15.60:37358) 2024-04-14T23:49:14.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.984+0000 7ff12a000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff12c152cf0 0x7ff12c1f28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:14.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.984+0000 7ff12b400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff12c159450 0x7ff12c1f2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:14.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.984+0000 7ff12aa00640 1 -- 172.21.15.60:0/1858867371 learned_addr learned my addr 172.21.15.60:0/1858867371 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:49:14.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.985+0000 7ff12b400640 1 -- 172.21.15.60:0/1858867371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff12c152cf0 msgr2=0x7ff12c1f28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:14.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.985+0000 7ff12b400640 1 --2- 172.21.15.60:0/1858867371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff12c152cf0 0x7ff12c1f28d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:14.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.985+0000 7ff12b400640 1 -- 172.21.15.60:0/1858867371 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff12c0c63c0 msgr2=0x7ff12c151d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:14.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.985+0000 7ff12b400640 1 --2- 172.21.15.60:0/1858867371 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff12c0c63c0 0x7ff12c151d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:14.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.985+0000 7ff12b400640 1 -- 172.21.15.60:0/1858867371 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff12c1f4800 con 0x7ff12c159450 2024-04-14T23:49:14.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.985+0000 7ff12a000640 1 --2- 172.21.15.60:0/1858867371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff12c152cf0 0x7ff12c1f28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:49:14.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.985+0000 7ff12b400640 1 --2- 172.21.15.60:0/1858867371 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff12c159450 0x7ff12c1f2e10 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff12c1598e0 tx=0x7ff12400cb90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:14.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.985+0000 7ff123400640 1 -- 172.21.15.60:0/1858867371 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff124003b70 con 0x7ff12c159450 2024-04-14T23:49:14.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.985+0000 7ff123400640 1 -- 172.21.15.60:0/1858867371 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff124003d10 con 0x7ff12c159450 2024-04-14T23:49:14.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.985+0000 7ff123400640 1 -- 172.21.15.60:0/1858867371 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff124005310 con 0x7ff12c159450 2024-04-14T23:49:14.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.986+0000 7ff131e00640 1 -- 172.21.15.60:0/1858867371 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff12c1f4aa0 con 0x7ff12c159450 2024-04-14T23:49:14.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.986+0000 7ff131e00640 1 -- 172.21.15.60:0/1858867371 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff12c1f5060 con 0x7ff12c159450 2024-04-14T23:49:14.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.988+0000 7ff131e00640 1 -- 172.21.15.60:0/1858867371 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff12c076970 con 0x7ff12c159450 2024-04-14T23:49:14.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.988+0000 7ff123400640 1 -- 172.21.15.60:0/1858867371 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7ff1240054b0 con 0x7ff12c159450 2024-04-14T23:49:14.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.990+0000 7ff123400640 1 --2- 172.21.15.60:0/1858867371 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff10c06b990 0x7ff10c06de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:14.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.990+0000 7ff123400640 1 -- 172.21.15.60:0/1858867371 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ff12408a500 con 0x7ff12c159450 2024-04-14T23:49:14.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.990+0000 7ff12aa00640 1 --2- 172.21.15.60:0/1858867371 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff10c06b990 0x7ff10c06de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:14.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.990+0000 7ff12aa00640 1 --2- 172.21.15.60:0/1858867371 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff10c06b990 0x7ff10c06de50 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff11c009560 tx=0x7ff11c009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:14.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:14.994+0000 7ff123400640 1 -- 172.21.15.60:0/1858867371 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff124058a20 con 0x7ff12c159450 2024-04-14T23:49:15.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.200+0000 7ff131e00640 1 -- 172.21.15.60:0/1858867371 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff12c06ecc0 con 0x7ff12c159450 2024-04-14T23:49:15.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.202+0000 7ff123400640 1 -- 172.21.15.60:0/1858867371 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7ff12405cac0 con 0x7ff12c159450 2024-04-14T23:49:15.206 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2024-04-14T23:49:15.209 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.204+0000 7ff131e00640 1 -- 172.21.15.60:0/1858867371 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff10c06b990 msgr2=0x7ff10c06de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:15.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.204+0000 7ff131e00640 1 --2- 172.21.15.60:0/1858867371 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff10c06b990 0x7ff10c06de50 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff11c009560 tx=0x7ff11c009190 comp rx=0 tx=0).stop 2024-04-14T23:49:15.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.204+0000 7ff131e00640 1 -- 172.21.15.60:0/1858867371 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff12c159450 msgr2=0x7ff12c1f2e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:15.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.204+0000 7ff131e00640 1 --2- 172.21.15.60:0/1858867371 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff12c159450 0x7ff12c1f2e10 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff12c1598e0 tx=0x7ff12400cb90 comp rx=0 tx=0).stop 2024-04-14T23:49:15.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.204+0000 7ff131e00640 1 -- 172.21.15.60:0/1858867371 shutdown_connections 2024-04-14T23:49:15.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.204+0000 7ff131e00640 1 --2- 172.21.15.60:0/1858867371 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff10c06b990 0x7ff10c06de50 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:15.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.204+0000 7ff131e00640 1 --2- 172.21.15.60:0/1858867371 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff12c159450 0x7ff12c1f2e10 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:15.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.204+0000 7ff131e00640 1 --2- 172.21.15.60:0/1858867371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff12c152cf0 0x7ff12c1f28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:15.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.204+0000 7ff131e00640 1 --2- 172.21.15.60:0/1858867371 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff12c0c63c0 0x7ff12c151d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:15.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.204+0000 7ff131e00640 1 -- 172.21.15.60:0/1858867371 >> 172.21.15.60:0/1858867371 conn(0x7ff12c0cc170 msgr2=0x7ff12c15c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:49:15.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.204+0000 7ff131e00640 1 -- 172.21.15.60:0/1858867371 shutdown_connections 2024-04-14T23:49:15.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.204+0000 7ff131e00640 1 -- 172.21.15.60:0/1858867371 wait complete. 2024-04-14T23:49:15.232 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:49:15.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.447+0000 7f7089c00640 1 Processor -- start 2024-04-14T23:49:15.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.447+0000 7f7089c00640 1 -- start start 2024-04-14T23:49:15.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.448+0000 7f7089c00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f70840cd540 0x7f708415ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:15.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.448+0000 7f7089c00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f70840cb750 0x7f7084154b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:15.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.448+0000 7f7089c00640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70840b3f50 con 0x7f70840cd160 2024-04-14T23:49:15.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.448+0000 7f7089c00640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f708415b370 con 0x7f70840cb750 2024-04-14T23:49:15.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.448+0000 7f7089c00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f70840b39b0 con 0x7f70840cd540 2024-04-14T23:49:15.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.450+0000 7f7083400640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f70840cb750 0x7f7084154b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:15.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.450+0000 7f7083400640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f70840cb750 0x7f7084154b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37370/0 (socket says 172.21.15.60:37370) 2024-04-14T23:49:15.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.450+0000 7f7082000640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f70840cd540 0x7f708415ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:15.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.450+0000 7f7083400640 1 -- 172.21.15.60:0/1706638170 learned_addr learned my addr 172.21.15.60:0/1706638170 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:49:15.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.450+0000 7f7082000640 1 -- 172.21.15.60:0/1706638170 >> v2:172.21.15.60:3300/0 conn(0x7f70840cb750 msgr2=0x7f7084154b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:49:15.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.450+0000 7f7082000640 1 --2- 172.21.15.60:0/1706638170 >> v2:172.21.15.60:3300/0 conn(0x7f70840cb750 0x7f7084154b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:15.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.450+0000 7f7082000640 1 -- 172.21.15.60:0/1706638170 >> v1:172.21.15.186:6789/0 conn(0x7f70840cd160 legacy=0x7f70840b3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:15.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.450+0000 7f7082000640 1 -- 172.21.15.60:0/1706638170 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70840b3c70 con 0x7f70840cd540 2024-04-14T23:49:15.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.450+0000 7f7081600640 1 -- 172.21.15.60:0/1706638170 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2434157254 0 0) 0x7f70840b3f50 con 0x7f70840cd160 2024-04-14T23:49:15.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.451+0000 7f7082000640 1 --2- 172.21.15.60:0/1706638170 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f70840cd540 0x7f708415ae30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f706c0097a0 tx=0x7f706c02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5348d797d96275f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:15.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.451+0000 7f7081600640 1 -- 172.21.15.60:0/1706638170 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f706c035020 con 0x7f70840cd540 2024-04-14T23:49:15.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.451+0000 7f7081600640 1 -- 172.21.15.60:0/1706638170 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f706c031810 con 0x7f70840cd540 2024-04-14T23:49:15.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.451+0000 7f7081600640 1 -- 172.21.15.60:0/1706638170 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f706c031be0 con 0x7f70840cd540 2024-04-14T23:49:15.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.452+0000 7f7089c00640 1 -- 172.21.15.60:0/1706638170 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f70840cd540 msgr2=0x7f708415ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:15.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.452+0000 7f7089c00640 1 --2- 172.21.15.60:0/1706638170 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f70840cd540 0x7f708415ae30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f706c0097a0 tx=0x7f706c02b8b0 comp rx=0 tx=0).stop 2024-04-14T23:49:15.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.452+0000 7f7089c00640 1 -- 172.21.15.60:0/1706638170 shutdown_connections 2024-04-14T23:49:15.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.452+0000 7f7089c00640 1 --2- 172.21.15.60:0/1706638170 >> v2:172.21.15.60:3300/0 conn(0x7f70840cb750 0x7f7084154b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:15.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.452+0000 7f7089c00640 1 --2- 172.21.15.60:0/1706638170 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f70840cd540 0x7f708415ae30 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:15.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.452+0000 7f7089c00640 1 -- 172.21.15.60:0/1706638170 >> 172.21.15.60:0/1706638170 conn(0x7f7084066f10 msgr2=0x7f70840672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:49:15.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.452+0000 7f7089c00640 1 -- 172.21.15.60:0/1706638170 shutdown_connections 2024-04-14T23:49:15.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.452+0000 7f7089c00640 1 -- 172.21.15.60:0/1706638170 wait complete. 2024-04-14T23:49:15.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.454+0000 7f7089c00640 1 Processor -- start 2024-04-14T23:49:15.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.454+0000 7f7089c00640 1 -- start start 2024-04-14T23:49:15.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.455+0000 7f7089c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70840cb750 0x7f708414bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:15.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.455+0000 7f7089c00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f70840cd160 0x7f70841f0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:15.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.455+0000 7f7089c00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f70840cd540 0x7f7084159c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:15.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.455+0000 7f7089c00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f708415ea70 con 0x7f70840cd160 2024-04-14T23:49:15.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.455+0000 7f7089c00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f708415e790 con 0x7f70840cb750 2024-04-14T23:49:15.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.455+0000 7f7089c00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f70840ccb20 con 0x7f70840cd540 2024-04-14T23:49:15.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.455+0000 7f7082000640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f70840cd160 0x7f70841f0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:15.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.455+0000 7f7082a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70840cb750 0x7f708414bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:15.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.455+0000 7f7082000640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f70840cd160 0x7f70841f0dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37372/0 (socket says 172.21.15.60:37372) 2024-04-14T23:49:15.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.455+0000 7f7082000640 1 -- 172.21.15.60:0/4250110781 learned_addr learned my addr 172.21.15.60:0/4250110781 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:49:15.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.455+0000 7f7083400640 1 --2- 172.21.15.60:0/4250110781 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f70840cd540 0x7f7084159c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:15.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.456+0000 7f7082000640 1 -- 172.21.15.60:0/4250110781 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f70840cd540 msgr2=0x7f7084159c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:15.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.456+0000 7f7082000640 1 --2- 172.21.15.60:0/4250110781 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f70840cd540 0x7f7084159c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:15.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.456+0000 7f7082000640 1 -- 172.21.15.60:0/4250110781 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70840cb750 msgr2=0x7f708414bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:15.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.456+0000 7f7082000640 1 --2- 172.21.15.60:0/4250110781 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70840cb750 0x7f708414bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:15.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.456+0000 7f7082000640 1 -- 172.21.15.60:0/4250110781 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70840ca2e0 con 0x7f70840cd160 2024-04-14T23:49:15.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.456+0000 7f7082000640 1 --2- 172.21.15.60:0/4250110781 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f70840cd160 0x7f70841f0dc0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f706c0098c0 tx=0x7f706c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:15.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.457+0000 7f707aa00640 1 -- 172.21.15.60:0/4250110781 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f706c041070 con 0x7f70840cd160 2024-04-14T23:49:15.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.457+0000 7f707aa00640 1 -- 172.21.15.60:0/4250110781 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f706c03ad00 con 0x7f70840cd160 2024-04-14T23:49:15.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.457+0000 7f707aa00640 1 -- 172.21.15.60:0/4250110781 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f706c035040 con 0x7f70840cd160 2024-04-14T23:49:15.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.457+0000 7f7089c00640 1 -- 172.21.15.60:0/4250110781 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70840d0130 con 0x7f70840cd160 2024-04-14T23:49:15.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.457+0000 7f7089c00640 1 -- 172.21.15.60:0/4250110781 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70841f48c0 con 0x7f70840cd160 2024-04-14T23:49:15.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.458+0000 7f7089c00640 1 -- 172.21.15.60:0/4250110781 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7084068bc0 con 0x7f70840cd160 2024-04-14T23:49:15.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.459+0000 7f707aa00640 1 -- 172.21.15.60:0/4250110781 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f706c038020 con 0x7f70840cd160 2024-04-14T23:49:15.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.460+0000 7f707aa00640 1 --2- 172.21.15.60:0/4250110781 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f706006b990 0x7f706006de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:15.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.460+0000 7f707aa00640 1 -- 172.21.15.60:0/4250110781 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f706c0ac450 con 0x7f70840cd160 2024-04-14T23:49:15.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.461+0000 7f7082a00640 1 --2- 172.21.15.60:0/4250110781 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f706006b990 0x7f706006de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:15.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.461+0000 7f7082a00640 1 --2- 172.21.15.60:0/4250110781 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f706006b990 0x7f706006de50 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7074000f60 tx=0x7f7074009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:15.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.465+0000 7f707aa00640 1 -- 172.21.15.60:0/4250110781 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7084068bc0 con 0x7f70840cd160 2024-04-14T23:49:15.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.706+0000 7f7089c00640 1 -- 172.21.15.60:0/4250110781 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f70841e7fc0 con 0x7f70840cd160 2024-04-14T23:49:15.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.707+0000 7f707aa00640 1 -- 172.21.15.60:0/4250110781 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6827 (secure 0 0 0) 0x7f706c07a970 con 0x7f70840cd160 2024-04-14T23:49:15.712 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:49:15.712 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/78303888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":78303888},{"type":"v1","addr":"172.21.15.60:6833","nonce":78303888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:48:49.256965+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.60:6835/2677133122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2677133122},{"type":"v1","addr":"172.21.15.60:6835","nonce":2677133122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4279,"qdb_cluster":[4279]},"id":1}]} 2024-04-14T23:49:15.713 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 8 2024-04-14T23:49:15.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.710+0000 7f7089c00640 1 -- 172.21.15.60:0/4250110781 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f706006b990 msgr2=0x7f706006de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:15.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.710+0000 7f7089c00640 1 --2- 172.21.15.60:0/4250110781 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f706006b990 0x7f706006de50 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7074000f60 tx=0x7f7074009040 comp rx=0 tx=0).stop 2024-04-14T23:49:15.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.710+0000 7f7089c00640 1 -- 172.21.15.60:0/4250110781 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f70840cd160 msgr2=0x7f70841f0dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:15.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.710+0000 7f7089c00640 1 --2- 172.21.15.60:0/4250110781 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f70840cd160 0x7f70841f0dc0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f706c0098c0 tx=0x7f706c0046c0 comp rx=0 tx=0).stop 2024-04-14T23:49:15.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.710+0000 7f7089c00640 1 -- 172.21.15.60:0/4250110781 shutdown_connections 2024-04-14T23:49:15.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.710+0000 7f7089c00640 1 --2- 172.21.15.60:0/4250110781 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f706006b990 0x7f706006de50 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:15.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.710+0000 7f7089c00640 1 --2- 172.21.15.60:0/4250110781 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f70840cd540 0x7f7084159c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:15.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.710+0000 7f7089c00640 1 --2- 172.21.15.60:0/4250110781 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f70840cd160 0x7f70841f0dc0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:15.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.710+0000 7f7089c00640 1 --2- 172.21.15.60:0/4250110781 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70840cb750 0x7f708414bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:15.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.710+0000 7f7089c00640 1 -- 172.21.15.60:0/4250110781 >> 172.21.15.60:0/4250110781 conn(0x7f7084066f10 msgr2=0x7f708415c1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:49:15.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.710+0000 7f7089c00640 1 -- 172.21.15.60:0/4250110781 shutdown_connections 2024-04-14T23:49:15.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.710+0000 7f7089c00640 1 -- 172.21.15.60:0/4250110781 wait complete. 2024-04-14T23:49:15.735 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-14T23:49:15.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.946+0000 7f69a2200640 1 Processor -- start 2024-04-14T23:49:15.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.946+0000 7f69a2200640 1 -- start start 2024-04-14T23:49:15.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.948+0000 7f69a2200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f699c073320 0x7f699c0c6950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:15.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.948+0000 7f69a2200640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f699c152fb0 con 0x7f699c0b1e00 2024-04-14T23:49:15.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.948+0000 7f69a2200640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f699c159310 con 0x7f699c0c3d90 2024-04-14T23:49:15.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.948+0000 7f69a2200640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f699c0c6540 con 0x7f699c073320 2024-04-14T23:49:15.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.948+0000 7f69a0e00640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f699c0b1e00 0x7f699c0c7fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:51184/0 (socket says 172.21.15.60:51184) 2024-04-14T23:49:15.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.948+0000 7f69a0e00640 1 -- 172.21.15.60:0/1389755663 learned_addr learned my addr 172.21.15.60:0/1389755663 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:49:15.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.949+0000 7f699be00640 1 --2- 172.21.15.60:0/1389755663 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f699c073320 0x7f699c0c6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:15.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.949+0000 7f699b400640 1 -- 172.21.15.60:0/1389755663 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2338606009 0 0) 0x7f699c152fb0 con 0x7f699c0b1e00 2024-04-14T23:49:15.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.949+0000 7f699b400640 1 -- 172.21.15.60:0/1389755663 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6984003520 con 0x7f699c0b1e00 2024-04-14T23:49:15.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.949+0000 7f699be00640 1 -- 172.21.15.60:0/1389755663 >> v1:172.21.15.186:6789/0 conn(0x7f699c0c3d90 legacy=0x7f699c0c41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:15.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.949+0000 7f699b400640 1 -- 172.21.15.60:0/1389755663 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3235829547 0 0) 0x7f699c159310 con 0x7f699c0c3d90 2024-04-14T23:49:15.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.949+0000 7f699be00640 1 -- 172.21.15.60:0/1389755663 >> v1:172.21.15.60:6789/0 conn(0x7f699c0b1e00 legacy=0x7f699c0c7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:15.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.949+0000 7f699be00640 1 -- 172.21.15.60:0/1389755663 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f699c0c38b0 con 0x7f699c073320 2024-04-14T23:49:15.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.950+0000 7f699be00640 1 --2- 172.21.15.60:0/1389755663 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f699c073320 0x7f699c0c6950 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f699c0c3a80 tx=0x7f698802b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0b52fa12bc2e02b server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:15.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.951+0000 7f699b400640 1 -- 172.21.15.60:0/1389755663 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6988034070 con 0x7f699c073320 2024-04-14T23:49:15.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.951+0000 7f699b400640 1 -- 172.21.15.60:0/1389755663 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6988030880 con 0x7f699c073320 2024-04-14T23:49:15.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.951+0000 7f699b400640 1 -- 172.21.15.60:0/1389755663 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6988030c50 con 0x7f699c073320 2024-04-14T23:49:15.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.951+0000 7f69a2200640 1 -- 172.21.15.60:0/1389755663 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f699c073320 msgr2=0x7f699c0c6950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:15.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.951+0000 7f69a2200640 1 --2- 172.21.15.60:0/1389755663 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f699c073320 0x7f699c0c6950 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f699c0c3a80 tx=0x7f698802b9c0 comp rx=0 tx=0).stop 2024-04-14T23:49:15.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.951+0000 7f69a2200640 1 -- 172.21.15.60:0/1389755663 shutdown_connections 2024-04-14T23:49:15.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.951+0000 7f69a2200640 1 --2- 172.21.15.60:0/1389755663 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f699c073320 0x7f699c0c6950 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:15.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.951+0000 7f69a2200640 1 -- 172.21.15.60:0/1389755663 >> 172.21.15.60:0/1389755663 conn(0x7f699c0c8b50 msgr2=0x7f699c0c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:49:15.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.952+0000 7f69a2200640 1 -- 172.21.15.60:0/1389755663 shutdown_connections 2024-04-14T23:49:15.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.952+0000 7f69a2200640 1 -- 172.21.15.60:0/1389755663 wait complete. 2024-04-14T23:49:15.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.953+0000 7f69a2200640 1 Processor -- start 2024-04-14T23:49:15.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.953+0000 7f69a2200640 1 -- start start 2024-04-14T23:49:15.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.954+0000 7f69a2200640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f699c073320 0x7f699c1f47a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:15.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.954+0000 7f69a2200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f699c0b1e00 0x7f699c1f1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:15.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.954+0000 7f69a2200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f699c0c3d90 0x7f699c0c7f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:15.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.955+0000 7f69a2200640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f699c159890 con 0x7f699c073320 2024-04-14T23:49:15.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.955+0000 7f69a2200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f699c0c5f80 con 0x7f699c0b1e00 2024-04-14T23:49:15.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.955+0000 7f69a2200640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f699c0c6260 con 0x7f699c0c3d90 2024-04-14T23:49:15.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.955+0000 7f69a0e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f699c073320 0x7f699c1f47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:15.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.955+0000 7f69a0e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f699c073320 0x7f699c1f47a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37380/0 (socket says 172.21.15.60:37380) 2024-04-14T23:49:15.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.955+0000 7f69a0e00640 1 -- 172.21.15.60:0/1990191780 learned_addr learned my addr 172.21.15.60:0/1990191780 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:49:15.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.955+0000 7f699be00640 1 --2- 172.21.15.60:0/1990191780 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f699c0b1e00 0x7f699c1f1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:15.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.955+0000 7f69a1800640 1 --2- 172.21.15.60:0/1990191780 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f699c0c3d90 0x7f699c0c7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:15.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.955+0000 7f69a0e00640 1 -- 172.21.15.60:0/1990191780 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f699c0c3d90 msgr2=0x7f699c0c7f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:15.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.955+0000 7f69a0e00640 1 --2- 172.21.15.60:0/1990191780 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f699c0c3d90 0x7f699c0c7f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:15.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.955+0000 7f69a0e00640 1 -- 172.21.15.60:0/1990191780 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f699c0b1e00 msgr2=0x7f699c1f1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:15.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.955+0000 7f69a0e00640 1 --2- 172.21.15.60:0/1990191780 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f699c0b1e00 0x7f699c1f1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:15.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.955+0000 7f69a0e00640 1 -- 172.21.15.60:0/1990191780 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f699c1f4ce0 con 0x7f699c073320 2024-04-14T23:49:15.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.955+0000 7f69a1800640 1 --2- 172.21.15.60:0/1990191780 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f699c0c3d90 0x7f699c0c7f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-14T23:49:15.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.956+0000 7f69a0e00640 1 --2- 172.21.15.60:0/1990191780 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f699c073320 0x7f699c1f47a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f699000c460 tx=0x7f699000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:15.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.956+0000 7f6998c00640 1 -- 172.21.15.60:0/1990191780 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6990017070 con 0x7f699c073320 2024-04-14T23:49:15.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.956+0000 7f6998c00640 1 -- 172.21.15.60:0/1990191780 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6990005ce0 con 0x7f699c073320 2024-04-14T23:49:15.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.956+0000 7f6998c00640 1 -- 172.21.15.60:0/1990191780 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6990007290 con 0x7f699c073320 2024-04-14T23:49:15.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.956+0000 7f69a2200640 1 -- 172.21.15.60:0/1990191780 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f699c15c890 con 0x7f699c073320 2024-04-14T23:49:15.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.956+0000 7f69a2200640 1 -- 172.21.15.60:0/1990191780 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f699c0af6e0 con 0x7f699c073320 2024-04-14T23:49:15.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.957+0000 7f69a2200640 1 -- 172.21.15.60:0/1990191780 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f699c04f540 con 0x7f699c073320 2024-04-14T23:49:15.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.958+0000 7f6998c00640 1 -- 172.21.15.60:0/1990191780 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f699001a070 con 0x7f699c073320 2024-04-14T23:49:15.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.959+0000 7f6998c00640 1 --2- 172.21.15.60:0/1990191780 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f697c06b9c0 0x7f697c06de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:49:15.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.959+0000 7f6998c00640 1 -- 172.21.15.60:0/1990191780 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f6990089c20 con 0x7f699c073320 2024-04-14T23:49:15.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.959+0000 7f699be00640 1 --2- 172.21.15.60:0/1990191780 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f697c06b9c0 0x7f697c06de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:49:15.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.960+0000 7f699be00640 1 --2- 172.21.15.60:0/1990191780 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f697c06b9c0 0x7f697c06de80 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6988036000 tx=0x7f6988008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:49:15.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:15.963+0000 7f6998c00640 1 -- 172.21.15.60:0/1990191780 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f69900580b0 con 0x7f699c073320 2024-04-14T23:49:16.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:16.165+0000 7f69a2200640 1 -- 172.21.15.60:0/1990191780 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f699c0b21e0 con 0x7f699c073320 2024-04-14T23:49:16.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:16.167+0000 7f6998c00640 1 -- 172.21.15.60:0/1990191780 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16605 (secure 0 0 0) 0x7f699005c150 con 0x7f699c073320 2024-04-14T23:49:16.172 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:49:16.172 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":21,"fsid":"5847db56-a76d-4486-b563-d83f1186af92","created":"2024-04-14T23:48:23.900003+0000","modified":"2024-04-14T23:48:44.219352+0000","last_up_change":"2024-04-14T23:48:34.203573+0000","last_in_change":"2024-04-14T23:48:26.345499+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-14T23:48:36.247372+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-14T23:48:40.274195+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-14T23:48:41.722183+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b2b2b755-7f42-4f9b-af9a-a178f7cc5a52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6817","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6818","nonce":2536528895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6822","nonce":2536528895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6827","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6828","nonce":2536528895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6825","nonce":2536528895}]},"public_addr":"172.21.15.60:6818/2536528895","cluster_addr":"172.21.15.60:6822/2536528895","heartbeat_back_addr":"172.21.15.60:6828/2536528895","heartbeat_front_addr":"172.21.15.60:6825/2536528895","state":["exists","up"]},{"osd":1,"uuid":"321418cd-5ab7-41a5-b9c8-b3fc12f68ebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6801","nonce":3115695676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6803","nonce":3115695676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6807","nonce":3115695676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6805","nonce":3115695676}]},"public_addr":"172.21.15.60:6801/3115695676","cluster_addr":"172.21.15.60:6803/3115695676","heartbeat_back_addr":"172.21.15.60:6807/3115695676","heartbeat_front_addr":"172.21.15.60:6805/3115695676","state":["exists","up"]},{"osd":2,"uuid":"07874e9f-3a38-41dc-be9f-2b75e84e53d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6819","nonce":3475168698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6824","nonce":3475168698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6831","nonce":3475168698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6829","nonce":3475168698}]},"public_addr":"172.21.15.60:6819/3475168698","cluster_addr":"172.21.15.60:6824/3475168698","heartbeat_back_addr":"172.21.15.60:6831/3475168698","heartbeat_front_addr":"172.21.15.60:6829/3475168698","state":["exists","up"]},{"osd":3,"uuid":"7e4ef6c9-b17a-47c6-a320-f5d16a6fe05e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6809","nonce":1730813674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6811","nonce":1730813674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6815","nonce":1730813674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6813","nonce":1730813674}]},"public_addr":"172.21.15.60:6809/1730813674","cluster_addr":"172.21.15.60:6811/1730813674","heartbeat_back_addr":"172.21.15.60:6815/1730813674","heartbeat_front_addr":"172.21.15.60:6813/1730813674","state":["exists","up"]},{"osd":4,"uuid":"759a3efc-dade-44c5-b57a-1623499aba50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6805","nonce":1464587038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6810","nonce":1464587038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6819","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6822","nonce":1464587038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6813","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6816","nonce":1464587038}]},"public_addr":"172.21.15.186:6805/1464587038","cluster_addr":"172.21.15.186:6810/1464587038","heartbeat_back_addr":"172.21.15.186:6822/1464587038","heartbeat_front_addr":"172.21.15.186:6816/1464587038","state":["exists","up"]},{"osd":5,"uuid":"24cd58c1-0cf1-4077-ab33-84c39c6907ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6801","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6803","nonce":1697098067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6809","nonce":1697098067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6820","nonce":1697098067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6815","nonce":1697098067}]},"public_addr":"172.21.15.186:6803/1697098067","cluster_addr":"172.21.15.186:6809/1697098067","heartbeat_back_addr":"172.21.15.186:6820/1697098067","heartbeat_front_addr":"172.21.15.186:6815/1697098067","state":["exists","up"]},{"osd":6,"uuid":"6add38f1-65e9-4b0d-927a-4ab77f5e2323","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6804","nonce":1837108559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6807","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6811","nonce":1837108559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6823","nonce":1837108559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6817","nonce":1837108559}]},"public_addr":"172.21.15.186:6804/1837108559","cluster_addr":"172.21.15.186:6811/1837108559","heartbeat_back_addr":"172.21.15.186:6823/1837108559","heartbeat_front_addr":"172.21.15.186:6817/1837108559","state":["exists","up"]},{"osd":7,"uuid":"a2326b6f-7601-4dda-a456-67c443ea5929","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6825","nonce":842537875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6827","nonce":842537875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6831","nonce":842537875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6829","nonce":842537875}]},"public_addr":"172.21.15.186:6825/842537875","cluster_addr":"172.21.15.186:6827/842537875","heartbeat_back_addr":"172.21.15.186:6831/842537875","heartbeat_front_addr":"172.21.15.186:6829/842537875","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.975583+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.976806+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.722725+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.763705+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.502140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.528972+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.517961+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.513350+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-14T23:49:16.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:16.169+0000 7f69a2200640 1 -- 172.21.15.60:0/1990191780 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f697c06b9c0 msgr2=0x7f697c06de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:16.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:16.169+0000 7f69a2200640 1 --2- 172.21.15.60:0/1990191780 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f697c06b9c0 0x7f697c06de80 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6988036000 tx=0x7f6988008030 comp rx=0 tx=0).stop 2024-04-14T23:49:16.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:16.169+0000 7f69a2200640 1 -- 172.21.15.60:0/1990191780 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f699c073320 msgr2=0x7f699c1f47a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:49:16.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:16.169+0000 7f69a2200640 1 --2- 172.21.15.60:0/1990191780 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f699c073320 0x7f699c1f47a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f699000c460 tx=0x7f699000c920 comp rx=0 tx=0).stop 2024-04-14T23:49:16.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:16.169+0000 7f69a2200640 1 -- 172.21.15.60:0/1990191780 shutdown_connections 2024-04-14T23:49:16.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:16.169+0000 7f69a2200640 1 --2- 172.21.15.60:0/1990191780 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f697c06b9c0 0x7f697c06de80 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:16.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:16.170+0000 7f69a2200640 1 --2- 172.21.15.60:0/1990191780 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f699c0c3d90 0x7f699c0c7f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:16.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:16.170+0000 7f69a2200640 1 --2- 172.21.15.60:0/1990191780 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f699c0b1e00 0x7f699c1f1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:16.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:16.170+0000 7f69a2200640 1 --2- 172.21.15.60:0/1990191780 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f699c073320 0x7f699c1f47a0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:49:16.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:16.170+0000 7f69a2200640 1 -- 172.21.15.60:0/1990191780 >> 172.21.15.60:0/1990191780 conn(0x7f699c0c8b50 msgr2=0x7f699c0c41e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:49:16.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:16.170+0000 7f69a2200640 1 -- 172.21.15.60:0/1990191780 shutdown_connections 2024-04-14T23:49:16.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:49:16.170+0000 7f69a2200640 1 -- 172.21.15.60:0/1990191780 wait complete. 2024-04-14T23:49:16.195 DEBUG:tasks.mds_thrash:Yielding 2024-04-14T23:49:16.195 INFO:teuthology.run_tasks:Running task workunit... 2024-04-14T23:49:16.207 INFO:tasks.workunit:Pulling workunits from ref a9a752df26c63acad72e1b3569fd79a515ca0765 2024-04-14T23:49:16.208 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-14T23:49:16.208 INFO:tasks.workunit:timeout=6h 2024-04-14T23:49:16.208 INFO:tasks.workunit:cleanup=True 2024-04-14T23:49:16.208 DEBUG:teuthology.orchestra.run.smithi060:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-14T23:49:16.209 INFO:tasks.mds_thrash.fs.[cephfs]:starting mds_do_thrash for fs cephfs 2024-04-14T23:49:16.209 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 86 secs before thrashing 2024-04-14T23:49:16.265 INFO:teuthology.orchestra.run.smithi060.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-04-14T23:49:16.265 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-04-14T23:49:16.265 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 2bh/43d Inode: 1 Links: 2 2024-04-14T23:49:16.265 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-14T23:49:16.265 INFO:teuthology.orchestra.run.smithi060.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-04-14T23:49:16.265 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-04-14T23:49:16.265 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2024-04-14 23:48:48.209228850 +0000 2024-04-14T23:49:16.266 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2024-04-14 23:49:06.302540887 +0000 2024-04-14T23:49:16.266 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: 2024-04-14 23:48:48.209228850 +0000 2024-04-14T23:49:16.266 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-04-14T23:49:16.266 DEBUG:teuthology.orchestra.run.smithi060:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-04-14T23:49:16.377 DEBUG:teuthology.orchestra.run.smithi060:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout a9a752df26c63acad72e1b3569fd79a515ca0765 2024-04-14T23:49:16.441 INFO:tasks.workunit.client.0.smithi060.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-14T23:49:44.098 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-14T23:49:44.106 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-14T23:50:14.240 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-14T23:50:14.248 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-14T23:50:30.214 INFO:tasks.workunit.client.0.smithi060.stderr:Updating files: 63% (7898/12473) Updating files: 64% (7983/12473) Updating files: 65% (8108/12473) Updating files: 66% (8233/12473) Updating files: 67% (8357/12473) Updating files: 68% (8482/12473) Updating files: 69% (8607/12473) Updating files: 70% (8732/12473) Updating files: 71% (8856/12473) Updating files: 72% (8981/12473) Updating files: 73% (9106/12473) Updating files: 74% (9231/12473) Updating files: 75% (9355/12473) Updating files: 76% (9480/12473) Updating files: 77% (9605/12473) Updating files: 78% (9729/12473) Updating files: 79% (9854/12473) Updating files: 80% (9979/12473) Updating files: 81% (10104/12473) Updating files: 82% (10228/12473) Updating files: 83% (10353/12473) Updating files: 84% (10478/12473) Updating files: 85% (10603/12473) Updating files: 86% (10727/12473) Updating files: 87% (10852/12473) Updating files: 88% (10977/12473) Updating files: 89% (11101/12473) Updating files: 90% (11226/12473) Updating files: 91% (11351/12473) Updating files: 92% (11476/12473) Updating files: 93% (11600/12473) Updating files: 94% (11725/12473) Updating files: 95% (11850/12473) Updating files: 96% (11975/12473) Updating files: 97% (12099/12473) Updating files: 98% (12224/12473) Updating files: 99% (12349/12473) Updating files: 100% (12473/12473) Updating files: 100% (12473/12473), done. 2024-04-14T23:50:30.716 INFO:tasks.workunit.client.0.smithi060.stderr:Note: switching to 'a9a752df26c63acad72e1b3569fd79a515ca0765'. 2024-04-14T23:50:30.716 INFO:tasks.workunit.client.0.smithi060.stderr: 2024-04-14T23:50:30.716 INFO:tasks.workunit.client.0.smithi060.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-14T23:50:30.716 INFO:tasks.workunit.client.0.smithi060.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-14T23:50:30.716 INFO:tasks.workunit.client.0.smithi060.stderr:state without impacting any branches by switching back to a branch. 2024-04-14T23:50:30.716 INFO:tasks.workunit.client.0.smithi060.stderr: 2024-04-14T23:50:30.716 INFO:tasks.workunit.client.0.smithi060.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-14T23:50:30.716 INFO:tasks.workunit.client.0.smithi060.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-14T23:50:30.716 INFO:tasks.workunit.client.0.smithi060.stderr: 2024-04-14T23:50:30.716 INFO:tasks.workunit.client.0.smithi060.stderr: git switch -c 2024-04-14T23:50:30.716 INFO:tasks.workunit.client.0.smithi060.stderr: 2024-04-14T23:50:30.716 INFO:tasks.workunit.client.0.smithi060.stderr:Or undo this operation with: 2024-04-14T23:50:30.717 INFO:tasks.workunit.client.0.smithi060.stderr: 2024-04-14T23:50:30.717 INFO:tasks.workunit.client.0.smithi060.stderr: git switch - 2024-04-14T23:50:30.717 INFO:tasks.workunit.client.0.smithi060.stderr: 2024-04-14T23:50:30.717 INFO:tasks.workunit.client.0.smithi060.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-14T23:50:30.717 INFO:tasks.workunit.client.0.smithi060.stderr: 2024-04-14T23:50:30.717 INFO:tasks.workunit.client.0.smithi060.stderr:HEAD is now at a9a752df26c Merge pull request #56871 from zdover23/wip-doc-2024-04-14-backport-56870-to-squid 2024-04-14T23:50:30.725 DEBUG:teuthology.orchestra.run.smithi060:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-14T23:50:30.787 INFO:tasks.workunit.client.0.smithi060.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-14T23:50:30.795 INFO:tasks.workunit.client.0.smithi060.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-14T23:50:30.795 INFO:tasks.workunit.client.0.smithi060.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-14T23:50:31.597 INFO:tasks.workunit.client.0.smithi060.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-14T23:50:31.710 INFO:tasks.workunit.client.0.smithi060.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-14T23:50:31.794 INFO:tasks.workunit.client.0.smithi060.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-14T23:50:31.800 INFO:tasks.workunit.client.0.smithi060.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-14T23:50:31.800 INFO:tasks.workunit.client.0.smithi060.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-14T23:50:31.888 INFO:tasks.workunit.client.0.smithi060.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-14T23:50:31.898 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-04-14T23:50:31.898 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-14T23:50:31.954 INFO:tasks.workunit:Running workunits matching fs/snaps on client.0... 2024-04-14T23:50:31.955 INFO:tasks.workunit:Running workunit fs/snaps/snap-rm-diff.sh... 2024-04-14T23:50:31.955 DEBUG:teuthology.orchestra.run.smithi060:workunit test fs/snaps/snap-rm-diff.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a9a752df26c63acad72e1b3569fd79a515ca0765 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snap-rm-diff.sh 2024-04-14T23:50:32.023 INFO:tasks.workunit.client.0.smithi060.stderr:+ wget -q http://download.ceph.com/qa/linux-2.6.33.tar.bz2 2024-04-14T23:50:34.673 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir foo 2024-04-14T23:50:34.688 INFO:tasks.workunit.client.0.smithi060.stderr:+ cp linux-2.6.33.tar.bz2 foo 2024-04-14T23:50:34.805 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir foo/.snap/barsnap 2024-04-14T23:50:34.844 INFO:tasks.workunit.client.0.smithi060.stderr:+ rm foo/linux-2.6.33.tar.bz2 2024-04-14T23:50:34.861 INFO:tasks.workunit.client.0.smithi060.stderr:+ diff -q foo/.snap/barsnap/linux-2.6.33.tar.bz2 linux-2.6.33.tar.bz2 2024-04-14T23:50:36.662 INFO:tasks.workunit.client.0.smithi060.stdout:passed: files are identical 2024-04-14T23:50:36.662 INFO:tasks.workunit.client.0.smithi060.stderr:+ echo 'passed: files are identical' 2024-04-14T23:50:36.662 INFO:tasks.workunit.client.0.smithi060.stderr:+ rmdir foo/.snap/barsnap 2024-04-14T23:50:37.263 INFO:tasks.workunit.client.0.smithi060.stdout:OK 2024-04-14T23:50:37.263 INFO:tasks.workunit.client.0.smithi060.stderr:+ echo OK 2024-04-14T23:50:37.264 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-14T23:50:37.265 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-14T23:50:37.394 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-1.sh... 2024-04-14T23:50:37.395 DEBUG:teuthology.orchestra.run.smithi060:workunit test fs/snaps/snaptest-1.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a9a752df26c63acad72e1b3569fd79a515ca0765 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-1.sh 2024-04-14T23:50:37.473 INFO:tasks.workunit.client.0.smithi060.stderr:+ echo 1 2024-04-14T23:50:37.474 INFO:tasks.workunit.client.0.smithi060.stderr:+ echo 2 2024-04-14T23:50:37.476 INFO:tasks.workunit.client.0.smithi060.stderr:+ echo 3 2024-04-14T23:50:37.478 INFO:tasks.workunit.client.0.smithi060.stderr:+ '[' -e file4 ']' 2024-04-14T23:50:37.478 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir .snap/snap1 2024-04-14T23:50:37.519 INFO:tasks.workunit.client.0.smithi060.stderr:+ echo 4 2024-04-14T23:50:37.523 INFO:tasks.workunit.client.0.smithi060.stderr:++ ls 2024-04-14T23:50:37.533 INFO:tasks.workunit.client.0.smithi060.stderr:+ now='file1 2024-04-14T23:50:37.533 INFO:tasks.workunit.client.0.smithi060.stderr:file2 2024-04-14T23:50:37.533 INFO:tasks.workunit.client.0.smithi060.stderr:file3 2024-04-14T23:50:37.533 INFO:tasks.workunit.client.0.smithi060.stderr:file4' 2024-04-14T23:50:37.534 INFO:tasks.workunit.client.0.smithi060.stderr:++ ls .snap/snap1 2024-04-14T23:50:37.544 INFO:tasks.workunit.client.0.smithi060.stderr:+ then='file1 2024-04-14T23:50:37.544 INFO:tasks.workunit.client.0.smithi060.stderr:file2 2024-04-14T23:50:37.544 INFO:tasks.workunit.client.0.smithi060.stderr:file3' 2024-04-14T23:50:37.544 INFO:tasks.workunit.client.0.smithi060.stderr:+ rmdir .snap/snap1 2024-04-14T23:50:37.556 INFO:tasks.workunit.client.0.smithi060.stderr:+ '[' 'file1 2024-04-14T23:50:37.556 INFO:tasks.workunit.client.0.smithi060.stderr:file2 2024-04-14T23:50:37.556 INFO:tasks.workunit.client.0.smithi060.stderr:file3 2024-04-14T23:50:37.556 INFO:tasks.workunit.client.0.smithi060.stderr:file4' = 'file1 2024-04-14T23:50:37.556 INFO:tasks.workunit.client.0.smithi060.stderr:file2 2024-04-14T23:50:37.556 INFO:tasks.workunit.client.0.smithi060.stderr:file3' ']' 2024-04-14T23:50:37.556 INFO:tasks.workunit.client.0.smithi060.stderr:+ echo 1 2024-04-14T23:50:37.571 INFO:tasks.workunit.client.0.smithi060.stderr:+ echo 2 2024-04-14T23:50:37.586 INFO:tasks.workunit.client.0.smithi060.stderr:+ echo 3 2024-04-14T23:50:37.603 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir .snap/snap1 2024-04-14T23:50:37.638 INFO:tasks.workunit.client.0.smithi060.stderr:+ echo 4 2024-04-14T23:50:37.650 INFO:tasks.workunit.client.0.smithi060.stderr:+ rmdir .snap/snap1 2024-04-14T23:50:37.668 INFO:tasks.workunit.client.0.smithi060.stderr:+ rm file1 file2 file3 file4 2024-04-14T23:50:37.682 INFO:tasks.workunit.client.0.smithi060.stderr:+ echo OK 2024-04-14T23:50:37.682 INFO:tasks.workunit.client.0.smithi060.stdout:OK 2024-04-14T23:50:37.684 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-14T23:50:37.684 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-14T23:50:37.784 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-2.sh... 2024-04-14T23:50:37.785 DEBUG:teuthology.orchestra.run.smithi060:workunit test fs/snaps/snaptest-2.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a9a752df26c63acad72e1b3569fd79a515ca0765 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-2.sh 2024-04-14T23:50:37.862 INFO:tasks.workunit.client.0.smithi060.stdout:Create dir 100 to 199 ... 2024-04-14T23:50:37.875 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 100 2024-04-14T23:50:37.905 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 101 2024-04-14T23:50:37.946 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 102 2024-04-14T23:50:37.987 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 103 2024-04-14T23:50:38.022 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 104 2024-04-14T23:50:38.076 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 105 2024-04-14T23:50:38.118 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 106 2024-04-14T23:50:38.154 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 107 2024-04-14T23:50:38.193 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 108 2024-04-14T23:50:38.232 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 109 2024-04-14T23:50:38.273 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 110 2024-04-14T23:50:38.316 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 111 2024-04-14T23:50:38.427 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 112 2024-04-14T23:50:38.467 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 113 2024-04-14T23:50:38.508 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 114 2024-04-14T23:50:38.548 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 115 2024-04-14T23:50:38.588 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 116 2024-04-14T23:50:38.629 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 117 2024-04-14T23:50:38.669 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 118 2024-04-14T23:50:38.707 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 119 2024-04-14T23:50:38.746 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 120 2024-04-14T23:50:38.784 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 121 2024-04-14T23:50:38.824 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 122 2024-04-14T23:50:39.127 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 123 2024-04-14T23:50:39.454 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 124 2024-04-14T23:50:39.503 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 125 2024-04-14T23:50:39.549 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 126 2024-04-14T23:50:39.591 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 127 2024-04-14T23:50:39.622 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 128 2024-04-14T23:50:39.653 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 129 2024-04-14T23:50:39.669 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 130 2024-04-14T23:50:40.170 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 131 2024-04-14T23:50:40.205 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 132 2024-04-14T23:50:40.237 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 133 2024-04-14T23:50:40.278 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 134 2024-04-14T23:50:40.320 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 135 2024-04-14T23:50:40.358 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 136 2024-04-14T23:50:40.397 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 137 2024-04-14T23:50:40.438 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 138 2024-04-14T23:50:40.484 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 139 2024-04-14T23:50:40.537 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 140 2024-04-14T23:50:40.830 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 141 2024-04-14T23:50:40.872 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 142 2024-04-14T23:50:40.928 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 143 2024-04-14T23:50:40.981 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 144 2024-04-14T23:50:41.676 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 145 2024-04-14T23:50:41.718 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 146 2024-04-14T23:50:41.758 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 147 2024-04-14T23:50:42.210 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:50:42.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.437+0000 7fc892400640 1 Processor -- start 2024-04-14T23:50:42.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.437+0000 7fc892400640 1 -- start start 2024-04-14T23:50:42.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.439+0000 7fc892400640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc88c0c5380 0x7fc88c15a980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:50:42.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.439+0000 7fc892400640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc88c06b4b0 con 0x7fc88c070d50 2024-04-14T23:50:42.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.439+0000 7fc892400640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc88c06b790 con 0x7fc88c0c4fa0 2024-04-14T23:50:42.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.439+0000 7fc892400640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fc88c0c69f0 con 0x7fc88c0c5380 2024-04-14T23:50:42.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.439+0000 7fc891a00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc88c0c5380 0x7fc88c15a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:50:42.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.439+0000 7fc891000640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fc88c0c4fa0 0x7fc88c0c77b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.60:46970/0 (socket says 172.21.15.60:46970) 2024-04-14T23:50:42.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.439+0000 7fc891000640 1 -- 172.21.15.60:0/242441883 learned_addr learned my addr 172.21.15.60:0/242441883 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:50:42.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.440+0000 7fc88b400640 1 -- 172.21.15.60:0/242441883 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 884520627 0 0) 0x7fc88c06b790 con 0x7fc88c0c4fa0 2024-04-14T23:50:42.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.440+0000 7fc88b400640 1 -- 172.21.15.60:0/242441883 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc874003520 con 0x7fc88c0c4fa0 2024-04-14T23:50:42.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.440+0000 7fc891a00640 1 -- 172.21.15.60:0/242441883 >> v1:172.21.15.186:6789/0 conn(0x7fc88c0c4fa0 legacy=0x7fc88c0c77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:50:42.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.440+0000 7fc891a00640 1 -- 172.21.15.60:0/242441883 >> v1:172.21.15.60:6789/0 conn(0x7fc88c070d50 legacy=0x7fc88c15a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:50:42.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.440+0000 7fc891a00640 1 -- 172.21.15.60:0/242441883 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc88c0c6cd0 con 0x7fc88c0c5380 2024-04-14T23:50:42.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.441+0000 7fc891a00640 1 --2- 172.21.15.60:0/242441883 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc88c0c5380 0x7fc88c15a980 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fc884009d90 tx=0x7fc88402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af51d1470bc0948c server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:50:42.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.441+0000 7fc88b400640 1 -- 172.21.15.60:0/242441883 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc884039070 con 0x7fc88c0c5380 2024-04-14T23:50:42.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.441+0000 7fc88b400640 1 -- 172.21.15.60:0/242441883 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc88402cd00 con 0x7fc88c0c5380 2024-04-14T23:50:42.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.441+0000 7fc88b400640 1 -- 172.21.15.60:0/242441883 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc884034660 con 0x7fc88c0c5380 2024-04-14T23:50:42.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.442+0000 7fc892400640 1 -- 172.21.15.60:0/242441883 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc88c0c5380 msgr2=0x7fc88c15a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:50:42.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.442+0000 7fc892400640 1 --2- 172.21.15.60:0/242441883 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc88c0c5380 0x7fc88c15a980 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fc884009d90 tx=0x7fc88402bf50 comp rx=0 tx=0).stop 2024-04-14T23:50:42.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.442+0000 7fc892400640 1 -- 172.21.15.60:0/242441883 shutdown_connections 2024-04-14T23:50:42.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.442+0000 7fc892400640 1 --2- 172.21.15.60:0/242441883 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc88c0c5380 0x7fc88c15a980 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:42.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.442+0000 7fc892400640 1 -- 172.21.15.60:0/242441883 >> 172.21.15.60:0/242441883 conn(0x7fc88c0c8b50 msgr2=0x7fc88c0c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:50:42.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.442+0000 7fc892400640 1 -- 172.21.15.60:0/242441883 shutdown_connections 2024-04-14T23:50:42.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.442+0000 7fc892400640 1 -- 172.21.15.60:0/242441883 wait complete. 2024-04-14T23:50:42.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.444+0000 7fc892400640 1 Processor -- start 2024-04-14T23:50:42.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.444+0000 7fc892400640 1 -- start start 2024-04-14T23:50:42.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.445+0000 7fc892400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc88c070d50 0x7fc88c0c01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:50:42.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.445+0000 7fc892400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc88c0c4fa0 0x7fc88c0c15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:50:42.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.445+0000 7fc892400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc88c0c5380 0x7fc88c0c1b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:50:42.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.445+0000 7fc892400640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fc88c06b1c0 con 0x7fc88c0c4fa0 2024-04-14T23:50:42.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.445+0000 7fc892400640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc88c0c7290 con 0x7fc88c070d50 2024-04-14T23:50:42.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.445+0000 7fc892400640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fc88c0c99d0 con 0x7fc88c0c5380 2024-04-14T23:50:42.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.446+0000 7fc891000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc88c070d50 0x7fc88c0c01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:50:42.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.446+0000 7fc891000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc88c070d50 0x7fc88c0c01c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:47720/0 (socket says 172.21.15.60:47720) 2024-04-14T23:50:42.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.446+0000 7fc891000640 1 -- 172.21.15.60:0/2238345684 learned_addr learned my addr 172.21.15.60:0/2238345684 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:50:42.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.446+0000 7fc88be00640 1 --2- 172.21.15.60:0/2238345684 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc88c0c4fa0 0x7fc88c0c15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:50:42.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.446+0000 7fc891a00640 1 --2- 172.21.15.60:0/2238345684 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc88c0c5380 0x7fc88c0c1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:50:42.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.446+0000 7fc88be00640 1 -- 172.21.15.60:0/2238345684 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc88c0c5380 msgr2=0x7fc88c0c1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:50:42.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.446+0000 7fc88be00640 1 --2- 172.21.15.60:0/2238345684 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc88c0c5380 0x7fc88c0c1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:42.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.446+0000 7fc88be00640 1 -- 172.21.15.60:0/2238345684 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc88c070d50 msgr2=0x7fc88c0c01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:50:42.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.446+0000 7fc88be00640 1 --2- 172.21.15.60:0/2238345684 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc88c070d50 0x7fc88c0c01c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:42.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.446+0000 7fc88be00640 1 -- 172.21.15.60:0/2238345684 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc88c1f0680 con 0x7fc88c0c4fa0 2024-04-14T23:50:42.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.446+0000 7fc891000640 1 --2- 172.21.15.60:0/2238345684 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc88c070d50 0x7fc88c0c01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:50:42.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.446+0000 7fc88be00640 1 --2- 172.21.15.60:0/2238345684 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc88c0c4fa0 0x7fc88c0c15d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc88c0cd020 tx=0x7fc87800c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:50:42.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.447+0000 7fc888c00640 1 -- 172.21.15.60:0/2238345684 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8780052d0 con 0x7fc88c0c4fa0 2024-04-14T23:50:42.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.447+0000 7fc888c00640 1 -- 172.21.15.60:0/2238345684 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc878005490 con 0x7fc88c0c4fa0 2024-04-14T23:50:42.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.447+0000 7fc888c00640 1 -- 172.21.15.60:0/2238345684 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc878007260 con 0x7fc88c0c4fa0 2024-04-14T23:50:42.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.447+0000 7fc892400640 1 -- 172.21.15.60:0/2238345684 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc88c1f0960 con 0x7fc88c0c4fa0 2024-04-14T23:50:42.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.447+0000 7fc892400640 1 -- 172.21.15.60:0/2238345684 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc88c1f0f20 con 0x7fc88c0c4fa0 2024-04-14T23:50:42.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.448+0000 7fc892400640 1 -- 172.21.15.60:0/2238345684 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc88c04f540 con 0x7fc88c0c4fa0 2024-04-14T23:50:42.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.449+0000 7fc888c00640 1 -- 172.21.15.60:0/2238345684 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7fc878007400 con 0x7fc88c0c4fa0 2024-04-14T23:50:42.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.450+0000 7fc888c00640 1 --2- 172.21.15.60:0/2238345684 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fc86c06b990 0x7fc86c06de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:50:42.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.450+0000 7fc888c00640 1 -- 172.21.15.60:0/2238345684 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5881+0+0 (secure 0 0 0) 0x7fc878089cd0 con 0x7fc88c0c4fa0 2024-04-14T23:50:42.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.450+0000 7fc891000640 1 --2- 172.21.15.60:0/2238345684 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fc86c06b990 0x7fc86c06de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:50:42.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.451+0000 7fc891000640 1 --2- 172.21.15.60:0/2238345684 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fc86c06b990 0x7fc86c06de50 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc8800011e0 tx=0x7fc880009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:50:42.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.455+0000 7fc888c00640 1 -- 172.21.15.60:0/2238345684 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc8780580b0 con 0x7fc88c0c4fa0 2024-04-14T23:50:42.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.696+0000 7fc892400640 1 -- 172.21.15.60:0/2238345684 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc88c071580 con 0x7fc88c0c4fa0 2024-04-14T23:50:42.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.698+0000 7fc888c00640 1 -- 172.21.15.60:0/2238345684 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6827 (secure 0 0 0) 0x7fc87805c150 con 0x7fc88c0c4fa0 2024-04-14T23:50:42.704 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:50:42.704 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/78303888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":78303888},{"type":"v1","addr":"172.21.15.60:6833","nonce":78303888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:48:49.256965+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.60:6835/2677133122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2677133122},{"type":"v1","addr":"172.21.15.60:6835","nonce":2677133122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4279,"qdb_cluster":[4279]},"id":1}]} 2024-04-14T23:50:42.704 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 8 2024-04-14T23:50:42.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.701+0000 7fc892400640 1 -- 172.21.15.60:0/2238345684 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fc86c06b990 msgr2=0x7fc86c06de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:50:42.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.701+0000 7fc892400640 1 --2- 172.21.15.60:0/2238345684 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fc86c06b990 0x7fc86c06de50 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc8800011e0 tx=0x7fc880009040 comp rx=0 tx=0).stop 2024-04-14T23:50:42.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.701+0000 7fc892400640 1 -- 172.21.15.60:0/2238345684 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc88c0c4fa0 msgr2=0x7fc88c0c15d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:50:42.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.701+0000 7fc892400640 1 --2- 172.21.15.60:0/2238345684 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc88c0c4fa0 0x7fc88c0c15d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc88c0cd020 tx=0x7fc87800c820 comp rx=0 tx=0).stop 2024-04-14T23:50:42.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.701+0000 7fc892400640 1 -- 172.21.15.60:0/2238345684 shutdown_connections 2024-04-14T23:50:42.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.701+0000 7fc892400640 1 --2- 172.21.15.60:0/2238345684 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fc86c06b990 0x7fc86c06de50 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:42.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.701+0000 7fc892400640 1 --2- 172.21.15.60:0/2238345684 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc88c0c5380 0x7fc88c0c1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:42.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.701+0000 7fc892400640 1 --2- 172.21.15.60:0/2238345684 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc88c0c4fa0 0x7fc88c0c15d0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:42.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.701+0000 7fc892400640 1 --2- 172.21.15.60:0/2238345684 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc88c070d50 0x7fc88c0c01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:42.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.701+0000 7fc892400640 1 -- 172.21.15.60:0/2238345684 >> 172.21.15.60:0/2238345684 conn(0x7fc88c0c8b50 msgr2=0x7fc88c0cb910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:50:42.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.701+0000 7fc892400640 1 -- 172.21.15.60:0/2238345684 shutdown_connections 2024-04-14T23:50:42.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:42.701+0000 7fc892400640 1 -- 172.21.15.60:0/2238345684 wait complete. 2024-04-14T23:50:42.714 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 148 2024-04-14T23:50:42.727 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=0) 2024-04-14T23:50:42.728 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-14T23:50:42.728 INFO:teuthology.orchestra.run:waiting for 300 2024-04-14T23:50:42.749 INFO:tasks.ceph.mds.c:Stopped 2024-04-14T23:50:42.750 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:50:43.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.004+0000 7f2c87e00640 1 Processor -- start 2024-04-14T23:50:43.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.004+0000 7f2c87e00640 1 -- start start 2024-04-14T23:50:43.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.006+0000 7f2c87e00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2c88154b20 0x7f2c880cbf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:50:43.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.006+0000 7f2c87e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2c880cc720 0x7f2c880d0230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:50:43.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.006+0000 7f2c87e00640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c8806cda0 con 0x7f2c8815e930 2024-04-14T23:50:43.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.006+0000 7f2c87e00640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f2c8806d2f0 con 0x7f2c88154b20 2024-04-14T23:50:43.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.006+0000 7f2c87e00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f2c880cbc20 con 0x7f2c880cc720 2024-04-14T23:50:43.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.006+0000 7f2c86a00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2c88154b20 0x7f2c880cbf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:50:43.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.006+0000 7f2c86a00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2c88154b20 0x7f2c880cbf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53562/0 (socket says 172.21.15.60:53562) 2024-04-14T23:50:43.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.006+0000 7f2c86a00640 1 -- 172.21.15.60:0/1625423713 learned_addr learned my addr 172.21.15.60:0/1625423713 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:50:43.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.006+0000 7f2c86000640 1 --2- 172.21.15.60:0/1625423713 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2c880cc720 0x7f2c880d0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:50:43.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.007+0000 7f2c86a00640 1 -- 172.21.15.60:0/1625423713 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2c880cc720 msgr2=0x7f2c880d0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:50:43.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.007+0000 7f2c86a00640 1 --2- 172.21.15.60:0/1625423713 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2c880cc720 0x7f2c880d0230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:43.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.007+0000 7f2c86a00640 1 -- 172.21.15.60:0/1625423713 >> v1:172.21.15.186:6789/0 conn(0x7f2c8815e930 legacy=0x7f2c8815ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:50:43.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.007+0000 7f2c86a00640 1 -- 172.21.15.60:0/1625423713 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c8806d010 con 0x7f2c88154b20 2024-04-14T23:50:43.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.007+0000 7f2c85600640 1 -- 172.21.15.60:0/1625423713 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3470506639 0 0) 0x7f2c8806cda0 con 0x7f2c8815e930 2024-04-14T23:50:43.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.008+0000 7f2c86a00640 1 --2- 172.21.15.60:0/1625423713 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c88154b20 0x7f2c880cbf00 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2c78009e50 tx=0x7f2c7802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f08aaac87d7dc054 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:50:43.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.008+0000 7f2c85600640 1 -- 172.21.15.60:0/1625423713 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c7802cb00 con 0x7f2c88154b20 2024-04-14T23:50:43.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.008+0000 7f2c85600640 1 -- 172.21.15.60:0/1625423713 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c7802cca0 con 0x7f2c88154b20 2024-04-14T23:50:43.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.008+0000 7f2c85600640 1 -- 172.21.15.60:0/1625423713 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c78034610 con 0x7f2c88154b20 2024-04-14T23:50:43.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.009+0000 7f2c87e00640 1 -- 172.21.15.60:0/1625423713 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c88154b20 msgr2=0x7f2c880cbf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:50:43.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.009+0000 7f2c87e00640 1 --2- 172.21.15.60:0/1625423713 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c88154b20 0x7f2c880cbf00 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2c78009e50 tx=0x7f2c7802bf60 comp rx=0 tx=0).stop 2024-04-14T23:50:43.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.009+0000 7f2c87e00640 1 -- 172.21.15.60:0/1625423713 shutdown_connections 2024-04-14T23:50:43.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.009+0000 7f2c87e00640 1 --2- 172.21.15.60:0/1625423713 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2c880cc720 0x7f2c880d0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:43.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.009+0000 7f2c87e00640 1 --2- 172.21.15.60:0/1625423713 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c88154b20 0x7f2c880cbf00 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:43.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.009+0000 7f2c87e00640 1 -- 172.21.15.60:0/1625423713 >> 172.21.15.60:0/1625423713 conn(0x7f2c8806ddb0 msgr2=0x7f2c88066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:50:43.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.009+0000 7f2c87e00640 1 -- 172.21.15.60:0/1625423713 shutdown_connections 2024-04-14T23:50:43.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.009+0000 7f2c87e00640 1 -- 172.21.15.60:0/1625423713 wait complete. 2024-04-14T23:50:43.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.011+0000 7f2c87e00640 1 Processor -- start 2024-04-14T23:50:43.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.011+0000 7f2c87e00640 1 -- start start 2024-04-14T23:50:43.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.012+0000 7f2c87e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c880cc720 0x7f2c88156690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:50:43.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.012+0000 7f2c87e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c88154b20 0x7f2c88155330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:50:43.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.012+0000 7f2c87e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2c8815e930 0x7f2c8814aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:50:43.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.012+0000 7f2c87e00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f2c88155080 con 0x7f2c880cc720 2024-04-14T23:50:43.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.012+0000 7f2c87e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2c88154f00 con 0x7f2c88154b20 2024-04-14T23:50:43.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.012+0000 7f2c87e00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f2c8815b430 con 0x7f2c8815e930 2024-04-14T23:50:43.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.013+0000 7f2c86a00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c880cc720 0x7f2c88156690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:50:43.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.013+0000 7f2c86a00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c880cc720 0x7f2c88156690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53568/0 (socket says 172.21.15.60:53568) 2024-04-14T23:50:43.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.013+0000 7f2c86000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c88154b20 0x7f2c88155330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:50:43.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.013+0000 7f2c86a00640 1 -- 172.21.15.60:0/2309779647 learned_addr learned my addr 172.21.15.60:0/2309779647 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:50:43.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.013+0000 7f2c87400640 1 --2- 172.21.15.60:0/2309779647 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2c8815e930 0x7f2c8814aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:50:43.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.013+0000 7f2c86a00640 1 -- 172.21.15.60:0/2309779647 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2c8815e930 msgr2=0x7f2c8814aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:50:43.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.013+0000 7f2c86a00640 1 --2- 172.21.15.60:0/2309779647 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2c8815e930 0x7f2c8814aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:43.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.013+0000 7f2c86a00640 1 -- 172.21.15.60:0/2309779647 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c88154b20 msgr2=0x7f2c88155330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:50:43.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.013+0000 7f2c86a00640 1 --2- 172.21.15.60:0/2309779647 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c88154b20 0x7f2c88155330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:43.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.013+0000 7f2c86a00640 1 -- 172.21.15.60:0/2309779647 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c880d0ad0 con 0x7f2c880cc720 2024-04-14T23:50:43.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.013+0000 7f2c86a00640 1 --2- 172.21.15.60:0/2309779647 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c880cc720 0x7f2c88156690 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f2c78009f70 tx=0x7f2c78004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:50:43.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.014+0000 7f2c7ea00640 1 -- 172.21.15.60:0/2309779647 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c78043020 con 0x7f2c880cc720 2024-04-14T23:50:43.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.014+0000 7f2c7ea00640 1 -- 172.21.15.60:0/2309779647 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c7802ce50 con 0x7f2c880cc720 2024-04-14T23:50:43.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.014+0000 7f2c7ea00640 1 -- 172.21.15.60:0/2309779647 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c78009510 con 0x7f2c880cc720 2024-04-14T23:50:43.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.014+0000 7f2c87e00640 1 -- 172.21.15.60:0/2309779647 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c8815df20 con 0x7f2c880cc720 2024-04-14T23:50:43.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.014+0000 7f2c87e00640 1 -- 172.21.15.60:0/2309779647 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c881f0fd0 con 0x7f2c880cc720 2024-04-14T23:50:43.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.015+0000 7f2c87e00640 1 -- 172.21.15.60:0/2309779647 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c8804f540 con 0x7f2c880cc720 2024-04-14T23:50:43.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.016+0000 7f2c7ea00640 1 -- 172.21.15.60:0/2309779647 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f2c78046020 con 0x7f2c880cc720 2024-04-14T23:50:43.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.017+0000 7f2c7ea00640 1 --2- 172.21.15.60:0/2309779647 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f2c6806b9c0 0x7f2c6806de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:50:43.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.017+0000 7f2c7ea00640 1 -- 172.21.15.60:0/2309779647 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5881+0+0 (secure 0 0 0) 0x7f2c780acaf0 con 0x7f2c880cc720 2024-04-14T23:50:43.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.018+0000 7f2c86000640 1 --2- 172.21.15.60:0/2309779647 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f2c6806b9c0 0x7f2c6806de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:50:43.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.018+0000 7f2c86000640 1 --2- 172.21.15.60:0/2309779647 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f2c6806b9c0 0x7f2c6806de80 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2c880d0d00 tx=0x7f2c74009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:50:43.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.022+0000 7f2c7ea00640 1 -- 172.21.15.60:0/2309779647 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2c7804d070 con 0x7f2c880cc720 2024-04-14T23:50:43.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.272+0000 7f2c87e00640 1 -- 172.21.15.60:0/2309779647 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2c881523e0 con 0x7f2c880cc720 2024-04-14T23:50:43.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.274+0000 7f2c7ea00640 1 -- 172.21.15.60:0/2309779647 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6827 (secure 0 0 0) 0x7f2c7807aff0 con 0x7f2c880cc720 2024-04-14T23:50:43.280 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:50:43.280 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/78303888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":78303888},{"type":"v1","addr":"172.21.15.60:6833","nonce":78303888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:48:49.256965+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.60:6835/2677133122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2677133122},{"type":"v1","addr":"172.21.15.60:6835","nonce":2677133122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4279,"qdb_cluster":[4279]},"id":1}]} 2024-04-14T23:50:43.280 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 8 2024-04-14T23:50:43.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.276+0000 7f2c87e00640 1 -- 172.21.15.60:0/2309779647 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f2c6806b9c0 msgr2=0x7f2c6806de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:50:43.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.276+0000 7f2c87e00640 1 --2- 172.21.15.60:0/2309779647 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f2c6806b9c0 0x7f2c6806de80 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2c880d0d00 tx=0x7f2c74009190 comp rx=0 tx=0).stop 2024-04-14T23:50:43.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.276+0000 7f2c87e00640 1 -- 172.21.15.60:0/2309779647 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c880cc720 msgr2=0x7f2c88156690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:50:43.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.276+0000 7f2c87e00640 1 --2- 172.21.15.60:0/2309779647 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c880cc720 0x7f2c88156690 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f2c78009f70 tx=0x7f2c78004a40 comp rx=0 tx=0).stop 2024-04-14T23:50:43.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.276+0000 7f2c87e00640 1 -- 172.21.15.60:0/2309779647 shutdown_connections 2024-04-14T23:50:43.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.276+0000 7f2c87e00640 1 --2- 172.21.15.60:0/2309779647 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f2c6806b9c0 0x7f2c6806de80 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:43.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.276+0000 7f2c87e00640 1 --2- 172.21.15.60:0/2309779647 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2c8815e930 0x7f2c8814aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:43.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.276+0000 7f2c87e00640 1 --2- 172.21.15.60:0/2309779647 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c88154b20 0x7f2c88155330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:43.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.276+0000 7f2c87e00640 1 --2- 172.21.15.60:0/2309779647 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c880cc720 0x7f2c88156690 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:43.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.276+0000 7f2c87e00640 1 -- 172.21.15.60:0/2309779647 >> 172.21.15.60:0/2309779647 conn(0x7f2c8806ddb0 msgr2=0x7f2c880c52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:50:43.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.277+0000 7f2c87e00640 1 -- 172.21.15.60:0/2309779647 shutdown_connections 2024-04-14T23:50:43.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:43.277+0000 7f2c87e00640 1 -- 172.21.15.60:0/2309779647 wait complete. 2024-04-14T23:50:43.303 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-14T23:50:44.395 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-14T23:50:44.403 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-14T23:50:45.304 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:50:45.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.530+0000 7fbb1fe00640 1 Processor -- start 2024-04-14T23:50:45.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.530+0000 7fbb1fe00640 1 -- start start 2024-04-14T23:50:45.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.531+0000 7fbb1fe00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbb20073320 0x7fbb200c6950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:50:45.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.531+0000 7fbb1fe00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb20152fb0 con 0x7fbb200b1e00 2024-04-14T23:50:45.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.531+0000 7fbb1fe00640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb20159310 con 0x7fbb200c3d90 2024-04-14T23:50:45.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.531+0000 7fbb1fe00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fbb200c6540 con 0x7fbb20073320 2024-04-14T23:50:45.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.532+0000 7fbb1ea00640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fbb200b1e00 0x7fbb200c7fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53666/0 (socket says 172.21.15.60:53666) 2024-04-14T23:50:45.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.532+0000 7fbb1ea00640 1 -- 172.21.15.60:0/3942109312 learned_addr learned my addr 172.21.15.60:0/3942109312 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:50:45.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.532+0000 7fbb1e000640 1 --2- 172.21.15.60:0/3942109312 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbb20073320 0x7fbb200c6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:50:45.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.532+0000 7fbb1d600640 1 -- 172.21.15.60:0/3942109312 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2862810963 0 0) 0x7fbb20152fb0 con 0x7fbb200b1e00 2024-04-14T23:50:45.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.532+0000 7fbb1d600640 1 -- 172.21.15.60:0/3942109312 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb04003520 con 0x7fbb200b1e00 2024-04-14T23:50:45.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.532+0000 7fbb1d600640 1 -- 172.21.15.60:0/3942109312 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1773762207 0 0) 0x7fbb20159310 con 0x7fbb200c3d90 2024-04-14T23:50:45.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.532+0000 7fbb1d600640 1 -- 172.21.15.60:0/3942109312 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb20152fb0 con 0x7fbb200c3d90 2024-04-14T23:50:45.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.532+0000 7fbb1e000640 1 -- 172.21.15.60:0/3942109312 >> v1:172.21.15.186:6789/0 conn(0x7fbb200c3d90 legacy=0x7fbb200c41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:50:45.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.532+0000 7fbb1e000640 1 -- 172.21.15.60:0/3942109312 >> v1:172.21.15.60:6789/0 conn(0x7fbb200b1e00 legacy=0x7fbb200c7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:50:45.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.533+0000 7fbb1e000640 1 -- 172.21.15.60:0/3942109312 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb200c38b0 con 0x7fbb20073320 2024-04-14T23:50:45.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.534+0000 7fbb1e000640 1 --2- 172.21.15.60:0/3942109312 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbb20073320 0x7fbb200c6950 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fbb200c3a80 tx=0x7fbb0802b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a43bd12e1560f25 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:50:45.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.534+0000 7fbb1d600640 1 -- 172.21.15.60:0/3942109312 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb08034070 con 0x7fbb20073320 2024-04-14T23:50:45.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.534+0000 7fbb1d600640 1 -- 172.21.15.60:0/3942109312 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb08030880 con 0x7fbb20073320 2024-04-14T23:50:45.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.534+0000 7fbb1d600640 1 -- 172.21.15.60:0/3942109312 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb08030c50 con 0x7fbb20073320 2024-04-14T23:50:45.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.535+0000 7fbb1fe00640 1 -- 172.21.15.60:0/3942109312 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbb20073320 msgr2=0x7fbb200c6950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:50:45.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.535+0000 7fbb1fe00640 1 --2- 172.21.15.60:0/3942109312 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbb20073320 0x7fbb200c6950 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fbb200c3a80 tx=0x7fbb0802b9c0 comp rx=0 tx=0).stop 2024-04-14T23:50:45.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.535+0000 7fbb1fe00640 1 -- 172.21.15.60:0/3942109312 shutdown_connections 2024-04-14T23:50:45.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.535+0000 7fbb1fe00640 1 --2- 172.21.15.60:0/3942109312 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbb20073320 0x7fbb200c6950 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:45.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.535+0000 7fbb1fe00640 1 -- 172.21.15.60:0/3942109312 >> 172.21.15.60:0/3942109312 conn(0x7fbb200c8b50 msgr2=0x7fbb200c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:50:45.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.535+0000 7fbb1fe00640 1 -- 172.21.15.60:0/3942109312 shutdown_connections 2024-04-14T23:50:45.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.535+0000 7fbb1fe00640 1 -- 172.21.15.60:0/3942109312 wait complete. 2024-04-14T23:50:45.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.538+0000 7fbb1fe00640 1 Processor -- start 2024-04-14T23:50:45.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.538+0000 7fbb1fe00640 1 -- start start 2024-04-14T23:50:45.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.539+0000 7fbb1fe00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbb20073320 0x7fbb20071760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:50:45.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.539+0000 7fbb1fe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb200b1e00 0x7fbb2014fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:50:45.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.539+0000 7fbb1fe00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb200c3d90 0x7fbb201e3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:50:45.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.539+0000 7fbb1fe00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fbb20159890 con 0x7fbb200c3d90 2024-04-14T23:50:45.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.539+0000 7fbb1fe00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbb200c5f80 con 0x7fbb200b1e00 2024-04-14T23:50:45.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.539+0000 7fbb1fe00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fbb200c6260 con 0x7fbb20073320 2024-04-14T23:50:45.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.539+0000 7fbb1f400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb200c3d90 0x7fbb201e3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:50:45.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.539+0000 7fbb1ea00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbb20073320 0x7fbb20071760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:50:45.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.539+0000 7fbb1f400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb200c3d90 0x7fbb201e3d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53572/0 (socket says 172.21.15.60:53572) 2024-04-14T23:50:45.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.539+0000 7fbb1f400640 1 -- 172.21.15.60:0/2151040763 learned_addr learned my addr 172.21.15.60:0/2151040763 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:50:45.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.540+0000 7fbb1e000640 1 --2- 172.21.15.60:0/2151040763 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb200b1e00 0x7fbb2014fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:50:45.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.540+0000 7fbb1f400640 1 -- 172.21.15.60:0/2151040763 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbb20073320 msgr2=0x7fbb20071760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:50:45.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.540+0000 7fbb1f400640 1 --2- 172.21.15.60:0/2151040763 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbb20073320 0x7fbb20071760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:45.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.540+0000 7fbb1f400640 1 -- 172.21.15.60:0/2151040763 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb200b1e00 msgr2=0x7fbb2014fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:50:45.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.540+0000 7fbb1f400640 1 --2- 172.21.15.60:0/2151040763 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb200b1e00 0x7fbb2014fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:45.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.540+0000 7fbb1f400640 1 -- 172.21.15.60:0/2151040763 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb20071ca0 con 0x7fbb200c3d90 2024-04-14T23:50:45.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.540+0000 7fbb1f400640 1 --2- 172.21.15.60:0/2151040763 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb200c3d90 0x7fbb201e3d50 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fbb1800c400 tx=0x7fbb1800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:50:45.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.541+0000 7fbb17400640 1 -- 172.21.15.60:0/2151040763 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb18017070 con 0x7fbb200c3d90 2024-04-14T23:50:45.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.541+0000 7fbb17400640 1 -- 172.21.15.60:0/2151040763 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb180052d0 con 0x7fbb200c3d90 2024-04-14T23:50:45.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.541+0000 7fbb17400640 1 -- 172.21.15.60:0/2151040763 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb180037f0 con 0x7fbb200c3d90 2024-04-14T23:50:45.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.541+0000 7fbb1fe00640 1 -- 172.21.15.60:0/2151040763 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb20072bb0 con 0x7fbb200c3d90 2024-04-14T23:50:45.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.541+0000 7fbb1fe00640 1 -- 172.21.15.60:0/2151040763 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb201e3270 con 0x7fbb200c3d90 2024-04-14T23:50:45.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.542+0000 7fbb1fe00640 1 -- 172.21.15.60:0/2151040763 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb2004f540 con 0x7fbb200c3d90 2024-04-14T23:50:45.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.543+0000 7fbb17400640 1 -- 172.21.15.60:0/2151040763 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7fbb1801a070 con 0x7fbb200c3d90 2024-04-14T23:50:45.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.544+0000 7fbb17400640 1 --2- 172.21.15.60:0/2151040763 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fbafc06b9c0 0x7fbafc06de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:50:45.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.544+0000 7fbb17400640 1 -- 172.21.15.60:0/2151040763 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5881+0+0 (secure 0 0 0) 0x7fbb18089c20 con 0x7fbb200c3d90 2024-04-14T23:50:45.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.544+0000 7fbb1ea00640 1 --2- 172.21.15.60:0/2151040763 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fbafc06b9c0 0x7fbafc06de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:50:45.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.545+0000 7fbb1ea00640 1 --2- 172.21.15.60:0/2151040763 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fbafc06b9c0 0x7fbafc06de80 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fbb10001560 tx=0x7fbb100019f0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:50:45.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.551+0000 7fbb17400640 1 -- 172.21.15.60:0/2151040763 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fbb18058090 con 0x7fbb200c3d90 2024-04-14T23:50:45.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.788+0000 7fbb1fe00640 1 -- 172.21.15.60:0/2151040763 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbb200b21e0 con 0x7fbb200c3d90 2024-04-14T23:50:45.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.789+0000 7fbb17400640 1 -- 172.21.15.60:0/2151040763 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6827 (secure 0 0 0) 0x7fbb1805c130 con 0x7fbb200c3d90 2024-04-14T23:50:45.795 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:50:45.795 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/78303888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":78303888},{"type":"v1","addr":"172.21.15.60:6833","nonce":78303888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:48:49.256965+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.60:6835/2677133122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2677133122},{"type":"v1","addr":"172.21.15.60:6835","nonce":2677133122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4279,"qdb_cluster":[4279]},"id":1}]} 2024-04-14T23:50:45.796 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 8 2024-04-14T23:50:45.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.792+0000 7fbb1fe00640 1 -- 172.21.15.60:0/2151040763 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fbafc06b9c0 msgr2=0x7fbafc06de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:50:45.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.792+0000 7fbb1fe00640 1 --2- 172.21.15.60:0/2151040763 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fbafc06b9c0 0x7fbafc06de80 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fbb10001560 tx=0x7fbb100019f0 comp rx=0 tx=0).stop 2024-04-14T23:50:45.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.792+0000 7fbb1fe00640 1 -- 172.21.15.60:0/2151040763 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb200c3d90 msgr2=0x7fbb201e3d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:50:45.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.792+0000 7fbb1fe00640 1 --2- 172.21.15.60:0/2151040763 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb200c3d90 0x7fbb201e3d50 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fbb1800c400 tx=0x7fbb1800c8c0 comp rx=0 tx=0).stop 2024-04-14T23:50:45.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.792+0000 7fbb1fe00640 1 -- 172.21.15.60:0/2151040763 shutdown_connections 2024-04-14T23:50:45.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.792+0000 7fbb1fe00640 1 --2- 172.21.15.60:0/2151040763 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fbafc06b9c0 0x7fbafc06de80 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:45.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.792+0000 7fbb1fe00640 1 --2- 172.21.15.60:0/2151040763 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb200c3d90 0x7fbb201e3d50 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:45.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.792+0000 7fbb1fe00640 1 --2- 172.21.15.60:0/2151040763 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb200b1e00 0x7fbb2014fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:45.799 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.792+0000 7fbb1fe00640 1 --2- 172.21.15.60:0/2151040763 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbb20073320 0x7fbb20071760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:45.799 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.792+0000 7fbb1fe00640 1 -- 172.21.15.60:0/2151040763 >> 172.21.15.60:0/2151040763 conn(0x7fbb200c8b50 msgr2=0x7fbb200c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:50:45.799 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.792+0000 7fbb1fe00640 1 -- 172.21.15.60:0/2151040763 shutdown_connections 2024-04-14T23:50:45.799 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:45.792+0000 7fbb1fe00640 1 -- 172.21.15.60:0/2151040763 wait complete. 2024-04-14T23:50:45.818 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-14T23:50:47.819 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:50:48.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.043+0000 7fded2e00640 1 Processor -- start 2024-04-14T23:50:48.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.043+0000 7fded2e00640 1 -- start start 2024-04-14T23:50:48.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.045+0000 7fded2e00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fdecc0b1e00 0x7fdecc0c6950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:50:48.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.045+0000 7fded2e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdecc073320 0x7fdecc0c7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:50:48.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.045+0000 7fded2e00640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdecc152fb0 con 0x7fdecc0c84e0 2024-04-14T23:50:48.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.045+0000 7fded2e00640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fdecc0c6540 con 0x7fdecc0b1e00 2024-04-14T23:50:48.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.045+0000 7fded2e00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fdecc0b21e0 con 0x7fdecc073320 2024-04-14T23:50:48.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.045+0000 7fded1a00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fdecc0b1e00 0x7fdecc0c6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:50:48.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.045+0000 7fded1000640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdecc073320 0x7fdecc0c7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:50:48.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.045+0000 7fded2400640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fdecc0c84e0 0x7fdecc0c3d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.60:47002/0 (socket says 172.21.15.60:47002) 2024-04-14T23:50:48.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.045+0000 7fded2400640 1 -- 172.21.15.60:0/2499681250 learned_addr learned my addr 172.21.15.60:0/2499681250 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:50:48.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.046+0000 7fdecbe00640 1 -- 172.21.15.60:0/2499681250 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3032010693 0 0) 0x7fdecc152fb0 con 0x7fdecc0c84e0 2024-04-14T23:50:48.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.046+0000 7fdecbe00640 1 -- 172.21.15.60:0/2499681250 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdeb4003520 con 0x7fdecc0c84e0 2024-04-14T23:50:48.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.046+0000 7fded1000640 1 -- 172.21.15.60:0/2499681250 >> v2:172.21.15.186:3300/0 conn(0x7fdecc0b1e00 msgr2=0x7fdecc0c6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:50:48.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.046+0000 7fded1000640 1 --2- 172.21.15.60:0/2499681250 >> v2:172.21.15.186:3300/0 conn(0x7fdecc0b1e00 0x7fdecc0c6950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:48.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.046+0000 7fded1000640 1 -- 172.21.15.60:0/2499681250 >> v1:172.21.15.186:6789/0 conn(0x7fdecc0c84e0 legacy=0x7fdecc0c3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:50:48.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.046+0000 7fded1000640 1 -- 172.21.15.60:0/2499681250 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdecc159310 con 0x7fdecc073320 2024-04-14T23:50:48.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.046+0000 7fded1a00640 1 --2- 172.21.15.60:0/2499681250 >> v2:172.21.15.186:3300/0 conn(0x7fdecc0b1e00 0x7fdecc0c6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:50:48.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.047+0000 7fded1000640 1 --2- 172.21.15.60:0/2499681250 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdecc073320 0x7fdecc0c7fa0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fdeb80097a0 tx=0x7fdeb802b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f6e104ce02849ceb server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:50:48.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.047+0000 7fdecbe00640 1 -- 172.21.15.60:0/2499681250 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb8034070 con 0x7fdecc073320 2024-04-14T23:50:48.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.048+0000 7fdecbe00640 1 -- 172.21.15.60:0/2499681250 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdeb8030800 con 0x7fdecc073320 2024-04-14T23:50:48.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.048+0000 7fdecbe00640 1 -- 172.21.15.60:0/2499681250 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb8030bd0 con 0x7fdecc073320 2024-04-14T23:50:48.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.048+0000 7fded2e00640 1 -- 172.21.15.60:0/2499681250 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdecc073320 msgr2=0x7fdecc0c7fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:50:48.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.048+0000 7fded2e00640 1 --2- 172.21.15.60:0/2499681250 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdecc073320 0x7fdecc0c7fa0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fdeb80097a0 tx=0x7fdeb802b940 comp rx=0 tx=0).stop 2024-04-14T23:50:48.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.048+0000 7fded2e00640 1 -- 172.21.15.60:0/2499681250 shutdown_connections 2024-04-14T23:50:48.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.048+0000 7fded2e00640 1 --2- 172.21.15.60:0/2499681250 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdecc073320 0x7fdecc0c7fa0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:48.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.048+0000 7fded2e00640 1 --2- 172.21.15.60:0/2499681250 >> v2:172.21.15.186:3300/0 conn(0x7fdecc0b1e00 0x7fdecc0c6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:48.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.048+0000 7fded2e00640 1 -- 172.21.15.60:0/2499681250 >> 172.21.15.60:0/2499681250 conn(0x7fdecc0c8b50 msgr2=0x7fdecc0c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:50:48.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.048+0000 7fded2e00640 1 -- 172.21.15.60:0/2499681250 shutdown_connections 2024-04-14T23:50:48.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.049+0000 7fded2e00640 1 -- 172.21.15.60:0/2499681250 wait complete. 2024-04-14T23:50:48.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.050+0000 7fded2e00640 1 Processor -- start 2024-04-14T23:50:48.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.050+0000 7fded2e00640 1 -- start start 2024-04-14T23:50:48.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.051+0000 7fded2e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdecc073320 0x7fdecc155bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:50:48.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.051+0000 7fded2e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdecc0b1e00 0x7fdecc1e4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:50:48.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.051+0000 7fded2e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdecc0c84e0 0x7fdecc153d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:50:48.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.051+0000 7fded2e00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fdecc0c38b0 con 0x7fdecc0c84e0 2024-04-14T23:50:48.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.051+0000 7fded2e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdecc159890 con 0x7fdecc0b1e00 2024-04-14T23:50:48.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.051+0000 7fded2e00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fdecc0c5f80 con 0x7fdecc073320 2024-04-14T23:50:48.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.052+0000 7fded1a00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdecc073320 0x7fdecc155bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:50:48.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.052+0000 7fded2400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdecc0c84e0 0x7fdecc153d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:50:48.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.052+0000 7fded1a00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdecc073320 0x7fdecc155bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.60:60270/0 (socket says 172.21.15.60:60270) 2024-04-14T23:50:48.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.052+0000 7fded1a00640 1 -- 172.21.15.60:0/2103406240 learned_addr learned my addr 172.21.15.60:0/2103406240 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:50:48.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.052+0000 7fded1000640 1 --2- 172.21.15.60:0/2103406240 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdecc0b1e00 0x7fdecc1e4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:50:48.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.052+0000 7fded2400640 1 -- 172.21.15.60:0/2103406240 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdecc073320 msgr2=0x7fdecc155bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:50:48.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.052+0000 7fded2400640 1 --2- 172.21.15.60:0/2103406240 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdecc073320 0x7fdecc155bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:48.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.052+0000 7fded2400640 1 -- 172.21.15.60:0/2103406240 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdecc0b1e00 msgr2=0x7fdecc1e4600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:50:48.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.052+0000 7fded2400640 1 --2- 172.21.15.60:0/2103406240 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdecc0b1e00 0x7fdecc1e4600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:48.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.052+0000 7fded2400640 1 -- 172.21.15.60:0/2103406240 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdecc1558d0 con 0x7fdecc0c84e0 2024-04-14T23:50:48.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.052+0000 7fded1a00640 1 --2- 172.21.15.60:0/2103406240 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdecc073320 0x7fdecc155bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:50:48.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.052+0000 7fded2400640 1 --2- 172.21.15.60:0/2103406240 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdecc0c84e0 0x7fdecc153d50 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fdec400ca80 tx=0x7fdec400cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:50:48.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.053+0000 7fdec9600640 1 -- 172.21.15.60:0/2103406240 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdec4017070 con 0x7fdecc0c84e0 2024-04-14T23:50:48.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.053+0000 7fdec9600640 1 -- 172.21.15.60:0/2103406240 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdec400ac30 con 0x7fdecc0c84e0 2024-04-14T23:50:48.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.053+0000 7fdec9600640 1 -- 172.21.15.60:0/2103406240 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdec4012600 con 0x7fdecc0c84e0 2024-04-14T23:50:48.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.053+0000 7fded2e00640 1 -- 172.21.15.60:0/2103406240 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdecc1e37a0 con 0x7fdecc0c84e0 2024-04-14T23:50:48.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.053+0000 7fded2e00640 1 -- 172.21.15.60:0/2103406240 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdecc1e3d60 con 0x7fdecc0c84e0 2024-04-14T23:50:48.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.054+0000 7fded2e00640 1 -- 172.21.15.60:0/2103406240 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdecc04f540 con 0x7fdecc0c84e0 2024-04-14T23:50:48.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.055+0000 7fdec9600640 1 -- 172.21.15.60:0/2103406240 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7fdec401a070 con 0x7fdecc0c84e0 2024-04-14T23:50:48.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.056+0000 7fdec9600640 1 --2- 172.21.15.60:0/2103406240 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fdeac06b990 0x7fdeac06de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:50:48.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.056+0000 7fdec9600640 1 -- 172.21.15.60:0/2103406240 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5881+0+0 (secure 0 0 0) 0x7fdec408a6b0 con 0x7fdecc0c84e0 2024-04-14T23:50:48.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.056+0000 7fded1a00640 1 --2- 172.21.15.60:0/2103406240 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fdeac06b990 0x7fdeac06de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:50:48.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.057+0000 7fded1a00640 1 --2- 172.21.15.60:0/2103406240 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fdeac06b990 0x7fdeac06de50 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fdec00095e0 tx=0x7fdec0009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:50:48.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.060+0000 7fdec9600640 1 -- 172.21.15.60:0/2103406240 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fdecc1558d0 con 0x7fdecc0c84e0 2024-04-14T23:50:48.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.289+0000 7fded2e00640 1 -- 172.21.15.60:0/2103406240 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdecc0c6250 con 0x7fdecc0c84e0 2024-04-14T23:50:48.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.290+0000 7fdec9600640 1 -- 172.21.15.60:0/2103406240 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6827 (secure 0 0 0) 0x7fdec400ae20 con 0x7fdecc0c84e0 2024-04-14T23:50:48.296 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:50:48.296 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/78303888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":78303888},{"type":"v1","addr":"172.21.15.60:6833","nonce":78303888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:48:49.256965+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.60:6835/2677133122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2677133122},{"type":"v1","addr":"172.21.15.60:6835","nonce":2677133122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4279,"qdb_cluster":[4279]},"id":1}]} 2024-04-14T23:50:48.296 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 8 2024-04-14T23:50:48.298 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.292+0000 7fded2e00640 1 -- 172.21.15.60:0/2103406240 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fdeac06b990 msgr2=0x7fdeac06de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:50:48.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.292+0000 7fded2e00640 1 --2- 172.21.15.60:0/2103406240 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fdeac06b990 0x7fdeac06de50 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fdec00095e0 tx=0x7fdec0009190 comp rx=0 tx=0).stop 2024-04-14T23:50:48.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.292+0000 7fded2e00640 1 -- 172.21.15.60:0/2103406240 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdecc0c84e0 msgr2=0x7fdecc153d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:50:48.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.292+0000 7fded2e00640 1 --2- 172.21.15.60:0/2103406240 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdecc0c84e0 0x7fdecc153d50 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fdec400ca80 tx=0x7fdec400cf40 comp rx=0 tx=0).stop 2024-04-14T23:50:48.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.292+0000 7fded2e00640 1 -- 172.21.15.60:0/2103406240 shutdown_connections 2024-04-14T23:50:48.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.292+0000 7fded2e00640 1 --2- 172.21.15.60:0/2103406240 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fdeac06b990 0x7fdeac06de50 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:48.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.292+0000 7fded2e00640 1 --2- 172.21.15.60:0/2103406240 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdecc0c84e0 0x7fdecc153d50 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:48.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.292+0000 7fded2e00640 1 --2- 172.21.15.60:0/2103406240 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdecc0b1e00 0x7fdecc1e4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:48.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.292+0000 7fded2e00640 1 --2- 172.21.15.60:0/2103406240 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdecc073320 0x7fdecc155bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:48.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.292+0000 7fded2e00640 1 -- 172.21.15.60:0/2103406240 >> 172.21.15.60:0/2103406240 conn(0x7fdecc0c8b50 msgr2=0x7fdecc0cb800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:50:48.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.293+0000 7fded2e00640 1 -- 172.21.15.60:0/2103406240 shutdown_connections 2024-04-14T23:50:48.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:48.293+0000 7fded2e00640 1 -- 172.21.15.60:0/2103406240 wait complete. 2024-04-14T23:50:48.318 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-14T23:50:50.319 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:50:50.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.546+0000 7f9f21000640 1 Processor -- start 2024-04-14T23:50:50.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.546+0000 7f9f21000640 1 -- start start 2024-04-14T23:50:50.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.547+0000 7f9f21000640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9f1c073320 0x7f9f1c0c6950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:50:50.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.547+0000 7f9f21000640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f9f1c0c3d90 0x7f9f1c0c41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:50:50.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.548+0000 7f9f21000640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f1c152fb0 con 0x7f9f1c0b1e00 2024-04-14T23:50:50.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.548+0000 7f9f21000640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f9f1c0c6540 con 0x7f9f1c073320 2024-04-14T23:50:50.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.548+0000 7f9f21000640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f9f1c0b21e0 con 0x7f9f1c0c3d90 2024-04-14T23:50:50.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.548+0000 7f9f1aa00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9f1c073320 0x7f9f1c0c6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:50:50.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.548+0000 7f9f1aa00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9f1c073320 0x7f9f1c0c6950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53598/0 (socket says 172.21.15.60:53598) 2024-04-14T23:50:50.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.548+0000 7f9f1aa00640 1 -- 172.21.15.60:0/1318092329 learned_addr learned my addr 172.21.15.60:0/1318092329 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:50:50.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.548+0000 7f9f1be00640 1 --2- 172.21.15.60:0/1318092329 >> v2:172.21.15.186:3300/0 conn(0x7f9f1c0c3d90 0x7f9f1c0c41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:50:50.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.549+0000 7f9f1aa00640 1 -- 172.21.15.60:0/1318092329 >> v2:172.21.15.186:3300/0 conn(0x7f9f1c0c3d90 msgr2=0x7f9f1c0c41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:50:50.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.549+0000 7f9f1aa00640 1 --2- 172.21.15.60:0/1318092329 >> v2:172.21.15.186:3300/0 conn(0x7f9f1c0c3d90 0x7f9f1c0c41c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:50.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.549+0000 7f9f1aa00640 1 -- 172.21.15.60:0/1318092329 >> v1:172.21.15.60:6789/0 conn(0x7f9f1c0b1e00 legacy=0x7f9f1c0c7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:50:50.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.549+0000 7f9f1aa00640 1 -- 172.21.15.60:0/1318092329 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f1c159310 con 0x7f9f1c073320 2024-04-14T23:50:50.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.549+0000 7f9f1be00640 1 --2- 172.21.15.60:0/1318092329 >> v2:172.21.15.186:3300/0 conn(0x7f9f1c0c3d90 0x7f9f1c0c41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:50:50.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.550+0000 7f9f1aa00640 1 --2- 172.21.15.60:0/1318092329 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f1c073320 0x7f9f1c0c6950 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9f1c0c7c70 tx=0x7f9f0402ba40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c468937849b462a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:50:50.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.550+0000 7f9f1a000640 1 -- 172.21.15.60:0/1318092329 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f0402f860 con 0x7f9f1c073320 2024-04-14T23:50:50.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.550+0000 7f9f1a000640 1 -- 172.21.15.60:0/1318092329 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f0402fa00 con 0x7f9f1c073320 2024-04-14T23:50:50.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.550+0000 7f9f1a000640 1 -- 172.21.15.60:0/1318092329 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f0402fdd0 con 0x7f9f1c073320 2024-04-14T23:50:50.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.551+0000 7f9f21000640 1 -- 172.21.15.60:0/1318092329 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f1c073320 msgr2=0x7f9f1c0c6950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:50:50.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.551+0000 7f9f21000640 1 --2- 172.21.15.60:0/1318092329 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f1c073320 0x7f9f1c0c6950 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9f1c0c7c70 tx=0x7f9f0402ba40 comp rx=0 tx=0).stop 2024-04-14T23:50:50.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.551+0000 7f9f21000640 1 -- 172.21.15.60:0/1318092329 shutdown_connections 2024-04-14T23:50:50.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.551+0000 7f9f21000640 1 --2- 172.21.15.60:0/1318092329 >> v2:172.21.15.186:3300/0 conn(0x7f9f1c0c3d90 0x7f9f1c0c41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:50.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.551+0000 7f9f21000640 1 --2- 172.21.15.60:0/1318092329 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f1c073320 0x7f9f1c0c6950 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:50.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.551+0000 7f9f21000640 1 -- 172.21.15.60:0/1318092329 >> 172.21.15.60:0/1318092329 conn(0x7f9f1c0c8b50 msgr2=0x7f9f1c0c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:50:50.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.551+0000 7f9f21000640 1 -- 172.21.15.60:0/1318092329 shutdown_connections 2024-04-14T23:50:50.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.551+0000 7f9f21000640 1 -- 172.21.15.60:0/1318092329 wait complete. 2024-04-14T23:50:50.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.553+0000 7f9f21000640 1 Processor -- start 2024-04-14T23:50:50.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.553+0000 7f9f21000640 1 -- start start 2024-04-14T23:50:50.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.554+0000 7f9f21000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f1c073320 0x7f9f1c0c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:50:50.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.554+0000 7f9f21000640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9f1c0b1e00 0x7f9f1c0c0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:50:50.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.554+0000 7f9f21000640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f1c0c3d90 0x7f9f1c0b0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:50:50.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.554+0000 7f9f21000640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f9f1c0c38b0 con 0x7f9f1c0c3d90 2024-04-14T23:50:50.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.554+0000 7f9f21000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9f1c159890 con 0x7f9f1c073320 2024-04-14T23:50:50.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.554+0000 7f9f21000640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f9f1c1595b0 con 0x7f9f1c0b1e00 2024-04-14T23:50:50.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.554+0000 7f9f1be00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f1c0c3d90 0x7f9f1c0b0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:50:50.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.554+0000 7f9f1b400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f1c073320 0x7f9f1c0c2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:50:50.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.554+0000 7f9f1aa00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9f1c0b1e00 0x7f9f1c0c0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:50:50.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.554+0000 7f9f1be00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f1c0c3d90 0x7f9f1c0b0c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53600/0 (socket says 172.21.15.60:53600) 2024-04-14T23:50:50.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.554+0000 7f9f1be00640 1 -- 172.21.15.60:0/3538214426 learned_addr learned my addr 172.21.15.60:0/3538214426 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:50:50.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.555+0000 7f9f1aa00640 1 -- 172.21.15.60:0/3538214426 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f1c073320 msgr2=0x7f9f1c0c2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:50:50.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.555+0000 7f9f1aa00640 1 --2- 172.21.15.60:0/3538214426 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f1c073320 0x7f9f1c0c2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:50.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.555+0000 7f9f1aa00640 1 -- 172.21.15.60:0/3538214426 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f1c0c3d90 msgr2=0x7f9f1c0b0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:50:50.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.555+0000 7f9f1aa00640 1 --2- 172.21.15.60:0/3538214426 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f1c0c3d90 0x7f9f1c0b0c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:50.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.555+0000 7f9f1aa00640 1 -- 172.21.15.60:0/3538214426 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f1c0aea00 con 0x7f9f1c0b1e00 2024-04-14T23:50:50.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.555+0000 7f9f1b400640 1 --2- 172.21.15.60:0/3538214426 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f1c073320 0x7f9f1c0c2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:50:50.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.555+0000 7f9f1be00640 1 --2- 172.21.15.60:0/3538214426 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f1c0c3d90 0x7f9f1c0b0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:50:50.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.555+0000 7f9f1aa00640 1 --2- 172.21.15.60:0/3538214426 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9f1c0b1e00 0x7f9f1c0c0750 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9f04033000 tx=0x7f9f04004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:50:50.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.556+0000 7f9f13400640 1 -- 172.21.15.60:0/3538214426 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f04037bc0 con 0x7f9f1c0b1e00 2024-04-14T23:50:50.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.556+0000 7f9f13400640 1 -- 172.21.15.60:0/3538214426 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f04046070 con 0x7f9f1c0b1e00 2024-04-14T23:50:50.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.556+0000 7f9f13400640 1 -- 172.21.15.60:0/3538214426 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f04041400 con 0x7f9f1c0b1e00 2024-04-14T23:50:50.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.556+0000 7f9f21000640 1 -- 172.21.15.60:0/3538214426 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f1c0c2d60 con 0x7f9f1c0b1e00 2024-04-14T23:50:50.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.556+0000 7f9f21000640 1 -- 172.21.15.60:0/3538214426 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f1c0aeca0 con 0x7f9f1c0b1e00 2024-04-14T23:50:50.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.557+0000 7f9f21000640 1 -- 172.21.15.60:0/3538214426 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f1c0c34c0 con 0x7f9f1c0b1e00 2024-04-14T23:50:50.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.558+0000 7f9f13400640 1 -- 172.21.15.60:0/3538214426 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f9f04040070 con 0x7f9f1c0b1e00 2024-04-14T23:50:50.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.559+0000 7f9f13400640 1 --2- 172.21.15.60:0/3538214426 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f9ef806b9c0 0x7f9ef806de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:50:50.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.559+0000 7f9f13400640 1 -- 172.21.15.60:0/3538214426 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 5853+0+0 (secure 0 0 0) 0x7f9f040acd70 con 0x7f9f1c0b1e00 2024-04-14T23:50:50.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.559+0000 7f9f1b400640 1 --2- 172.21.15.60:0/3538214426 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f9ef806b9c0 0x7f9ef806de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:50:50.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.560+0000 7f9f1b400640 1 --2- 172.21.15.60:0/3538214426 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f9ef806b9c0 0x7f9ef806de80 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c00f290 tx=0x7f9f0c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:50:50.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.564+0000 7f9f13400640 1 -- 172.21.15.60:0/3538214426 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f9f1c0c34c0 con 0x7f9f1c0b1e00 2024-04-14T23:50:50.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.803+0000 7f9f21000640 1 -- 172.21.15.60:0/3538214426 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9f1c04f540 con 0x7f9f1c0b1e00 2024-04-14T23:50:50.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.804+0000 7f9f13400640 1 -- 172.21.15.60:0/3538214426 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6827 (secure 0 0 0) 0x7f9f04033030 con 0x7f9f1c0b1e00 2024-04-14T23:50:50.810 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:50:50.810 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/78303888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":78303888},{"type":"v1","addr":"172.21.15.60:6833","nonce":78303888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:48:49.256965+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.60:6835/2677133122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2677133122},{"type":"v1","addr":"172.21.15.60:6835","nonce":2677133122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4279,"qdb_cluster":[4279]},"id":1}]} 2024-04-14T23:50:50.810 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 8 2024-04-14T23:50:50.812 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.806+0000 7f9f21000640 1 -- 172.21.15.60:0/3538214426 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f9ef806b9c0 msgr2=0x7f9ef806de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:50:50.813 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.806+0000 7f9f21000640 1 --2- 172.21.15.60:0/3538214426 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f9ef806b9c0 0x7f9ef806de80 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c00f290 tx=0x7f9f0c009040 comp rx=0 tx=0).stop 2024-04-14T23:50:50.813 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.806+0000 7f9f21000640 1 -- 172.21.15.60:0/3538214426 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9f1c0b1e00 msgr2=0x7f9f1c0c0750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:50:50.813 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.807+0000 7f9f21000640 1 --2- 172.21.15.60:0/3538214426 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9f1c0b1e00 0x7f9f1c0c0750 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9f04033000 tx=0x7f9f04004730 comp rx=0 tx=0).stop 2024-04-14T23:50:50.813 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.807+0000 7f9f21000640 1 -- 172.21.15.60:0/3538214426 shutdown_connections 2024-04-14T23:50:50.813 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.807+0000 7f9f21000640 1 --2- 172.21.15.60:0/3538214426 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f9ef806b9c0 0x7f9ef806de80 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:50.813 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.807+0000 7f9f21000640 1 --2- 172.21.15.60:0/3538214426 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f1c0c3d90 0x7f9f1c0b0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:50.813 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.807+0000 7f9f21000640 1 --2- 172.21.15.60:0/3538214426 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9f1c0b1e00 0x7f9f1c0c0750 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:50.813 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.807+0000 7f9f21000640 1 --2- 172.21.15.60:0/3538214426 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f1c073320 0x7f9f1c0c2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:50.813 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.807+0000 7f9f21000640 1 -- 172.21.15.60:0/3538214426 >> 172.21.15.60:0/3538214426 conn(0x7f9f1c0c8b50 msgr2=0x7f9f1c152690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:50:50.813 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.807+0000 7f9f21000640 1 -- 172.21.15.60:0/3538214426 shutdown_connections 2024-04-14T23:50:50.813 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:50.807+0000 7f9f21000640 1 -- 172.21.15.60:0/3538214426 wait complete. 2024-04-14T23:50:50.834 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-14T23:50:52.835 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:50:53.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.060+0000 7ffa4ea00640 1 Processor -- start 2024-04-14T23:50:53.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.060+0000 7ffa4ea00640 1 -- start start 2024-04-14T23:50:53.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.061+0000 7ffa4ea00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ffa480c8360 0x7ffa4815ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:50:53.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.061+0000 7ffa4ea00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ffa480c8740 0x7ffa4806f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:50:53.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.061+0000 7ffa4ea00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffa480c75c0 0x7ffa48158e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:50:53.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.062+0000 7ffa4ea00640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7ffa480c6b70 con 0x7ffa480c8360 2024-04-14T23:50:53.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.062+0000 7ffa4ea00640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7ffa4815f7c0 con 0x7ffa480c8740 2024-04-14T23:50:53.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.062+0000 7ffa4ea00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7ffa4815faa0 con 0x7ffa480c75c0 2024-04-14T23:50:53.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.062+0000 7ffa47e00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ffa480c8360 0x7ffa4815ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:50:53.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.062+0000 7ffa4ca00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffa480c75c0 0x7ffa48158e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:50:53.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.062+0000 7ffa47e00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ffa480c8360 0x7ffa4815ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41374/0 (socket says 172.21.15.60:41374) 2024-04-14T23:50:53.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.062+0000 7ffa47e00640 1 -- 172.21.15.60:0/2934313794 learned_addr learned my addr 172.21.15.60:0/2934313794 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:50:53.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.062+0000 7ffa47400640 1 --2- 172.21.15.60:0/2934313794 >> v2:172.21.15.186:3300/0 conn(0x7ffa480c8740 0x7ffa4806f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:50:53.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.062+0000 7ffa47e00640 1 -- 172.21.15.60:0/2934313794 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffa480c75c0 msgr2=0x7ffa48158e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:50:53.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.062+0000 7ffa47e00640 1 --2- 172.21.15.60:0/2934313794 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffa480c75c0 0x7ffa48158e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:53.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.062+0000 7ffa47e00640 1 -- 172.21.15.60:0/2934313794 >> v2:172.21.15.186:3300/0 conn(0x7ffa480c8740 msgr2=0x7ffa4806f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:50:53.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.062+0000 7ffa47e00640 1 --2- 172.21.15.60:0/2934313794 >> v2:172.21.15.186:3300/0 conn(0x7ffa480c8740 0x7ffa4806f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:53.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.063+0000 7ffa47e00640 1 -- 172.21.15.60:0/2934313794 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa4815fd80 con 0x7ffa480c8360 2024-04-14T23:50:53.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.064+0000 7ffa47e00640 1 --2- 172.21.15.60:0/2934313794 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa480c8360 0x7ffa4815ef80 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ffa48075690 tx=0x7ffa3802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d3c050529eba3bb server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:50:53.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.064+0000 7ffa46a00640 1 -- 172.21.15.60:0/2934313794 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa38034070 con 0x7ffa480c8360 2024-04-14T23:50:53.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.064+0000 7ffa46a00640 1 -- 172.21.15.60:0/2934313794 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa38030860 con 0x7ffa480c8360 2024-04-14T23:50:53.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.064+0000 7ffa46a00640 1 -- 172.21.15.60:0/2934313794 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa38030c30 con 0x7ffa480c8360 2024-04-14T23:50:53.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.065+0000 7ffa4ea00640 1 -- 172.21.15.60:0/2934313794 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa480c8360 msgr2=0x7ffa4815ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:50:53.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.065+0000 7ffa4ea00640 1 --2- 172.21.15.60:0/2934313794 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa480c8360 0x7ffa4815ef80 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ffa48075690 tx=0x7ffa3802b9a0 comp rx=0 tx=0).stop 2024-04-14T23:50:53.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.065+0000 7ffa4ea00640 1 -- 172.21.15.60:0/2934313794 shutdown_connections 2024-04-14T23:50:53.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.065+0000 7ffa4ea00640 1 --2- 172.21.15.60:0/2934313794 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffa480c75c0 0x7ffa48158e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:53.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.065+0000 7ffa4ea00640 1 --2- 172.21.15.60:0/2934313794 >> v2:172.21.15.186:3300/0 conn(0x7ffa480c8740 0x7ffa4806f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:53.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.065+0000 7ffa4ea00640 1 --2- 172.21.15.60:0/2934313794 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa480c8360 0x7ffa4815ef80 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:53.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.065+0000 7ffa4ea00640 1 -- 172.21.15.60:0/2934313794 >> 172.21.15.60:0/2934313794 conn(0x7ffa480cc170 msgr2=0x7ffa480cc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:50:53.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.065+0000 7ffa4ea00640 1 -- 172.21.15.60:0/2934313794 shutdown_connections 2024-04-14T23:50:53.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.065+0000 7ffa4ea00640 1 -- 172.21.15.60:0/2934313794 wait complete. 2024-04-14T23:50:53.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.066+0000 7ffa4ea00640 1 Processor -- start 2024-04-14T23:50:53.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.067+0000 7ffa4ea00640 1 -- start start 2024-04-14T23:50:53.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.067+0000 7ffa4ea00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa480c75c0 0x7ffa481f8750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:50:53.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.067+0000 7ffa4ea00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffa480c8360 0x7ffa481f51e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:50:53.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.068+0000 7ffa4ea00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffa480c8740 0x7ffa4815d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:50:53.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.068+0000 7ffa4ea00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ffa4815f4c0 con 0x7ffa480c75c0 2024-04-14T23:50:53.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.068+0000 7ffa4ea00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ffa480c70a0 con 0x7ffa480c8740 2024-04-14T23:50:53.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.068+0000 7ffa4ea00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7ffa48071030 con 0x7ffa480c8360 2024-04-14T23:50:53.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.068+0000 7ffa47e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa480c75c0 0x7ffa481f8750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:50:53.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.068+0000 7ffa47e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa480c75c0 0x7ffa481f8750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41382/0 (socket says 172.21.15.60:41382) 2024-04-14T23:50:53.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.068+0000 7ffa47e00640 1 -- 172.21.15.60:0/3108403355 learned_addr learned my addr 172.21.15.60:0/3108403355 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:50:53.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.068+0000 7ffa47400640 1 --2- 172.21.15.60:0/3108403355 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffa480c8360 0x7ffa481f51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:50:53.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.068+0000 7ffa4ca00640 1 --2- 172.21.15.60:0/3108403355 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffa480c8740 0x7ffa4815d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:50:53.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.068+0000 7ffa47e00640 1 -- 172.21.15.60:0/3108403355 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffa480c8360 msgr2=0x7ffa481f51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:50:53.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.068+0000 7ffa47e00640 1 --2- 172.21.15.60:0/3108403355 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffa480c8360 0x7ffa481f51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:53.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.068+0000 7ffa47e00640 1 -- 172.21.15.60:0/3108403355 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffa480c8740 msgr2=0x7ffa4815d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:50:53.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.068+0000 7ffa47e00640 1 --2- 172.21.15.60:0/3108403355 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffa480c8740 0x7ffa4815d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:53.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.068+0000 7ffa47e00640 1 -- 172.21.15.60:0/3108403355 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa481f5720 con 0x7ffa480c75c0 2024-04-14T23:50:53.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.069+0000 7ffa47e00640 1 --2- 172.21.15.60:0/3108403355 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa480c75c0 0x7ffa481f8750 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ffa3802b9e0 tx=0x7ffa38004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:50:53.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.069+0000 7ffa3fe00640 1 -- 172.21.15.60:0/3108403355 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa38048070 con 0x7ffa480c75c0 2024-04-14T23:50:53.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.069+0000 7ffa3fe00640 1 -- 172.21.15.60:0/3108403355 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa38030660 con 0x7ffa480c75c0 2024-04-14T23:50:53.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.069+0000 7ffa3fe00640 1 -- 172.21.15.60:0/3108403355 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa38034040 con 0x7ffa480c75c0 2024-04-14T23:50:53.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.069+0000 7ffa4ea00640 1 -- 172.21.15.60:0/3108403355 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa4815bce0 con 0x7ffa480c75c0 2024-04-14T23:50:53.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.069+0000 7ffa4ea00640 1 -- 172.21.15.60:0/3108403355 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa4815c280 con 0x7ffa480c75c0 2024-04-14T23:50:53.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.070+0000 7ffa4ea00640 1 -- 172.21.15.60:0/3108403355 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa4815c4c0 con 0x7ffa480c75c0 2024-04-14T23:50:53.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.071+0000 7ffa3fe00640 1 -- 172.21.15.60:0/3108403355 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7ffa3804b070 con 0x7ffa480c75c0 2024-04-14T23:50:53.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.072+0000 7ffa3fe00640 1 --2- 172.21.15.60:0/3108403355 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ffa2806b9c0 0x7ffa2806de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:50:53.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.072+0000 7ffa3fe00640 1 -- 172.21.15.60:0/3108403355 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5853+0+0 (secure 0 0 0) 0x7ffa380ac840 con 0x7ffa480c75c0 2024-04-14T23:50:53.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.073+0000 7ffa47400640 1 --2- 172.21.15.60:0/3108403355 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ffa2806b9c0 0x7ffa2806de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:50:53.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.074+0000 7ffa47400640 1 --2- 172.21.15.60:0/3108403355 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ffa2806b9c0 0x7ffa2806de80 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ffa48072bc0 tx=0x7ffa34009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:50:53.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.078+0000 7ffa3fe00640 1 -- 172.21.15.60:0/3108403355 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ffa38030830 con 0x7ffa480c75c0 2024-04-14T23:50:53.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.322+0000 7ffa4ea00640 1 -- 172.21.15.60:0/3108403355 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffa48076970 con 0x7ffa480c75c0 2024-04-14T23:50:53.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.323+0000 7ffa3fe00640 1 -- 172.21.15.60:0/3108403355 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6827 (secure 0 0 0) 0x7ffa3807adf0 con 0x7ffa480c75c0 2024-04-14T23:50:53.329 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:50:53.329 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/78303888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":78303888},{"type":"v1","addr":"172.21.15.60:6833","nonce":78303888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:48:49.256965+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.60:6835/2677133122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2677133122},{"type":"v1","addr":"172.21.15.60:6835","nonce":2677133122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4279,"qdb_cluster":[4279]},"id":1}]} 2024-04-14T23:50:53.329 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 8 2024-04-14T23:50:53.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.326+0000 7ffa4ea00640 1 -- 172.21.15.60:0/3108403355 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ffa2806b9c0 msgr2=0x7ffa2806de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:50:53.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.326+0000 7ffa4ea00640 1 --2- 172.21.15.60:0/3108403355 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ffa2806b9c0 0x7ffa2806de80 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ffa48072bc0 tx=0x7ffa34009190 comp rx=0 tx=0).stop 2024-04-14T23:50:53.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.326+0000 7ffa4ea00640 1 -- 172.21.15.60:0/3108403355 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa480c75c0 msgr2=0x7ffa481f8750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:50:53.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.326+0000 7ffa4ea00640 1 --2- 172.21.15.60:0/3108403355 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa480c75c0 0x7ffa481f8750 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ffa3802b9e0 tx=0x7ffa38004730 comp rx=0 tx=0).stop 2024-04-14T23:50:53.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.326+0000 7ffa4ea00640 1 -- 172.21.15.60:0/3108403355 shutdown_connections 2024-04-14T23:50:53.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.326+0000 7ffa4ea00640 1 --2- 172.21.15.60:0/3108403355 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ffa2806b9c0 0x7ffa2806de80 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:53.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.326+0000 7ffa4ea00640 1 --2- 172.21.15.60:0/3108403355 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffa480c8740 0x7ffa4815d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:53.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.326+0000 7ffa4ea00640 1 --2- 172.21.15.60:0/3108403355 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffa480c8360 0x7ffa481f51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:53.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.326+0000 7ffa4ea00640 1 --2- 172.21.15.60:0/3108403355 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa480c75c0 0x7ffa481f8750 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:53.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.326+0000 7ffa4ea00640 1 -- 172.21.15.60:0/3108403355 >> 172.21.15.60:0/3108403355 conn(0x7ffa480cc170 msgr2=0x7ffa48071210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:50:53.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.326+0000 7ffa4ea00640 1 -- 172.21.15.60:0/3108403355 shutdown_connections 2024-04-14T23:50:53.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:53.326+0000 7ffa4ea00640 1 -- 172.21.15.60:0/3108403355 wait complete. 2024-04-14T23:50:53.353 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-14T23:50:55.354 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:50:55.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.581+0000 7f3dd2600640 1 Processor -- start 2024-04-14T23:50:55.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.581+0000 7f3dd2600640 1 -- start start 2024-04-14T23:50:55.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.582+0000 7f3dd2600640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3dcc073320 0x7f3dcc0c6950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:50:55.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.582+0000 7f3dd2600640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3dcc0c3d90 0x7f3dcc0c41e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:50:55.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.582+0000 7f3dd2600640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3dcc152fb0 con 0x7f3dcc0b1e00 2024-04-14T23:50:55.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.582+0000 7f3dd2600640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f3dcc0c6540 con 0x7f3dcc0c3d90 2024-04-14T23:50:55.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.582+0000 7f3dd2600640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f3dcc0b21e0 con 0x7f3dcc073320 2024-04-14T23:50:55.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.583+0000 7f3dcbe00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3dcc073320 0x7f3dcc0c6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:50:55.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.583+0000 7f3dd1200640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f3dcc0b1e00 0x7f3dcc0c7fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.60:58084/0 (socket says 172.21.15.60:58084) 2024-04-14T23:50:55.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.583+0000 7f3dd1200640 1 -- 172.21.15.60:0/2281917004 learned_addr learned my addr 172.21.15.60:0/2281917004 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:50:55.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.583+0000 7f3dd1c00640 1 --2- 172.21.15.60:0/2281917004 >> v2:172.21.15.60:3300/0 conn(0x7f3dcc0c3d90 0x7f3dcc0c41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:50:55.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.583+0000 7f3dcbe00640 1 -- 172.21.15.60:0/2281917004 >> v2:172.21.15.60:3300/0 conn(0x7f3dcc0c3d90 msgr2=0x7f3dcc0c41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:50:55.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.583+0000 7f3dcbe00640 1 --2- 172.21.15.60:0/2281917004 >> v2:172.21.15.60:3300/0 conn(0x7f3dcc0c3d90 0x7f3dcc0c41e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:55.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.583+0000 7f3dcb400640 1 -- 172.21.15.60:0/2281917004 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1552482958 0 0) 0x7f3dcc152fb0 con 0x7f3dcc0b1e00 2024-04-14T23:50:55.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.583+0000 7f3dcbe00640 1 -- 172.21.15.60:0/2281917004 >> v1:172.21.15.186:6789/0 conn(0x7f3dcc0b1e00 legacy=0x7f3dcc0c7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:50:55.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.583+0000 7f3dcbe00640 1 -- 172.21.15.60:0/2281917004 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dcc159310 con 0x7f3dcc073320 2024-04-14T23:50:55.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.585+0000 7f3dcbe00640 1 --2- 172.21.15.60:0/2281917004 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3dcc073320 0x7f3dcc0c6950 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3dc00097a0 tx=0x7f3dc002b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2b4821386c058a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:50:55.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.585+0000 7f3dcb400640 1 -- 172.21.15.60:0/2281917004 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dc0034070 con 0x7f3dcc073320 2024-04-14T23:50:55.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.585+0000 7f3dcb400640 1 -- 172.21.15.60:0/2281917004 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3dc0030800 con 0x7f3dcc073320 2024-04-14T23:50:55.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.585+0000 7f3dcb400640 1 -- 172.21.15.60:0/2281917004 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dc0030bd0 con 0x7f3dcc073320 2024-04-14T23:50:55.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.586+0000 7f3dd2600640 1 -- 172.21.15.60:0/2281917004 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3dcc073320 msgr2=0x7f3dcc0c6950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:50:55.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.586+0000 7f3dd2600640 1 --2- 172.21.15.60:0/2281917004 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3dcc073320 0x7f3dcc0c6950 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3dc00097a0 tx=0x7f3dc002b940 comp rx=0 tx=0).stop 2024-04-14T23:50:55.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.586+0000 7f3dd2600640 1 -- 172.21.15.60:0/2281917004 shutdown_connections 2024-04-14T23:50:55.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.586+0000 7f3dd2600640 1 --2- 172.21.15.60:0/2281917004 >> v2:172.21.15.60:3300/0 conn(0x7f3dcc0c3d90 0x7f3dcc0c41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:55.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.586+0000 7f3dd2600640 1 --2- 172.21.15.60:0/2281917004 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3dcc073320 0x7f3dcc0c6950 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:55.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.586+0000 7f3dd2600640 1 -- 172.21.15.60:0/2281917004 >> 172.21.15.60:0/2281917004 conn(0x7f3dcc0c8b50 msgr2=0x7f3dcc0c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:50:55.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.586+0000 7f3dd2600640 1 -- 172.21.15.60:0/2281917004 shutdown_connections 2024-04-14T23:50:55.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.586+0000 7f3dd2600640 1 -- 172.21.15.60:0/2281917004 wait complete. 2024-04-14T23:50:55.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.588+0000 7f3dd2600640 1 Processor -- start 2024-04-14T23:50:55.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.588+0000 7f3dd2600640 1 -- start start 2024-04-14T23:50:55.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.590+0000 7f3dd2600640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3dcc073320 0x7f3dcc154e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:50:55.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.590+0000 7f3dd2600640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3dcc0b1e00 0x7f3dcc1566e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:50:55.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.590+0000 7f3dd2600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3dcc0c3d90 0x7f3dcc0afcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:50:55.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.590+0000 7f3dd2600640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f3dcc0c38b0 con 0x7f3dcc073320 2024-04-14T23:50:55.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.590+0000 7f3dd2600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3dcc159890 con 0x7f3dcc0c3d90 2024-04-14T23:50:55.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.590+0000 7f3dd2600640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f3dcc1595b0 con 0x7f3dcc0b1e00 2024-04-14T23:50:55.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.590+0000 7f3dd1200640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3dcc073320 0x7f3dcc154e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:50:55.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.590+0000 7f3dd1200640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3dcc073320 0x7f3dcc154e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41406/0 (socket says 172.21.15.60:41406) 2024-04-14T23:50:55.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.590+0000 7f3dd1200640 1 -- 172.21.15.60:0/3785861121 learned_addr learned my addr 172.21.15.60:0/3785861121 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:50:55.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.590+0000 7f3dcbe00640 1 --2- 172.21.15.60:0/3785861121 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3dcc0b1e00 0x7f3dcc1566e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:50:55.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.590+0000 7f3dd1c00640 1 --2- 172.21.15.60:0/3785861121 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3dcc0c3d90 0x7f3dcc0afcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:50:55.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.590+0000 7f3dd1200640 1 -- 172.21.15.60:0/3785861121 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3dcc0b1e00 msgr2=0x7f3dcc1566e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:50:55.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.590+0000 7f3dd1200640 1 --2- 172.21.15.60:0/3785861121 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3dcc0b1e00 0x7f3dcc1566e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:55.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.590+0000 7f3dd1200640 1 -- 172.21.15.60:0/3785861121 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3dcc0c3d90 msgr2=0x7f3dcc0afcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:50:55.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.590+0000 7f3dd1200640 1 --2- 172.21.15.60:0/3785861121 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3dcc0c3d90 0x7f3dcc0afcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:55.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.590+0000 7f3dd1200640 1 -- 172.21.15.60:0/3785861121 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dcc156150 con 0x7f3dcc073320 2024-04-14T23:50:55.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.590+0000 7f3dd1200640 1 --2- 172.21.15.60:0/3785861121 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3dcc073320 0x7f3dcc154e80 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc00c460 tx=0x7f3dbc00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:50:55.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.590+0000 7f3dc8c00640 1 -- 172.21.15.60:0/3785861121 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dbc018070 con 0x7f3dcc073320 2024-04-14T23:50:55.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.590+0000 7f3dc8c00640 1 -- 172.21.15.60:0/3785861121 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3dbc007500 con 0x7f3dcc073320 2024-04-14T23:50:55.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.590+0000 7f3dc8c00640 1 -- 172.21.15.60:0/3785861121 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dbc004a20 con 0x7f3dcc073320 2024-04-14T23:50:55.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.590+0000 7f3dd2600640 1 -- 172.21.15.60:0/3785861121 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3dcc157ec0 con 0x7f3dcc073320 2024-04-14T23:50:55.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.591+0000 7f3dd2600640 1 -- 172.21.15.60:0/3785861121 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3dcc1540c0 con 0x7f3dcc073320 2024-04-14T23:50:55.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.592+0000 7f3dd2600640 1 -- 172.21.15.60:0/3785861121 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3dcc04f540 con 0x7f3dcc073320 2024-04-14T23:50:55.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.593+0000 7f3dc8c00640 1 -- 172.21.15.60:0/3785861121 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f3dbc004bc0 con 0x7f3dcc073320 2024-04-14T23:50:55.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.596+0000 7f3dc8c00640 1 --2- 172.21.15.60:0/3785861121 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f3db006b9c0 0x7f3db006de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:50:55.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.596+0000 7f3dc8c00640 1 -- 172.21.15.60:0/3785861121 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5853+0+0 (secure 0 0 0) 0x7f3dbc089a50 con 0x7f3dcc073320 2024-04-14T23:50:55.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.596+0000 7f3dcbe00640 1 --2- 172.21.15.60:0/3785861121 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f3db006b9c0 0x7f3db006de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:50:55.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.596+0000 7f3dcbe00640 1 --2- 172.21.15.60:0/3785861121 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f3db006b9c0 0x7f3db006de80 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f3dc0036000 tx=0x7f3dc0008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:50:55.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.601+0000 7f3dc8c00640 1 -- 172.21.15.60:0/3785861121 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3dbc058010 con 0x7f3dcc073320 2024-04-14T23:50:55.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.837+0000 7f3dd2600640 1 -- 172.21.15.60:0/3785861121 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3dcc0c99e0 con 0x7f3dcc073320 2024-04-14T23:50:55.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.838+0000 7f3dc8c00640 1 -- 172.21.15.60:0/3785861121 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6827 (secure 0 0 0) 0x7f3dbc05c0b0 con 0x7f3dcc073320 2024-04-14T23:50:55.844 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:50:55.845 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/78303888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":78303888},{"type":"v1","addr":"172.21.15.60:6833","nonce":78303888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:48:49.256965+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.60:6835/2677133122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2677133122},{"type":"v1","addr":"172.21.15.60:6835","nonce":2677133122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4279,"qdb_cluster":[4279]},"id":1}]} 2024-04-14T23:50:55.845 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 8 2024-04-14T23:50:55.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.841+0000 7f3dd2600640 1 -- 172.21.15.60:0/3785861121 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f3db006b9c0 msgr2=0x7f3db006de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:50:55.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.841+0000 7f3dd2600640 1 --2- 172.21.15.60:0/3785861121 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f3db006b9c0 0x7f3db006de80 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f3dc0036000 tx=0x7f3dc0008030 comp rx=0 tx=0).stop 2024-04-14T23:50:55.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.841+0000 7f3dd2600640 1 -- 172.21.15.60:0/3785861121 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3dcc073320 msgr2=0x7f3dcc154e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:50:55.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.841+0000 7f3dd2600640 1 --2- 172.21.15.60:0/3785861121 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3dcc073320 0x7f3dcc154e80 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc00c460 tx=0x7f3dbc00c920 comp rx=0 tx=0).stop 2024-04-14T23:50:55.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.841+0000 7f3dd2600640 1 -- 172.21.15.60:0/3785861121 shutdown_connections 2024-04-14T23:50:55.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.841+0000 7f3dd2600640 1 --2- 172.21.15.60:0/3785861121 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f3db006b9c0 0x7f3db006de80 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:55.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.841+0000 7f3dd2600640 1 --2- 172.21.15.60:0/3785861121 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3dcc0c3d90 0x7f3dcc0afcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:55.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.841+0000 7f3dd2600640 1 --2- 172.21.15.60:0/3785861121 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3dcc0b1e00 0x7f3dcc1566e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:55.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.841+0000 7f3dd2600640 1 --2- 172.21.15.60:0/3785861121 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3dcc073320 0x7f3dcc154e80 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:55.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.841+0000 7f3dd2600640 1 -- 172.21.15.60:0/3785861121 >> 172.21.15.60:0/3785861121 conn(0x7f3dcc0c8b50 msgr2=0x7f3dcc0b0b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:50:55.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.841+0000 7f3dd2600640 1 -- 172.21.15.60:0/3785861121 shutdown_connections 2024-04-14T23:50:55.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:55.841+0000 7f3dd2600640 1 -- 172.21.15.60:0/3785861121 wait complete. 2024-04-14T23:50:55.864 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-14T23:50:57.865 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:50:58.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.094+0000 7f6651600640 1 Processor -- start 2024-04-14T23:50:58.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.094+0000 7f6651600640 1 -- start start 2024-04-14T23:50:58.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.095+0000 7f6651600640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f664c0c4fa0 0x7f664c0c77b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:50:58.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.095+0000 7f6651600640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f664c070d50 0x7f664c0c7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:50:58.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.095+0000 7f6651600640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f664c0c5380 0x7f664c15a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:50:58.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.095+0000 7f6651600640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f664c06b4b0 con 0x7f664c0c5380 2024-04-14T23:50:58.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.095+0000 7f6651600640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f664c06b790 con 0x7f664c0c4fa0 2024-04-14T23:50:58.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.095+0000 7f6651600640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f664c0c69f0 con 0x7f664c070d50 2024-04-14T23:50:58.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.096+0000 7f6643e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f664c070d50 0x7f664c0c7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:50:58.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.096+0000 7f664be00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f664c0c4fa0 0x7f664c0c77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:50:58.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.096+0000 7f6643e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f664c070d50 0x7f664c0c7cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.60:44836/0 (socket says 172.21.15.60:44836) 2024-04-14T23:50:58.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.096+0000 7f664be00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f664c0c4fa0 0x7f664c0c77b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:46422/0 (socket says 172.21.15.60:46422) 2024-04-14T23:50:58.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.096+0000 7f6650c00640 1 --2- 172.21.15.60:0/3356933051 >> v2:172.21.15.60:3300/0 conn(0x7f664c0c5380 0x7f664c15a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:50:58.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.096+0000 7f6643e00640 1 -- 172.21.15.60:0/3356933051 learned_addr learned my addr 172.21.15.60:0/3356933051 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:50:58.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.096+0000 7f664be00640 1 -- 172.21.15.60:0/3356933051 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f664c070d50 msgr2=0x7f664c0c7cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:50:58.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.096+0000 7f664be00640 1 --2- 172.21.15.60:0/3356933051 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f664c070d50 0x7f664c0c7cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:58.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.096+0000 7f664be00640 1 -- 172.21.15.60:0/3356933051 >> v2:172.21.15.60:3300/0 conn(0x7f664c0c5380 msgr2=0x7f664c15a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:50:58.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.096+0000 7f664be00640 1 --2- 172.21.15.60:0/3356933051 >> v2:172.21.15.60:3300/0 conn(0x7f664c0c5380 0x7f664c15a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:58.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.096+0000 7f664be00640 1 -- 172.21.15.60:0/3356933051 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f664c0c6cd0 con 0x7f664c0c4fa0 2024-04-14T23:50:58.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.096+0000 7f6643e00640 1 --2- 172.21.15.60:0/3356933051 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f664c070d50 0x7f664c0c7cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:50:58.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.098+0000 7f664be00640 1 --2- 172.21.15.60:0/3356933051 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f664c0c4fa0 0x7f664c0c77b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f664c0b1ff0 tx=0x7f663c02da50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25c3665667da40e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:50:58.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.098+0000 7f664b400640 1 -- 172.21.15.60:0/3356933051 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f663c031870 con 0x7f664c0c4fa0 2024-04-14T23:50:58.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.098+0000 7f664b400640 1 -- 172.21.15.60:0/3356933051 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f663c031a10 con 0x7f664c0c4fa0 2024-04-14T23:50:58.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.098+0000 7f664b400640 1 -- 172.21.15.60:0/3356933051 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f663c031de0 con 0x7f664c0c4fa0 2024-04-14T23:50:58.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.099+0000 7f6651600640 1 -- 172.21.15.60:0/3356933051 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f664c0c4fa0 msgr2=0x7f664c0c77b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:50:58.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.099+0000 7f6651600640 1 --2- 172.21.15.60:0/3356933051 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f664c0c4fa0 0x7f664c0c77b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f664c0b1ff0 tx=0x7f663c02da50 comp rx=0 tx=0).stop 2024-04-14T23:50:58.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.099+0000 7f6651600640 1 -- 172.21.15.60:0/3356933051 shutdown_connections 2024-04-14T23:50:58.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.099+0000 7f6651600640 1 --2- 172.21.15.60:0/3356933051 >> v2:172.21.15.60:3300/0 conn(0x7f664c0c5380 0x7f664c15a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:58.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.099+0000 7f6651600640 1 --2- 172.21.15.60:0/3356933051 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f664c070d50 0x7f664c0c7cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:58.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.099+0000 7f6651600640 1 --2- 172.21.15.60:0/3356933051 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f664c0c4fa0 0x7f664c0c77b0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:58.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.099+0000 7f6651600640 1 -- 172.21.15.60:0/3356933051 >> 172.21.15.60:0/3356933051 conn(0x7f664c0c8b50 msgr2=0x7f664c0c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:50:58.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.099+0000 7f6651600640 1 -- 172.21.15.60:0/3356933051 shutdown_connections 2024-04-14T23:50:58.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.099+0000 7f6651600640 1 -- 172.21.15.60:0/3356933051 wait complete. 2024-04-14T23:50:58.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.101+0000 7f6651600640 1 Processor -- start 2024-04-14T23:50:58.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.101+0000 7f6651600640 1 -- start start 2024-04-14T23:50:58.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.103+0000 7f6651600640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f664c070d50 0x7f664c0c1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:50:58.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.103+0000 7f6651600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f664c0c4fa0 0x7f664c0c2020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:50:58.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.103+0000 7f6651600640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f664c0c5380 0x7f664c152180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:50:58.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.103+0000 7f6651600640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f664c06b1c0 con 0x7f664c0c5380 2024-04-14T23:50:58.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.103+0000 7f6651600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f664c0c7290 con 0x7f664c0c4fa0 2024-04-14T23:50:58.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.103+0000 7f6651600640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f664c0c99d0 con 0x7f664c070d50 2024-04-14T23:50:58.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.103+0000 7f664be00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f664c070d50 0x7f664c0c1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:50:58.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.103+0000 7f6643e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f664c0c4fa0 0x7f664c0c2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:50:58.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.103+0000 7f664be00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f664c070d50 0x7f664c0c1ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.60:44840/0 (socket says 172.21.15.60:44840) 2024-04-14T23:50:58.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.103+0000 7f664be00640 1 -- 172.21.15.60:0/2167962915 learned_addr learned my addr 172.21.15.60:0/2167962915 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:50:58.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.103+0000 7f6650c00640 1 --2- 172.21.15.60:0/2167962915 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f664c0c5380 0x7f664c152180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:50:58.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.103+0000 7f6643e00640 1 -- 172.21.15.60:0/2167962915 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f664c070d50 msgr2=0x7f664c0c1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:50:58.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.103+0000 7f6643e00640 1 --2- 172.21.15.60:0/2167962915 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f664c070d50 0x7f664c0c1ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:58.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.103+0000 7f6643e00640 1 -- 172.21.15.60:0/2167962915 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f664c0c5380 msgr2=0x7f664c152180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:50:58.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.103+0000 7f6643e00640 1 --2- 172.21.15.60:0/2167962915 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f664c0c5380 0x7f664c152180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:58.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.103+0000 7f6643e00640 1 -- 172.21.15.60:0/2167962915 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f664c1ecdb0 con 0x7f664c0c4fa0 2024-04-14T23:50:58.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.104+0000 7f6643e00640 1 --2- 172.21.15.60:0/2167962915 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f664c0c4fa0 0x7f664c0c2020 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f664c0cd020 tx=0x7f663400ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:50:58.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.104+0000 7f6648c00640 1 -- 172.21.15.60:0/2167962915 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6634017070 con 0x7f664c0c4fa0 2024-04-14T23:50:58.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.104+0000 7f6648c00640 1 -- 172.21.15.60:0/2167962915 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f663400c940 con 0x7f664c0c4fa0 2024-04-14T23:50:58.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.104+0000 7f6648c00640 1 -- 172.21.15.60:0/2167962915 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f663400cd30 con 0x7f664c0c4fa0 2024-04-14T23:50:58.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.104+0000 7f6651600640 1 -- 172.21.15.60:0/2167962915 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f664c0aa4c0 con 0x7f664c0c4fa0 2024-04-14T23:50:58.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.104+0000 7f6651600640 1 -- 172.21.15.60:0/2167962915 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f664c0aa780 con 0x7f664c0c4fa0 2024-04-14T23:50:58.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.105+0000 7f6651600640 1 -- 172.21.15.60:0/2167962915 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f664c04f540 con 0x7f664c0c4fa0 2024-04-14T23:50:58.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.106+0000 7f6648c00640 1 -- 172.21.15.60:0/2167962915 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f663401a070 con 0x7f664c0c4fa0 2024-04-14T23:50:58.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.107+0000 7f6648c00640 1 --2- 172.21.15.60:0/2167962915 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f662806b990 0x7f662806de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:50:58.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.107+0000 7f6648c00640 1 -- 172.21.15.60:0/2167962915 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5853+0+0 (secure 0 0 0) 0x7f663408a690 con 0x7f664c0c4fa0 2024-04-14T23:50:58.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.107+0000 7f664be00640 1 --2- 172.21.15.60:0/2167962915 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f662806b990 0x7f662806de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:50:58.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.108+0000 7f664be00640 1 --2- 172.21.15.60:0/2167962915 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f662806b990 0x7f662806de50 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f663c009560 tx=0x7f663c008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:50:58.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.112+0000 7f6648c00640 1 -- 172.21.15.60:0/2167962915 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6634058b20 con 0x7f664c0c4fa0 2024-04-14T23:50:58.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.539+0000 7f6651600640 1 -- 172.21.15.60:0/2167962915 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f664c0cbab0 con 0x7f664c0c4fa0 2024-04-14T23:50:58.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.540+0000 7f6648c00640 1 -- 172.21.15.60:0/2167962915 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6827 (secure 0 0 0) 0x7f663405cbc0 con 0x7f664c0c4fa0 2024-04-14T23:50:58.545 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:50:58.545 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/78303888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":78303888},{"type":"v1","addr":"172.21.15.60:6833","nonce":78303888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:48:49.256965+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.60:6835/2677133122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2677133122},{"type":"v1","addr":"172.21.15.60:6835","nonce":2677133122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4279,"qdb_cluster":[4279]},"id":1}]} 2024-04-14T23:50:58.545 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 8 2024-04-14T23:50:58.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.542+0000 7f6651600640 1 -- 172.21.15.60:0/2167962915 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f662806b990 msgr2=0x7f662806de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:50:58.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.542+0000 7f6651600640 1 --2- 172.21.15.60:0/2167962915 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f662806b990 0x7f662806de50 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f663c009560 tx=0x7f663c008030 comp rx=0 tx=0).stop 2024-04-14T23:50:58.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.542+0000 7f6651600640 1 -- 172.21.15.60:0/2167962915 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f664c0c4fa0 msgr2=0x7f664c0c2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:50:58.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.542+0000 7f6651600640 1 --2- 172.21.15.60:0/2167962915 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f664c0c4fa0 0x7f664c0c2020 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f664c0cd020 tx=0x7f663400ecd0 comp rx=0 tx=0).stop 2024-04-14T23:50:58.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.543+0000 7f6651600640 1 -- 172.21.15.60:0/2167962915 shutdown_connections 2024-04-14T23:50:58.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.543+0000 7f6651600640 1 --2- 172.21.15.60:0/2167962915 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f662806b990 0x7f662806de50 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:58.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.543+0000 7f6651600640 1 --2- 172.21.15.60:0/2167962915 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f664c0c5380 0x7f664c152180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:58.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.543+0000 7f6651600640 1 --2- 172.21.15.60:0/2167962915 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f664c0c4fa0 0x7f664c0c2020 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:58.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.543+0000 7f6651600640 1 --2- 172.21.15.60:0/2167962915 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f664c070d50 0x7f664c0c1ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:50:58.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.543+0000 7f6651600640 1 -- 172.21.15.60:0/2167962915 >> 172.21.15.60:0/2167962915 conn(0x7f664c0c8b50 msgr2=0x7f664c0c83b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:50:58.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.543+0000 7f6651600640 1 -- 172.21.15.60:0/2167962915 shutdown_connections 2024-04-14T23:50:58.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:50:58.543+0000 7f6651600640 1 -- 172.21.15.60:0/2167962915 wait complete. 2024-04-14T23:50:58.572 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-14T23:51:00.573 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:51:00.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:00.801+0000 7f12f5c00640 1 Processor -- start 2024-04-14T23:51:00.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:00.801+0000 7f12f5c00640 1 -- start start 2024-04-14T23:51:00.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:00.803+0000 7f12f5c00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12f00c8df0 0x7f12f00c93d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:00.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:00.803+0000 7f12f5c00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f12f00c9910 0x7f12f00c9d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:00.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:00.803+0000 7f12f5c00640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12f01530a0 con 0x7f12f0152cc0 2024-04-14T23:51:00.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:00.803+0000 7f12f5c00640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f12f00c4f80 con 0x7f12f00c9910 2024-04-14T23:51:00.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:00.803+0000 7f12f5c00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f12f00717b0 con 0x7f12f00c8df0 2024-04-14T23:51:00.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:00.803+0000 7f12efe00640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f12f0152cc0 0x7f12f0070270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.60:58094/0 (socket says 172.21.15.60:58094) 2024-04-14T23:51:00.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:00.803+0000 7f12ef400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12f00c8df0 0x7f12f00c93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:00.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:00.803+0000 7f12efe00640 1 -- 172.21.15.60:0/1929231655 learned_addr learned my addr 172.21.15.60:0/1929231655 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:51:00.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:00.803+0000 7f12f5200640 1 --2- 172.21.15.60:0/1929231655 >> v2:172.21.15.60:3300/0 conn(0x7f12f00c9910 0x7f12f00c9d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:00.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:00.803+0000 7f12eea00640 1 -- 172.21.15.60:0/1929231655 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1496797896 0 0) 0x7f12f01530a0 con 0x7f12f0152cc0 2024-04-14T23:51:00.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:00.804+0000 7f12ef400640 1 -- 172.21.15.60:0/1929231655 >> v2:172.21.15.60:3300/0 conn(0x7f12f00c9910 msgr2=0x7f12f00c9d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:51:00.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:00.804+0000 7f12ef400640 1 --2- 172.21.15.60:0/1929231655 >> v2:172.21.15.60:3300/0 conn(0x7f12f00c9910 0x7f12f00c9d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:00.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:00.804+0000 7f12ef400640 1 -- 172.21.15.60:0/1929231655 >> v1:172.21.15.186:6789/0 conn(0x7f12f0152cc0 legacy=0x7f12f0070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:00.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:00.804+0000 7f12ef400640 1 -- 172.21.15.60:0/1929231655 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12f01595f0 con 0x7f12f00c8df0 2024-04-14T23:51:00.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:00.805+0000 7f12ef400640 1 --2- 172.21.15.60:0/1929231655 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12f00c8df0 0x7f12f00c93d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f12dc0097a0 tx=0x7f12dc02b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1264eb23757f7748 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:51:00.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:00.805+0000 7f12eea00640 1 -- 172.21.15.60:0/1929231655 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12dc034070 con 0x7f12f00c8df0 2024-04-14T23:51:00.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:00.805+0000 7f12eea00640 1 -- 172.21.15.60:0/1929231655 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12dc030800 con 0x7f12f00c8df0 2024-04-14T23:51:00.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:00.805+0000 7f12eea00640 1 -- 172.21.15.60:0/1929231655 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12dc030bd0 con 0x7f12f00c8df0 2024-04-14T23:51:00.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:00.806+0000 7f12f5c00640 1 -- 172.21.15.60:0/1929231655 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12f00c8df0 msgr2=0x7f12f00c93d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:00.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:00.806+0000 7f12f5c00640 1 --2- 172.21.15.60:0/1929231655 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12f00c8df0 0x7f12f00c93d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f12dc0097a0 tx=0x7f12dc02b940 comp rx=0 tx=0).stop 2024-04-14T23:51:00.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:00.806+0000 7f12f5c00640 1 -- 172.21.15.60:0/1929231655 shutdown_connections 2024-04-14T23:51:00.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:00.806+0000 7f12f5c00640 1 --2- 172.21.15.60:0/1929231655 >> v2:172.21.15.60:3300/0 conn(0x7f12f00c9910 0x7f12f00c9d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:00.812 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:00.806+0000 7f12f5c00640 1 --2- 172.21.15.60:0/1929231655 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12f00c8df0 0x7f12f00c93d0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:00.812 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:00.806+0000 7f12f5c00640 1 -- 172.21.15.60:0/1929231655 >> 172.21.15.60:0/1929231655 conn(0x7f12f0066f10 msgr2=0x7f12f0067320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:51:00.812 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:00.806+0000 7f12f5c00640 1 -- 172.21.15.60:0/1929231655 shutdown_connections 2024-04-14T23:51:00.812 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:00.806+0000 7f12f5c00640 1 -- 172.21.15.60:0/1929231655 wait complete. 2024-04-14T23:51:00.812 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:00.808+0000 7f12f5c00640 1 Processor -- start 2024-04-14T23:51:00.812 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:00.808+0000 7f12f5c00640 1 -- start start 2024-04-14T23:51:00.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:00.809+0000 7f12f5c00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12f00c8df0 0x7f12f015b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:00.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:00.809+0000 7f12f5c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12f00c9910 0x7f12f01f0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:00.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:00.809+0000 7f12f5c00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12f0152cc0 0x7f12f015c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:00.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:00.809+0000 7f12f5c00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f12f006f900 con 0x7f12f0152cc0 2024-04-14T23:51:00.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:00.809+0000 7f12f5c00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f12f01598d0 con 0x7f12f00c9910 2024-04-14T23:51:00.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:00.809+0000 7f12f5c00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f12f00b0e70 con 0x7f12f00c8df0 2024-04-14T23:51:00.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:00.809+0000 7f12ef400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12f00c9910 0x7f12f01f0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:00.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:00.809+0000 7f12f5200640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12f0152cc0 0x7f12f015c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:00.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:00.809+0000 7f12efe00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12f00c8df0 0x7f12f015b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:00.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:00.809+0000 7f12ef400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12f00c9910 0x7f12f01f0bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:46442/0 (socket says 172.21.15.60:46442) 2024-04-14T23:51:00.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:00.809+0000 7f12ef400640 1 -- 172.21.15.60:0/2047569060 learned_addr learned my addr 172.21.15.60:0/2047569060 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:51:00.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:00.810+0000 7f12ef400640 1 -- 172.21.15.60:0/2047569060 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12f00c8df0 msgr2=0x7f12f015b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:00.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:00.810+0000 7f12ef400640 1 --2- 172.21.15.60:0/2047569060 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12f00c8df0 0x7f12f015b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:00.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:00.810+0000 7f12ef400640 1 -- 172.21.15.60:0/2047569060 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12f0152cc0 msgr2=0x7f12f015c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:00.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:00.810+0000 7f12ef400640 1 --2- 172.21.15.60:0/2047569060 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12f0152cc0 0x7f12f015c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:00.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:00.810+0000 7f12ef400640 1 -- 172.21.15.60:0/2047569060 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12f006c500 con 0x7f12f00c9910 2024-04-14T23:51:00.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:00.810+0000 7f12f5200640 1 --2- 172.21.15.60:0/2047569060 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12f0152cc0 0x7f12f015c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:51:00.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:00.810+0000 7f12ef400640 1 --2- 172.21.15.60:0/2047569060 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12f00c9910 0x7f12f01f0bc0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f12dc02bf50 tx=0x7f12dc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:51:00.817 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:00.810+0000 7f12e7e00640 1 -- 172.21.15.60:0/2047569060 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12dc034040 con 0x7f12f00c9910 2024-04-14T23:51:00.817 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:00.811+0000 7f12e7e00640 1 -- 172.21.15.60:0/2047569060 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12dc030600 con 0x7f12f00c9910 2024-04-14T23:51:00.817 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:00.811+0000 7f12e7e00640 1 -- 172.21.15.60:0/2047569060 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12dc006e10 con 0x7f12f00c9910 2024-04-14T23:51:00.817 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:00.811+0000 7f12f5c00640 1 -- 172.21.15.60:0/2047569060 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12f01f4150 con 0x7f12f00c9910 2024-04-14T23:51:00.817 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:00.811+0000 7f12f5c00640 1 -- 172.21.15.60:0/2047569060 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12f01f43f0 con 0x7f12f00c9910 2024-04-14T23:51:00.817 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:00.812+0000 7f12f5c00640 1 -- 172.21.15.60:0/2047569060 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12f004f540 con 0x7f12f00c9910 2024-04-14T23:51:00.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:00.813+0000 7f12e7e00640 1 -- 172.21.15.60:0/2047569060 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f12dc037070 con 0x7f12f00c9910 2024-04-14T23:51:00.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:00.814+0000 7f12e7e00640 1 --2- 172.21.15.60:0/2047569060 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f12d006b9c0 0x7f12d006de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:00.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:00.814+0000 7f12e7e00640 1 -- 172.21.15.60:0/2047569060 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 5911+0+0 (secure 0 0 0) 0x7f12dc0acff0 con 0x7f12f00c9910 2024-04-14T23:51:00.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:00.814+0000 7f12efe00640 1 --2- 172.21.15.60:0/2047569060 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f12d006b9c0 0x7f12d006de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:00.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:00.815+0000 7f12efe00640 1 --2- 172.21.15.60:0/2047569060 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f12d006b9c0 0x7f12d006de80 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f12e0000f60 tx=0x7f12e0007e40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:51:00.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:00.818+0000 7f12e7e00640 1 -- 172.21.15.60:0/2047569060 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f12dc07b550 con 0x7f12f00c9910 2024-04-14T23:51:01.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.059+0000 7f12f5c00640 1 -- 172.21.15.60:0/2047569060 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f12f0070050 con 0x7f12f00c9910 2024-04-14T23:51:01.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.060+0000 7f12e7e00640 1 -- 172.21.15.60:0/2047569060 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5997 (secure 0 0 0) 0x7f12dc07f5f0 con 0x7f12f00c9910 2024-04-14T23:51:01.065 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:51:01.065 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:50:58.881832+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4276},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4276":{"gid":4276,"name":"a","rank":0,"incarnation":9,"state":"up:replay","state_seq":1,"addr":"172.21.15.60:6833/78303888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":78303888},{"type":"v1","addr":"172.21.15.60:6833","nonce":78303888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-14T23:51:01.065 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 9 2024-04-14T23:51:01.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.062+0000 7f12f5c00640 1 -- 172.21.15.60:0/2047569060 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f12d006b9c0 msgr2=0x7f12d006de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:01.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.062+0000 7f12f5c00640 1 --2- 172.21.15.60:0/2047569060 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f12d006b9c0 0x7f12d006de80 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f12e0000f60 tx=0x7f12e0007e40 comp rx=0 tx=0).stop 2024-04-14T23:51:01.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.062+0000 7f12f5c00640 1 -- 172.21.15.60:0/2047569060 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12f00c9910 msgr2=0x7f12f01f0bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:01.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.062+0000 7f12f5c00640 1 --2- 172.21.15.60:0/2047569060 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12f00c9910 0x7f12f01f0bc0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f12dc02bf50 tx=0x7f12dc0046c0 comp rx=0 tx=0).stop 2024-04-14T23:51:01.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.063+0000 7f12f5c00640 1 -- 172.21.15.60:0/2047569060 shutdown_connections 2024-04-14T23:51:01.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.063+0000 7f12f5c00640 1 --2- 172.21.15.60:0/2047569060 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f12d006b9c0 0x7f12d006de80 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:01.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.063+0000 7f12f5c00640 1 --2- 172.21.15.60:0/2047569060 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12f0152cc0 0x7f12f015c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:01.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.063+0000 7f12f5c00640 1 --2- 172.21.15.60:0/2047569060 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12f00c9910 0x7f12f01f0bc0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:01.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.063+0000 7f12f5c00640 1 --2- 172.21.15.60:0/2047569060 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12f00c8df0 0x7f12f015b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:01.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.063+0000 7f12f5c00640 1 -- 172.21.15.60:0/2047569060 >> 172.21.15.60:0/2047569060 conn(0x7f12f0066f10 msgr2=0x7f12f00672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:51:01.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.063+0000 7f12f5c00640 1 -- 172.21.15.60:0/2047569060 shutdown_connections 2024-04-14T23:51:01.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.063+0000 7f12f5c00640 1 -- 172.21.15.60:0/2047569060 wait complete. 2024-04-14T23:51:01.089 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c down, removed from mdsmap 2024-04-14T23:51:01.089 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-04-14T23:51:01.089 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:51:01.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.304+0000 7f2c6be00640 1 Processor -- start 2024-04-14T23:51:01.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.305+0000 7f2c6be00640 1 -- start start 2024-04-14T23:51:01.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.306+0000 7f2c6be00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2c64152bf0 0x7f2c6406eba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:01.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.306+0000 7f2c6be00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2c6406fa80 0x7f2c6406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:01.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.306+0000 7f2c6be00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2c6415beb0 0x7f2c6415c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:01.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.306+0000 7f2c6be00640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f2c640c9eb0 con 0x7f2c6406fa80 2024-04-14T23:51:01.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.306+0000 7f2c6be00640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f2c640cb500 con 0x7f2c64152bf0 2024-04-14T23:51:01.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.306+0000 7f2c6be00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f2c640cb7c0 con 0x7f2c6415beb0 2024-04-14T23:51:01.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.307+0000 7f2c69400640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2c64152bf0 0x7f2c6406eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:01.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.307+0000 7f2c68a00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2c6406fa80 0x7f2c6406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:01.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.307+0000 7f2c69e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2c6415beb0 0x7f2c6415c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:01.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.307+0000 7f2c69400640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2c64152bf0 0x7f2c6406eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:46452/0 (socket says 172.21.15.60:46452) 2024-04-14T23:51:01.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.307+0000 7f2c69400640 1 -- 172.21.15.60:0/1697310344 learned_addr learned my addr 172.21.15.60:0/1697310344 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:51:01.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.307+0000 7f2c69400640 1 -- 172.21.15.60:0/1697310344 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2c6415beb0 msgr2=0x7f2c6415c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:51:01.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.307+0000 7f2c69400640 1 --2- 172.21.15.60:0/1697310344 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2c6415beb0 0x7f2c6415c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:01.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.307+0000 7f2c69400640 1 -- 172.21.15.60:0/1697310344 >> v2:172.21.15.60:3300/0 conn(0x7f2c6406fa80 msgr2=0x7f2c6406f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:51:01.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.307+0000 7f2c69400640 1 --2- 172.21.15.60:0/1697310344 >> v2:172.21.15.60:3300/0 conn(0x7f2c6406fa80 0x7f2c6406f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:01.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.307+0000 7f2c69400640 1 -- 172.21.15.60:0/1697310344 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c640c98f0 con 0x7f2c64152bf0 2024-04-14T23:51:01.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.308+0000 7f2c69400640 1 --2- 172.21.15.60:0/1697310344 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c64152bf0 0x7f2c6406eba0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2c64071a90 tx=0x7f2c5402bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9218c414e781ce79 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:51:01.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.309+0000 7f2c63e00640 1 -- 172.21.15.60:0/1697310344 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c5402ca70 con 0x7f2c64152bf0 2024-04-14T23:51:01.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.309+0000 7f2c63e00640 1 -- 172.21.15.60:0/1697310344 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c5402cc10 con 0x7f2c64152bf0 2024-04-14T23:51:01.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.309+0000 7f2c63e00640 1 -- 172.21.15.60:0/1697310344 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c540343f0 con 0x7f2c64152bf0 2024-04-14T23:51:01.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.309+0000 7f2c6be00640 1 -- 172.21.15.60:0/1697310344 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c64152bf0 msgr2=0x7f2c6406eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:01.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.309+0000 7f2c6be00640 1 --2- 172.21.15.60:0/1697310344 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c64152bf0 0x7f2c6406eba0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2c64071a90 tx=0x7f2c5402bed0 comp rx=0 tx=0).stop 2024-04-14T23:51:01.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.309+0000 7f2c6be00640 1 -- 172.21.15.60:0/1697310344 shutdown_connections 2024-04-14T23:51:01.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.309+0000 7f2c6be00640 1 --2- 172.21.15.60:0/1697310344 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2c6415beb0 0x7f2c6415c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:01.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.309+0000 7f2c6be00640 1 --2- 172.21.15.60:0/1697310344 >> v2:172.21.15.60:3300/0 conn(0x7f2c6406fa80 0x7f2c6406f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:01.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.309+0000 7f2c6be00640 1 --2- 172.21.15.60:0/1697310344 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c64152bf0 0x7f2c6406eba0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:01.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.309+0000 7f2c6be00640 1 -- 172.21.15.60:0/1697310344 >> 172.21.15.60:0/1697310344 conn(0x7f2c64070d20 msgr2=0x7f2c64071100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:51:01.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.310+0000 7f2c6be00640 1 -- 172.21.15.60:0/1697310344 shutdown_connections 2024-04-14T23:51:01.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.310+0000 7f2c6be00640 1 -- 172.21.15.60:0/1697310344 wait complete. 2024-04-14T23:51:01.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.311+0000 7f2c6be00640 1 Processor -- start 2024-04-14T23:51:01.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.311+0000 7f2c6be00640 1 -- start start 2024-04-14T23:51:01.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.312+0000 7f2c6be00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c6406fa80 0x7f2c641f26d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:01.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.312+0000 7f2c6be00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2c64152bf0 0x7f2c640b3340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:01.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.312+0000 7f2c6be00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c6415beb0 0x7f2c6414e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:01.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.312+0000 7f2c6be00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f2c64152fd0 con 0x7f2c6415beb0 2024-04-14T23:51:01.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.312+0000 7f2c6be00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2c640c9030 con 0x7f2c6406fa80 2024-04-14T23:51:01.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.312+0000 7f2c6be00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f2c6415c860 con 0x7f2c64152bf0 2024-04-14T23:51:01.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.313+0000 7f2c69400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c6406fa80 0x7f2c641f26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:01.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.313+0000 7f2c68a00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2c64152bf0 0x7f2c640b3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:01.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.313+0000 7f2c69e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c6415beb0 0x7f2c6414e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:01.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.313+0000 7f2c68a00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2c64152bf0 0x7f2c640b3340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.60:44874/0 (socket says 172.21.15.60:44874) 2024-04-14T23:51:01.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.313+0000 7f2c69400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c6406fa80 0x7f2c641f26d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:46460/0 (socket says 172.21.15.60:46460) 2024-04-14T23:51:01.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.313+0000 7f2c68a00640 1 -- 172.21.15.60:0/1738282249 learned_addr learned my addr 172.21.15.60:0/1738282249 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:51:01.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.313+0000 7f2c69e00640 1 -- 172.21.15.60:0/1738282249 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2c64152bf0 msgr2=0x7f2c640b3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:01.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.313+0000 7f2c69e00640 1 --2- 172.21.15.60:0/1738282249 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2c64152bf0 0x7f2c640b3340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:01.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.313+0000 7f2c69e00640 1 -- 172.21.15.60:0/1738282249 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c6406fa80 msgr2=0x7f2c641f26d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:01.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.313+0000 7f2c69e00640 1 --2- 172.21.15.60:0/1738282249 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c6406fa80 0x7f2c641f26d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:01.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.313+0000 7f2c69e00640 1 -- 172.21.15.60:0/1738282249 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c641f2c10 con 0x7f2c6415beb0 2024-04-14T23:51:01.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.313+0000 7f2c69400640 1 --2- 172.21.15.60:0/1738282249 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c6406fa80 0x7f2c641f26d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:51:01.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.314+0000 7f2c69e00640 1 --2- 172.21.15.60:0/1738282249 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c6415beb0 0x7f2c6414e420 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f2c5c00e7f0 tx=0x7f2c5c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:51:01.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.314+0000 7f2c61600640 1 -- 172.21.15.60:0/1738282249 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c5c017070 con 0x7f2c6415beb0 2024-04-14T23:51:01.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.314+0000 7f2c61600640 1 -- 172.21.15.60:0/1738282249 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c5c00c730 con 0x7f2c6415beb0 2024-04-14T23:51:01.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.314+0000 7f2c61600640 1 -- 172.21.15.60:0/1738282249 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c5c00cb60 con 0x7f2c6415beb0 2024-04-14T23:51:01.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.314+0000 7f2c6be00640 1 -- 172.21.15.60:0/1738282249 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c640b3bd0 con 0x7f2c6415beb0 2024-04-14T23:51:01.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.314+0000 7f2c6be00640 1 -- 172.21.15.60:0/1738282249 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c640b1fa0 con 0x7f2c6415beb0 2024-04-14T23:51:01.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.315+0000 7f2c6be00640 1 -- 172.21.15.60:0/1738282249 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c640684b0 con 0x7f2c6415beb0 2024-04-14T23:51:01.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.316+0000 7f2c61600640 1 -- 172.21.15.60:0/1738282249 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f2c5c01a070 con 0x7f2c6415beb0 2024-04-14T23:51:01.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.318+0000 7f2c61600640 1 --2- 172.21.15.60:0/1738282249 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f2c4406b990 0x7f2c4406de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:01.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.318+0000 7f2c61600640 1 -- 172.21.15.60:0/1738282249 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 5911+0+0 (secure 0 0 0) 0x7f2c5c08a360 con 0x7f2c6415beb0 2024-04-14T23:51:01.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.318+0000 7f2c69400640 1 --2- 172.21.15.60:0/1738282249 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f2c4406b990 0x7f2c4406de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:01.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.319+0000 7f2c69400640 1 --2- 172.21.15.60:0/1738282249 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f2c4406b990 0x7f2c4406de50 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f2c54035000 tx=0x7f2c54030000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:51:01.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.322+0000 7f2c61600640 1 -- 172.21.15.60:0/1738282249 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2c5c058720 con 0x7f2c6415beb0 2024-04-14T23:51:01.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.560+0000 7f2c6be00640 1 -- 172.21.15.60:0/1738282249 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2c640cd550 con 0x7f2c6415beb0 2024-04-14T23:51:01.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.561+0000 7f2c61600640 1 -- 172.21.15.60:0/1738282249 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5997 (secure 0 0 0) 0x7f2c5c05c7c0 con 0x7f2c6415beb0 2024-04-14T23:51:01.567 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:51:01.567 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:50:58.881832+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4276},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4276":{"gid":4276,"name":"a","rank":0,"incarnation":9,"state":"up:replay","state_seq":1,"addr":"172.21.15.60:6833/78303888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":78303888},{"type":"v1","addr":"172.21.15.60:6833","nonce":78303888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-14T23:51:01.567 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 9 2024-04-14T23:51:01.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.564+0000 7f2c6be00640 1 -- 172.21.15.60:0/1738282249 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f2c4406b990 msgr2=0x7f2c4406de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:01.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.564+0000 7f2c6be00640 1 --2- 172.21.15.60:0/1738282249 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f2c4406b990 0x7f2c4406de50 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f2c54035000 tx=0x7f2c54030000 comp rx=0 tx=0).stop 2024-04-14T23:51:01.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.564+0000 7f2c6be00640 1 -- 172.21.15.60:0/1738282249 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c6415beb0 msgr2=0x7f2c6414e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:01.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.564+0000 7f2c6be00640 1 --2- 172.21.15.60:0/1738282249 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c6415beb0 0x7f2c6414e420 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f2c5c00e7f0 tx=0x7f2c5c00ecb0 comp rx=0 tx=0).stop 2024-04-14T23:51:01.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.564+0000 7f2c6be00640 1 -- 172.21.15.60:0/1738282249 shutdown_connections 2024-04-14T23:51:01.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.564+0000 7f2c6be00640 1 --2- 172.21.15.60:0/1738282249 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f2c4406b990 0x7f2c4406de50 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:01.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.564+0000 7f2c6be00640 1 --2- 172.21.15.60:0/1738282249 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c6415beb0 0x7f2c6414e420 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:01.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.564+0000 7f2c6be00640 1 --2- 172.21.15.60:0/1738282249 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2c64152bf0 0x7f2c640b3340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:01.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.564+0000 7f2c6be00640 1 --2- 172.21.15.60:0/1738282249 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c6406fa80 0x7f2c641f26d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:01.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.564+0000 7f2c6be00640 1 -- 172.21.15.60:0/1738282249 >> 172.21.15.60:0/1738282249 conn(0x7f2c64070d20 msgr2=0x7f2c64071100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:51:01.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.564+0000 7f2c6be00640 1 -- 172.21.15.60:0/1738282249 shutdown_connections 2024-04-14T23:51:01.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:01.564+0000 7f2c6be00640 1 -- 172.21.15.60:0/1738282249 wait complete. 2024-04-14T23:51:01.588 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 9, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-14T23:48:42.700188+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 9, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4276": { "addr": "172.21.15.60:6833/78303888", "addrs": { "addrvec": [ { "addr": "172.21.15.60:6832", "nonce": 78303888, "type": "v2" }, { "addr": "172.21.15.60:6833", "nonce": 78303888, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4276, "incarnation": 9, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 26, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-14T23:50:58.881832+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4276 } } } ], "standbys": [ { "addr": "172.21.15.60:6837/1527815634", "addrs": { "addrvec": [ { "addr": "172.21.15.60:6836", "nonce": 1527815634, "type": "v2" }, { "addr": "172.21.15.60:6837", "nonce": 1527815634, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4197, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6838/3958319486", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6835", "nonce": 3958319486, "type": "v2" }, { "addr": "172.21.15.186:6838", "nonce": 3958319486, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4220, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6837/3910951185", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6834", "nonce": 3910951185, "type": "v2" }, { "addr": "172.21.15.186:6837", "nonce": 3910951185, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4264, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6839/2155020404", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6836", "nonce": 2155020404, "type": "v2" }, { "addr": "172.21.15.186:6839", "nonce": 2155020404, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4267, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-14T23:51:03.589 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:51:03.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.876+0000 7f6dbf200640 1 Processor -- start 2024-04-14T23:51:03.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.876+0000 7f6dbf200640 1 -- start start 2024-04-14T23:51:03.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.879+0000 7f6dbf200640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6db80cc460 0x7f6db80d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:03.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.879+0000 7f6dbf200640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6db8159910 con 0x7f6db80d09e0 2024-04-14T23:51:03.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.879+0000 7f6dbf200640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6db80ccfc0 con 0x7f6db8159530 2024-04-14T23:51:03.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.879+0000 7f6dbf200640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f6db80c5b00 con 0x7f6db80cc460 2024-04-14T23:51:03.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.879+0000 7f6db7e00640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f6db8159530 0x7f6db80c53f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.60:47830/0 (socket says 172.21.15.60:47830) 2024-04-14T23:51:03.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.879+0000 7f6db7e00640 1 -- 172.21.15.60:0/1357616498 learned_addr learned my addr 172.21.15.60:0/1357616498 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:51:03.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.880+0000 7f6dafe00640 1 --2- 172.21.15.60:0/1357616498 >> v2:172.21.15.186:3300/0 conn(0x7f6db80cc460 0x7f6db80d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:03.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.880+0000 7f6db7400640 1 -- 172.21.15.60:0/1357616498 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2889402691 0 0) 0x7f6db80ccfc0 con 0x7f6db8159530 2024-04-14T23:51:03.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.880+0000 7f6db7400640 1 -- 172.21.15.60:0/1357616498 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6d9c003520 con 0x7f6db8159530 2024-04-14T23:51:03.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.881+0000 7f6db7400640 1 -- 172.21.15.60:0/1357616498 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 828437810 0 0) 0x7f6d9c003520 con 0x7f6db8159530 2024-04-14T23:51:03.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.881+0000 7f6db7400640 1 -- 172.21.15.60:0/1357616498 >> v2:172.21.15.186:3300/0 conn(0x7f6db80cc460 msgr2=0x7f6db80d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:51:03.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.881+0000 7f6db7400640 1 --2- 172.21.15.60:0/1357616498 >> v2:172.21.15.186:3300/0 conn(0x7f6db80cc460 0x7f6db80d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:03.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.881+0000 7f6db7400640 1 -- 172.21.15.60:0/1357616498 >> v1:172.21.15.60:6789/0 conn(0x7f6db80d09e0 legacy=0x7f6db815f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:03.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.882+0000 7f6db7400640 1 -- 172.21.15.60:0/1357616498 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6db815fd50 con 0x7f6db8159530 2024-04-14T23:51:03.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.882+0000 7f6db7400640 1 -- 172.21.15.60:0/1357616498 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2814038680 0 0) 0x7f6da8001670 con 0x7f6db8159530 2024-04-14T23:51:03.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.882+0000 7f6db7400640 1 -- 172.21.15.60:0/1357616498 >> v1:172.21.15.186:6789/0 conn(0x7f6db8159530 legacy=0x7f6db80c53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:03.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.882+0000 7f6db7400640 1 --2- 172.21.15.60:0/1357616498 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d9c0039b0 0x7f6d9c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:03.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.882+0000 7f6db7400640 1 --2- 172.21.15.60:0/1357616498 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6d9c004800 0x7f6d9c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:03.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.882+0000 7f6db7400640 1 --2- 172.21.15.60:0/1357616498 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6d9c0051c0 0x7f6d9c01c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:03.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.882+0000 7f6db7400640 1 -- 172.21.15.60:0/1357616498 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f6db80c5b00 con 0x7f6d9c0039b0 2024-04-14T23:51:03.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.882+0000 7f6db7400640 1 -- 172.21.15.60:0/1357616498 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6d9c01cf90 con 0x7f6d9c0051c0 2024-04-14T23:51:03.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.882+0000 7f6db7400640 1 -- 172.21.15.60:0/1357616498 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f6d9c01d190 con 0x7f6d9c004800 2024-04-14T23:51:03.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.883+0000 7f6db7e00640 1 --2- 172.21.15.60:0/1357616498 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d9c0039b0 0x7f6d9c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:03.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.884+0000 7f6dbd200640 1 --2- 172.21.15.60:0/1357616498 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6d9c0051c0 0x7f6d9c01c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:03.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.884+0000 7f6dafe00640 1 --2- 172.21.15.60:0/1357616498 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6d9c004800 0x7f6d9c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:03.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.885+0000 7f6dbd200640 1 -- 172.21.15.60:0/1357616498 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6d9c004800 msgr2=0x7f6d9c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:51:03.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.885+0000 7f6dbd200640 1 --2- 172.21.15.60:0/1357616498 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6d9c004800 0x7f6d9c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:03.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.885+0000 7f6dbd200640 1 -- 172.21.15.60:0/1357616498 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d9c0039b0 msgr2=0x7f6d9c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:51:03.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.885+0000 7f6dbd200640 1 --2- 172.21.15.60:0/1357616498 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d9c0039b0 0x7f6d9c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:03.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.885+0000 7f6dbd200640 1 -- 172.21.15.60:0/1357616498 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d9c003520 con 0x7f6d9c0051c0 2024-04-14T23:51:03.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.885+0000 7f6db7e00640 1 --2- 172.21.15.60:0/1357616498 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d9c0039b0 0x7f6d9c003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-14T23:51:03.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.887+0000 7f6dbd200640 1 -- 172.21.15.60:0/1357616498 reap_dead start 2024-04-14T23:51:03.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.887+0000 7f6dbd200640 1 --2- 172.21.15.60:0/1357616498 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6d9c0051c0 0x7f6d9c01c9d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6db0001f60 tx=0x7f6d9c005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a08d3bd62e747fbc server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:51:03.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.888+0000 7f6db7400640 1 -- 172.21.15.60:0/1357616498 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6db0001b30 con 0x7f6d9c0051c0 2024-04-14T23:51:03.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.888+0000 7f6db7400640 1 -- 172.21.15.60:0/1357616498 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6db0003b90 con 0x7f6d9c0051c0 2024-04-14T23:51:03.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.888+0000 7f6db7400640 1 -- 172.21.15.60:0/1357616498 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6db0005070 con 0x7f6d9c0051c0 2024-04-14T23:51:03.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.889+0000 7f6dbf200640 1 -- 172.21.15.60:0/1357616498 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6d9c0051c0 msgr2=0x7f6d9c01c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:03.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.889+0000 7f6dbf200640 1 --2- 172.21.15.60:0/1357616498 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6d9c0051c0 0x7f6d9c01c9d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6db0001f60 tx=0x7f6d9c005830 comp rx=0 tx=0).stop 2024-04-14T23:51:03.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.889+0000 7f6dbf200640 1 -- 172.21.15.60:0/1357616498 shutdown_connections 2024-04-14T23:51:03.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.889+0000 7f6dbf200640 1 --2- 172.21.15.60:0/1357616498 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6d9c0051c0 0x7f6d9c01c9d0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:03.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.889+0000 7f6dbf200640 1 -- 172.21.15.60:0/1357616498 >> 172.21.15.60:0/1357616498 conn(0x7f6db806bfd0 msgr2=0x7f6db806cb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:51:03.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.889+0000 7f6dbf200640 1 -- 172.21.15.60:0/1357616498 shutdown_connections 2024-04-14T23:51:03.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.889+0000 7f6dbf200640 1 -- 172.21.15.60:0/1357616498 wait complete. 2024-04-14T23:51:03.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.892+0000 7f6dbf200640 1 Processor -- start 2024-04-14T23:51:03.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.893+0000 7f6dbf200640 1 -- start start 2024-04-14T23:51:03.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.894+0000 7f6dbf200640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d9c0051c0 0x7f6db815dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:03.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.894+0000 7f6dbf200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6db815ac70 0x7f6db814ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:03.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.894+0000 7f6dbf200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6db80c7970 0x7f6db80cf890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:03.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.894+0000 7f6dbf200640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f6db80cd7f0 con 0x7f6d9c0051c0 2024-04-14T23:51:03.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.894+0000 7f6dbf200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6db80cd510 con 0x7f6db80c7970 2024-04-14T23:51:03.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.894+0000 7f6dbf200640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f6db80cdd10 con 0x7f6db815ac70 2024-04-14T23:51:03.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.895+0000 7f6dbd200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6db80c7970 0x7f6db80cf890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:03.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.895+0000 7f6dbd200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6db80c7970 0x7f6db80cf890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:33124/0 (socket says 172.21.15.60:33124) 2024-04-14T23:51:03.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.895+0000 7f6dbd200640 1 -- 172.21.15.60:0/1835992759 learned_addr learned my addr 172.21.15.60:0/1835992759 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:51:03.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.895+0000 7f6db7e00640 1 --2- 172.21.15.60:0/1835992759 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d9c0051c0 0x7f6db815dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:03.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.896+0000 7f6dbd200640 1 -- 172.21.15.60:0/1835992759 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6db815ac70 msgr2=0x7f6db814ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:03.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.896+0000 7f6dafe00640 1 --2- 172.21.15.60:0/1835992759 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6db815ac70 0x7f6db814ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:03.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.896+0000 7f6dbd200640 1 --2- 172.21.15.60:0/1835992759 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6db815ac70 0x7f6db814ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:03.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.896+0000 7f6dbd200640 1 -- 172.21.15.60:0/1835992759 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d9c0051c0 msgr2=0x7f6db815dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:03.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.896+0000 7f6dbd200640 1 --2- 172.21.15.60:0/1835992759 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d9c0051c0 0x7f6db815dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:03.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.896+0000 7f6dbd200640 1 -- 172.21.15.60:0/1835992759 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6db80c8330 con 0x7f6db80c7970 2024-04-14T23:51:03.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.896+0000 7f6db7e00640 1 --2- 172.21.15.60:0/1835992759 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d9c0051c0 0x7f6db815dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-14T23:51:03.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.896+0000 7f6dafe00640 1 --2- 172.21.15.60:0/1835992759 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6db815ac70 0x7f6db814ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-14T23:51:03.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.896+0000 7f6dbd200640 1 --2- 172.21.15.60:0/1835992759 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6db80c7970 0x7f6db80cf890 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6db0002050 tx=0x7f6db00015b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:51:03.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.897+0000 7f6db4c00640 1 -- 172.21.15.60:0/1835992759 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6db0046070 con 0x7f6db80c7970 2024-04-14T23:51:03.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.897+0000 7f6db4c00640 1 -- 172.21.15.60:0/1835992759 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6db0006b50 con 0x7f6db80c7970 2024-04-14T23:51:03.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.897+0000 7f6db4c00640 1 -- 172.21.15.60:0/1835992759 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6db002d500 con 0x7f6db80c7970 2024-04-14T23:51:03.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.897+0000 7f6dbf200640 1 -- 172.21.15.60:0/1835992759 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6db815e130 con 0x7f6db80c7970 2024-04-14T23:51:03.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.897+0000 7f6dbf200640 1 -- 172.21.15.60:0/1835992759 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6db815f030 con 0x7f6db80c7970 2024-04-14T23:51:03.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.899+0000 7f6dbf200640 1 -- 172.21.15.60:0/1835992759 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6db80b3650 con 0x7f6db80c7970 2024-04-14T23:51:03.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.901+0000 7f6db4c00640 1 -- 172.21.15.60:0/1835992759 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f6db0049070 con 0x7f6db80c7970 2024-04-14T23:51:03.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.903+0000 7f6db4c00640 1 --2- 172.21.15.60:0/1835992759 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f6d9406ba90 0x7f6d9406df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:03.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.904+0000 7f6db4c00640 1 -- 172.21.15.60:0/1835992759 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 5911+0+0 (secure 0 0 0) 0x7f6db00ac160 con 0x7f6db80c7970 2024-04-14T23:51:03.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.906+0000 7f6db7e00640 1 --2- 172.21.15.60:0/1835992759 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f6d9406ba90 0x7f6d9406df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:03.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.906+0000 7f6db7e00640 1 --2- 172.21.15.60:0/1835992759 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f6d9406ba90 0x7f6d9406df50 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6da8005a10 tx=0x7f6da8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:51:03.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:03.909+0000 7f6db4c00640 1 -- 172.21.15.60:0/1835992759 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6db80b3650 con 0x7f6db80c7970 2024-04-14T23:51:04.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:04.149+0000 7f6dbf200640 1 -- 172.21.15.60:0/1835992759 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6db8076970 con 0x7f6db80c7970 2024-04-14T23:51:04.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:04.150+0000 7f6db4c00640 1 -- 172.21.15.60:0/1835992759 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5997 (secure 0 0 0) 0x7f6db80c8330 con 0x7f6db80c7970 2024-04-14T23:51:04.155 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:51:04.155 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:50:58.881832+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4276},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4276":{"gid":4276,"name":"a","rank":0,"incarnation":9,"state":"up:replay","state_seq":1,"addr":"172.21.15.60:6833/78303888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":78303888},{"type":"v1","addr":"172.21.15.60:6833","nonce":78303888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-14T23:51:04.155 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 9 2024-04-14T23:51:04.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:04.153+0000 7f6dbf200640 1 -- 172.21.15.60:0/1835992759 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f6d9406ba90 msgr2=0x7f6d9406df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:04.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:04.153+0000 7f6dbf200640 1 --2- 172.21.15.60:0/1835992759 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f6d9406ba90 0x7f6d9406df50 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6da8005a10 tx=0x7f6da8009040 comp rx=0 tx=0).stop 2024-04-14T23:51:04.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:04.153+0000 7f6dbf200640 1 -- 172.21.15.60:0/1835992759 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6db80c7970 msgr2=0x7f6db80cf890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:04.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:04.153+0000 7f6dbf200640 1 --2- 172.21.15.60:0/1835992759 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6db80c7970 0x7f6db80cf890 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6db0002050 tx=0x7f6db00015b0 comp rx=0 tx=0).stop 2024-04-14T23:51:04.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:04.153+0000 7f6dbf200640 1 -- 172.21.15.60:0/1835992759 shutdown_connections 2024-04-14T23:51:04.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:04.153+0000 7f6dbf200640 1 --2- 172.21.15.60:0/1835992759 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f6d9406ba90 0x7f6d9406df50 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:04.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:04.153+0000 7f6dbf200640 1 --2- 172.21.15.60:0/1835992759 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6db80c7970 0x7f6db80cf890 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:04.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:04.153+0000 7f6dbf200640 1 --2- 172.21.15.60:0/1835992759 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6db815ac70 0x7f6db814ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:04.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:04.153+0000 7f6dbf200640 1 --2- 172.21.15.60:0/1835992759 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d9c0051c0 0x7f6db815dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:04.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:04.153+0000 7f6dbf200640 1 -- 172.21.15.60:0/1835992759 >> 172.21.15.60:0/1835992759 conn(0x7f6db806bfd0 msgr2=0x7f6db806cb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:51:04.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:04.153+0000 7f6dbf200640 1 -- 172.21.15.60:0/1835992759 shutdown_connections 2024-04-14T23:51:04.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:04.153+0000 7f6dbf200640 1 -- 172.21.15.60:0/1835992759 wait complete. 2024-04-14T23:51:04.175 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-14T23:51:06.176 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:51:06.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.400+0000 7ff4a0000640 1 Processor -- start 2024-04-14T23:51:06.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.401+0000 7ff4a0000640 1 -- start start 2024-04-14T23:51:06.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.403+0000 7ff4a0000640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff4980c63c0 0x7ff4980c6b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:06.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.403+0000 7ff4a0000640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff49815c9b0 0x7ff4980c70b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:06.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.403+0000 7ff4a0000640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4981530d0 con 0x7ff498152cf0 2024-04-14T23:51:06.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.403+0000 7ff4a0000640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7ff4980751f0 con 0x7ff49815c9b0 2024-04-14T23:51:06.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.403+0000 7ff4a0000640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7ff4980c8370 con 0x7ff4980c63c0 2024-04-14T23:51:06.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.403+0000 7ff49d600640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7ff498152cf0 0x7ff49815c2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.60:47842/0 (socket says 172.21.15.60:47842) 2024-04-14T23:51:06.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.403+0000 7ff49e000640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff49815c9b0 0x7ff4980c70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:06.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.403+0000 7ff49d600640 1 -- 172.21.15.60:0/2012104499 learned_addr learned my addr 172.21.15.60:0/2012104499 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:51:06.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.403+0000 7ff49cc00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff4980c63c0 0x7ff4980c6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:06.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.403+0000 7ff497e00640 1 -- 172.21.15.60:0/2012104499 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 692206672 0 0) 0x7ff4981530d0 con 0x7ff498152cf0 2024-04-14T23:51:06.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.404+0000 7ff49cc00640 1 -- 172.21.15.60:0/2012104499 >> v2:172.21.15.60:3300/0 conn(0x7ff49815c9b0 msgr2=0x7ff4980c70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:51:06.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.404+0000 7ff49cc00640 1 --2- 172.21.15.60:0/2012104499 >> v2:172.21.15.60:3300/0 conn(0x7ff49815c9b0 0x7ff4980c70b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:06.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.404+0000 7ff49cc00640 1 -- 172.21.15.60:0/2012104499 >> v1:172.21.15.186:6789/0 conn(0x7ff498152cf0 legacy=0x7ff49815c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:06.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.404+0000 7ff49cc00640 1 -- 172.21.15.60:0/2012104499 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4980c3810 con 0x7ff4980c63c0 2024-04-14T23:51:06.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.405+0000 7ff49cc00640 1 --2- 172.21.15.60:0/2012104499 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff4980c63c0 0x7ff4980c6b70 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff4840097a0 tx=0x7ff48402b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43f28db8c2a67f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:51:06.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.405+0000 7ff497e00640 1 -- 172.21.15.60:0/2012104499 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff484034070 con 0x7ff4980c63c0 2024-04-14T23:51:06.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.405+0000 7ff497e00640 1 -- 172.21.15.60:0/2012104499 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff484030610 con 0x7ff4980c63c0 2024-04-14T23:51:06.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.405+0000 7ff497e00640 1 -- 172.21.15.60:0/2012104499 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4840309e0 con 0x7ff4980c63c0 2024-04-14T23:51:06.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.405+0000 7ff4a0000640 1 -- 172.21.15.60:0/2012104499 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff4980c63c0 msgr2=0x7ff4980c6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:06.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.406+0000 7ff4a0000640 1 --2- 172.21.15.60:0/2012104499 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff4980c63c0 0x7ff4980c6b70 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff4840097a0 tx=0x7ff48402b960 comp rx=0 tx=0).stop 2024-04-14T23:51:06.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.406+0000 7ff4a0000640 1 -- 172.21.15.60:0/2012104499 shutdown_connections 2024-04-14T23:51:06.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.406+0000 7ff4a0000640 1 --2- 172.21.15.60:0/2012104499 >> v2:172.21.15.60:3300/0 conn(0x7ff49815c9b0 0x7ff4980c70b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:06.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.406+0000 7ff4a0000640 1 --2- 172.21.15.60:0/2012104499 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff4980c63c0 0x7ff4980c6b70 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:06.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.406+0000 7ff4a0000640 1 -- 172.21.15.60:0/2012104499 >> 172.21.15.60:0/2012104499 conn(0x7ff4980cc170 msgr2=0x7ff4980cc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:51:06.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.406+0000 7ff4a0000640 1 -- 172.21.15.60:0/2012104499 shutdown_connections 2024-04-14T23:51:06.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.406+0000 7ff4a0000640 1 -- 172.21.15.60:0/2012104499 wait complete. 2024-04-14T23:51:06.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.408+0000 7ff4a0000640 1 Processor -- start 2024-04-14T23:51:06.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.408+0000 7ff4a0000640 1 -- start start 2024-04-14T23:51:06.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.409+0000 7ff4a0000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff4980c63c0 0x7ff4981efc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:06.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.409+0000 7ff4a0000640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff498152cf0 0x7ff49813f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:06.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.409+0000 7ff4a0000640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff49815c9b0 0x7ff4981406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:06.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.409+0000 7ff4a0000640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff4980c88f0 con 0x7ff498152cf0 2024-04-14T23:51:06.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.409+0000 7ff4a0000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff4980c8610 con 0x7ff4980c63c0 2024-04-14T23:51:06.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.409+0000 7ff4a0000640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7ff498159450 con 0x7ff49815c9b0 2024-04-14T23:51:06.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.409+0000 7ff49cc00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff498152cf0 0x7ff49813f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:06.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.409+0000 7ff49cc00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff498152cf0 0x7ff49813f940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57164/0 (socket says 172.21.15.60:57164) 2024-04-14T23:51:06.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.409+0000 7ff49cc00640 1 -- 172.21.15.60:0/4177526513 learned_addr learned my addr 172.21.15.60:0/4177526513 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:51:06.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.409+0000 7ff49e000640 1 --2- 172.21.15.60:0/4177526513 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff49815c9b0 0x7ff4981406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:06.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.409+0000 7ff49d600640 1 --2- 172.21.15.60:0/4177526513 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff4980c63c0 0x7ff4981efc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:06.415 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.410+0000 7ff49e000640 1 -- 172.21.15.60:0/4177526513 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff4980c63c0 msgr2=0x7ff4981efc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:06.415 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.410+0000 7ff49e000640 1 --2- 172.21.15.60:0/4177526513 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff4980c63c0 0x7ff4981efc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:06.415 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.410+0000 7ff49e000640 1 -- 172.21.15.60:0/4177526513 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff498152cf0 msgr2=0x7ff49813f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:06.415 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.410+0000 7ff49e000640 1 --2- 172.21.15.60:0/4177526513 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff498152cf0 0x7ff49813f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:06.416 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.410+0000 7ff49e000640 1 -- 172.21.15.60:0/4177526513 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff498151d60 con 0x7ff49815c9b0 2024-04-14T23:51:06.416 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.410+0000 7ff49e000640 1 --2- 172.21.15.60:0/4177526513 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff49815c9b0 0x7ff4981406d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff49000e820 tx=0x7ff49000ece0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:51:06.416 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.410+0000 7ff495600640 1 -- 172.21.15.60:0/4177526513 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff490017070 con 0x7ff49815c9b0 2024-04-14T23:51:06.416 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.410+0000 7ff495600640 1 -- 172.21.15.60:0/4177526513 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff49000c730 con 0x7ff49815c9b0 2024-04-14T23:51:06.416 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.410+0000 7ff495600640 1 -- 172.21.15.60:0/4177526513 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff49000cb40 con 0x7ff49815c9b0 2024-04-14T23:51:06.416 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.410+0000 7ff4a0000640 1 -- 172.21.15.60:0/4177526513 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff498152000 con 0x7ff49815c9b0 2024-04-14T23:51:06.416 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.410+0000 7ff4a0000640 1 -- 172.21.15.60:0/4177526513 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4981ee930 con 0x7ff49815c9b0 2024-04-14T23:51:06.416 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.411+0000 7ff4a0000640 1 -- 172.21.15.60:0/4177526513 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff498076970 con 0x7ff49815c9b0 2024-04-14T23:51:06.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.413+0000 7ff495600640 1 -- 172.21.15.60:0/4177526513 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7ff49001a070 con 0x7ff49815c9b0 2024-04-14T23:51:06.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.413+0000 7ff495600640 1 --2- 172.21.15.60:0/4177526513 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff47806b9c0 0x7ff47806de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:06.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.414+0000 7ff495600640 1 -- 172.21.15.60:0/4177526513 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 5911+0+0 (secure 0 0 0) 0x7ff49008a360 con 0x7ff49815c9b0 2024-04-14T23:51:06.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.414+0000 7ff49d600640 1 --2- 172.21.15.60:0/4177526513 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff47806b9c0 0x7ff47806de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:06.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.414+0000 7ff49d600640 1 --2- 172.21.15.60:0/4177526513 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff47806b9c0 0x7ff47806de80 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff48c001920 tx=0x7ff48c0018b0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:51:06.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.418+0000 7ff495600640 1 -- 172.21.15.60:0/4177526513 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff490058720 con 0x7ff49815c9b0 2024-04-14T23:51:06.665 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.660+0000 7ff4a0000640 1 -- 172.21.15.60:0/4177526513 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff498145940 con 0x7ff49815c9b0 2024-04-14T23:51:06.666 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.662+0000 7ff495600640 1 -- 172.21.15.60:0/4177526513 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+6003 (secure 0 0 0) 0x7ff49005c7c0 con 0x7ff49815c9b0 2024-04-14T23:51:06.667 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:51:06.667 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":10,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:51:04.079247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4276},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4276":{"gid":4276,"name":"a","rank":0,"incarnation":9,"state":"up:reconnect","state_seq":39,"addr":"172.21.15.60:6833/78303888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":78303888},{"type":"v1","addr":"172.21.15.60:6833","nonce":78303888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-14T23:51:06.667 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 10 2024-04-14T23:51:06.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.664+0000 7ff4a0000640 1 -- 172.21.15.60:0/4177526513 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff47806b9c0 msgr2=0x7ff47806de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:06.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.664+0000 7ff4a0000640 1 --2- 172.21.15.60:0/4177526513 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff47806b9c0 0x7ff47806de80 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff48c001920 tx=0x7ff48c0018b0 comp rx=0 tx=0).stop 2024-04-14T23:51:06.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.664+0000 7ff4a0000640 1 -- 172.21.15.60:0/4177526513 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff49815c9b0 msgr2=0x7ff4981406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:06.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.664+0000 7ff4a0000640 1 --2- 172.21.15.60:0/4177526513 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff49815c9b0 0x7ff4981406d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff49000e820 tx=0x7ff49000ece0 comp rx=0 tx=0).stop 2024-04-14T23:51:06.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.665+0000 7ff4a0000640 1 -- 172.21.15.60:0/4177526513 shutdown_connections 2024-04-14T23:51:06.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.665+0000 7ff4a0000640 1 --2- 172.21.15.60:0/4177526513 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff47806b9c0 0x7ff47806de80 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:06.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.665+0000 7ff4a0000640 1 --2- 172.21.15.60:0/4177526513 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff49815c9b0 0x7ff4981406d0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:06.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.665+0000 7ff4a0000640 1 --2- 172.21.15.60:0/4177526513 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff498152cf0 0x7ff49813f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:06.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.665+0000 7ff4a0000640 1 --2- 172.21.15.60:0/4177526513 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff4980c63c0 0x7ff4981efc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:06.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.665+0000 7ff4a0000640 1 -- 172.21.15.60:0/4177526513 >> 172.21.15.60:0/4177526513 conn(0x7ff4980cc170 msgr2=0x7ff49806e280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:51:06.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.665+0000 7ff4a0000640 1 -- 172.21.15.60:0/4177526513 shutdown_connections 2024-04-14T23:51:06.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:06.665+0000 7ff4a0000640 1 -- 172.21.15.60:0/4177526513 wait complete. 2024-04-14T23:51:06.688 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-14T23:51:08.689 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:51:08.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:08.920+0000 7f184f200640 1 Processor -- start 2024-04-14T23:51:08.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:08.920+0000 7f184f200640 1 -- start start 2024-04-14T23:51:08.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:08.921+0000 7f184f200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f18480b1e00 0x7f18480c6950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:08.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:08.922+0000 7f184f200640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f18480c3d90 0x7f18480c41e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:08.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:08.922+0000 7f184f200640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1848152fb0 con 0x7f1848073320 2024-04-14T23:51:08.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:08.922+0000 7f184f200640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f18480c6540 con 0x7f18480c3d90 2024-04-14T23:51:08.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:08.922+0000 7f184f200640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f18480b21e0 con 0x7f18480b1e00 2024-04-14T23:51:08.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:08.922+0000 7f184e800640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f18480c3d90 0x7f18480c41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:08.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:08.922+0000 7f184e800640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f18480c3d90 0x7f18480c41e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57180/0 (socket says 172.21.15.60:57180) 2024-04-14T23:51:08.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:08.922+0000 7f184e800640 1 -- 172.21.15.60:0/1324230923 learned_addr learned my addr 172.21.15.60:0/1324230923 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:51:08.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:08.922+0000 7f184de00640 1 --2- 172.21.15.60:0/1324230923 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f18480b1e00 0x7f18480c6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:08.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:08.922+0000 7f184e800640 1 -- 172.21.15.60:0/1324230923 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f18480b1e00 msgr2=0x7f18480c6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:51:08.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:08.922+0000 7f184e800640 1 --2- 172.21.15.60:0/1324230923 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f18480b1e00 0x7f18480c6950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:08.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:08.922+0000 7f184e800640 1 -- 172.21.15.60:0/1324230923 >> v1:172.21.15.186:6789/0 conn(0x7f1848073320 legacy=0x7f18480c7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:08.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:08.922+0000 7f184e800640 1 -- 172.21.15.60:0/1324230923 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1848159310 con 0x7f18480c3d90 2024-04-14T23:51:08.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:08.924+0000 7f184e800640 1 --2- 172.21.15.60:0/1324230923 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18480c3d90 0x7f18480c41e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1844009e50 tx=0x7f184402bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c022e9f5e7d39709 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:51:08.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:08.924+0000 7f184ca00640 1 -- 172.21.15.60:0/1324230923 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f184402cb90 con 0x7f18480c3d90 2024-04-14T23:51:08.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:08.924+0000 7f184ca00640 1 -- 172.21.15.60:0/1324230923 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f184402cd30 con 0x7f18480c3d90 2024-04-14T23:51:08.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:08.924+0000 7f184ca00640 1 -- 172.21.15.60:0/1324230923 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1844034610 con 0x7f18480c3d90 2024-04-14T23:51:08.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:08.924+0000 7f184f200640 1 -- 172.21.15.60:0/1324230923 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18480c3d90 msgr2=0x7f18480c41e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:08.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:08.924+0000 7f184f200640 1 --2- 172.21.15.60:0/1324230923 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18480c3d90 0x7f18480c41e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1844009e50 tx=0x7f184402bff0 comp rx=0 tx=0).stop 2024-04-14T23:51:08.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:08.925+0000 7f184f200640 1 -- 172.21.15.60:0/1324230923 shutdown_connections 2024-04-14T23:51:08.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:08.925+0000 7f184f200640 1 --2- 172.21.15.60:0/1324230923 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18480c3d90 0x7f18480c41e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:08.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:08.925+0000 7f184f200640 1 --2- 172.21.15.60:0/1324230923 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f18480b1e00 0x7f18480c6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:08.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:08.925+0000 7f184f200640 1 -- 172.21.15.60:0/1324230923 >> 172.21.15.60:0/1324230923 conn(0x7f18480c8b50 msgr2=0x7f18480c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:51:08.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:08.925+0000 7f184f200640 1 -- 172.21.15.60:0/1324230923 shutdown_connections 2024-04-14T23:51:08.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:08.925+0000 7f184f200640 1 -- 172.21.15.60:0/1324230923 wait complete. 2024-04-14T23:51:08.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:08.927+0000 7f184f200640 1 Processor -- start 2024-04-14T23:51:08.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:08.927+0000 7f184f200640 1 -- start start 2024-04-14T23:51:08.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:08.928+0000 7f184f200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1848073320 0x7f18481e3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:08.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:08.928+0000 7f184f200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f18480b1e00 0x7f18480cb420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:08.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:08.928+0000 7f184f200640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18480c3d90 0x7f18481e1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:08.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:08.928+0000 7f184f200640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f18480c38b0 con 0x7f18480c3d90 2024-04-14T23:51:08.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:08.928+0000 7f184f200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1848159890 con 0x7f1848073320 2024-04-14T23:51:08.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:08.928+0000 7f184f200640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f18481595b0 con 0x7f18480b1e00 2024-04-14T23:51:08.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:08.928+0000 7f184e800640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18480c3d90 0x7f18481e1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:08.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:08.928+0000 7f184e800640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18480c3d90 0x7f18481e1ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57192/0 (socket says 172.21.15.60:57192) 2024-04-14T23:51:08.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:08.928+0000 7f184e800640 1 -- 172.21.15.60:0/4128285398 learned_addr learned my addr 172.21.15.60:0/4128285398 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:51:08.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:08.928+0000 7f184d400640 1 --2- 172.21.15.60:0/4128285398 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f18480b1e00 0x7f18480cb420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:08.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:08.928+0000 7f184de00640 1 --2- 172.21.15.60:0/4128285398 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1848073320 0x7f18481e3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:08.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:08.928+0000 7f184e800640 1 -- 172.21.15.60:0/4128285398 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f18480b1e00 msgr2=0x7f18480cb420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:08.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:08.928+0000 7f184e800640 1 --2- 172.21.15.60:0/4128285398 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f18480b1e00 0x7f18480cb420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:08.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:08.928+0000 7f184e800640 1 -- 172.21.15.60:0/4128285398 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1848073320 msgr2=0x7f18481e3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:08.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:08.928+0000 7f184e800640 1 --2- 172.21.15.60:0/4128285398 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1848073320 0x7f18481e3ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:08.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:08.928+0000 7f184e800640 1 -- 172.21.15.60:0/4128285398 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18481f06b0 con 0x7f18480c3d90 2024-04-14T23:51:08.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:08.929+0000 7f184d400640 1 --2- 172.21.15.60:0/4128285398 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f18480b1e00 0x7f18480cb420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:51:08.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:08.929+0000 7f184e800640 1 --2- 172.21.15.60:0/4128285398 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18480c3d90 0x7f18481e1ce0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1844009f70 tx=0x7f1844004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:51:08.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:08.929+0000 7f1842000640 1 -- 172.21.15.60:0/4128285398 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1844043020 con 0x7f18480c3d90 2024-04-14T23:51:08.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:08.929+0000 7f1842000640 1 -- 172.21.15.60:0/4128285398 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f184403e450 con 0x7f18480c3d90 2024-04-14T23:51:08.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:08.929+0000 7f1842000640 1 -- 172.21.15.60:0/4128285398 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1844009510 con 0x7f18480c3d90 2024-04-14T23:51:08.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:08.929+0000 7f184f200640 1 -- 172.21.15.60:0/4128285398 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1848072f70 con 0x7f18480c3d90 2024-04-14T23:51:08.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:08.929+0000 7f184f200640 1 -- 172.21.15.60:0/4128285398 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18481f0c10 con 0x7f18480c3d90 2024-04-14T23:51:08.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:08.930+0000 7f184f200640 1 -- 172.21.15.60:0/4128285398 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f184804f540 con 0x7f18480c3d90 2024-04-14T23:51:08.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:08.931+0000 7f1842000640 1 -- 172.21.15.60:0/4128285398 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f1844046020 con 0x7f18480c3d90 2024-04-14T23:51:08.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:08.932+0000 7f1842000640 1 --2- 172.21.15.60:0/4128285398 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f182c06b9c0 0x7f182c06de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:08.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:08.932+0000 7f1842000640 1 -- 172.21.15.60:0/4128285398 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 5911+0+0 (secure 0 0 0) 0x7f18440ace90 con 0x7f18480c3d90 2024-04-14T23:51:08.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:08.933+0000 7f184de00640 1 --2- 172.21.15.60:0/4128285398 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f182c06b9c0 0x7f182c06de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:08.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:08.934+0000 7f184de00640 1 --2- 172.21.15.60:0/4128285398 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f182c06b9c0 0x7f182c06de80 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f183c009b40 tx=0x7f183c009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:51:08.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:08.938+0000 7f1842000640 1 -- 172.21.15.60:0/4128285398 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f184404d030 con 0x7f18480c3d90 2024-04-14T23:51:09.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:09.178+0000 7f184f200640 1 -- 172.21.15.60:0/4128285398 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f18480c99e0 con 0x7f18480c3d90 2024-04-14T23:51:09.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:09.180+0000 7f1842000640 1 -- 172.21.15.60:0/4128285398 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+6003 (secure 0 0 0) 0x7f18481f06b0 con 0x7f18480c3d90 2024-04-14T23:51:09.185 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:51:09.185 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":10,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:51:04.079247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4276},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4276":{"gid":4276,"name":"a","rank":0,"incarnation":9,"state":"up:reconnect","state_seq":39,"addr":"172.21.15.60:6833/78303888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":78303888},{"type":"v1","addr":"172.21.15.60:6833","nonce":78303888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-14T23:51:09.185 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 10 2024-04-14T23:51:09.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:09.182+0000 7f184f200640 1 -- 172.21.15.60:0/4128285398 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f182c06b9c0 msgr2=0x7f182c06de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:09.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:09.182+0000 7f184f200640 1 --2- 172.21.15.60:0/4128285398 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f182c06b9c0 0x7f182c06de80 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f183c009b40 tx=0x7f183c009190 comp rx=0 tx=0).stop 2024-04-14T23:51:09.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:09.183+0000 7f184f200640 1 -- 172.21.15.60:0/4128285398 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18480c3d90 msgr2=0x7f18481e1ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:09.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:09.183+0000 7f184f200640 1 --2- 172.21.15.60:0/4128285398 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18480c3d90 0x7f18481e1ce0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1844009f70 tx=0x7f1844004a40 comp rx=0 tx=0).stop 2024-04-14T23:51:09.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:09.183+0000 7f184f200640 1 -- 172.21.15.60:0/4128285398 shutdown_connections 2024-04-14T23:51:09.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:09.183+0000 7f184f200640 1 --2- 172.21.15.60:0/4128285398 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f182c06b9c0 0x7f182c06de80 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:09.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:09.183+0000 7f184f200640 1 --2- 172.21.15.60:0/4128285398 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18480c3d90 0x7f18481e1ce0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:09.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:09.183+0000 7f184f200640 1 --2- 172.21.15.60:0/4128285398 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f18480b1e00 0x7f18480cb420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:09.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:09.183+0000 7f184f200640 1 --2- 172.21.15.60:0/4128285398 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1848073320 0x7f18481e3ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:09.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:09.183+0000 7f184f200640 1 -- 172.21.15.60:0/4128285398 >> 172.21.15.60:0/4128285398 conn(0x7f18480c8b50 msgr2=0x7f1848070d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:51:09.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:09.183+0000 7f184f200640 1 -- 172.21.15.60:0/4128285398 shutdown_connections 2024-04-14T23:51:09.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:09.183+0000 7f184f200640 1 -- 172.21.15.60:0/4128285398 wait complete. 2024-04-14T23:51:09.209 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-14T23:51:11.210 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:51:11.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.440+0000 7efcf5400640 1 Processor -- start 2024-04-14T23:51:11.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.440+0000 7efcf5400640 1 -- start start 2024-04-14T23:51:11.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.441+0000 7efcf5400640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7efcf00cd540 0x7efcf015ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:11.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.441+0000 7efcf5400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efcf00cb750 0x7efcf0154b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:11.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.442+0000 7efcf5400640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efcf00b3f50 con 0x7efcf00cd160 2024-04-14T23:51:11.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.442+0000 7efcf5400640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7efcf015b370 con 0x7efcf00cd540 2024-04-14T23:51:11.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.442+0000 7efcf5400640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7efcf00b39b0 con 0x7efcf00cb750 2024-04-14T23:51:11.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.442+0000 7efcede00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7efcf00cd540 0x7efcf015ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:11.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.442+0000 7efcede00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7efcf00cd540 0x7efcf015ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57194/0 (socket says 172.21.15.60:57194) 2024-04-14T23:51:11.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.442+0000 7efcede00640 1 -- 172.21.15.60:0/3121953604 learned_addr learned my addr 172.21.15.60:0/3121953604 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:51:11.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.442+0000 7efcee800640 1 --1- 172.21.15.60:0/3121953604 >> v1:172.21.15.186:6789/0 conn(0x7efcf00cd160 0x7efcf00b3190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.60:47846/0 (socket says 172.21.15.60:47846) 2024-04-14T23:51:11.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.442+0000 7efcef200640 1 --2- 172.21.15.60:0/3121953604 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efcf00cb750 0x7efcf0154b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:11.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.442+0000 7efcede00640 1 -- 172.21.15.60:0/3121953604 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efcf00cb750 msgr2=0x7efcf0154b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:51:11.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.442+0000 7efcede00640 1 --2- 172.21.15.60:0/3121953604 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efcf00cb750 0x7efcf0154b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:11.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.442+0000 7efcede00640 1 -- 172.21.15.60:0/3121953604 >> v1:172.21.15.186:6789/0 conn(0x7efcf00cd160 legacy=0x7efcf00b3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:11.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.442+0000 7efcede00640 1 -- 172.21.15.60:0/3121953604 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcf00b3c70 con 0x7efcf00cd540 2024-04-14T23:51:11.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.444+0000 7efcede00640 1 --2- 172.21.15.60:0/3121953604 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcf00cd540 0x7efcf015ae30 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7efce0009870 tx=0x7efce002ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c0e2618d9973340 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:51:11.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.444+0000 7efced400640 1 -- 172.21.15.60:0/3121953604 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efce002f830 con 0x7efcf00cd540 2024-04-14T23:51:11.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.444+0000 7efced400640 1 -- 172.21.15.60:0/3121953604 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efce002f9d0 con 0x7efcf00cd540 2024-04-14T23:51:11.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.444+0000 7efced400640 1 -- 172.21.15.60:0/3121953604 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efce002fda0 con 0x7efcf00cd540 2024-04-14T23:51:11.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.445+0000 7efcf5400640 1 -- 172.21.15.60:0/3121953604 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcf00cd540 msgr2=0x7efcf015ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:11.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.445+0000 7efcf5400640 1 --2- 172.21.15.60:0/3121953604 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcf00cd540 0x7efcf015ae30 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7efce0009870 tx=0x7efce002ba10 comp rx=0 tx=0).stop 2024-04-14T23:51:11.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.445+0000 7efcf5400640 1 -- 172.21.15.60:0/3121953604 shutdown_connections 2024-04-14T23:51:11.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.445+0000 7efcf5400640 1 --2- 172.21.15.60:0/3121953604 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efcf00cb750 0x7efcf0154b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:11.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.445+0000 7efcf5400640 1 --2- 172.21.15.60:0/3121953604 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcf00cd540 0x7efcf015ae30 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:11.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.445+0000 7efcf5400640 1 -- 172.21.15.60:0/3121953604 >> 172.21.15.60:0/3121953604 conn(0x7efcf0066f10 msgr2=0x7efcf00672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:51:11.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.445+0000 7efcf5400640 1 -- 172.21.15.60:0/3121953604 shutdown_connections 2024-04-14T23:51:11.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.445+0000 7efcf5400640 1 -- 172.21.15.60:0/3121953604 wait complete. 2024-04-14T23:51:11.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.447+0000 7efcf5400640 1 Processor -- start 2024-04-14T23:51:11.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.447+0000 7efcf5400640 1 -- start start 2024-04-14T23:51:11.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.448+0000 7efcf5400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcf00cb750 0x7efcf0155940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:11.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.448+0000 7efcf5400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efcf00cd160 0x7efcf0151270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:11.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.448+0000 7efcf5400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efcf00cd540 0x7efcf00cfba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:11.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.448+0000 7efcf5400640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7efcf015ea70 con 0x7efcf00cb750 2024-04-14T23:51:11.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.448+0000 7efcf5400640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7efcf015e790 con 0x7efcf00cd160 2024-04-14T23:51:11.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.448+0000 7efcf5400640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7efcf00ccb20 con 0x7efcf00cd540 2024-04-14T23:51:11.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.448+0000 7efcee800640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcf00cb750 0x7efcf0155940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:11.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.448+0000 7efcee800640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcf00cb750 0x7efcf0155940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57204/0 (socket says 172.21.15.60:57204) 2024-04-14T23:51:11.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.448+0000 7efcee800640 1 -- 172.21.15.60:0/2732269120 learned_addr learned my addr 172.21.15.60:0/2732269120 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:51:11.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.448+0000 7efcef200640 1 --2- 172.21.15.60:0/2732269120 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efcf00cd540 0x7efcf00cfba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:11.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.448+0000 7efcede00640 1 --2- 172.21.15.60:0/2732269120 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efcf00cd160 0x7efcf0151270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:11.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.449+0000 7efcee800640 1 -- 172.21.15.60:0/2732269120 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efcf00cd540 msgr2=0x7efcf00cfba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:11.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.449+0000 7efcee800640 1 --2- 172.21.15.60:0/2732269120 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efcf00cd540 0x7efcf00cfba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:11.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.449+0000 7efcee800640 1 -- 172.21.15.60:0/2732269120 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efcf00cd160 msgr2=0x7efcf0151270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:11.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.449+0000 7efcee800640 1 --2- 172.21.15.60:0/2732269120 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efcf00cd160 0x7efcf0151270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:11.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.449+0000 7efcee800640 1 -- 172.21.15.60:0/2732269120 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcf00ce9c0 con 0x7efcf00cb750 2024-04-14T23:51:11.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.449+0000 7efcee800640 1 --2- 172.21.15.60:0/2732269120 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcf00cb750 0x7efcf0155940 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7efcd800c4d0 tx=0x7efcd800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:51:11.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.450+0000 7efce6a00640 1 -- 172.21.15.60:0/2732269120 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcd8018070 con 0x7efcf00cb750 2024-04-14T23:51:11.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.450+0000 7efce6a00640 1 -- 172.21.15.60:0/2732269120 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcd8007480 con 0x7efcf00cb750 2024-04-14T23:51:11.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.450+0000 7efce6a00640 1 -- 172.21.15.60:0/2732269120 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcd80047e0 con 0x7efcf00cb750 2024-04-14T23:51:11.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.450+0000 7efcf5400640 1 -- 172.21.15.60:0/2732269120 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcf00cec60 con 0x7efcf00cb750 2024-04-14T23:51:11.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.450+0000 7efcf5400640 1 -- 172.21.15.60:0/2732269120 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcf015ad20 con 0x7efcf00cb750 2024-04-14T23:51:11.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.451+0000 7efcf5400640 1 -- 172.21.15.60:0/2732269120 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efcf006d380 con 0x7efcf00cb750 2024-04-14T23:51:11.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.452+0000 7efce6a00640 1 -- 172.21.15.60:0/2732269120 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7efcd8004980 con 0x7efcf00cb750 2024-04-14T23:51:11.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.454+0000 7efce6a00640 1 --2- 172.21.15.60:0/2732269120 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7efccc06b990 0x7efccc06de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:11.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.454+0000 7efce6a00640 1 -- 172.21.15.60:0/2732269120 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 5911+0+0 (secure 0 0 0) 0x7efcd8089b00 con 0x7efcf00cb750 2024-04-14T23:51:11.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.454+0000 7efcede00640 1 --2- 172.21.15.60:0/2732269120 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7efccc06b990 0x7efccc06de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:11.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.455+0000 7efcede00640 1 --2- 172.21.15.60:0/2732269120 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7efccc06b990 0x7efccc06de50 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7efce0036000 tx=0x7efce0008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:51:11.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.459+0000 7efce6a00640 1 -- 172.21.15.60:0/2732269120 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7efcf006d380 con 0x7efcf00cb750 2024-04-14T23:51:11.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.701+0000 7efcf5400640 1 -- 172.21.15.60:0/2732269120 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efcf0076970 con 0x7efcf00cb750 2024-04-14T23:51:11.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.702+0000 7efce6a00640 1 -- 172.21.15.60:0/2732269120 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+6000 (secure 0 0 0) 0x7efcd8089e20 con 0x7efcf00cb750 2024-04-14T23:51:11.707 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:51:11.707 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":11,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":11,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:51:09.965711+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4276},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4276":{"gid":4276,"name":"a","rank":0,"incarnation":9,"state":"up:rejoin","state_seq":41,"addr":"172.21.15.60:6833/78303888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":78303888},{"type":"v1","addr":"172.21.15.60:6833","nonce":78303888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-14T23:51:11.707 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 11 2024-04-14T23:51:11.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.705+0000 7efcf5400640 1 -- 172.21.15.60:0/2732269120 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7efccc06b990 msgr2=0x7efccc06de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:11.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.705+0000 7efcf5400640 1 --2- 172.21.15.60:0/2732269120 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7efccc06b990 0x7efccc06de50 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7efce0036000 tx=0x7efce0008030 comp rx=0 tx=0).stop 2024-04-14T23:51:11.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.705+0000 7efcf5400640 1 -- 172.21.15.60:0/2732269120 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcf00cb750 msgr2=0x7efcf0155940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:11.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.705+0000 7efcf5400640 1 --2- 172.21.15.60:0/2732269120 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcf00cb750 0x7efcf0155940 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7efcd800c4d0 tx=0x7efcd800c990 comp rx=0 tx=0).stop 2024-04-14T23:51:11.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.705+0000 7efcf5400640 1 -- 172.21.15.60:0/2732269120 shutdown_connections 2024-04-14T23:51:11.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.705+0000 7efcf5400640 1 --2- 172.21.15.60:0/2732269120 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7efccc06b990 0x7efccc06de50 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:11.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.705+0000 7efcf5400640 1 --2- 172.21.15.60:0/2732269120 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efcf00cd540 0x7efcf00cfba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:11.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.705+0000 7efcf5400640 1 --2- 172.21.15.60:0/2732269120 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efcf00cd160 0x7efcf0151270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:11.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.705+0000 7efcf5400640 1 --2- 172.21.15.60:0/2732269120 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcf00cb750 0x7efcf0155940 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:11.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.705+0000 7efcf5400640 1 -- 172.21.15.60:0/2732269120 >> 172.21.15.60:0/2732269120 conn(0x7efcf0066f10 msgr2=0x7efcf015d750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:51:11.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.705+0000 7efcf5400640 1 -- 172.21.15.60:0/2732269120 shutdown_connections 2024-04-14T23:51:11.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:11.705+0000 7efcf5400640 1 -- 172.21.15.60:0/2732269120 wait complete. 2024-04-14T23:51:11.730 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-14T23:51:13.731 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:51:13.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:13.959+0000 7f90a3000640 1 Processor -- start 2024-04-14T23:51:13.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:13.959+0000 7f90a3000640 1 -- start start 2024-04-14T23:51:13.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:13.961+0000 7f90a3000640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f909c071030 0x7f909c071a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:13.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:13.961+0000 7f90a3000640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f909c06e210 0x7f909c06e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:13.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:13.961+0000 7f90a3000640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f909c157e40 con 0x7f909c151120 2024-04-14T23:51:13.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:13.961+0000 7f90a3000640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f909c072d60 con 0x7f909c06e210 2024-04-14T23:51:13.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:13.961+0000 7f90a3000640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f909c072150 con 0x7f909c071030 2024-04-14T23:51:13.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:13.962+0000 7f90a1000640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f909c06e210 0x7f909c06e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:13.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:13.962+0000 7f909be00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f909c071030 0x7f909c071a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:13.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:13.962+0000 7f909b400640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f909c151120 0x7f909c0c4a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:41842/0 (socket says 172.21.15.60:41842) 2024-04-14T23:51:13.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:13.962+0000 7f909b400640 1 -- 172.21.15.60:0/2406278357 learned_addr learned my addr 172.21.15.60:0/2406278357 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:51:13.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:13.962+0000 7f90a1000640 1 -- 172.21.15.60:0/2406278357 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f909c071030 msgr2=0x7f909c071a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:51:13.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:13.962+0000 7f90a1000640 1 --2- 172.21.15.60:0/2406278357 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f909c071030 0x7f909c071a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:13.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:13.962+0000 7f90a1000640 1 -- 172.21.15.60:0/2406278357 >> v1:172.21.15.60:6789/0 conn(0x7f909c151120 legacy=0x7f909c0c4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:13.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:13.962+0000 7f90a1000640 1 -- 172.21.15.60:0/2406278357 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f909c072430 con 0x7f909c06e210 2024-04-14T23:51:13.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:13.962+0000 7f909aa00640 1 -- 172.21.15.60:0/2406278357 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3850626285 0 0) 0x7f909c157e40 con 0x7f909c151120 2024-04-14T23:51:13.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:13.963+0000 7f90a1000640 1 --2- 172.21.15.60:0/2406278357 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f909c06e210 0x7f909c06e6a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f90940097d0 tx=0x7f909402b930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a91de3fbae5053dc server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:51:13.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:13.964+0000 7f909aa00640 1 -- 172.21.15.60:0/2406278357 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9094035020 con 0x7f909c06e210 2024-04-14T23:51:13.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:13.964+0000 7f909aa00640 1 -- 172.21.15.60:0/2406278357 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9094031890 con 0x7f909c06e210 2024-04-14T23:51:13.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:13.964+0000 7f909aa00640 1 -- 172.21.15.60:0/2406278357 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9094031c60 con 0x7f909c06e210 2024-04-14T23:51:13.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:13.964+0000 7f90a3000640 1 -- 172.21.15.60:0/2406278357 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f909c06e210 msgr2=0x7f909c06e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:13.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:13.964+0000 7f90a3000640 1 --2- 172.21.15.60:0/2406278357 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f909c06e210 0x7f909c06e6a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f90940097d0 tx=0x7f909402b930 comp rx=0 tx=0).stop 2024-04-14T23:51:13.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:13.964+0000 7f90a3000640 1 -- 172.21.15.60:0/2406278357 shutdown_connections 2024-04-14T23:51:13.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:13.964+0000 7f90a3000640 1 --2- 172.21.15.60:0/2406278357 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f909c06e210 0x7f909c06e6a0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:13.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:13.964+0000 7f90a3000640 1 --2- 172.21.15.60:0/2406278357 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f909c071030 0x7f909c071a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:13.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:13.964+0000 7f90a3000640 1 -- 172.21.15.60:0/2406278357 >> 172.21.15.60:0/2406278357 conn(0x7f909c0cc170 msgr2=0x7f909c0cc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:51:13.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:13.965+0000 7f90a3000640 1 -- 172.21.15.60:0/2406278357 shutdown_connections 2024-04-14T23:51:13.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:13.965+0000 7f90a3000640 1 -- 172.21.15.60:0/2406278357 wait complete. 2024-04-14T23:51:13.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:13.966+0000 7f90a3000640 1 Processor -- start 2024-04-14T23:51:13.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:13.966+0000 7f90a3000640 1 -- start start 2024-04-14T23:51:13.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:13.967+0000 7f90a3000640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f909c06e210 0x7f909c145dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:13.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:13.967+0000 7f90a3000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f909c071030 0x7f909c15a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:13.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:13.967+0000 7f90a3000640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f909c151120 0x7f909c15a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:13.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:13.967+0000 7f90a3000640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f909c151560 con 0x7f909c06e210 2024-04-14T23:51:13.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:13.967+0000 7f90a3000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f909c0729f0 con 0x7f909c071030 2024-04-14T23:51:13.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:13.967+0000 7f90a3000640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f909c148010 con 0x7f909c151120 2024-04-14T23:51:13.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:13.968+0000 7f909be00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f909c06e210 0x7f909c145dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:13.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:13.968+0000 7f909be00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f909c06e210 0x7f909c145dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40398/0 (socket says 172.21.15.60:40398) 2024-04-14T23:51:13.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:13.968+0000 7f909be00640 1 -- 172.21.15.60:0/4220029190 learned_addr learned my addr 172.21.15.60:0/4220029190 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:51:13.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:13.968+0000 7f90a1000640 1 --2- 172.21.15.60:0/4220029190 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f909c151120 0x7f909c15a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:13.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:13.968+0000 7f909b400640 1 --2- 172.21.15.60:0/4220029190 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f909c071030 0x7f909c15a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:13.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:13.968+0000 7f909b400640 1 -- 172.21.15.60:0/4220029190 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f909c151120 msgr2=0x7f909c15a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:13.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:13.968+0000 7f909b400640 1 --2- 172.21.15.60:0/4220029190 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f909c151120 0x7f909c15a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:13.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:13.968+0000 7f909b400640 1 -- 172.21.15.60:0/4220029190 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f909c06e210 msgr2=0x7f909c145dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:13.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:13.968+0000 7f909b400640 1 --2- 172.21.15.60:0/4220029190 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f909c06e210 0x7f909c145dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:13.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:13.968+0000 7f909b400640 1 -- 172.21.15.60:0/4220029190 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f909c0cb9c0 con 0x7f909c071030 2024-04-14T23:51:13.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:13.968+0000 7f90a1000640 1 --2- 172.21.15.60:0/4220029190 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f909c151120 0x7f909c15a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:51:13.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:13.969+0000 7f909b400640 1 --2- 172.21.15.60:0/4220029190 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f909c071030 0x7f909c15a330 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f908c00c780 tx=0x7f908c00cc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:51:13.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:13.969+0000 7f9093e00640 1 -- 172.21.15.60:0/4220029190 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f908c018070 con 0x7f909c071030 2024-04-14T23:51:13.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:13.969+0000 7f9093e00640 1 -- 172.21.15.60:0/4220029190 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f908c00ce30 con 0x7f909c071030 2024-04-14T23:51:13.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:13.969+0000 7f90a3000640 1 -- 172.21.15.60:0/4220029190 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f909c159390 con 0x7f909c071030 2024-04-14T23:51:13.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:13.969+0000 7f9093e00640 1 -- 172.21.15.60:0/4220029190 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f908c00a860 con 0x7f909c071030 2024-04-14T23:51:13.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:13.969+0000 7f90a3000640 1 -- 172.21.15.60:0/4220029190 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f909c147d30 con 0x7f909c071030 2024-04-14T23:51:13.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:13.970+0000 7f90a3000640 1 -- 172.21.15.60:0/4220029190 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f909c076970 con 0x7f909c071030 2024-04-14T23:51:13.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:13.971+0000 7f9093e00640 1 -- 172.21.15.60:0/4220029190 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f908c01b070 con 0x7f909c071030 2024-04-14T23:51:13.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:13.972+0000 7f9093e00640 1 --2- 172.21.15.60:0/4220029190 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f907806b9c0 0x7f907806de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:13.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:13.972+0000 7f9093e00640 1 -- 172.21.15.60:0/4220029190 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 5911+0+0 (secure 0 0 0) 0x7f908c089df0 con 0x7f909c071030 2024-04-14T23:51:13.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:13.972+0000 7f909be00640 1 --2- 172.21.15.60:0/4220029190 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f907806b9c0 0x7f907806de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:13.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:13.973+0000 7f909be00640 1 --2- 172.21.15.60:0/4220029190 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f907806b9c0 0x7f907806de80 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f909c146860 tx=0x7f9084009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:51:13.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:13.977+0000 7f9093e00640 1 -- 172.21.15.60:0/4220029190 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f909c159390 con 0x7f909c071030 2024-04-14T23:51:14.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:14.219+0000 7f90a3000640 1 -- 172.21.15.60:0/4220029190 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f909c0c32f0 con 0x7f909c071030 2024-04-14T23:51:14.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:14.220+0000 7f9093e00640 1 -- 172.21.15.60:0/4220029190 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+6000 (secure 0 0 0) 0x7f908c058240 con 0x7f909c071030 2024-04-14T23:51:14.225 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:51:14.225 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":11,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":11,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:51:09.965711+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4276},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4276":{"gid":4276,"name":"a","rank":0,"incarnation":9,"state":"up:rejoin","state_seq":41,"addr":"172.21.15.60:6833/78303888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":78303888},{"type":"v1","addr":"172.21.15.60:6833","nonce":78303888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-14T23:51:14.225 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 11 2024-04-14T23:51:14.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:14.222+0000 7f90a3000640 1 -- 172.21.15.60:0/4220029190 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f907806b9c0 msgr2=0x7f907806de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:14.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:14.222+0000 7f90a3000640 1 --2- 172.21.15.60:0/4220029190 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f907806b9c0 0x7f907806de80 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f909c146860 tx=0x7f9084009190 comp rx=0 tx=0).stop 2024-04-14T23:51:14.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:14.222+0000 7f90a3000640 1 -- 172.21.15.60:0/4220029190 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f909c071030 msgr2=0x7f909c15a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:14.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:14.222+0000 7f90a3000640 1 --2- 172.21.15.60:0/4220029190 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f909c071030 0x7f909c15a330 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f908c00c780 tx=0x7f908c00cc40 comp rx=0 tx=0).stop 2024-04-14T23:51:14.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:14.222+0000 7f90a3000640 1 -- 172.21.15.60:0/4220029190 shutdown_connections 2024-04-14T23:51:14.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:14.222+0000 7f90a3000640 1 --2- 172.21.15.60:0/4220029190 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f907806b9c0 0x7f907806de80 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:14.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:14.222+0000 7f90a3000640 1 --2- 172.21.15.60:0/4220029190 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f909c151120 0x7f909c15a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:14.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:14.222+0000 7f90a3000640 1 --2- 172.21.15.60:0/4220029190 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f909c071030 0x7f909c15a330 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:14.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:14.222+0000 7f90a3000640 1 --2- 172.21.15.60:0/4220029190 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f909c06e210 0x7f909c145dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:14.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:14.222+0000 7f90a3000640 1 -- 172.21.15.60:0/4220029190 >> 172.21.15.60:0/4220029190 conn(0x7f909c0cc170 msgr2=0x7f909c0c9f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:51:14.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:14.223+0000 7f90a3000640 1 -- 172.21.15.60:0/4220029190 shutdown_connections 2024-04-14T23:51:14.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:14.223+0000 7f90a3000640 1 -- 172.21.15.60:0/4220029190 wait complete. 2024-04-14T23:51:14.249 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 11, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-14T23:48:42.700188+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 11, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4276": { "addr": "172.21.15.60:6833/78303888", "addrs": { "addrvec": [ { "addr": "172.21.15.60:6832", "nonce": 78303888, "type": "v2" }, { "addr": "172.21.15.60:6833", "nonce": 78303888, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4276, "incarnation": 9, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:rejoin", "state_seq": 41 } }, "last_failure": 0, "last_failure_osd_epoch": 26, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-14T23:51:09.965711+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4276 } } } ], "standbys": [ { "addr": "172.21.15.60:6837/1527815634", "addrs": { "addrvec": [ { "addr": "172.21.15.60:6836", "nonce": 1527815634, "type": "v2" }, { "addr": "172.21.15.60:6837", "nonce": 1527815634, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4197, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6838/3958319486", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6835", "nonce": 3958319486, "type": "v2" }, { "addr": "172.21.15.186:6838", "nonce": 3958319486, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4220, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6837/3910951185", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6834", "nonce": 3910951185, "type": "v2" }, { "addr": "172.21.15.186:6837", "nonce": 3910951185, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4264, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6839/2155020404", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6836", "nonce": 2155020404, "type": "v2" }, { "addr": "172.21.15.186:6839", "nonce": 2155020404, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4267, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-14T23:51:14.547 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-14T23:51:14.555 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-14T23:51:16.250 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:51:16.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.478+0000 7efc04e00640 1 Processor -- start 2024-04-14T23:51:16.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.478+0000 7efc04e00640 1 -- start start 2024-04-14T23:51:16.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.480+0000 7efc04e00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7efc000c4fa0 0x7efc000c77b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:16.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.480+0000 7efc04e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efc00070d50 0x7efc000c7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:16.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.480+0000 7efc04e00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7efc000c5380 0x7efc0015a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:16.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.480+0000 7efc04e00640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7efc0006b4b0 con 0x7efc000c5380 2024-04-14T23:51:16.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.480+0000 7efc04e00640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7efc0006b790 con 0x7efc000c4fa0 2024-04-14T23:51:16.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.480+0000 7efc04e00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7efc000c69f0 con 0x7efc00070d50 2024-04-14T23:51:16.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.480+0000 7efbfea00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efc00070d50 0x7efc000c7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:16.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.480+0000 7efbffe00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7efc000c5380 0x7efc0015a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:16.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.480+0000 7efbff400640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7efc000c4fa0 0x7efc000c77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:16.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.480+0000 7efbfea00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efc00070d50 0x7efc000c7cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.60:60554/0 (socket says 172.21.15.60:60554) 2024-04-14T23:51:16.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.480+0000 7efbffe00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7efc000c5380 0x7efc0015a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40406/0 (socket says 172.21.15.60:40406) 2024-04-14T23:51:16.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.480+0000 7efbff400640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7efc000c4fa0 0x7efc000c77b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:54268/0 (socket says 172.21.15.60:54268) 2024-04-14T23:51:16.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.480+0000 7efbfea00640 1 -- 172.21.15.60:0/1980168792 learned_addr learned my addr 172.21.15.60:0/1980168792 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:51:16.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.481+0000 7efbfea00640 1 -- 172.21.15.60:0/1980168792 >> v2:172.21.15.186:3300/0 conn(0x7efc000c4fa0 msgr2=0x7efc000c77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:51:16.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.481+0000 7efbfea00640 1 --2- 172.21.15.60:0/1980168792 >> v2:172.21.15.186:3300/0 conn(0x7efc000c4fa0 0x7efc000c77b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:16.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.481+0000 7efbfea00640 1 -- 172.21.15.60:0/1980168792 >> v2:172.21.15.60:3300/0 conn(0x7efc000c5380 msgr2=0x7efc0015a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:51:16.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.481+0000 7efbfea00640 1 --2- 172.21.15.60:0/1980168792 >> v2:172.21.15.60:3300/0 conn(0x7efc000c5380 0x7efc0015a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:16.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.481+0000 7efbfea00640 1 -- 172.21.15.60:0/1980168792 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc000c6cd0 con 0x7efc00070d50 2024-04-14T23:51:16.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.481+0000 7efbffe00640 1 --2- 172.21.15.60:0/1980168792 >> v2:172.21.15.60:3300/0 conn(0x7efc000c5380 0x7efc0015a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:51:16.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.481+0000 7efbff400640 1 --2- 172.21.15.60:0/1980168792 >> v2:172.21.15.186:3300/0 conn(0x7efc000c4fa0 0x7efc000c77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:51:16.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.482+0000 7efbfea00640 1 --2- 172.21.15.60:0/1980168792 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efc00070d50 0x7efc000c7cf0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7efc000b1ff0 tx=0x7efbec02b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb3137a4ebb6e59c server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:51:16.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.482+0000 7efbfe000640 1 -- 172.21.15.60:0/1980168792 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbec035020 con 0x7efc00070d50 2024-04-14T23:51:16.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.482+0000 7efbfe000640 1 -- 172.21.15.60:0/1980168792 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efbec031870 con 0x7efc00070d50 2024-04-14T23:51:16.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.482+0000 7efbfe000640 1 -- 172.21.15.60:0/1980168792 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbec031c40 con 0x7efc00070d50 2024-04-14T23:51:16.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.483+0000 7efc04e00640 1 -- 172.21.15.60:0/1980168792 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efc00070d50 msgr2=0x7efc000c7cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:16.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.483+0000 7efc04e00640 1 --2- 172.21.15.60:0/1980168792 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efc00070d50 0x7efc000c7cf0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7efc000b1ff0 tx=0x7efbec02b910 comp rx=0 tx=0).stop 2024-04-14T23:51:16.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.483+0000 7efc04e00640 1 -- 172.21.15.60:0/1980168792 shutdown_connections 2024-04-14T23:51:16.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.483+0000 7efc04e00640 1 --2- 172.21.15.60:0/1980168792 >> v2:172.21.15.60:3300/0 conn(0x7efc000c5380 0x7efc0015a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:16.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.483+0000 7efc04e00640 1 --2- 172.21.15.60:0/1980168792 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efc00070d50 0x7efc000c7cf0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:16.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.483+0000 7efc04e00640 1 --2- 172.21.15.60:0/1980168792 >> v2:172.21.15.186:3300/0 conn(0x7efc000c4fa0 0x7efc000c77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:16.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.483+0000 7efc04e00640 1 -- 172.21.15.60:0/1980168792 >> 172.21.15.60:0/1980168792 conn(0x7efc000c8b50 msgr2=0x7efc000c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:51:16.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.483+0000 7efc04e00640 1 -- 172.21.15.60:0/1980168792 shutdown_connections 2024-04-14T23:51:16.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.483+0000 7efc04e00640 1 -- 172.21.15.60:0/1980168792 wait complete. 2024-04-14T23:51:16.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.485+0000 7efc04e00640 1 Processor -- start 2024-04-14T23:51:16.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.485+0000 7efc04e00640 1 -- start start 2024-04-14T23:51:16.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.486+0000 7efc04e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efc00070d50 0x7efc001e33d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:16.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.486+0000 7efc04e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efc000c4fa0 0x7efc00152d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:16.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.486+0000 7efc04e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc000c5380 0x7efc00153240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:16.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.486+0000 7efc04e00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7efc0006b1c0 con 0x7efc000c5380 2024-04-14T23:51:16.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.486+0000 7efc04e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7efc000c7290 con 0x7efc00070d50 2024-04-14T23:51:16.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.486+0000 7efc04e00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7efc000c99d0 con 0x7efc000c4fa0 2024-04-14T23:51:16.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.486+0000 7efbffe00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc000c5380 0x7efc00153240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:16.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.487+0000 7efbff400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efc00070d50 0x7efc001e33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:16.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.487+0000 7efbffe00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc000c5380 0x7efc00153240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40410/0 (socket says 172.21.15.60:40410) 2024-04-14T23:51:16.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.487+0000 7efbffe00640 1 -- 172.21.15.60:0/1649377375 learned_addr learned my addr 172.21.15.60:0/1649377375 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:51:16.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.487+0000 7efbfea00640 1 --2- 172.21.15.60:0/1649377375 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efc000c4fa0 0x7efc00152d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:16.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.487+0000 7efbffe00640 1 -- 172.21.15.60:0/1649377375 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efc000c4fa0 msgr2=0x7efc00152d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:16.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.487+0000 7efbffe00640 1 --2- 172.21.15.60:0/1649377375 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efc000c4fa0 0x7efc00152d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:16.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.487+0000 7efbffe00640 1 -- 172.21.15.60:0/1649377375 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efc00070d50 msgr2=0x7efc001e33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:16.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.487+0000 7efbffe00640 1 --2- 172.21.15.60:0/1649377375 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efc00070d50 0x7efc001e33d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:16.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.487+0000 7efbffe00640 1 -- 172.21.15.60:0/1649377375 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc00072930 con 0x7efc000c5380 2024-04-14T23:51:16.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.487+0000 7efbff400640 1 --2- 172.21.15.60:0/1649377375 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efc00070d50 0x7efc001e33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:51:16.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.487+0000 7efbffe00640 1 --2- 172.21.15.60:0/1649377375 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc000c5380 0x7efc00153240 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7efbf800efc0 tx=0x7efbf8009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:51:16.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.488+0000 7efbf7400640 1 -- 172.21.15.60:0/1649377375 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbf8018030 con 0x7efc000c5380 2024-04-14T23:51:16.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.488+0000 7efbf7400640 1 -- 172.21.15.60:0/1649377375 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efbf800cde0 con 0x7efc000c5380 2024-04-14T23:51:16.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.488+0000 7efbf7400640 1 -- 172.21.15.60:0/1649377375 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbf8003bb0 con 0x7efc000c5380 2024-04-14T23:51:16.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.488+0000 7efc04e00640 1 -- 172.21.15.60:0/1649377375 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc000673a0 con 0x7efc000c5380 2024-04-14T23:51:16.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.488+0000 7efc04e00640 1 -- 172.21.15.60:0/1649377375 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc00150910 con 0x7efc000c5380 2024-04-14T23:51:16.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.489+0000 7efc04e00640 1 -- 172.21.15.60:0/1649377375 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc0004f540 con 0x7efc000c5380 2024-04-14T23:51:16.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.490+0000 7efbf7400640 1 -- 172.21.15.60:0/1649377375 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7efbf801b070 con 0x7efc000c5380 2024-04-14T23:51:16.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.491+0000 7efbf7400640 1 --2- 172.21.15.60:0/1649377375 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7efbe006b990 0x7efbe006de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:16.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.491+0000 7efbf7400640 1 -- 172.21.15.60:0/1649377375 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 5911+0+0 (secure 0 0 0) 0x7efbf808b320 con 0x7efc000c5380 2024-04-14T23:51:16.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.491+0000 7efbff400640 1 --2- 172.21.15.60:0/1649377375 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7efbe006b990 0x7efbe006de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:16.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.492+0000 7efbff400640 1 --2- 172.21.15.60:0/1649377375 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7efbe006b990 0x7efbe006de50 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7efbf0009560 tx=0x7efbf0009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:51:16.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.496+0000 7efbf7400640 1 -- 172.21.15.60:0/1649377375 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7efc00072930 con 0x7efc000c5380 2024-04-14T23:51:16.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.738+0000 7efc04e00640 1 -- 172.21.15.60:0/1649377375 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efc000cbbd0 con 0x7efc000c5380 2024-04-14T23:51:16.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.739+0000 7efbf7400640 1 -- 172.21.15.60:0/1649377375 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+6000 (secure 0 0 0) 0x7efbf8010040 con 0x7efc000c5380 2024-04-14T23:51:16.744 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:51:16.745 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":11,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":11,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:51:09.965711+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4276},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4276":{"gid":4276,"name":"a","rank":0,"incarnation":9,"state":"up:rejoin","state_seq":41,"addr":"172.21.15.60:6833/78303888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":78303888},{"type":"v1","addr":"172.21.15.60:6833","nonce":78303888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-14T23:51:16.745 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 11 2024-04-14T23:51:16.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.742+0000 7efc04e00640 1 -- 172.21.15.60:0/1649377375 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7efbe006b990 msgr2=0x7efbe006de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:16.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.742+0000 7efc04e00640 1 --2- 172.21.15.60:0/1649377375 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7efbe006b990 0x7efbe006de50 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7efbf0009560 tx=0x7efbf0009190 comp rx=0 tx=0).stop 2024-04-14T23:51:16.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.742+0000 7efc04e00640 1 -- 172.21.15.60:0/1649377375 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc000c5380 msgr2=0x7efc00153240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:16.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.742+0000 7efc04e00640 1 --2- 172.21.15.60:0/1649377375 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc000c5380 0x7efc00153240 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7efbf800efc0 tx=0x7efbf8009f90 comp rx=0 tx=0).stop 2024-04-14T23:51:16.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.742+0000 7efc04e00640 1 -- 172.21.15.60:0/1649377375 shutdown_connections 2024-04-14T23:51:16.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.742+0000 7efc04e00640 1 --2- 172.21.15.60:0/1649377375 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7efbe006b990 0x7efbe006de50 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:16.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.742+0000 7efc04e00640 1 --2- 172.21.15.60:0/1649377375 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc000c5380 0x7efc00153240 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:16.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.742+0000 7efc04e00640 1 --2- 172.21.15.60:0/1649377375 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efc000c4fa0 0x7efc00152d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:16.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.742+0000 7efc04e00640 1 --2- 172.21.15.60:0/1649377375 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efc00070d50 0x7efc001e33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:16.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.742+0000 7efc04e00640 1 -- 172.21.15.60:0/1649377375 >> 172.21.15.60:0/1649377375 conn(0x7efc000c8b50 msgr2=0x7efc000c8360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:51:16.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.742+0000 7efc04e00640 1 -- 172.21.15.60:0/1649377375 shutdown_connections 2024-04-14T23:51:16.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:16.742+0000 7efc04e00640 1 -- 172.21.15.60:0/1649377375 wait complete. 2024-04-14T23:51:16.767 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-14T23:51:18.768 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:51:18.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:18.993+0000 7f1bca400640 1 Processor -- start 2024-04-14T23:51:18.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:18.994+0000 7f1bca400640 1 -- start start 2024-04-14T23:51:19.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:18.995+0000 7f1bca400640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f1bc40c6950 0x7f1bc406b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:19.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:18.995+0000 7f1bca400640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bc406b850 con 0x7f1bc40c7080 2024-04-14T23:51:19.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:18.995+0000 7f1bca400640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bc406bb30 con 0x7f1bc415ba70 2024-04-14T23:51:19.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:18.995+0000 7f1bca400640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f1bc406c740 con 0x7f1bc40c6950 2024-04-14T23:51:19.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:18.995+0000 7f1bc3e00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f1bc40c6950 0x7f1bc406b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:19.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:18.995+0000 7f1bc9000640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f1bc40c7080 0x7f1bc415b360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:41848/0 (socket says 172.21.15.60:41848) 2024-04-14T23:51:19.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:18.996+0000 7f1bc9000640 1 -- 172.21.15.60:0/2899669803 learned_addr learned my addr 172.21.15.60:0/2899669803 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:51:19.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:18.996+0000 7f1bc3e00640 1 -- 172.21.15.60:0/2899669803 >> v1:172.21.15.186:6789/0 conn(0x7f1bc415ba70 legacy=0x7f1bc4070d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:19.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:18.996+0000 7f1bc3e00640 1 -- 172.21.15.60:0/2899669803 >> v1:172.21.15.60:6789/0 conn(0x7f1bc40c7080 legacy=0x7f1bc415b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:19.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:18.996+0000 7f1bc3e00640 1 -- 172.21.15.60:0/2899669803 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bc406be10 con 0x7f1bc40c6950 2024-04-14T23:51:19.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:18.997+0000 7f1bc3e00640 1 --2- 172.21.15.60:0/2899669803 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1bc40c6950 0x7f1bc406b1c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1bb00097a0 tx=0x7f1bb002b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6040d71e00d433a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:51:19.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:18.997+0000 7f1bc3400640 1 -- 172.21.15.60:0/2899669803 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bb002f810 con 0x7f1bc40c6950 2024-04-14T23:51:19.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:18.997+0000 7f1bc3400640 1 -- 172.21.15.60:0/2899669803 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bb002f9b0 con 0x7f1bc40c6950 2024-04-14T23:51:19.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:18.997+0000 7f1bc3400640 1 -- 172.21.15.60:0/2899669803 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bb002fd80 con 0x7f1bc40c6950 2024-04-14T23:51:19.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:18.998+0000 7f1bca400640 1 -- 172.21.15.60:0/2899669803 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1bc40c6950 msgr2=0x7f1bc406b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:19.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:18.998+0000 7f1bca400640 1 --2- 172.21.15.60:0/2899669803 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1bc40c6950 0x7f1bc406b1c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1bb00097a0 tx=0x7f1bb002b9f0 comp rx=0 tx=0).stop 2024-04-14T23:51:19.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:18.998+0000 7f1bca400640 1 -- 172.21.15.60:0/2899669803 shutdown_connections 2024-04-14T23:51:19.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:18.998+0000 7f1bca400640 1 --2- 172.21.15.60:0/2899669803 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1bc40c6950 0x7f1bc406b1c0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:19.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:18.998+0000 7f1bca400640 1 -- 172.21.15.60:0/2899669803 >> 172.21.15.60:0/2899669803 conn(0x7f1bc40c8b50 msgr2=0x7f1bc40c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:51:19.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:18.998+0000 7f1bca400640 1 -- 172.21.15.60:0/2899669803 shutdown_connections 2024-04-14T23:51:19.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:18.998+0000 7f1bca400640 1 -- 172.21.15.60:0/2899669803 wait complete. 2024-04-14T23:51:19.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:19.000+0000 7f1bca400640 1 Processor -- start 2024-04-14T23:51:19.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:19.001+0000 7f1bca400640 1 -- start start 2024-04-14T23:51:19.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:19.002+0000 7f1bca400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1bc40c6950 0x7f1bc41f4700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:19.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:19.002+0000 7f1bca400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1bc40c7080 0x7f1bc41e7490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:19.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:19.002+0000 7f1bca400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1bc415ba70 0x7f1bc41f0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:19.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:19.002+0000 7f1bca400640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f1bc40c7460 con 0x7f1bc40c6950 2024-04-14T23:51:19.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:19.002+0000 7f1bca400640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1bc406c3d0 con 0x7f1bc415ba70 2024-04-14T23:51:19.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:19.002+0000 7f1bca400640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f1bc40c75e0 con 0x7f1bc40c7080 2024-04-14T23:51:19.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:19.002+0000 7f1bc9000640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1bc40c6950 0x7f1bc41f4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:19.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:19.002+0000 7f1bc9000640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1bc40c6950 0x7f1bc41f4700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40424/0 (socket says 172.21.15.60:40424) 2024-04-14T23:51:19.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:19.002+0000 7f1bc9000640 1 -- 172.21.15.60:0/2199295089 learned_addr learned my addr 172.21.15.60:0/2199295089 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:51:19.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:19.002+0000 7f1bc9a00640 1 --2- 172.21.15.60:0/2199295089 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1bc415ba70 0x7f1bc41f0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:19.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:19.002+0000 7f1bc3e00640 1 --2- 172.21.15.60:0/2199295089 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1bc40c7080 0x7f1bc41e7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:19.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:19.003+0000 7f1bc9000640 1 -- 172.21.15.60:0/2199295089 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1bc40c7080 msgr2=0x7f1bc41e7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:19.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:19.003+0000 7f1bc9000640 1 --2- 172.21.15.60:0/2199295089 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1bc40c7080 0x7f1bc41e7490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:19.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:19.003+0000 7f1bc9000640 1 -- 172.21.15.60:0/2199295089 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1bc415ba70 msgr2=0x7f1bc41f0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:19.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:19.003+0000 7f1bc9000640 1 --2- 172.21.15.60:0/2199295089 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1bc415ba70 0x7f1bc41f0ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:19.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:19.003+0000 7f1bc9000640 1 -- 172.21.15.60:0/2199295089 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bc40aea20 con 0x7f1bc40c6950 2024-04-14T23:51:19.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:19.003+0000 7f1bc9000640 1 --2- 172.21.15.60:0/2199295089 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1bc40c6950 0x7f1bc41f4700 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1bb800c430 tx=0x7f1bb800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:51:19.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:19.004+0000 7f1bc0c00640 1 -- 172.21.15.60:0/2199295089 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bb8017070 con 0x7f1bc40c6950 2024-04-14T23:51:19.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:19.004+0000 7f1bc0c00640 1 -- 172.21.15.60:0/2199295089 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bb8005ce0 con 0x7f1bc40c6950 2024-04-14T23:51:19.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:19.004+0000 7f1bc0c00640 1 -- 172.21.15.60:0/2199295089 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bb80072f0 con 0x7f1bc40c6950 2024-04-14T23:51:19.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:19.004+0000 7f1bca400640 1 -- 172.21.15.60:0/2199295089 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bc40aed00 con 0x7f1bc40c6950 2024-04-14T23:51:19.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:19.004+0000 7f1bca400640 1 -- 172.21.15.60:0/2199295089 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bc41574e0 con 0x7f1bc40c6950 2024-04-14T23:51:19.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:19.005+0000 7f1bca400640 1 -- 172.21.15.60:0/2199295089 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bc404f540 con 0x7f1bc40c6950 2024-04-14T23:51:19.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:19.006+0000 7f1bc0c00640 1 -- 172.21.15.60:0/2199295089 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f1bb801a070 con 0x7f1bc40c6950 2024-04-14T23:51:19.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:19.007+0000 7f1bc0c00640 1 --2- 172.21.15.60:0/2199295089 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f1ba406b990 0x7f1ba406de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:19.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:19.008+0000 7f1bc0c00640 1 -- 172.21.15.60:0/2199295089 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 5911+0+0 (secure 0 0 0) 0x7f1bb8089de0 con 0x7f1bc40c6950 2024-04-14T23:51:19.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:19.008+0000 7f1bc3e00640 1 --2- 172.21.15.60:0/2199295089 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f1ba406b990 0x7f1ba406de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:19.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:19.009+0000 7f1bc3e00640 1 --2- 172.21.15.60:0/2199295089 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f1ba406b990 0x7f1ba406de50 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1bb0036000 tx=0x7f1bb0008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:51:19.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:19.013+0000 7f1bc0c00640 1 -- 172.21.15.60:0/2199295089 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1bb80581a0 con 0x7f1bc40c6950 2024-04-14T23:51:19.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:19.255+0000 7f1bca400640 1 -- 172.21.15.60:0/2199295089 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1bc40c7970 con 0x7f1bc40c6950 2024-04-14T23:51:19.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:19.256+0000 7f1bc0c00640 1 -- 172.21.15.60:0/2199295089 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+6006 (secure 0 0 0) 0x7f1bb805c240 con 0x7f1bc40c6950 2024-04-14T23:51:19.261 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:51:19.261 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:51:17.000890+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4276},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4276":{"gid":4276,"name":"a","rank":0,"incarnation":9,"state":"up:clientreplay","state_seq":43,"addr":"172.21.15.60:6833/78303888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":78303888},{"type":"v1","addr":"172.21.15.60:6833","nonce":78303888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-14T23:51:19.262 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 12 2024-04-14T23:51:19.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:19.259+0000 7f1bca400640 1 -- 172.21.15.60:0/2199295089 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f1ba406b990 msgr2=0x7f1ba406de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:19.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:19.259+0000 7f1bca400640 1 --2- 172.21.15.60:0/2199295089 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f1ba406b990 0x7f1ba406de50 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1bb0036000 tx=0x7f1bb0008030 comp rx=0 tx=0).stop 2024-04-14T23:51:19.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:19.259+0000 7f1bca400640 1 -- 172.21.15.60:0/2199295089 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1bc40c6950 msgr2=0x7f1bc41f4700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:19.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:19.259+0000 7f1bca400640 1 --2- 172.21.15.60:0/2199295089 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1bc40c6950 0x7f1bc41f4700 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1bb800c430 tx=0x7f1bb800c8f0 comp rx=0 tx=0).stop 2024-04-14T23:51:19.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:19.259+0000 7f1bca400640 1 -- 172.21.15.60:0/2199295089 shutdown_connections 2024-04-14T23:51:19.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:19.259+0000 7f1bca400640 1 --2- 172.21.15.60:0/2199295089 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f1ba406b990 0x7f1ba406de50 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:19.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:19.259+0000 7f1bca400640 1 --2- 172.21.15.60:0/2199295089 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1bc415ba70 0x7f1bc41f0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:19.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:19.259+0000 7f1bca400640 1 --2- 172.21.15.60:0/2199295089 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1bc40c7080 0x7f1bc41e7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:19.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:19.259+0000 7f1bca400640 1 --2- 172.21.15.60:0/2199295089 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1bc40c6950 0x7f1bc41f4700 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:19.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:19.259+0000 7f1bca400640 1 -- 172.21.15.60:0/2199295089 >> 172.21.15.60:0/2199295089 conn(0x7f1bc40c8b50 msgr2=0x7f1bc4072120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:51:19.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:19.259+0000 7f1bca400640 1 -- 172.21.15.60:0/2199295089 shutdown_connections 2024-04-14T23:51:19.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:19.259+0000 7f1bca400640 1 -- 172.21.15.60:0/2199295089 wait complete. 2024-04-14T23:51:19.285 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-14T23:51:21.286 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:51:21.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.509+0000 7fd98cc00640 1 Processor -- start 2024-04-14T23:51:21.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.509+0000 7fd98cc00640 1 -- start start 2024-04-14T23:51:21.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.510+0000 7fd98cc00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd9880b1e00 0x7fd9880c6950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:21.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.511+0000 7fd98cc00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd9880c3d90 0x7fd9880c41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:21.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.511+0000 7fd98cc00640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd988152fb0 con 0x7fd988073320 2024-04-14T23:51:21.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.511+0000 7fd98cc00640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fd9880c6540 con 0x7fd9880b1e00 2024-04-14T23:51:21.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.511+0000 7fd98cc00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fd9880b21e0 con 0x7fd9880c3d90 2024-04-14T23:51:21.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.511+0000 7fd987400640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd9880b1e00 0x7fd9880c6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:21.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.511+0000 7fd987400640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd9880b1e00 0x7fd9880c6950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:54292/0 (socket says 172.21.15.60:54292) 2024-04-14T23:51:21.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.511+0000 7fd986a00640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fd988073320 0x7fd9880c7fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.60:34058/0 (socket says 172.21.15.60:34058) 2024-04-14T23:51:21.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.511+0000 7fd987400640 1 -- 172.21.15.60:0/143218389 learned_addr learned my addr 172.21.15.60:0/143218389 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:51:21.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.511+0000 7fd987e00640 1 --2- 172.21.15.60:0/143218389 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd9880c3d90 0x7fd9880c41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:21.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.511+0000 7fd987400640 1 -- 172.21.15.60:0/143218389 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd9880c3d90 msgr2=0x7fd9880c41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:51:21.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.511+0000 7fd987400640 1 --2- 172.21.15.60:0/143218389 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd9880c3d90 0x7fd9880c41c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:21.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.511+0000 7fd987400640 1 -- 172.21.15.60:0/143218389 >> v1:172.21.15.186:6789/0 conn(0x7fd988073320 legacy=0x7fd9880c7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:21.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.511+0000 7fd987400640 1 -- 172.21.15.60:0/143218389 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd988159310 con 0x7fd9880b1e00 2024-04-14T23:51:21.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.511+0000 7fd986000640 1 -- 172.21.15.60:0/143218389 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2192744753 0 0) 0x7fd988152fb0 con 0x7fd988073320 2024-04-14T23:51:21.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.512+0000 7fd987400640 1 --2- 172.21.15.60:0/143218389 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9880b1e00 0x7fd9880c6950 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd978009870 tx=0x7fd97802ba20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb9330906a6ac5b server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:51:21.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.513+0000 7fd986000640 1 -- 172.21.15.60:0/143218389 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd97802f840 con 0x7fd9880b1e00 2024-04-14T23:51:21.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.513+0000 7fd986000640 1 -- 172.21.15.60:0/143218389 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd97802f9e0 con 0x7fd9880b1e00 2024-04-14T23:51:21.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.513+0000 7fd986000640 1 -- 172.21.15.60:0/143218389 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd97802fdb0 con 0x7fd9880b1e00 2024-04-14T23:51:21.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.514+0000 7fd98cc00640 1 -- 172.21.15.60:0/143218389 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9880b1e00 msgr2=0x7fd9880c6950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:21.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.514+0000 7fd98cc00640 1 --2- 172.21.15.60:0/143218389 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9880b1e00 0x7fd9880c6950 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd978009870 tx=0x7fd97802ba20 comp rx=0 tx=0).stop 2024-04-14T23:51:21.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.514+0000 7fd98cc00640 1 -- 172.21.15.60:0/143218389 shutdown_connections 2024-04-14T23:51:21.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.514+0000 7fd98cc00640 1 --2- 172.21.15.60:0/143218389 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd9880c3d90 0x7fd9880c41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:21.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.514+0000 7fd98cc00640 1 --2- 172.21.15.60:0/143218389 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9880b1e00 0x7fd9880c6950 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:21.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.514+0000 7fd98cc00640 1 -- 172.21.15.60:0/143218389 >> 172.21.15.60:0/143218389 conn(0x7fd9880c8b50 msgr2=0x7fd9880c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:51:21.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.514+0000 7fd98cc00640 1 -- 172.21.15.60:0/143218389 shutdown_connections 2024-04-14T23:51:21.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.514+0000 7fd98cc00640 1 -- 172.21.15.60:0/143218389 wait complete. 2024-04-14T23:51:21.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.516+0000 7fd98cc00640 1 Processor -- start 2024-04-14T23:51:21.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.516+0000 7fd98cc00640 1 -- start start 2024-04-14T23:51:21.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.518+0000 7fd98cc00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd988073320 0x7fd988157c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:21.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.518+0000 7fd98cc00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9880b1e00 0x7fd9880afcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:21.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.518+0000 7fd98cc00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd9880c3d90 0x7fd9881f0920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:21.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.518+0000 7fd98cc00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fd9880c38b0 con 0x7fd9880b1e00 2024-04-14T23:51:21.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.518+0000 7fd98cc00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd988159890 con 0x7fd988073320 2024-04-14T23:51:21.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.518+0000 7fd98cc00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fd9881595b0 con 0x7fd9880c3d90 2024-04-14T23:51:21.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.518+0000 7fd986a00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9880b1e00 0x7fd9880afcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:21.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.518+0000 7fd987400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd988073320 0x7fd988157c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:21.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.518+0000 7fd986a00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9880b1e00 0x7fd9880afcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40426/0 (socket says 172.21.15.60:40426) 2024-04-14T23:51:21.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.518+0000 7fd986a00640 1 -- 172.21.15.60:0/3072977530 learned_addr learned my addr 172.21.15.60:0/3072977530 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:51:21.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.518+0000 7fd987e00640 1 --2- 172.21.15.60:0/3072977530 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd9880c3d90 0x7fd9881f0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:21.524 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.519+0000 7fd987400640 1 -- 172.21.15.60:0/3072977530 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd9880c3d90 msgr2=0x7fd9881f0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:21.525 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.519+0000 7fd987400640 1 --2- 172.21.15.60:0/3072977530 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd9880c3d90 0x7fd9881f0920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:21.525 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.519+0000 7fd987400640 1 -- 172.21.15.60:0/3072977530 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9880b1e00 msgr2=0x7fd9880afcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:21.525 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.519+0000 7fd987400640 1 --2- 172.21.15.60:0/3072977530 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9880b1e00 0x7fd9880afcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:21.525 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.519+0000 7fd987400640 1 -- 172.21.15.60:0/3072977530 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd988155f60 con 0x7fd988073320 2024-04-14T23:51:21.525 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.519+0000 7fd987e00640 1 --2- 172.21.15.60:0/3072977530 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd9880c3d90 0x7fd9881f0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:51:21.525 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.519+0000 7fd987400640 1 --2- 172.21.15.60:0/3072977530 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd988073320 0x7fd988157c90 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd978033000 tx=0x7fd978004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:51:21.525 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.519+0000 7fd97f400640 1 -- 172.21.15.60:0/3072977530 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd978037bc0 con 0x7fd988073320 2024-04-14T23:51:21.525 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.519+0000 7fd97f400640 1 -- 172.21.15.60:0/3072977530 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd978046070 con 0x7fd988073320 2024-04-14T23:51:21.525 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.519+0000 7fd98cc00640 1 -- 172.21.15.60:0/3072977530 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9881f4d80 con 0x7fd988073320 2024-04-14T23:51:21.525 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.519+0000 7fd97f400640 1 -- 172.21.15.60:0/3072977530 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd978041400 con 0x7fd988073320 2024-04-14T23:51:21.525 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.520+0000 7fd98cc00640 1 -- 172.21.15.60:0/3072977530 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9881f3fc0 con 0x7fd988073320 2024-04-14T23:51:21.526 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.521+0000 7fd98cc00640 1 -- 172.21.15.60:0/3072977530 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9880c8940 con 0x7fd988073320 2024-04-14T23:51:21.526 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.522+0000 7fd97f400640 1 -- 172.21.15.60:0/3072977530 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7fd978040070 con 0x7fd988073320 2024-04-14T23:51:21.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.523+0000 7fd97f400640 1 --2- 172.21.15.60:0/3072977530 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fd96806b990 0x7fd96806de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:21.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.523+0000 7fd97f400640 1 -- 172.21.15.60:0/3072977530 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 5911+0+0 (secure 0 0 0) 0x7fd9780ad930 con 0x7fd988073320 2024-04-14T23:51:21.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.523+0000 7fd986a00640 1 --2- 172.21.15.60:0/3072977530 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fd96806b990 0x7fd96806de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:21.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.524+0000 7fd986a00640 1 --2- 172.21.15.60:0/3072977530 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fd96806b990 0x7fd96806de50 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd974007fa0 tx=0x7fd974007f10 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:51:21.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.526+0000 7fd97f400640 1 -- 172.21.15.60:0/3072977530 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd9880c8940 con 0x7fd988073320 2024-04-14T23:51:21.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.770+0000 7fd98cc00640 1 -- 172.21.15.60:0/3072977530 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd98804f540 con 0x7fd988073320 2024-04-14T23:51:21.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.771+0000 7fd97f400640 1 -- 172.21.15.60:0/3072977530 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+6006 (secure 0 0 0) 0x7fd978033030 con 0x7fd988073320 2024-04-14T23:51:21.776 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:51:21.776 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:51:17.000890+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4276},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4276":{"gid":4276,"name":"a","rank":0,"incarnation":9,"state":"up:clientreplay","state_seq":43,"addr":"172.21.15.60:6833/78303888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":78303888},{"type":"v1","addr":"172.21.15.60:6833","nonce":78303888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-14T23:51:21.776 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 12 2024-04-14T23:51:21.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.774+0000 7fd98cc00640 1 -- 172.21.15.60:0/3072977530 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fd96806b990 msgr2=0x7fd96806de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:21.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.774+0000 7fd98cc00640 1 --2- 172.21.15.60:0/3072977530 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fd96806b990 0x7fd96806de50 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd974007fa0 tx=0x7fd974007f10 comp rx=0 tx=0).stop 2024-04-14T23:51:21.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.774+0000 7fd98cc00640 1 -- 172.21.15.60:0/3072977530 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd988073320 msgr2=0x7fd988157c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:21.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.774+0000 7fd98cc00640 1 --2- 172.21.15.60:0/3072977530 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd988073320 0x7fd988157c90 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd978033000 tx=0x7fd978004730 comp rx=0 tx=0).stop 2024-04-14T23:51:21.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.774+0000 7fd98cc00640 1 -- 172.21.15.60:0/3072977530 shutdown_connections 2024-04-14T23:51:21.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.774+0000 7fd98cc00640 1 --2- 172.21.15.60:0/3072977530 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fd96806b990 0x7fd96806de50 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:21.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.774+0000 7fd98cc00640 1 --2- 172.21.15.60:0/3072977530 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd9880c3d90 0x7fd9881f0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:21.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.774+0000 7fd98cc00640 1 --2- 172.21.15.60:0/3072977530 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9880b1e00 0x7fd9880afcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:21.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.774+0000 7fd98cc00640 1 --2- 172.21.15.60:0/3072977530 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd988073320 0x7fd988157c90 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:21.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.774+0000 7fd98cc00640 1 -- 172.21.15.60:0/3072977530 >> 172.21.15.60:0/3072977530 conn(0x7fd9880c8b50 msgr2=0x7fd9880729c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:51:21.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.774+0000 7fd98cc00640 1 -- 172.21.15.60:0/3072977530 shutdown_connections 2024-04-14T23:51:21.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:21.774+0000 7fd98cc00640 1 -- 172.21.15.60:0/3072977530 wait complete. 2024-04-14T23:51:21.799 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-14T23:51:23.800 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:51:24.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.015+0000 7f74c1a00640 1 Processor -- start 2024-04-14T23:51:24.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.015+0000 7f74c1a00640 1 -- start start 2024-04-14T23:51:24.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.017+0000 7f74c1a00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f74bc0c5380 0x7f74bc15a980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:24.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.017+0000 7f74c1a00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74bc06b4b0 con 0x7f74bc070d50 2024-04-14T23:51:24.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.017+0000 7f74c1a00640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74bc06b790 con 0x7f74bc0c4fa0 2024-04-14T23:51:24.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.017+0000 7f74c1a00640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f74bc0c69f0 con 0x7f74bc0c5380 2024-04-14T23:51:24.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.017+0000 7f74c1000640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f74bc0c5380 0x7f74bc15a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:24.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.017+0000 7f74b3e00640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f74bc070d50 0x7f74bc15a270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:43954/0 (socket says 172.21.15.60:43954) 2024-04-14T23:51:24.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.017+0000 7f74bbe00640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f74bc0c4fa0 0x7f74bc0c77b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.60:37284/0 (socket says 172.21.15.60:37284) 2024-04-14T23:51:24.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.017+0000 7f74b3e00640 1 -- 172.21.15.60:0/594746788 learned_addr learned my addr 172.21.15.60:0/594746788 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:51:24.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.018+0000 7f74c1000640 1 -- 172.21.15.60:0/594746788 >> v1:172.21.15.186:6789/0 conn(0x7f74bc0c4fa0 legacy=0x7f74bc0c77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:24.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.018+0000 7f74c1000640 1 -- 172.21.15.60:0/594746788 >> v1:172.21.15.60:6789/0 conn(0x7f74bc070d50 legacy=0x7f74bc15a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:24.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.018+0000 7f74c1000640 1 -- 172.21.15.60:0/594746788 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74bc0c6cd0 con 0x7f74bc0c5380 2024-04-14T23:51:24.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.018+0000 7f74bb400640 1 -- 172.21.15.60:0/594746788 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2128660571 0 0) 0x7f74bc06b790 con 0x7f74bc0c4fa0 2024-04-14T23:51:24.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.019+0000 7f74c1000640 1 --2- 172.21.15.60:0/594746788 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74bc0c5380 0x7f74bc15a980 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f74b4009d90 tx=0x7f74b402bf40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=837b604623f34e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:51:24.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.020+0000 7f74bb400640 1 -- 172.21.15.60:0/594746788 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74b4039070 con 0x7f74bc0c5380 2024-04-14T23:51:24.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.020+0000 7f74bb400640 1 -- 172.21.15.60:0/594746788 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74b402ccf0 con 0x7f74bc0c5380 2024-04-14T23:51:24.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.020+0000 7f74bb400640 1 -- 172.21.15.60:0/594746788 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74b4034660 con 0x7f74bc0c5380 2024-04-14T23:51:24.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.020+0000 7f74c1a00640 1 -- 172.21.15.60:0/594746788 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74bc0c5380 msgr2=0x7f74bc15a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:24.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.020+0000 7f74c1a00640 1 --2- 172.21.15.60:0/594746788 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74bc0c5380 0x7f74bc15a980 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f74b4009d90 tx=0x7f74b402bf40 comp rx=0 tx=0).stop 2024-04-14T23:51:24.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.020+0000 7f74c1a00640 1 -- 172.21.15.60:0/594746788 shutdown_connections 2024-04-14T23:51:24.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.020+0000 7f74c1a00640 1 --2- 172.21.15.60:0/594746788 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74bc0c5380 0x7f74bc15a980 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:24.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.020+0000 7f74c1a00640 1 -- 172.21.15.60:0/594746788 >> 172.21.15.60:0/594746788 conn(0x7f74bc0c8b50 msgr2=0x7f74bc0c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:51:24.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.020+0000 7f74c1a00640 1 -- 172.21.15.60:0/594746788 shutdown_connections 2024-04-14T23:51:24.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.020+0000 7f74c1a00640 1 -- 172.21.15.60:0/594746788 wait complete. 2024-04-14T23:51:24.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.022+0000 7f74c1a00640 1 Processor -- start 2024-04-14T23:51:24.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.022+0000 7f74c1a00640 1 -- start start 2024-04-14T23:51:24.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.023+0000 7f74c1a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74bc070d50 0x7f74bc1ef4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:24.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.023+0000 7f74c1a00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74bc0c4fa0 0x7f74bc1f1810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:24.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.023+0000 7f74c1a00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74bc0c5380 0x7f74bc1f1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:24.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.023+0000 7f74c1a00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f74bc06b1c0 con 0x7f74bc0c5380 2024-04-14T23:51:24.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.023+0000 7f74c1a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f74bc0c7290 con 0x7f74bc070d50 2024-04-14T23:51:24.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.023+0000 7f74c1a00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f74bc0c99d0 con 0x7f74bc0c4fa0 2024-04-14T23:51:24.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.024+0000 7f74bbe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74bc070d50 0x7f74bc1ef4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:24.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.024+0000 7f74bbe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74bc070d50 0x7f74bc1ef4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:38222/0 (socket says 172.21.15.60:38222) 2024-04-14T23:51:24.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.024+0000 7f74b3e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74bc0c4fa0 0x7f74bc1f1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:24.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.024+0000 7f74bbe00640 1 -- 172.21.15.60:0/3849851096 learned_addr learned my addr 172.21.15.60:0/3849851096 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:51:24.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.024+0000 7f74c1000640 1 --2- 172.21.15.60:0/3849851096 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74bc0c5380 0x7f74bc1f1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:24.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.024+0000 7f74bbe00640 1 -- 172.21.15.60:0/3849851096 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74bc0c4fa0 msgr2=0x7f74bc1f1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:24.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.024+0000 7f74bbe00640 1 --2- 172.21.15.60:0/3849851096 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74bc0c4fa0 0x7f74bc1f1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:24.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.024+0000 7f74bbe00640 1 -- 172.21.15.60:0/3849851096 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74bc0c5380 msgr2=0x7f74bc1f1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:24.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.024+0000 7f74bbe00640 1 --2- 172.21.15.60:0/3849851096 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74bc0c5380 0x7f74bc1f1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:24.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.024+0000 7f74bbe00640 1 -- 172.21.15.60:0/3849851096 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74bc1570e0 con 0x7f74bc070d50 2024-04-14T23:51:24.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.024+0000 7f74bbe00640 1 --2- 172.21.15.60:0/3849851096 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74bc070d50 0x7f74bc1ef4a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f74ac00c4a0 tx=0x7f74ac00c960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:51:24.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.025+0000 7f74b8c00640 1 -- 172.21.15.60:0/3849851096 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74ac018070 con 0x7f74bc070d50 2024-04-14T23:51:24.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.025+0000 7f74b8c00640 1 -- 172.21.15.60:0/3849851096 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74ac0074e0 con 0x7f74bc070d50 2024-04-14T23:51:24.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.025+0000 7f74b8c00640 1 -- 172.21.15.60:0/3849851096 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74ac004a20 con 0x7f74bc070d50 2024-04-14T23:51:24.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.025+0000 7f74c1a00640 1 -- 172.21.15.60:0/3849851096 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74bc157380 con 0x7f74bc070d50 2024-04-14T23:51:24.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.025+0000 7f74c1a00640 1 -- 172.21.15.60:0/3849851096 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74bc1e4cd0 con 0x7f74bc070d50 2024-04-14T23:51:24.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.026+0000 7f74c1a00640 1 -- 172.21.15.60:0/3849851096 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74bc04f540 con 0x7f74bc070d50 2024-04-14T23:51:24.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.027+0000 7f74b8c00640 1 -- 172.21.15.60:0/3849851096 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f74ac01b070 con 0x7f74bc070d50 2024-04-14T23:51:24.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.028+0000 7f74b8c00640 1 --2- 172.21.15.60:0/3849851096 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f749806b9e0 0x7f749806dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:24.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.028+0000 7f74b8c00640 1 -- 172.21.15.60:0/3849851096 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 5911+0+0 (secure 0 0 0) 0x7f74ac089c70 con 0x7f74bc070d50 2024-04-14T23:51:24.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.028+0000 7f74b3e00640 1 --2- 172.21.15.60:0/3849851096 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f749806b9e0 0x7f749806dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:24.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.029+0000 7f74b3e00640 1 --2- 172.21.15.60:0/3849851096 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f749806b9e0 0x7f749806dea0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f74bc1e0430 tx=0x7f74a4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:51:24.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.033+0000 7f74b8c00640 1 -- 172.21.15.60:0/3849851096 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f74ac0581d0 con 0x7f74bc070d50 2024-04-14T23:51:24.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.277+0000 7f74c1a00640 1 -- 172.21.15.60:0/3849851096 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f74bc076aa0 con 0x7f74bc070d50 2024-04-14T23:51:24.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.278+0000 7f74b8c00640 1 -- 172.21.15.60:0/3849851096 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+6006 (secure 0 0 0) 0x7f74ac05c270 con 0x7f74bc070d50 2024-04-14T23:51:24.283 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:51:24.283 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:51:17.000890+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4276},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4276":{"gid":4276,"name":"a","rank":0,"incarnation":9,"state":"up:clientreplay","state_seq":43,"addr":"172.21.15.60:6833/78303888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":78303888},{"type":"v1","addr":"172.21.15.60:6833","nonce":78303888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-14T23:51:24.283 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 12 2024-04-14T23:51:24.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.280+0000 7f74c1a00640 1 -- 172.21.15.60:0/3849851096 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f749806b9e0 msgr2=0x7f749806dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:24.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.280+0000 7f74c1a00640 1 --2- 172.21.15.60:0/3849851096 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f749806b9e0 0x7f749806dea0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f74bc1e0430 tx=0x7f74a4009040 comp rx=0 tx=0).stop 2024-04-14T23:51:24.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.280+0000 7f74c1a00640 1 -- 172.21.15.60:0/3849851096 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74bc070d50 msgr2=0x7f74bc1ef4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:24.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.280+0000 7f74c1a00640 1 --2- 172.21.15.60:0/3849851096 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74bc070d50 0x7f74bc1ef4a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f74ac00c4a0 tx=0x7f74ac00c960 comp rx=0 tx=0).stop 2024-04-14T23:51:24.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.281+0000 7f74c1a00640 1 -- 172.21.15.60:0/3849851096 shutdown_connections 2024-04-14T23:51:24.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.281+0000 7f74c1a00640 1 --2- 172.21.15.60:0/3849851096 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f749806b9e0 0x7f749806dea0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:24.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.281+0000 7f74c1a00640 1 --2- 172.21.15.60:0/3849851096 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74bc0c5380 0x7f74bc1f1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:24.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.281+0000 7f74c1a00640 1 --2- 172.21.15.60:0/3849851096 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74bc0c4fa0 0x7f74bc1f1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:24.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.281+0000 7f74c1a00640 1 --2- 172.21.15.60:0/3849851096 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74bc070d50 0x7f74bc1ef4a0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:24.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.281+0000 7f74c1a00640 1 -- 172.21.15.60:0/3849851096 >> 172.21.15.60:0/3849851096 conn(0x7f74bc0c8b50 msgr2=0x7f74bc0c7b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:51:24.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.281+0000 7f74c1a00640 1 -- 172.21.15.60:0/3849851096 shutdown_connections 2024-04-14T23:51:24.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:24.281+0000 7f74c1a00640 1 -- 172.21.15.60:0/3849851096 wait complete. 2024-04-14T23:51:24.304 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-14T23:51:26.305 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:51:26.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.537+0000 7f3709400640 1 Processor -- start 2024-04-14T23:51:26.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.537+0000 7f3709400640 1 -- start start 2024-04-14T23:51:26.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.538+0000 7f3709400640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f37040b1e00 0x7f37040c6950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:26.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.538+0000 7f3709400640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3704073320 0x7f37040c7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:26.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.538+0000 7f3709400640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3704152fb0 con 0x7f37040c84e0 2024-04-14T23:51:26.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.538+0000 7f3709400640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f37040c6540 con 0x7f3704073320 2024-04-14T23:51:26.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.538+0000 7f3709400640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f37040b21e0 con 0x7f37040b1e00 2024-04-14T23:51:26.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.538+0000 7f3703400640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3704073320 0x7f37040c7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:26.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.538+0000 7f3703400640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3704073320 0x7f37040c7fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40208/0 (socket says 172.21.15.60:40208) 2024-04-14T23:51:26.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.538+0000 7f3703400640 1 -- 172.21.15.60:0/1867249724 learned_addr learned my addr 172.21.15.60:0/1867249724 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:51:26.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.539+0000 7f3703e00640 1 --2- 172.21.15.60:0/1867249724 >> v2:172.21.15.186:3300/0 conn(0x7f37040b1e00 0x7f37040c6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:26.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.539+0000 7f3703400640 1 -- 172.21.15.60:0/1867249724 >> v2:172.21.15.186:3300/0 conn(0x7f37040b1e00 msgr2=0x7f37040c6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:51:26.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.539+0000 7f3703400640 1 --2- 172.21.15.60:0/1867249724 >> v2:172.21.15.186:3300/0 conn(0x7f37040b1e00 0x7f37040c6950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:26.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.539+0000 7f3702a00640 1 -- 172.21.15.60:0/1867249724 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1693982040 0 0) 0x7f3704152fb0 con 0x7f37040c84e0 2024-04-14T23:51:26.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.539+0000 7f3703400640 1 -- 172.21.15.60:0/1867249724 >> v1:172.21.15.186:6789/0 conn(0x7f37040c84e0 legacy=0x7f37040c3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:26.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.539+0000 7f3703400640 1 -- 172.21.15.60:0/1867249724 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3704159310 con 0x7f3704073320 2024-04-14T23:51:26.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.540+0000 7f3703400640 1 --2- 172.21.15.60:0/1867249724 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3704073320 0x7f37040c7fa0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f37040c36e0 tx=0x7f36f402c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69558062b36b3843 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:51:26.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.541+0000 7f3702a00640 1 -- 172.21.15.60:0/1867249724 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36f402cbb0 con 0x7f3704073320 2024-04-14T23:51:26.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.541+0000 7f3702a00640 1 -- 172.21.15.60:0/1867249724 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36f402cd50 con 0x7f3704073320 2024-04-14T23:51:26.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.541+0000 7f3702a00640 1 -- 172.21.15.60:0/1867249724 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36f4034610 con 0x7f3704073320 2024-04-14T23:51:26.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.542+0000 7f3709400640 1 -- 172.21.15.60:0/1867249724 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3704073320 msgr2=0x7f37040c7fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:26.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.542+0000 7f3709400640 1 --2- 172.21.15.60:0/1867249724 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3704073320 0x7f37040c7fa0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f37040c36e0 tx=0x7f36f402c010 comp rx=0 tx=0).stop 2024-04-14T23:51:26.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.542+0000 7f3709400640 1 -- 172.21.15.60:0/1867249724 shutdown_connections 2024-04-14T23:51:26.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.542+0000 7f3709400640 1 --2- 172.21.15.60:0/1867249724 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3704073320 0x7f37040c7fa0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:26.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.542+0000 7f3709400640 1 --2- 172.21.15.60:0/1867249724 >> v2:172.21.15.186:3300/0 conn(0x7f37040b1e00 0x7f37040c6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:26.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.542+0000 7f3709400640 1 -- 172.21.15.60:0/1867249724 >> 172.21.15.60:0/1867249724 conn(0x7f37040c8b50 msgr2=0x7f37040c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:51:26.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.542+0000 7f3709400640 1 -- 172.21.15.60:0/1867249724 shutdown_connections 2024-04-14T23:51:26.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.542+0000 7f3709400640 1 -- 172.21.15.60:0/1867249724 wait complete. 2024-04-14T23:51:26.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.544+0000 7f3709400640 1 Processor -- start 2024-04-14T23:51:26.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.544+0000 7f3709400640 1 -- start start 2024-04-14T23:51:26.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.545+0000 7f3709400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3704073320 0x7f37040b0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:26.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.545+0000 7f3709400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f37040b1e00 0x7f3704157410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:26.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.545+0000 7f3709400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37040c84e0 0x7f37040aeb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:26.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.545+0000 7f3709400640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f37040c38b0 con 0x7f37040c84e0 2024-04-14T23:51:26.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.545+0000 7f3709400640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3704159890 con 0x7f37040b1e00 2024-04-14T23:51:26.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.545+0000 7f3709400640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f37040c5f80 con 0x7f3704073320 2024-04-14T23:51:26.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.545+0000 7f3703e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3704073320 0x7f37040b0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:26.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.545+0000 7f3703400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f37040b1e00 0x7f3704157410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:26.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.545+0000 7f3708a00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37040c84e0 0x7f37040aeb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:26.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.545+0000 7f3703e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3704073320 0x7f37040b0850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.60:58952/0 (socket says 172.21.15.60:58952) 2024-04-14T23:51:26.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.545+0000 7f3703e00640 1 -- 172.21.15.60:0/2516290316 learned_addr learned my addr 172.21.15.60:0/2516290316 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:51:26.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.546+0000 7f3703e00640 1 -- 172.21.15.60:0/2516290316 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f37040b1e00 msgr2=0x7f3704157410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:26.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.546+0000 7f3703e00640 1 --2- 172.21.15.60:0/2516290316 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f37040b1e00 0x7f3704157410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:26.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.546+0000 7f3703e00640 1 -- 172.21.15.60:0/2516290316 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37040c84e0 msgr2=0x7f37040aeb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:26.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.546+0000 7f3703e00640 1 --2- 172.21.15.60:0/2516290316 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37040c84e0 0x7f37040aeb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:26.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.546+0000 7f3703e00640 1 -- 172.21.15.60:0/2516290316 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3704155df0 con 0x7f3704073320 2024-04-14T23:51:26.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.546+0000 7f3708a00640 1 --2- 172.21.15.60:0/2516290316 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37040c84e0 0x7f37040aeb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:51:26.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.546+0000 7f3703e00640 1 --2- 172.21.15.60:0/2516290316 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3704073320 0x7f37040b0850 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f36ec00e9e0 tx=0x7f36ec00eea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:51:26.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.546+0000 7f36fbe00640 1 -- 172.21.15.60:0/2516290316 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36ec017070 con 0x7f3704073320 2024-04-14T23:51:26.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.546+0000 7f36fbe00640 1 -- 172.21.15.60:0/2516290316 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36ec00c9d0 con 0x7f3704073320 2024-04-14T23:51:26.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.546+0000 7f36fbe00640 1 -- 172.21.15.60:0/2516290316 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36ec00cdc0 con 0x7f3704073320 2024-04-14T23:51:26.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.546+0000 7f3709400640 1 -- 172.21.15.60:0/2516290316 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37041f10f0 con 0x7f3704073320 2024-04-14T23:51:26.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.547+0000 7f3709400640 1 -- 172.21.15.60:0/2516290316 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3704157950 con 0x7f3704073320 2024-04-14T23:51:26.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.548+0000 7f3709400640 1 -- 172.21.15.60:0/2516290316 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f370404f540 con 0x7f3704073320 2024-04-14T23:51:26.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.549+0000 7f36fbe00640 1 -- 172.21.15.60:0/2516290316 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f36ec01a070 con 0x7f3704073320 2024-04-14T23:51:26.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.549+0000 7f36fbe00640 1 --2- 172.21.15.60:0/2516290316 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f36e006b990 0x7f36e006de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:26.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.550+0000 7f36fbe00640 1 -- 172.21.15.60:0/2516290316 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 5911+0+0 (secure 0 0 0) 0x7f36ec08ac80 con 0x7f3704073320 2024-04-14T23:51:26.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.550+0000 7f3703400640 1 --2- 172.21.15.60:0/2516290316 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f36e006b990 0x7f36e006de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:26.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.551+0000 7f3703400640 1 --2- 172.21.15.60:0/2516290316 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f36e006b990 0x7f36e006de50 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f36f4035000 tx=0x7f36f4030000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:51:26.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.554+0000 7f36fbe00640 1 -- 172.21.15.60:0/2516290316 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f36ec0590d0 con 0x7f3704073320 2024-04-14T23:51:26.803 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.798+0000 7f3709400640 1 -- 172.21.15.60:0/2516290316 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f37040c99e0 con 0x7f3704073320 2024-04-14T23:51:26.803 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.799+0000 7f36fbe00640 1 -- 172.21.15.60:0/2516290316 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+6007 (secure 0 0 0) 0x7f36ec05d170 con 0x7f3704073320 2024-04-14T23:51:26.804 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:51:26.804 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:51:24.313553+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4276},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4276":{"gid":4276,"name":"a","rank":0,"incarnation":9,"state":"up:active","state_seq":45,"addr":"172.21.15.60:6833/78303888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":78303888},{"type":"v1","addr":"172.21.15.60:6833","nonce":78303888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4276,"qdb_cluster":[4276]},"id":1}]} 2024-04-14T23:51:26.804 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 13 2024-04-14T23:51:26.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.802+0000 7f3709400640 1 -- 172.21.15.60:0/2516290316 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f36e006b990 msgr2=0x7f36e006de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:26.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.802+0000 7f3709400640 1 --2- 172.21.15.60:0/2516290316 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f36e006b990 0x7f36e006de50 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f36f4035000 tx=0x7f36f4030000 comp rx=0 tx=0).stop 2024-04-14T23:51:26.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.802+0000 7f3709400640 1 -- 172.21.15.60:0/2516290316 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3704073320 msgr2=0x7f37040b0850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:26.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.802+0000 7f3709400640 1 --2- 172.21.15.60:0/2516290316 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3704073320 0x7f37040b0850 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f36ec00e9e0 tx=0x7f36ec00eea0 comp rx=0 tx=0).stop 2024-04-14T23:51:26.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.802+0000 7f3709400640 1 -- 172.21.15.60:0/2516290316 shutdown_connections 2024-04-14T23:51:26.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.802+0000 7f3709400640 1 --2- 172.21.15.60:0/2516290316 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f36e006b990 0x7f36e006de50 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:26.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.802+0000 7f3709400640 1 --2- 172.21.15.60:0/2516290316 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37040c84e0 0x7f37040aeb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:26.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.802+0000 7f3709400640 1 --2- 172.21.15.60:0/2516290316 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f37040b1e00 0x7f3704157410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:26.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.802+0000 7f3709400640 1 --2- 172.21.15.60:0/2516290316 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3704073320 0x7f37040b0850 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:26.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.802+0000 7f3709400640 1 -- 172.21.15.60:0/2516290316 >> 172.21.15.60:0/2516290316 conn(0x7f37040c8b50 msgr2=0x7f3704157f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:51:26.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.802+0000 7f3709400640 1 -- 172.21.15.60:0/2516290316 shutdown_connections 2024-04-14T23:51:26.808 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:26.802+0000 7f3709400640 1 -- 172.21.15.60:0/2516290316 wait complete. 2024-04-14T23:51:26.828 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=4279 2024-04-14T23:51:26.828 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 3 secs before reviving mds.c 2024-04-14T23:51:29.484 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 149 2024-04-14T23:51:29.829 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.c 2024-04-14T23:51:29.829 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-14T23:51:29.829 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-14T23:51:29.837 INFO:tasks.ceph.mds.c:Started 2024-04-14T23:51:29.838 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:51:30.015 INFO:tasks.ceph.mds.c.smithi060.stdout:starting mds.c at 2024-04-14T23:51:30.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.036+0000 7f8870e00640 1 Processor -- start 2024-04-14T23:51:30.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.036+0000 7f8870e00640 1 -- start start 2024-04-14T23:51:30.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.037+0000 7f8870e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f886c06f5d0 0x7f886c154980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:30.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.037+0000 7f8870e00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f886c0731e0 0x7f886c154ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:30.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.037+0000 7f8870e00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f886c15b0d0 con 0x7f886c06f1f0 2024-04-14T23:51:30.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.037+0000 7f8870e00640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f886c0c6e10 con 0x7f886c0731e0 2024-04-14T23:51:30.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.037+0000 7f8870e00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f886c15b370 con 0x7f886c06f5d0 2024-04-14T23:51:30.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.038+0000 7f8869e00640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f886c06f1f0 0x7f886c0c72a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:43962/0 (socket says 172.21.15.60:43962) 2024-04-14T23:51:30.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.038+0000 7f8869e00640 1 -- 172.21.15.60:0/3581300300 learned_addr learned my addr 172.21.15.60:0/3581300300 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:51:30.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.038+0000 7f8869400640 1 --2- 172.21.15.60:0/3581300300 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f886c06f5d0 0x7f886c154980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:30.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.038+0000 7f886a800640 1 --2- 172.21.15.60:0/3581300300 >> v2:172.21.15.186:3300/0 conn(0x7f886c0731e0 0x7f886c154ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:30.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.038+0000 7f8869400640 1 -- 172.21.15.60:0/3581300300 >> v2:172.21.15.186:3300/0 conn(0x7f886c0731e0 msgr2=0x7f886c154ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:51:30.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.038+0000 7f8869400640 1 --2- 172.21.15.60:0/3581300300 >> v2:172.21.15.186:3300/0 conn(0x7f886c0731e0 0x7f886c154ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:30.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.038+0000 7f8869400640 1 -- 172.21.15.60:0/3581300300 >> v1:172.21.15.60:6789/0 conn(0x7f886c06f1f0 legacy=0x7f886c0c72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:30.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.038+0000 7f8869400640 1 -- 172.21.15.60:0/3581300300 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f886c15b650 con 0x7f886c06f5d0 2024-04-14T23:51:30.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.038+0000 7f8868a00640 1 -- 172.21.15.60:0/3581300300 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2166511693 0 0) 0x7f886c15b0d0 con 0x7f886c06f1f0 2024-04-14T23:51:30.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.039+0000 7f8869400640 1 --2- 172.21.15.60:0/3581300300 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f886c06f5d0 0x7f886c154980 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f88580097a0 tx=0x7f885802b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=611bbc09433ea174 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:51:30.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.040+0000 7f8868a00640 1 -- 172.21.15.60:0/3581300300 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8858034070 con 0x7f886c06f5d0 2024-04-14T23:51:30.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.040+0000 7f8868a00640 1 -- 172.21.15.60:0/3581300300 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8858030610 con 0x7f886c06f5d0 2024-04-14T23:51:30.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.040+0000 7f8868a00640 1 -- 172.21.15.60:0/3581300300 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88580309e0 con 0x7f886c06f5d0 2024-04-14T23:51:30.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.040+0000 7f8870e00640 1 -- 172.21.15.60:0/3581300300 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f886c06f5d0 msgr2=0x7f886c154980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:30.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.040+0000 7f8870e00640 1 --2- 172.21.15.60:0/3581300300 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f886c06f5d0 0x7f886c154980 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f88580097a0 tx=0x7f885802b960 comp rx=0 tx=0).stop 2024-04-14T23:51:30.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.040+0000 7f8870e00640 1 -- 172.21.15.60:0/3581300300 shutdown_connections 2024-04-14T23:51:30.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.040+0000 7f8870e00640 1 --2- 172.21.15.60:0/3581300300 >> v2:172.21.15.186:3300/0 conn(0x7f886c0731e0 0x7f886c154ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:30.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.040+0000 7f8870e00640 1 --2- 172.21.15.60:0/3581300300 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f886c06f5d0 0x7f886c154980 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:30.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.040+0000 7f8870e00640 1 -- 172.21.15.60:0/3581300300 >> 172.21.15.60:0/3581300300 conn(0x7f886c0cc170 msgr2=0x7f886c0cc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:51:30.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.040+0000 7f8870e00640 1 -- 172.21.15.60:0/3581300300 shutdown_connections 2024-04-14T23:51:30.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.041+0000 7f8870e00640 1 -- 172.21.15.60:0/3581300300 wait complete. 2024-04-14T23:51:30.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.042+0000 7f8870e00640 1 Processor -- start 2024-04-14T23:51:30.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.042+0000 7f8870e00640 1 -- start start 2024-04-14T23:51:30.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.043+0000 7f8870e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f886c06f1f0 0x7f886c15c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:30.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.043+0000 7f8870e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f886c06f5d0 0x7f886c0c5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:30.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.043+0000 7f8870e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f886c0731e0 0x7f886c1f0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:30.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.043+0000 7f8870e00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f886c0c79b0 con 0x7f886c0731e0 2024-04-14T23:51:30.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.043+0000 7f8870e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f886c071180 con 0x7f886c06f5d0 2024-04-14T23:51:30.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.043+0000 7f8870e00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f886c0c7b30 con 0x7f886c06f1f0 2024-04-14T23:51:30.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.043+0000 7f886a800640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f886c0731e0 0x7f886c1f0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:30.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.044+0000 7f886a800640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f886c0731e0 0x7f886c1f0940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40250/0 (socket says 172.21.15.60:40250) 2024-04-14T23:51:30.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.044+0000 7f886a800640 1 -- 172.21.15.60:0/3847975424 learned_addr learned my addr 172.21.15.60:0/3847975424 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:51:30.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.044+0000 7f8869e00640 1 --2- 172.21.15.60:0/3847975424 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f886c06f1f0 0x7f886c15c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:30.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.044+0000 7f8869400640 1 --2- 172.21.15.60:0/3847975424 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f886c06f5d0 0x7f886c0c5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:30.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.044+0000 7f8869e00640 1 -- 172.21.15.60:0/3847975424 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f886c06f5d0 msgr2=0x7f886c0c5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:30.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.044+0000 7f8869e00640 1 --2- 172.21.15.60:0/3847975424 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f886c06f5d0 0x7f886c0c5440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:30.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.044+0000 7f8869e00640 1 -- 172.21.15.60:0/3847975424 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f886c0731e0 msgr2=0x7f886c1f0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:30.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.044+0000 7f8869e00640 1 --2- 172.21.15.60:0/3847975424 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f886c0731e0 0x7f886c1f0940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:30.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.044+0000 7f8869e00640 1 -- 172.21.15.60:0/3847975424 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f886c15ec00 con 0x7f886c06f1f0 2024-04-14T23:51:30.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.044+0000 7f8869e00640 1 --2- 172.21.15.60:0/3847975424 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f886c06f1f0 0x7f886c15c010 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f885c00c7b0 tx=0x7f885c00cc70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:51:30.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.045+0000 7f8862000640 1 -- 172.21.15.60:0/3847975424 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f885c018070 con 0x7f886c06f1f0 2024-04-14T23:51:30.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.045+0000 7f8862000640 1 -- 172.21.15.60:0/3847975424 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f885c00a640 con 0x7f886c06f1f0 2024-04-14T23:51:30.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.045+0000 7f8862000640 1 -- 172.21.15.60:0/3847975424 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f885c00ce60 con 0x7f886c06f1f0 2024-04-14T23:51:30.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.045+0000 7f8870e00640 1 -- 172.21.15.60:0/3847975424 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f886c15eee0 con 0x7f886c06f1f0 2024-04-14T23:51:30.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.045+0000 7f8870e00640 1 -- 172.21.15.60:0/3847975424 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f886c1f43e0 con 0x7f886c06f1f0 2024-04-14T23:51:30.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.046+0000 7f8870e00640 1 -- 172.21.15.60:0/3847975424 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f886c076970 con 0x7f886c06f1f0 2024-04-14T23:51:30.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.047+0000 7f8862000640 1 -- 172.21.15.60:0/3847975424 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f885c01b070 con 0x7f886c06f1f0 2024-04-14T23:51:30.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.048+0000 7f8862000640 1 --2- 172.21.15.60:0/3847975424 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f884c06b9e0 0x7f884c06dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:30.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.048+0000 7f8862000640 1 -- 172.21.15.60:0/3847975424 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 5911+0+0 (secure 0 0 0) 0x7f885c089fa0 con 0x7f886c06f1f0 2024-04-14T23:51:30.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.048+0000 7f8869400640 1 --2- 172.21.15.60:0/3847975424 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f884c06b9e0 0x7f884c06dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:30.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.049+0000 7f8869400640 1 --2- 172.21.15.60:0/3847975424 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f884c06b9e0 0x7f884c06dea0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8858036000 tx=0x7f885803b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:51:30.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.052+0000 7f8862000640 1 -- 172.21.15.60:0/3847975424 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f885c0583e0 con 0x7f886c06f1f0 2024-04-14T23:51:30.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.298+0000 7f8870e00640 1 -- 172.21.15.60:0/3847975424 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f886c0caa90 con 0x7f886c06f1f0 2024-04-14T23:51:30.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.299+0000 7f8862000640 1 -- 172.21.15.60:0/3847975424 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+6007 (secure 0 0 0) 0x7f885c05c480 con 0x7f886c06f1f0 2024-04-14T23:51:30.304 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:51:30.304 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:51:24.313553+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4276},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4276":{"gid":4276,"name":"a","rank":0,"incarnation":9,"state":"up:active","state_seq":45,"addr":"172.21.15.60:6833/78303888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":78303888},{"type":"v1","addr":"172.21.15.60:6833","nonce":78303888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4276,"qdb_cluster":[4276]},"id":1}]} 2024-04-14T23:51:30.304 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 13 2024-04-14T23:51:30.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.302+0000 7f8870e00640 1 -- 172.21.15.60:0/3847975424 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f884c06b9e0 msgr2=0x7f884c06dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:30.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.302+0000 7f8870e00640 1 --2- 172.21.15.60:0/3847975424 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f884c06b9e0 0x7f884c06dea0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8858036000 tx=0x7f885803b040 comp rx=0 tx=0).stop 2024-04-14T23:51:30.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.302+0000 7f8870e00640 1 -- 172.21.15.60:0/3847975424 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f886c06f1f0 msgr2=0x7f886c15c010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:30.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.302+0000 7f8870e00640 1 --2- 172.21.15.60:0/3847975424 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f886c06f1f0 0x7f886c15c010 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f885c00c7b0 tx=0x7f885c00cc70 comp rx=0 tx=0).stop 2024-04-14T23:51:30.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.303+0000 7f8870e00640 1 -- 172.21.15.60:0/3847975424 shutdown_connections 2024-04-14T23:51:30.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.303+0000 7f8870e00640 1 --2- 172.21.15.60:0/3847975424 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f884c06b9e0 0x7f884c06dea0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:30.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.303+0000 7f8870e00640 1 --2- 172.21.15.60:0/3847975424 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f886c0731e0 0x7f886c1f0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:30.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.303+0000 7f8870e00640 1 --2- 172.21.15.60:0/3847975424 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f886c06f5d0 0x7f886c0c5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:30.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.303+0000 7f8870e00640 1 --2- 172.21.15.60:0/3847975424 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f886c06f1f0 0x7f886c15c010 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:30.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.303+0000 7f8870e00640 1 -- 172.21.15.60:0/3847975424 >> 172.21.15.60:0/3847975424 conn(0x7f886c0cc170 msgr2=0x7f886c0cb570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:51:30.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.303+0000 7f8870e00640 1 -- 172.21.15.60:0/3847975424 shutdown_connections 2024-04-14T23:51:30.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:30.303+0000 7f8870e00640 1 -- 172.21.15.60:0/3847975424 wait complete. 2024-04-14T23:51:30.326 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2024-04-14T23:51:30.478 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 150 2024-04-14T23:51:30.507 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 151 2024-04-14T23:51:30.802 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 152 2024-04-14T23:51:30.842 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 153 2024-04-14T23:51:30.885 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 154 2024-04-14T23:51:30.923 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 155 2024-04-14T23:51:31.152 INFO:tasks.ceph.mds.a.smithi060.stderr:2024-04-14T23:51:31.147+0000 7f8ff8200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-14T23:51:31.228 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 156 2024-04-14T23:51:31.270 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 157 2024-04-14T23:51:31.871 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 158 2024-04-14T23:51:31.913 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 159 2024-04-14T23:51:31.954 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 160 2024-04-14T23:51:31.997 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 161 2024-04-14T23:51:32.037 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 162 2024-04-14T23:51:32.078 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 163 2024-04-14T23:51:32.118 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 164 2024-04-14T23:51:32.164 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 165 2024-04-14T23:51:32.208 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 166 2024-04-14T23:51:32.253 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 167 2024-04-14T23:51:32.300 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 168 2024-04-14T23:51:32.327 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:51:32.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.540+0000 7f50cfe00640 1 Processor -- start 2024-04-14T23:51:32.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.540+0000 7f50cfe00640 1 -- start start 2024-04-14T23:51:32.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.542+0000 7f50cfe00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f50d00b1e00 0x7f50d00c6950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:32.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.542+0000 7f50cfe00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f50d00c3d90 0x7f50d00c41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:32.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.542+0000 7f50cfe00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50d0152fb0 con 0x7f50d0073320 2024-04-14T23:51:32.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.542+0000 7f50cfe00640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f50d00c6540 con 0x7f50d00b1e00 2024-04-14T23:51:32.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.542+0000 7f50cfe00640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f50d00b21e0 con 0x7f50d00c3d90 2024-04-14T23:51:32.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.542+0000 7f50cea00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f50d00b1e00 0x7f50d00c6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:32.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.542+0000 7f50cea00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f50d00b1e00 0x7f50d00c6950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53462/0 (socket says 172.21.15.60:53462) 2024-04-14T23:51:32.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.542+0000 7f50cea00640 1 -- 172.21.15.60:0/2218501537 learned_addr learned my addr 172.21.15.60:0/2218501537 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:51:32.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.542+0000 7f50cf400640 1 --2- 172.21.15.60:0/2218501537 >> v2:172.21.15.186:3300/0 conn(0x7f50d00c3d90 0x7f50d00c41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:32.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.543+0000 7f50cea00640 1 -- 172.21.15.60:0/2218501537 >> v2:172.21.15.186:3300/0 conn(0x7f50d00c3d90 msgr2=0x7f50d00c41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:51:32.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.543+0000 7f50cea00640 1 --2- 172.21.15.60:0/2218501537 >> v2:172.21.15.186:3300/0 conn(0x7f50d00c3d90 0x7f50d00c41c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:32.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.543+0000 7f50cea00640 1 -- 172.21.15.60:0/2218501537 >> v1:172.21.15.60:6789/0 conn(0x7f50d0073320 legacy=0x7f50d00c7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:32.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.543+0000 7f50cea00640 1 -- 172.21.15.60:0/2218501537 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50d0159310 con 0x7f50d00b1e00 2024-04-14T23:51:32.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.544+0000 7f50cea00640 1 --2- 172.21.15.60:0/2218501537 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50d00b1e00 0x7f50d00c6950 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f50d00c7c70 tx=0x7f50c002ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe1c3b6466916f57 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:51:32.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.544+0000 7f50cd600640 1 -- 172.21.15.60:0/2218501537 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50c002f850 con 0x7f50d00b1e00 2024-04-14T23:51:32.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.544+0000 7f50cd600640 1 -- 172.21.15.60:0/2218501537 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50c002f9f0 con 0x7f50d00b1e00 2024-04-14T23:51:32.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.544+0000 7f50cd600640 1 -- 172.21.15.60:0/2218501537 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50c002fdc0 con 0x7f50d00b1e00 2024-04-14T23:51:32.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.545+0000 7f50cfe00640 1 -- 172.21.15.60:0/2218501537 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50d00b1e00 msgr2=0x7f50d00c6950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:32.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.545+0000 7f50cfe00640 1 --2- 172.21.15.60:0/2218501537 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50d00b1e00 0x7f50d00c6950 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f50d00c7c70 tx=0x7f50c002ba30 comp rx=0 tx=0).stop 2024-04-14T23:51:32.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.545+0000 7f50cfe00640 1 -- 172.21.15.60:0/2218501537 shutdown_connections 2024-04-14T23:51:32.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.545+0000 7f50cfe00640 1 --2- 172.21.15.60:0/2218501537 >> v2:172.21.15.186:3300/0 conn(0x7f50d00c3d90 0x7f50d00c41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:32.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.545+0000 7f50cfe00640 1 --2- 172.21.15.60:0/2218501537 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50d00b1e00 0x7f50d00c6950 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:32.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.545+0000 7f50cfe00640 1 -- 172.21.15.60:0/2218501537 >> 172.21.15.60:0/2218501537 conn(0x7f50d00c8b50 msgr2=0x7f50d00c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:51:32.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.545+0000 7f50cfe00640 1 -- 172.21.15.60:0/2218501537 shutdown_connections 2024-04-14T23:51:32.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.545+0000 7f50cfe00640 1 -- 172.21.15.60:0/2218501537 wait complete. 2024-04-14T23:51:32.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.547+0000 7f50cfe00640 1 Processor -- start 2024-04-14T23:51:32.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.547+0000 7f50cfe00640 1 -- start start 2024-04-14T23:51:32.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.548+0000 7f50cfe00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f50d0073320 0x7f50d01f0880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:32.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.548+0000 7f50cfe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f50d00b1e00 0x7f50d014ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:32.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.548+0000 7f50cfe00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50d00c3d90 0x7f50d0149400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:32.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.548+0000 7f50cfe00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f50d00c38b0 con 0x7f50d00c3d90 2024-04-14T23:51:32.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.548+0000 7f50cfe00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f50d0159890 con 0x7f50d00b1e00 2024-04-14T23:51:32.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.548+0000 7f50cfe00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f50d01595b0 con 0x7f50d0073320 2024-04-14T23:51:32.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.548+0000 7f50cea00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f50d0073320 0x7f50d01f0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:32.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.548+0000 7f50cf400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50d00c3d90 0x7f50d0149400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:32.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.549+0000 7f50cea00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f50d0073320 0x7f50d01f0880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.60:47978/0 (socket says 172.21.15.60:47978) 2024-04-14T23:51:32.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.549+0000 7f50cea00640 1 -- 172.21.15.60:0/308917219 learned_addr learned my addr 172.21.15.60:0/308917219 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:51:32.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.549+0000 7f50ce000640 1 --2- 172.21.15.60:0/308917219 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f50d00b1e00 0x7f50d014ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:32.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.549+0000 7f50cea00640 1 -- 172.21.15.60:0/308917219 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f50d00b1e00 msgr2=0x7f50d014ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:32.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.549+0000 7f50cea00640 1 --2- 172.21.15.60:0/308917219 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f50d00b1e00 0x7f50d014ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:32.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.549+0000 7f50cea00640 1 -- 172.21.15.60:0/308917219 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50d00c3d90 msgr2=0x7f50d0149400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:32.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.549+0000 7f50cea00640 1 --2- 172.21.15.60:0/308917219 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50d00c3d90 0x7f50d0149400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:32.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.549+0000 7f50cea00640 1 -- 172.21.15.60:0/308917219 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50d014fb90 con 0x7f50d0073320 2024-04-14T23:51:32.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.549+0000 7f50cea00640 1 --2- 172.21.15.60:0/308917219 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f50d0073320 0x7f50d01f0880 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f50c0000c00 tx=0x7f50c0004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:51:32.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.550+0000 7f50c7400640 1 -- 172.21.15.60:0/308917219 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50c0047070 con 0x7f50d0073320 2024-04-14T23:51:32.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.550+0000 7f50c7400640 1 -- 172.21.15.60:0/308917219 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50c0037d30 con 0x7f50d0073320 2024-04-14T23:51:32.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.550+0000 7f50cfe00640 1 -- 172.21.15.60:0/308917219 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50d014fe30 con 0x7f50d0073320 2024-04-14T23:51:32.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.550+0000 7f50c7400640 1 -- 172.21.15.60:0/308917219 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50c00426c0 con 0x7f50d0073320 2024-04-14T23:51:32.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.550+0000 7f50cfe00640 1 -- 172.21.15.60:0/308917219 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f50d0072d00 con 0x7f50d0073320 2024-04-14T23:51:32.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.551+0000 7f50cfe00640 1 -- 172.21.15.60:0/308917219 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50d006c7c0 con 0x7f50d0073320 2024-04-14T23:51:32.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.552+0000 7f50c7400640 1 -- 172.21.15.60:0/308917219 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f50c004a070 con 0x7f50d0073320 2024-04-14T23:51:32.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.553+0000 7f50c7400640 1 --2- 172.21.15.60:0/308917219 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f50b006b990 0x7f50b006de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:51:32.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.553+0000 7f50c7400640 1 -- 172.21.15.60:0/308917219 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 5911+0+0 (secure 0 0 0) 0x7f50c00acc70 con 0x7f50d0073320 2024-04-14T23:51:32.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.553+0000 7f50ce000640 1 --2- 172.21.15.60:0/308917219 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f50b006b990 0x7f50b006de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:51:32.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.554+0000 7f50ce000640 1 --2- 172.21.15.60:0/308917219 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f50b006b990 0x7f50b006de50 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f50d00cb010 tx=0x7f50bc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:51:32.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.557+0000 7f50c7400640 1 -- 172.21.15.60:0/308917219 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f50d006c7c0 con 0x7f50d0073320 2024-04-14T23:51:32.602 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 169 2024-04-14T23:51:32.629 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 170 2024-04-14T23:51:32.660 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 171 2024-04-14T23:51:32.697 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 172 2024-04-14T23:51:32.730 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 173 2024-04-14T23:51:32.756 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 174 2024-04-14T23:51:32.789 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 175 2024-04-14T23:51:32.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.811+0000 7f50cfe00640 1 -- 172.21.15.60:0/308917219 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f50d004f540 con 0x7f50d0073320 2024-04-14T23:51:32.817 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.812+0000 7f50c7400640 1 -- 172.21.15.60:0/308917219 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6832 (secure 0 0 0) 0x7f50c0033030 con 0x7f50d0073320 2024-04-14T23:51:32.817 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:51:32.817 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4839,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/1714133698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1714133698},{"type":"v1","addr":"172.21.15.60:6835","nonce":1714133698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":14}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:51:24.313553+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4276},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4276":{"gid":4276,"name":"a","rank":0,"incarnation":9,"state":"up:active","state_seq":45,"addr":"172.21.15.60:6833/78303888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":78303888},{"type":"v1","addr":"172.21.15.60:6833","nonce":78303888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4276,"qdb_cluster":[4276]},"id":1}]} 2024-04-14T23:51:32.817 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 14 2024-04-14T23:51:32.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.815+0000 7f50cfe00640 1 -- 172.21.15.60:0/308917219 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f50b006b990 msgr2=0x7f50b006de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:32.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.815+0000 7f50cfe00640 1 --2- 172.21.15.60:0/308917219 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f50b006b990 0x7f50b006de50 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f50d00cb010 tx=0x7f50bc009040 comp rx=0 tx=0).stop 2024-04-14T23:51:32.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.815+0000 7f50cfe00640 1 -- 172.21.15.60:0/308917219 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f50d0073320 msgr2=0x7f50d01f0880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:51:32.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.815+0000 7f50cfe00640 1 --2- 172.21.15.60:0/308917219 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f50d0073320 0x7f50d01f0880 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f50c0000c00 tx=0x7f50c0004730 comp rx=0 tx=0).stop 2024-04-14T23:51:32.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.815+0000 7f50cfe00640 1 -- 172.21.15.60:0/308917219 shutdown_connections 2024-04-14T23:51:32.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.815+0000 7f50cfe00640 1 --2- 172.21.15.60:0/308917219 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f50b006b990 0x7f50b006de50 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:32.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.815+0000 7f50cfe00640 1 --2- 172.21.15.60:0/308917219 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50d00c3d90 0x7f50d0149400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:32.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.815+0000 7f50cfe00640 1 --2- 172.21.15.60:0/308917219 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f50d00b1e00 0x7f50d014ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:32.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.815+0000 7f50cfe00640 1 --2- 172.21.15.60:0/308917219 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f50d0073320 0x7f50d01f0880 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:51:32.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.815+0000 7f50cfe00640 1 -- 172.21.15.60:0/308917219 >> 172.21.15.60:0/308917219 conn(0x7f50d00c8b50 msgr2=0x7f50d00c8110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:51:32.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.815+0000 7f50cfe00640 1 -- 172.21.15.60:0/308917219 shutdown_connections 2024-04-14T23:51:32.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:51:32.815+0000 7f50cfe00640 1 -- 172.21.15.60:0/308917219 wait complete. 2024-04-14T23:51:32.824 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 176 2024-04-14T23:51:32.840 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported in up:standby state 2024-04-14T23:51:32.840 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 42 secs before thrashing 2024-04-14T23:51:32.851 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 177 2024-04-14T23:51:32.888 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 178 2024-04-14T23:51:32.920 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 179 2024-04-14T23:51:32.961 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 180 2024-04-14T23:51:33.763 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 181 2024-04-14T23:51:34.095 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 182 2024-04-14T23:51:34.146 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 183 2024-04-14T23:51:34.203 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 184 2024-04-14T23:51:34.261 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 185 2024-04-14T23:51:34.318 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 186 2024-04-14T23:51:34.366 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 187 2024-04-14T23:51:34.421 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 188 2024-04-14T23:51:34.472 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 189 2024-04-14T23:51:34.519 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 190 2024-04-14T23:51:34.841 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 191 2024-04-14T23:51:34.878 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 192 2024-04-14T23:51:34.914 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 193 2024-04-14T23:51:35.896 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 194 2024-04-14T23:51:36.201 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 195 2024-04-14T23:51:36.501 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 196 2024-04-14T23:51:36.543 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 197 2024-04-14T23:51:36.584 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 198 2024-04-14T23:51:36.626 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 199 2024-04-14T23:51:36.667 INFO:tasks.workunit.client.0.smithi060.stdout:Take first snapshot .snap/test1 2024-04-14T23:51:36.837 INFO:tasks.workunit.client.0.smithi060.stdout:Create dir 200 to 299 ... 2024-04-14T23:51:36.841 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 200 2024-04-14T23:51:36.879 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 201 2024-04-14T23:51:36.916 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 202 2024-04-14T23:51:36.954 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 203 2024-04-14T23:51:36.993 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 204 2024-04-14T23:51:37.026 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 205 2024-04-14T23:51:37.066 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 206 2024-04-14T23:51:37.103 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 207 2024-04-14T23:51:37.141 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 208 2024-04-14T23:51:37.181 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 209 2024-04-14T23:51:37.211 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 210 2024-04-14T23:51:37.249 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 211 2024-04-14T23:51:37.286 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 212 2024-04-14T23:51:37.337 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 213 2024-04-14T23:51:37.387 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 214 2024-04-14T23:51:37.439 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 215 2024-04-14T23:51:37.482 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 216 2024-04-14T23:51:37.519 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 217 2024-04-14T23:51:37.554 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 218 2024-04-14T23:51:37.591 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 219 2024-04-14T23:51:37.720 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 220 2024-04-14T23:51:37.756 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 221 2024-04-14T23:51:37.795 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 222 2024-04-14T23:51:37.834 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 223 2024-04-14T23:51:37.873 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 224 2024-04-14T23:51:37.911 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 225 2024-04-14T23:51:37.949 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 226 2024-04-14T23:51:37.987 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 227 2024-04-14T23:51:38.031 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 228 2024-04-14T23:51:38.084 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 229 2024-04-14T23:51:38.137 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 230 2024-04-14T23:51:38.188 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 231 2024-04-14T23:51:38.240 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 232 2024-04-14T23:51:38.274 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 233 2024-04-14T23:51:38.575 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 234 2024-04-14T23:51:38.614 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 235 2024-04-14T23:51:38.653 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 236 2024-04-14T23:51:38.957 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 237 2024-04-14T23:51:38.996 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 238 2024-04-14T23:51:39.035 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 239 2024-04-14T23:51:39.074 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 240 2024-04-14T23:51:39.112 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 241 2024-04-14T23:51:39.394 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 242 2024-04-14T23:51:40.905 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 243 2024-04-14T23:51:41.517 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 244 2024-04-14T23:51:41.551 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 245 2024-04-14T23:51:41.590 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 246 2024-04-14T23:51:41.629 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 247 2024-04-14T23:51:41.661 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 248 2024-04-14T23:51:41.695 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 249 2024-04-14T23:51:41.733 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 250 2024-04-14T23:51:41.776 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 251 2024-04-14T23:51:44.448 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 252 2024-04-14T23:51:44.486 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 253 2024-04-14T23:51:44.524 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 254 2024-04-14T23:51:44.561 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 255 2024-04-14T23:51:44.601 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 256 2024-04-14T23:51:44.638 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 257 2024-04-14T23:51:44.675 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 258 2024-04-14T23:51:44.690 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-14T23:51:44.696 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-14T23:51:44.712 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 259 2024-04-14T23:51:44.737 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 260 2024-04-14T23:51:44.766 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 261 2024-04-14T23:51:44.785 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 262 2024-04-14T23:51:45.079 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 263 2024-04-14T23:51:45.120 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 264 2024-04-14T23:51:45.157 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 265 2024-04-14T23:51:45.195 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 266 2024-04-14T23:51:45.231 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 267 2024-04-14T23:51:45.270 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 268 2024-04-14T23:51:45.307 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 269 2024-04-14T23:51:45.343 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 270 2024-04-14T23:51:45.382 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 271 2024-04-14T23:51:45.419 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 272 2024-04-14T23:51:45.472 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 273 2024-04-14T23:51:45.503 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 274 2024-04-14T23:51:45.541 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 275 2024-04-14T23:51:45.579 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 276 2024-04-14T23:51:45.618 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 277 2024-04-14T23:51:45.656 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 278 2024-04-14T23:51:45.693 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 279 2024-04-14T23:51:45.730 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 280 2024-04-14T23:51:45.767 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 281 2024-04-14T23:51:45.804 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 282 2024-04-14T23:51:45.842 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 283 2024-04-14T23:51:45.879 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 284 2024-04-14T23:51:45.918 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 285 2024-04-14T23:51:45.955 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 286 2024-04-14T23:51:45.993 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 287 2024-04-14T23:51:46.032 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 288 2024-04-14T23:51:46.069 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 289 2024-04-14T23:51:46.107 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 290 2024-04-14T23:51:46.145 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 291 2024-04-14T23:51:46.183 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 292 2024-04-14T23:51:46.220 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 293 2024-04-14T23:51:46.259 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 294 2024-04-14T23:51:46.298 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 295 2024-04-14T23:51:46.327 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 296 2024-04-14T23:51:46.366 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 297 2024-04-14T23:51:46.393 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 298 2024-04-14T23:51:46.427 INFO:tasks.workunit.client.0.smithi060.stdout: create dir 299 2024-04-14T23:51:46.462 INFO:tasks.workunit.client.0.smithi060.stdout:Create a snapshot in every first level dir ... 2024-04-14T23:51:46.813 INFO:tasks.workunit.client.0.smithi060.stdout: create 100/.snap/snap-subdir-test 2024-04-14T23:51:46.860 INFO:tasks.workunit.client.0.smithi060.stdout: create 100/30 file after the snapshot 2024-04-14T23:51:46.861 INFO:tasks.workunit.client.0.smithi060.stdout: create 100/31 file after the snapshot 2024-04-14T23:51:46.862 INFO:tasks.workunit.client.0.smithi060.stdout: create 100/32 file after the snapshot 2024-04-14T23:51:46.864 INFO:tasks.workunit.client.0.smithi060.stdout: create 100/33 file after the snapshot 2024-04-14T23:51:46.866 INFO:tasks.workunit.client.0.smithi060.stdout: create 100/34 file after the snapshot 2024-04-14T23:51:46.868 INFO:tasks.workunit.client.0.smithi060.stdout: create 100/35 file after the snapshot 2024-04-14T23:51:46.870 INFO:tasks.workunit.client.0.smithi060.stdout: create 100/36 file after the snapshot 2024-04-14T23:51:46.872 INFO:tasks.workunit.client.0.smithi060.stdout: create 100/37 file after the snapshot 2024-04-14T23:51:46.874 INFO:tasks.workunit.client.0.smithi060.stdout: create 100/38 file after the snapshot 2024-04-14T23:51:46.877 INFO:tasks.workunit.client.0.smithi060.stdout: create 100/39 file after the snapshot 2024-04-14T23:51:46.879 INFO:tasks.workunit.client.0.smithi060.stdout: create 101/.snap/snap-subdir-test 2024-04-14T23:51:46.934 INFO:tasks.workunit.client.0.smithi060.stdout: create 101/30 file after the snapshot 2024-04-14T23:51:46.937 INFO:tasks.workunit.client.0.smithi060.stdout: create 101/31 file after the snapshot 2024-04-14T23:51:46.939 INFO:tasks.workunit.client.0.smithi060.stdout: create 101/32 file after the snapshot 2024-04-14T23:51:46.942 INFO:tasks.workunit.client.0.smithi060.stdout: create 101/33 file after the snapshot 2024-04-14T23:51:46.945 INFO:tasks.workunit.client.0.smithi060.stdout: create 101/34 file after the snapshot 2024-04-14T23:51:46.948 INFO:tasks.workunit.client.0.smithi060.stdout: create 101/35 file after the snapshot 2024-04-14T23:51:46.950 INFO:tasks.workunit.client.0.smithi060.stdout: create 101/36 file after the snapshot 2024-04-14T23:51:46.953 INFO:tasks.workunit.client.0.smithi060.stdout: create 101/37 file after the snapshot 2024-04-14T23:51:46.956 INFO:tasks.workunit.client.0.smithi060.stdout: create 101/38 file after the snapshot 2024-04-14T23:51:46.958 INFO:tasks.workunit.client.0.smithi060.stdout: create 101/39 file after the snapshot 2024-04-14T23:51:46.961 INFO:tasks.workunit.client.0.smithi060.stdout: create 102/.snap/snap-subdir-test 2024-04-14T23:51:47.031 INFO:tasks.workunit.client.0.smithi060.stdout: create 102/30 file after the snapshot 2024-04-14T23:51:47.033 INFO:tasks.workunit.client.0.smithi060.stdout: create 102/31 file after the snapshot 2024-04-14T23:51:47.036 INFO:tasks.workunit.client.0.smithi060.stdout: create 102/32 file after the snapshot 2024-04-14T23:51:47.038 INFO:tasks.workunit.client.0.smithi060.stdout: create 102/33 file after the snapshot 2024-04-14T23:51:47.041 INFO:tasks.workunit.client.0.smithi060.stdout: create 102/34 file after the snapshot 2024-04-14T23:51:47.044 INFO:tasks.workunit.client.0.smithi060.stdout: create 102/35 file after the snapshot 2024-04-14T23:51:47.047 INFO:tasks.workunit.client.0.smithi060.stdout: create 102/36 file after the snapshot 2024-04-14T23:51:47.049 INFO:tasks.workunit.client.0.smithi060.stdout: create 102/37 file after the snapshot 2024-04-14T23:51:47.052 INFO:tasks.workunit.client.0.smithi060.stdout: create 102/38 file after the snapshot 2024-04-14T23:51:47.055 INFO:tasks.workunit.client.0.smithi060.stdout: create 102/39 file after the snapshot 2024-04-14T23:51:47.057 INFO:tasks.workunit.client.0.smithi060.stdout: create 103/.snap/snap-subdir-test 2024-04-14T23:51:47.131 INFO:tasks.workunit.client.0.smithi060.stdout: create 103/30 file after the snapshot 2024-04-14T23:51:47.133 INFO:tasks.workunit.client.0.smithi060.stdout: create 103/31 file after the snapshot 2024-04-14T23:51:47.136 INFO:tasks.workunit.client.0.smithi060.stdout: create 103/32 file after the snapshot 2024-04-14T23:51:47.139 INFO:tasks.workunit.client.0.smithi060.stdout: create 103/33 file after the snapshot 2024-04-14T23:51:47.141 INFO:tasks.workunit.client.0.smithi060.stdout: create 103/34 file after the snapshot 2024-04-14T23:51:47.146 INFO:tasks.workunit.client.0.smithi060.stdout: create 103/35 file after the snapshot 2024-04-14T23:51:47.150 INFO:tasks.workunit.client.0.smithi060.stdout: create 103/36 file after the snapshot 2024-04-14T23:51:47.154 INFO:tasks.workunit.client.0.smithi060.stdout: create 103/37 file after the snapshot 2024-04-14T23:51:47.158 INFO:tasks.workunit.client.0.smithi060.stdout: create 103/38 file after the snapshot 2024-04-14T23:51:47.163 INFO:tasks.workunit.client.0.smithi060.stdout: create 103/39 file after the snapshot 2024-04-14T23:51:47.167 INFO:tasks.workunit.client.0.smithi060.stdout: create 104/.snap/snap-subdir-test 2024-04-14T23:51:47.236 INFO:tasks.workunit.client.0.smithi060.stdout: create 104/30 file after the snapshot 2024-04-14T23:51:47.241 INFO:tasks.workunit.client.0.smithi060.stdout: create 104/31 file after the snapshot 2024-04-14T23:51:47.245 INFO:tasks.workunit.client.0.smithi060.stdout: create 104/32 file after the snapshot 2024-04-14T23:51:47.249 INFO:tasks.workunit.client.0.smithi060.stdout: create 104/33 file after the snapshot 2024-04-14T23:51:47.252 INFO:tasks.workunit.client.0.smithi060.stdout: create 104/34 file after the snapshot 2024-04-14T23:51:47.255 INFO:tasks.workunit.client.0.smithi060.stdout: create 104/35 file after the snapshot 2024-04-14T23:51:47.258 INFO:tasks.workunit.client.0.smithi060.stdout: create 104/36 file after the snapshot 2024-04-14T23:51:47.261 INFO:tasks.workunit.client.0.smithi060.stdout: create 104/37 file after the snapshot 2024-04-14T23:51:47.264 INFO:tasks.workunit.client.0.smithi060.stdout: create 104/38 file after the snapshot 2024-04-14T23:51:47.267 INFO:tasks.workunit.client.0.smithi060.stdout: create 104/39 file after the snapshot 2024-04-14T23:51:47.269 INFO:tasks.workunit.client.0.smithi060.stdout: create 105/.snap/snap-subdir-test 2024-04-14T23:51:47.336 INFO:tasks.workunit.client.0.smithi060.stdout: create 105/30 file after the snapshot 2024-04-14T23:51:47.605 INFO:tasks.workunit.client.0.smithi060.stdout: create 105/31 file after the snapshot 2024-04-14T23:51:47.608 INFO:tasks.workunit.client.0.smithi060.stdout: create 105/32 file after the snapshot 2024-04-14T23:51:47.610 INFO:tasks.workunit.client.0.smithi060.stdout: create 105/33 file after the snapshot 2024-04-14T23:51:47.613 INFO:tasks.workunit.client.0.smithi060.stdout: create 105/34 file after the snapshot 2024-04-14T23:51:47.616 INFO:tasks.workunit.client.0.smithi060.stdout: create 105/35 file after the snapshot 2024-04-14T23:51:47.619 INFO:tasks.workunit.client.0.smithi060.stdout: create 105/36 file after the snapshot 2024-04-14T23:51:47.621 INFO:tasks.workunit.client.0.smithi060.stdout: create 105/37 file after the snapshot 2024-04-14T23:51:47.624 INFO:tasks.workunit.client.0.smithi060.stdout: create 105/38 file after the snapshot 2024-04-14T23:51:47.627 INFO:tasks.workunit.client.0.smithi060.stdout: create 105/39 file after the snapshot 2024-04-14T23:51:47.629 INFO:tasks.workunit.client.0.smithi060.stdout: create 106/.snap/snap-subdir-test 2024-04-14T23:51:47.697 INFO:tasks.workunit.client.0.smithi060.stdout: create 106/30 file after the snapshot 2024-04-14T23:51:47.700 INFO:tasks.workunit.client.0.smithi060.stdout: create 106/31 file after the snapshot 2024-04-14T23:51:47.703 INFO:tasks.workunit.client.0.smithi060.stdout: create 106/32 file after the snapshot 2024-04-14T23:51:47.705 INFO:tasks.workunit.client.0.smithi060.stdout: create 106/33 file after the snapshot 2024-04-14T23:51:47.708 INFO:tasks.workunit.client.0.smithi060.stdout: create 106/34 file after the snapshot 2024-04-14T23:51:47.711 INFO:tasks.workunit.client.0.smithi060.stdout: create 106/35 file after the snapshot 2024-04-14T23:51:47.714 INFO:tasks.workunit.client.0.smithi060.stdout: create 106/36 file after the snapshot 2024-04-14T23:51:47.717 INFO:tasks.workunit.client.0.smithi060.stdout: create 106/37 file after the snapshot 2024-04-14T23:51:47.719 INFO:tasks.workunit.client.0.smithi060.stdout: create 106/38 file after the snapshot 2024-04-14T23:51:47.722 INFO:tasks.workunit.client.0.smithi060.stdout: create 106/39 file after the snapshot 2024-04-14T23:51:47.725 INFO:tasks.workunit.client.0.smithi060.stdout: create 107/.snap/snap-subdir-test 2024-04-14T23:51:47.791 INFO:tasks.workunit.client.0.smithi060.stdout: create 107/30 file after the snapshot 2024-04-14T23:51:47.794 INFO:tasks.workunit.client.0.smithi060.stdout: create 107/31 file after the snapshot 2024-04-14T23:51:47.796 INFO:tasks.workunit.client.0.smithi060.stdout: create 107/32 file after the snapshot 2024-04-14T23:51:47.799 INFO:tasks.workunit.client.0.smithi060.stdout: create 107/33 file after the snapshot 2024-04-14T23:51:47.802 INFO:tasks.workunit.client.0.smithi060.stdout: create 107/34 file after the snapshot 2024-04-14T23:51:47.804 INFO:tasks.workunit.client.0.smithi060.stdout: create 107/35 file after the snapshot 2024-04-14T23:51:47.807 INFO:tasks.workunit.client.0.smithi060.stdout: create 107/36 file after the snapshot 2024-04-14T23:51:47.810 INFO:tasks.workunit.client.0.smithi060.stdout: create 107/37 file after the snapshot 2024-04-14T23:51:47.812 INFO:tasks.workunit.client.0.smithi060.stdout: create 107/38 file after the snapshot 2024-04-14T23:51:47.815 INFO:tasks.workunit.client.0.smithi060.stdout: create 107/39 file after the snapshot 2024-04-14T23:51:47.844 INFO:tasks.workunit.client.0.smithi060.stdout: create 108/.snap/snap-subdir-test 2024-04-14T23:51:48.426 INFO:tasks.workunit.client.0.smithi060.stdout: create 108/30 file after the snapshot 2024-04-14T23:51:48.429 INFO:tasks.workunit.client.0.smithi060.stdout: create 108/31 file after the snapshot 2024-04-14T23:51:48.432 INFO:tasks.workunit.client.0.smithi060.stdout: create 108/32 file after the snapshot 2024-04-14T23:51:48.434 INFO:tasks.workunit.client.0.smithi060.stdout: create 108/33 file after the snapshot 2024-04-14T23:51:48.437 INFO:tasks.workunit.client.0.smithi060.stdout: create 108/34 file after the snapshot 2024-04-14T23:51:48.440 INFO:tasks.workunit.client.0.smithi060.stdout: create 108/35 file after the snapshot 2024-04-14T23:51:48.443 INFO:tasks.workunit.client.0.smithi060.stdout: create 108/36 file after the snapshot 2024-04-14T23:51:48.445 INFO:tasks.workunit.client.0.smithi060.stdout: create 108/37 file after the snapshot 2024-04-14T23:51:48.448 INFO:tasks.workunit.client.0.smithi060.stdout: create 108/38 file after the snapshot 2024-04-14T23:51:48.452 INFO:tasks.workunit.client.0.smithi060.stdout: create 108/39 file after the snapshot 2024-04-14T23:51:48.454 INFO:tasks.workunit.client.0.smithi060.stdout: create 109/.snap/snap-subdir-test 2024-04-14T23:51:48.524 INFO:tasks.workunit.client.0.smithi060.stdout: create 109/30 file after the snapshot 2024-04-14T23:51:48.527 INFO:tasks.workunit.client.0.smithi060.stdout: create 109/31 file after the snapshot 2024-04-14T23:51:48.530 INFO:tasks.workunit.client.0.smithi060.stdout: create 109/32 file after the snapshot 2024-04-14T23:51:48.532 INFO:tasks.workunit.client.0.smithi060.stdout: create 109/33 file after the snapshot 2024-04-14T23:51:48.535 INFO:tasks.workunit.client.0.smithi060.stdout: create 109/34 file after the snapshot 2024-04-14T23:51:48.538 INFO:tasks.workunit.client.0.smithi060.stdout: create 109/35 file after the snapshot 2024-04-14T23:51:48.540 INFO:tasks.workunit.client.0.smithi060.stdout: create 109/36 file after the snapshot 2024-04-14T23:51:48.543 INFO:tasks.workunit.client.0.smithi060.stdout: create 109/37 file after the snapshot 2024-04-14T23:51:48.546 INFO:tasks.workunit.client.0.smithi060.stdout: create 109/38 file after the snapshot 2024-04-14T23:51:48.548 INFO:tasks.workunit.client.0.smithi060.stdout: create 109/39 file after the snapshot 2024-04-14T23:51:48.551 INFO:tasks.workunit.client.0.smithi060.stdout: create 110/.snap/snap-subdir-test 2024-04-14T23:51:48.611 INFO:tasks.workunit.client.0.smithi060.stdout: create 110/30 file after the snapshot 2024-04-14T23:51:48.614 INFO:tasks.workunit.client.0.smithi060.stdout: create 110/31 file after the snapshot 2024-04-14T23:51:48.617 INFO:tasks.workunit.client.0.smithi060.stdout: create 110/32 file after the snapshot 2024-04-14T23:51:48.620 INFO:tasks.workunit.client.0.smithi060.stdout: create 110/33 file after the snapshot 2024-04-14T23:51:48.623 INFO:tasks.workunit.client.0.smithi060.stdout: create 110/34 file after the snapshot 2024-04-14T23:51:48.626 INFO:tasks.workunit.client.0.smithi060.stdout: create 110/35 file after the snapshot 2024-04-14T23:51:48.628 INFO:tasks.workunit.client.0.smithi060.stdout: create 110/36 file after the snapshot 2024-04-14T23:51:48.631 INFO:tasks.workunit.client.0.smithi060.stdout: create 110/37 file after the snapshot 2024-04-14T23:51:48.634 INFO:tasks.workunit.client.0.smithi060.stdout: create 110/38 file after the snapshot 2024-04-14T23:51:48.637 INFO:tasks.workunit.client.0.smithi060.stdout: create 110/39 file after the snapshot 2024-04-14T23:51:48.639 INFO:tasks.workunit.client.0.smithi060.stdout: create 111/.snap/snap-subdir-test 2024-04-14T23:51:48.699 INFO:tasks.workunit.client.0.smithi060.stdout: create 111/30 file after the snapshot 2024-04-14T23:51:48.702 INFO:tasks.workunit.client.0.smithi060.stdout: create 111/31 file after the snapshot 2024-04-14T23:51:48.705 INFO:tasks.workunit.client.0.smithi060.stdout: create 111/32 file after the snapshot 2024-04-14T23:51:48.708 INFO:tasks.workunit.client.0.smithi060.stdout: create 111/33 file after the snapshot 2024-04-14T23:51:48.711 INFO:tasks.workunit.client.0.smithi060.stdout: create 111/34 file after the snapshot 2024-04-14T23:51:48.714 INFO:tasks.workunit.client.0.smithi060.stdout: create 111/35 file after the snapshot 2024-04-14T23:51:48.716 INFO:tasks.workunit.client.0.smithi060.stdout: create 111/36 file after the snapshot 2024-04-14T23:51:48.719 INFO:tasks.workunit.client.0.smithi060.stdout: create 111/37 file after the snapshot 2024-04-14T23:51:48.722 INFO:tasks.workunit.client.0.smithi060.stdout: create 111/38 file after the snapshot 2024-04-14T23:51:48.725 INFO:tasks.workunit.client.0.smithi060.stdout: create 111/39 file after the snapshot 2024-04-14T23:51:48.868 INFO:tasks.workunit.client.0.smithi060.stdout: create 112/.snap/snap-subdir-test 2024-04-14T23:51:49.114 INFO:tasks.workunit.client.0.smithi060.stdout: create 112/30 file after the snapshot 2024-04-14T23:51:49.117 INFO:tasks.workunit.client.0.smithi060.stdout: create 112/31 file after the snapshot 2024-04-14T23:51:49.120 INFO:tasks.workunit.client.0.smithi060.stdout: create 112/32 file after the snapshot 2024-04-14T23:51:49.123 INFO:tasks.workunit.client.0.smithi060.stdout: create 112/33 file after the snapshot 2024-04-14T23:51:49.126 INFO:tasks.workunit.client.0.smithi060.stdout: create 112/34 file after the snapshot 2024-04-14T23:51:49.129 INFO:tasks.workunit.client.0.smithi060.stdout: create 112/35 file after the snapshot 2024-04-14T23:51:49.132 INFO:tasks.workunit.client.0.smithi060.stdout: create 112/36 file after the snapshot 2024-04-14T23:51:49.135 INFO:tasks.workunit.client.0.smithi060.stdout: create 112/37 file after the snapshot 2024-04-14T23:51:49.138 INFO:tasks.workunit.client.0.smithi060.stdout: create 112/38 file after the snapshot 2024-04-14T23:51:49.140 INFO:tasks.workunit.client.0.smithi060.stdout: create 112/39 file after the snapshot 2024-04-14T23:51:49.143 INFO:tasks.workunit.client.0.smithi060.stdout: create 113/.snap/snap-subdir-test 2024-04-14T23:51:49.203 INFO:tasks.workunit.client.0.smithi060.stdout: create 113/30 file after the snapshot 2024-04-14T23:51:49.206 INFO:tasks.workunit.client.0.smithi060.stdout: create 113/31 file after the snapshot 2024-04-14T23:51:49.208 INFO:tasks.workunit.client.0.smithi060.stdout: create 113/32 file after the snapshot 2024-04-14T23:51:49.211 INFO:tasks.workunit.client.0.smithi060.stdout: create 113/33 file after the snapshot 2024-04-14T23:51:49.214 INFO:tasks.workunit.client.0.smithi060.stdout: create 113/34 file after the snapshot 2024-04-14T23:51:49.217 INFO:tasks.workunit.client.0.smithi060.stdout: create 113/35 file after the snapshot 2024-04-14T23:51:49.220 INFO:tasks.workunit.client.0.smithi060.stdout: create 113/36 file after the snapshot 2024-04-14T23:51:49.223 INFO:tasks.workunit.client.0.smithi060.stdout: create 113/37 file after the snapshot 2024-04-14T23:51:49.225 INFO:tasks.workunit.client.0.smithi060.stdout: create 113/38 file after the snapshot 2024-04-14T23:51:49.228 INFO:tasks.workunit.client.0.smithi060.stdout: create 113/39 file after the snapshot 2024-04-14T23:51:49.231 INFO:tasks.workunit.client.0.smithi060.stdout: create 114/.snap/snap-subdir-test 2024-04-14T23:51:49.291 INFO:tasks.workunit.client.0.smithi060.stdout: create 114/30 file after the snapshot 2024-04-14T23:51:49.294 INFO:tasks.workunit.client.0.smithi060.stdout: create 114/31 file after the snapshot 2024-04-14T23:51:49.297 INFO:tasks.workunit.client.0.smithi060.stdout: create 114/32 file after the snapshot 2024-04-14T23:51:49.300 INFO:tasks.workunit.client.0.smithi060.stdout: create 114/33 file after the snapshot 2024-04-14T23:51:49.303 INFO:tasks.workunit.client.0.smithi060.stdout: create 114/34 file after the snapshot 2024-04-14T23:51:49.306 INFO:tasks.workunit.client.0.smithi060.stdout: create 114/35 file after the snapshot 2024-04-14T23:51:49.309 INFO:tasks.workunit.client.0.smithi060.stdout: create 114/36 file after the snapshot 2024-04-14T23:51:49.311 INFO:tasks.workunit.client.0.smithi060.stdout: create 114/37 file after the snapshot 2024-04-14T23:51:49.314 INFO:tasks.workunit.client.0.smithi060.stdout: create 114/38 file after the snapshot 2024-04-14T23:51:49.317 INFO:tasks.workunit.client.0.smithi060.stdout: create 114/39 file after the snapshot 2024-04-14T23:51:49.320 INFO:tasks.workunit.client.0.smithi060.stdout: create 115/.snap/snap-subdir-test 2024-04-14T23:51:49.365 INFO:tasks.workunit.client.0.smithi060.stdout: create 115/30 file after the snapshot 2024-04-14T23:51:49.367 INFO:tasks.workunit.client.0.smithi060.stdout: create 115/31 file after the snapshot 2024-04-14T23:51:49.369 INFO:tasks.workunit.client.0.smithi060.stdout: create 115/32 file after the snapshot 2024-04-14T23:51:49.371 INFO:tasks.workunit.client.0.smithi060.stdout: create 115/33 file after the snapshot 2024-04-14T23:51:49.374 INFO:tasks.workunit.client.0.smithi060.stdout: create 115/34 file after the snapshot 2024-04-14T23:51:49.377 INFO:tasks.workunit.client.0.smithi060.stdout: create 115/35 file after the snapshot 2024-04-14T23:51:49.379 INFO:tasks.workunit.client.0.smithi060.stdout: create 115/36 file after the snapshot 2024-04-14T23:51:49.382 INFO:tasks.workunit.client.0.smithi060.stdout: create 115/37 file after the snapshot 2024-04-14T23:51:49.385 INFO:tasks.workunit.client.0.smithi060.stdout: create 115/38 file after the snapshot 2024-04-14T23:51:49.388 INFO:tasks.workunit.client.0.smithi060.stdout: create 115/39 file after the snapshot 2024-04-14T23:51:49.391 INFO:tasks.workunit.client.0.smithi060.stdout: create 116/.snap/snap-subdir-test 2024-04-14T23:51:49.474 INFO:tasks.workunit.client.0.smithi060.stdout: create 116/30 file after the snapshot 2024-04-14T23:51:49.476 INFO:tasks.workunit.client.0.smithi060.stdout: create 116/31 file after the snapshot 2024-04-14T23:51:49.479 INFO:tasks.workunit.client.0.smithi060.stdout: create 116/32 file after the snapshot 2024-04-14T23:51:49.482 INFO:tasks.workunit.client.0.smithi060.stdout: create 116/33 file after the snapshot 2024-04-14T23:51:49.485 INFO:tasks.workunit.client.0.smithi060.stdout: create 116/34 file after the snapshot 2024-04-14T23:51:49.488 INFO:tasks.workunit.client.0.smithi060.stdout: create 116/35 file after the snapshot 2024-04-14T23:51:49.491 INFO:tasks.workunit.client.0.smithi060.stdout: create 116/36 file after the snapshot 2024-04-14T23:51:49.493 INFO:tasks.workunit.client.0.smithi060.stdout: create 116/37 file after the snapshot 2024-04-14T23:51:49.496 INFO:tasks.workunit.client.0.smithi060.stdout: create 116/38 file after the snapshot 2024-04-14T23:51:49.499 INFO:tasks.workunit.client.0.smithi060.stdout: create 116/39 file after the snapshot 2024-04-14T23:51:49.502 INFO:tasks.workunit.client.0.smithi060.stdout: create 117/.snap/snap-subdir-test 2024-04-14T23:51:50.165 INFO:tasks.workunit.client.0.smithi060.stdout: create 117/30 file after the snapshot 2024-04-14T23:51:50.168 INFO:tasks.workunit.client.0.smithi060.stdout: create 117/31 file after the snapshot 2024-04-14T23:51:50.171 INFO:tasks.workunit.client.0.smithi060.stdout: create 117/32 file after the snapshot 2024-04-14T23:51:50.174 INFO:tasks.workunit.client.0.smithi060.stdout: create 117/33 file after the snapshot 2024-04-14T23:51:50.177 INFO:tasks.workunit.client.0.smithi060.stdout: create 117/34 file after the snapshot 2024-04-14T23:51:50.180 INFO:tasks.workunit.client.0.smithi060.stdout: create 117/35 file after the snapshot 2024-04-14T23:51:50.183 INFO:tasks.workunit.client.0.smithi060.stdout: create 117/36 file after the snapshot 2024-04-14T23:51:50.186 INFO:tasks.workunit.client.0.smithi060.stdout: create 117/37 file after the snapshot 2024-04-14T23:51:50.189 INFO:tasks.workunit.client.0.smithi060.stdout: create 117/38 file after the snapshot 2024-04-14T23:51:50.192 INFO:tasks.workunit.client.0.smithi060.stdout: create 117/39 file after the snapshot 2024-04-14T23:51:50.195 INFO:tasks.workunit.client.0.smithi060.stdout: create 118/.snap/snap-subdir-test 2024-04-14T23:51:50.263 INFO:tasks.workunit.client.0.smithi060.stdout: create 118/30 file after the snapshot 2024-04-14T23:51:50.266 INFO:tasks.workunit.client.0.smithi060.stdout: create 118/31 file after the snapshot 2024-04-14T23:51:50.269 INFO:tasks.workunit.client.0.smithi060.stdout: create 118/32 file after the snapshot 2024-04-14T23:51:50.272 INFO:tasks.workunit.client.0.smithi060.stdout: create 118/33 file after the snapshot 2024-04-14T23:51:50.275 INFO:tasks.workunit.client.0.smithi060.stdout: create 118/34 file after the snapshot 2024-04-14T23:51:50.278 INFO:tasks.workunit.client.0.smithi060.stdout: create 118/35 file after the snapshot 2024-04-14T23:51:50.281 INFO:tasks.workunit.client.0.smithi060.stdout: create 118/36 file after the snapshot 2024-04-14T23:51:50.284 INFO:tasks.workunit.client.0.smithi060.stdout: create 118/37 file after the snapshot 2024-04-14T23:51:50.287 INFO:tasks.workunit.client.0.smithi060.stdout: create 118/38 file after the snapshot 2024-04-14T23:51:50.290 INFO:tasks.workunit.client.0.smithi060.stdout: create 118/39 file after the snapshot 2024-04-14T23:51:50.293 INFO:tasks.workunit.client.0.smithi060.stdout: create 119/.snap/snap-subdir-test 2024-04-14T23:51:50.357 INFO:tasks.workunit.client.0.smithi060.stdout: create 119/30 file after the snapshot 2024-04-14T23:51:50.360 INFO:tasks.workunit.client.0.smithi060.stdout: create 119/31 file after the snapshot 2024-04-14T23:51:50.363 INFO:tasks.workunit.client.0.smithi060.stdout: create 119/32 file after the snapshot 2024-04-14T23:51:50.366 INFO:tasks.workunit.client.0.smithi060.stdout: create 119/33 file after the snapshot 2024-04-14T23:51:50.369 INFO:tasks.workunit.client.0.smithi060.stdout: create 119/34 file after the snapshot 2024-04-14T23:51:50.806 INFO:tasks.workunit.client.0.smithi060.stdout: create 119/35 file after the snapshot 2024-04-14T23:51:50.809 INFO:tasks.workunit.client.0.smithi060.stdout: create 119/36 file after the snapshot 2024-04-14T23:51:50.812 INFO:tasks.workunit.client.0.smithi060.stdout: create 119/37 file after the snapshot 2024-04-14T23:51:50.814 INFO:tasks.workunit.client.0.smithi060.stdout: create 119/38 file after the snapshot 2024-04-14T23:51:50.817 INFO:tasks.workunit.client.0.smithi060.stdout: create 119/39 file after the snapshot 2024-04-14T23:51:50.820 INFO:tasks.workunit.client.0.smithi060.stdout: create 120/.snap/snap-subdir-test 2024-04-14T23:51:50.884 INFO:tasks.workunit.client.0.smithi060.stdout: create 120/30 file after the snapshot 2024-04-14T23:51:50.887 INFO:tasks.workunit.client.0.smithi060.stdout: create 120/31 file after the snapshot 2024-04-14T23:51:50.890 INFO:tasks.workunit.client.0.smithi060.stdout: create 120/32 file after the snapshot 2024-04-14T23:51:50.893 INFO:tasks.workunit.client.0.smithi060.stdout: create 120/33 file after the snapshot 2024-04-14T23:51:50.896 INFO:tasks.workunit.client.0.smithi060.stdout: create 120/34 file after the snapshot 2024-04-14T23:51:50.899 INFO:tasks.workunit.client.0.smithi060.stdout: create 120/35 file after the snapshot 2024-04-14T23:51:50.902 INFO:tasks.workunit.client.0.smithi060.stdout: create 120/36 file after the snapshot 2024-04-14T23:51:50.905 INFO:tasks.workunit.client.0.smithi060.stdout: create 120/37 file after the snapshot 2024-04-14T23:51:50.908 INFO:tasks.workunit.client.0.smithi060.stdout: create 120/38 file after the snapshot 2024-04-14T23:51:50.911 INFO:tasks.workunit.client.0.smithi060.stdout: create 120/39 file after the snapshot 2024-04-14T23:51:50.913 INFO:tasks.workunit.client.0.smithi060.stdout: create 121/.snap/snap-subdir-test 2024-04-14T23:51:50.974 INFO:tasks.workunit.client.0.smithi060.stdout: create 121/30 file after the snapshot 2024-04-14T23:51:50.976 INFO:tasks.workunit.client.0.smithi060.stdout: create 121/31 file after the snapshot 2024-04-14T23:51:50.978 INFO:tasks.workunit.client.0.smithi060.stdout: create 121/32 file after the snapshot 2024-04-14T23:51:50.980 INFO:tasks.workunit.client.0.smithi060.stdout: create 121/33 file after the snapshot 2024-04-14T23:51:50.982 INFO:tasks.workunit.client.0.smithi060.stdout: create 121/34 file after the snapshot 2024-04-14T23:51:50.983 INFO:tasks.workunit.client.0.smithi060.stdout: create 121/35 file after the snapshot 2024-04-14T23:51:50.985 INFO:tasks.workunit.client.0.smithi060.stdout: create 121/36 file after the snapshot 2024-04-14T23:51:50.987 INFO:tasks.workunit.client.0.smithi060.stdout: create 121/37 file after the snapshot 2024-04-14T23:51:50.989 INFO:tasks.workunit.client.0.smithi060.stdout: create 121/38 file after the snapshot 2024-04-14T23:51:50.991 INFO:tasks.workunit.client.0.smithi060.stdout: create 121/39 file after the snapshot 2024-04-14T23:51:50.994 INFO:tasks.workunit.client.0.smithi060.stdout: create 122/.snap/snap-subdir-test 2024-04-14T23:51:51.059 INFO:tasks.workunit.client.0.smithi060.stdout: create 122/30 file after the snapshot 2024-04-14T23:51:51.061 INFO:tasks.workunit.client.0.smithi060.stdout: create 122/31 file after the snapshot 2024-04-14T23:51:51.064 INFO:tasks.workunit.client.0.smithi060.stdout: create 122/32 file after the snapshot 2024-04-14T23:51:51.067 INFO:tasks.workunit.client.0.smithi060.stdout: create 122/33 file after the snapshot 2024-04-14T23:51:51.069 INFO:tasks.workunit.client.0.smithi060.stdout: create 122/34 file after the snapshot 2024-04-14T23:51:51.072 INFO:tasks.workunit.client.0.smithi060.stdout: create 122/35 file after the snapshot 2024-04-14T23:51:51.075 INFO:tasks.workunit.client.0.smithi060.stdout: create 122/36 file after the snapshot 2024-04-14T23:51:51.078 INFO:tasks.workunit.client.0.smithi060.stdout: create 122/37 file after the snapshot 2024-04-14T23:51:51.081 INFO:tasks.workunit.client.0.smithi060.stdout: create 122/38 file after the snapshot 2024-04-14T23:51:51.084 INFO:tasks.workunit.client.0.smithi060.stdout: create 122/39 file after the snapshot 2024-04-14T23:51:51.087 INFO:tasks.workunit.client.0.smithi060.stdout: create 123/.snap/snap-subdir-test 2024-04-14T23:51:51.152 INFO:tasks.workunit.client.0.smithi060.stdout: create 123/30 file after the snapshot 2024-04-14T23:51:51.154 INFO:tasks.workunit.client.0.smithi060.stdout: create 123/31 file after the snapshot 2024-04-14T23:51:51.156 INFO:tasks.workunit.client.0.smithi060.stdout: create 123/32 file after the snapshot 2024-04-14T23:51:51.158 INFO:tasks.workunit.client.0.smithi060.stdout: create 123/33 file after the snapshot 2024-04-14T23:51:51.160 INFO:tasks.workunit.client.0.smithi060.stdout: create 123/34 file after the snapshot 2024-04-14T23:51:51.163 INFO:tasks.workunit.client.0.smithi060.stdout: create 123/35 file after the snapshot 2024-04-14T23:51:51.166 INFO:tasks.workunit.client.0.smithi060.stdout: create 123/36 file after the snapshot 2024-04-14T23:51:51.169 INFO:tasks.workunit.client.0.smithi060.stdout: create 123/37 file after the snapshot 2024-04-14T23:51:51.172 INFO:tasks.workunit.client.0.smithi060.stdout: create 123/38 file after the snapshot 2024-04-14T23:51:51.175 INFO:tasks.workunit.client.0.smithi060.stdout: create 123/39 file after the snapshot 2024-04-14T23:51:51.178 INFO:tasks.workunit.client.0.smithi060.stdout: create 124/.snap/snap-subdir-test 2024-04-14T23:51:51.242 INFO:tasks.workunit.client.0.smithi060.stdout: create 124/30 file after the snapshot 2024-04-14T23:51:51.245 INFO:tasks.workunit.client.0.smithi060.stdout: create 124/31 file after the snapshot 2024-04-14T23:51:51.246 INFO:tasks.workunit.client.0.smithi060.stdout: create 124/32 file after the snapshot 2024-04-14T23:51:51.248 INFO:tasks.workunit.client.0.smithi060.stdout: create 124/33 file after the snapshot 2024-04-14T23:51:51.251 INFO:tasks.workunit.client.0.smithi060.stdout: create 124/34 file after the snapshot 2024-04-14T23:51:51.253 INFO:tasks.workunit.client.0.smithi060.stdout: create 124/35 file after the snapshot 2024-04-14T23:51:51.256 INFO:tasks.workunit.client.0.smithi060.stdout: create 124/36 file after the snapshot 2024-04-14T23:51:51.259 INFO:tasks.workunit.client.0.smithi060.stdout: create 124/37 file after the snapshot 2024-04-14T23:51:51.262 INFO:tasks.workunit.client.0.smithi060.stdout: create 124/38 file after the snapshot 2024-04-14T23:51:51.264 INFO:tasks.workunit.client.0.smithi060.stdout: create 124/39 file after the snapshot 2024-04-14T23:51:51.267 INFO:tasks.workunit.client.0.smithi060.stdout: create 125/.snap/snap-subdir-test 2024-04-14T23:51:51.352 INFO:tasks.workunit.client.0.smithi060.stdout: create 125/30 file after the snapshot 2024-04-14T23:51:51.355 INFO:tasks.workunit.client.0.smithi060.stdout: create 125/31 file after the snapshot 2024-04-14T23:51:51.357 INFO:tasks.workunit.client.0.smithi060.stdout: create 125/32 file after the snapshot 2024-04-14T23:51:51.359 INFO:tasks.workunit.client.0.smithi060.stdout: create 125/33 file after the snapshot 2024-04-14T23:51:51.362 INFO:tasks.workunit.client.0.smithi060.stdout: create 125/34 file after the snapshot 2024-04-14T23:51:51.365 INFO:tasks.workunit.client.0.smithi060.stdout: create 125/35 file after the snapshot 2024-04-14T23:51:51.371 INFO:tasks.workunit.client.0.smithi060.stdout: create 125/36 file after the snapshot 2024-04-14T23:51:51.376 INFO:tasks.workunit.client.0.smithi060.stdout: create 125/37 file after the snapshot 2024-04-14T23:51:51.385 INFO:tasks.workunit.client.0.smithi060.stdout: create 125/38 file after the snapshot 2024-04-14T23:51:51.394 INFO:tasks.workunit.client.0.smithi060.stdout: create 125/39 file after the snapshot 2024-04-14T23:51:51.443 INFO:tasks.workunit.client.0.smithi060.stdout: create 126/.snap/snap-subdir-test 2024-04-14T23:51:52.358 INFO:tasks.workunit.client.0.smithi060.stdout: create 126/30 file after the snapshot 2024-04-14T23:51:52.360 INFO:tasks.workunit.client.0.smithi060.stdout: create 126/31 file after the snapshot 2024-04-14T23:51:52.363 INFO:tasks.workunit.client.0.smithi060.stdout: create 126/32 file after the snapshot 2024-04-14T23:51:52.366 INFO:tasks.workunit.client.0.smithi060.stdout: create 126/33 file after the snapshot 2024-04-14T23:51:52.368 INFO:tasks.workunit.client.0.smithi060.stdout: create 126/34 file after the snapshot 2024-04-14T23:51:52.372 INFO:tasks.workunit.client.0.smithi060.stdout: create 126/35 file after the snapshot 2024-04-14T23:51:52.376 INFO:tasks.workunit.client.0.smithi060.stdout: create 126/36 file after the snapshot 2024-04-14T23:51:52.381 INFO:tasks.workunit.client.0.smithi060.stdout: create 126/37 file after the snapshot 2024-04-14T23:51:52.385 INFO:tasks.workunit.client.0.smithi060.stdout: create 126/38 file after the snapshot 2024-04-14T23:51:52.389 INFO:tasks.workunit.client.0.smithi060.stdout: create 126/39 file after the snapshot 2024-04-14T23:51:52.394 INFO:tasks.workunit.client.0.smithi060.stdout: create 127/.snap/snap-subdir-test 2024-04-14T23:51:52.459 INFO:tasks.workunit.client.0.smithi060.stdout: create 127/30 file after the snapshot 2024-04-14T23:51:52.462 INFO:tasks.workunit.client.0.smithi060.stdout: create 127/31 file after the snapshot 2024-04-14T23:51:52.464 INFO:tasks.workunit.client.0.smithi060.stdout: create 127/32 file after the snapshot 2024-04-14T23:51:52.466 INFO:tasks.workunit.client.0.smithi060.stdout: create 127/33 file after the snapshot 2024-04-14T23:51:52.468 INFO:tasks.workunit.client.0.smithi060.stdout: create 127/34 file after the snapshot 2024-04-14T23:51:52.470 INFO:tasks.workunit.client.0.smithi060.stdout: create 127/35 file after the snapshot 2024-04-14T23:51:52.473 INFO:tasks.workunit.client.0.smithi060.stdout: create 127/36 file after the snapshot 2024-04-14T23:51:52.474 INFO:tasks.workunit.client.0.smithi060.stdout: create 127/37 file after the snapshot 2024-04-14T23:51:52.476 INFO:tasks.workunit.client.0.smithi060.stdout: create 127/38 file after the snapshot 2024-04-14T23:51:52.478 INFO:tasks.workunit.client.0.smithi060.stdout: create 127/39 file after the snapshot 2024-04-14T23:51:52.481 INFO:tasks.workunit.client.0.smithi060.stdout: create 128/.snap/snap-subdir-test 2024-04-14T23:51:52.551 INFO:tasks.workunit.client.0.smithi060.stdout: create 128/30 file after the snapshot 2024-04-14T23:51:52.554 INFO:tasks.workunit.client.0.smithi060.stdout: create 128/31 file after the snapshot 2024-04-14T23:51:52.556 INFO:tasks.workunit.client.0.smithi060.stdout: create 128/32 file after the snapshot 2024-04-14T23:51:52.559 INFO:tasks.workunit.client.0.smithi060.stdout: create 128/33 file after the snapshot 2024-04-14T23:51:52.562 INFO:tasks.workunit.client.0.smithi060.stdout: create 128/34 file after the snapshot 2024-04-14T23:51:52.564 INFO:tasks.workunit.client.0.smithi060.stdout: create 128/35 file after the snapshot 2024-04-14T23:51:52.567 INFO:tasks.workunit.client.0.smithi060.stdout: create 128/36 file after the snapshot 2024-04-14T23:51:52.570 INFO:tasks.workunit.client.0.smithi060.stdout: create 128/37 file after the snapshot 2024-04-14T23:51:52.572 INFO:tasks.workunit.client.0.smithi060.stdout: create 128/38 file after the snapshot 2024-04-14T23:51:52.575 INFO:tasks.workunit.client.0.smithi060.stdout: create 128/39 file after the snapshot 2024-04-14T23:51:52.578 INFO:tasks.workunit.client.0.smithi060.stdout: create 129/.snap/snap-subdir-test 2024-04-14T23:51:52.644 INFO:tasks.workunit.client.0.smithi060.stdout: create 129/30 file after the snapshot 2024-04-14T23:51:52.647 INFO:tasks.workunit.client.0.smithi060.stdout: create 129/31 file after the snapshot 2024-04-14T23:51:52.649 INFO:tasks.workunit.client.0.smithi060.stdout: create 129/32 file after the snapshot 2024-04-14T23:51:52.652 INFO:tasks.workunit.client.0.smithi060.stdout: create 129/33 file after the snapshot 2024-04-14T23:51:52.655 INFO:tasks.workunit.client.0.smithi060.stdout: create 129/34 file after the snapshot 2024-04-14T23:51:52.657 INFO:tasks.workunit.client.0.smithi060.stdout: create 129/35 file after the snapshot 2024-04-14T23:51:52.660 INFO:tasks.workunit.client.0.smithi060.stdout: create 129/36 file after the snapshot 2024-04-14T23:51:52.663 INFO:tasks.workunit.client.0.smithi060.stdout: create 129/37 file after the snapshot 2024-04-14T23:51:52.666 INFO:tasks.workunit.client.0.smithi060.stdout: create 129/38 file after the snapshot 2024-04-14T23:51:52.668 INFO:tasks.workunit.client.0.smithi060.stdout: create 129/39 file after the snapshot 2024-04-14T23:51:52.671 INFO:tasks.workunit.client.0.smithi060.stdout: create 130/.snap/snap-subdir-test 2024-04-14T23:51:52.731 INFO:tasks.workunit.client.0.smithi060.stdout: create 130/30 file after the snapshot 2024-04-14T23:51:52.733 INFO:tasks.workunit.client.0.smithi060.stdout: create 130/31 file after the snapshot 2024-04-14T23:51:52.736 INFO:tasks.workunit.client.0.smithi060.stdout: create 130/32 file after the snapshot 2024-04-14T23:51:52.739 INFO:tasks.workunit.client.0.smithi060.stdout: create 130/33 file after the snapshot 2024-04-14T23:51:52.742 INFO:tasks.workunit.client.0.smithi060.stdout: create 130/34 file after the snapshot 2024-04-14T23:51:52.745 INFO:tasks.workunit.client.0.smithi060.stdout: create 130/35 file after the snapshot 2024-04-14T23:51:52.747 INFO:tasks.workunit.client.0.smithi060.stdout: create 130/36 file after the snapshot 2024-04-14T23:51:52.750 INFO:tasks.workunit.client.0.smithi060.stdout: create 130/37 file after the snapshot 2024-04-14T23:51:52.753 INFO:tasks.workunit.client.0.smithi060.stdout: create 130/38 file after the snapshot 2024-04-14T23:51:52.756 INFO:tasks.workunit.client.0.smithi060.stdout: create 130/39 file after the snapshot 2024-04-14T23:51:52.758 INFO:tasks.workunit.client.0.smithi060.stdout: create 131/.snap/snap-subdir-test 2024-04-14T23:51:52.831 INFO:tasks.workunit.client.0.smithi060.stdout: create 131/30 file after the snapshot 2024-04-14T23:51:52.833 INFO:tasks.workunit.client.0.smithi060.stdout: create 131/31 file after the snapshot 2024-04-14T23:51:52.836 INFO:tasks.workunit.client.0.smithi060.stdout: create 131/32 file after the snapshot 2024-04-14T23:51:52.839 INFO:tasks.workunit.client.0.smithi060.stdout: create 131/33 file after the snapshot 2024-04-14T23:51:52.842 INFO:tasks.workunit.client.0.smithi060.stdout: create 131/34 file after the snapshot 2024-04-14T23:51:52.845 INFO:tasks.workunit.client.0.smithi060.stdout: create 131/35 file after the snapshot 2024-04-14T23:51:52.848 INFO:tasks.workunit.client.0.smithi060.stdout: create 131/36 file after the snapshot 2024-04-14T23:51:52.851 INFO:tasks.workunit.client.0.smithi060.stdout: create 131/37 file after the snapshot 2024-04-14T23:51:52.853 INFO:tasks.workunit.client.0.smithi060.stdout: create 131/38 file after the snapshot 2024-04-14T23:51:52.856 INFO:tasks.workunit.client.0.smithi060.stdout: create 131/39 file after the snapshot 2024-04-14T23:51:52.859 INFO:tasks.workunit.client.0.smithi060.stdout: create 132/.snap/snap-subdir-test 2024-04-14T23:51:52.932 INFO:tasks.workunit.client.0.smithi060.stdout: create 132/30 file after the snapshot 2024-04-14T23:51:52.935 INFO:tasks.workunit.client.0.smithi060.stdout: create 132/31 file after the snapshot 2024-04-14T23:51:52.938 INFO:tasks.workunit.client.0.smithi060.stdout: create 132/32 file after the snapshot 2024-04-14T23:51:52.941 INFO:tasks.workunit.client.0.smithi060.stdout: create 132/33 file after the snapshot 2024-04-14T23:51:52.943 INFO:tasks.workunit.client.0.smithi060.stdout: create 132/34 file after the snapshot 2024-04-14T23:51:52.946 INFO:tasks.workunit.client.0.smithi060.stdout: create 132/35 file after the snapshot 2024-04-14T23:51:52.949 INFO:tasks.workunit.client.0.smithi060.stdout: create 132/36 file after the snapshot 2024-04-14T23:51:52.952 INFO:tasks.workunit.client.0.smithi060.stdout: create 132/37 file after the snapshot 2024-04-14T23:51:52.955 INFO:tasks.workunit.client.0.smithi060.stdout: create 132/38 file after the snapshot 2024-04-14T23:51:52.957 INFO:tasks.workunit.client.0.smithi060.stdout: create 132/39 file after the snapshot 2024-04-14T23:51:52.960 INFO:tasks.workunit.client.0.smithi060.stdout: create 133/.snap/snap-subdir-test 2024-04-14T23:51:53.255 INFO:tasks.workunit.client.0.smithi060.stdout: create 133/30 file after the snapshot 2024-04-14T23:51:53.257 INFO:tasks.workunit.client.0.smithi060.stdout: create 133/31 file after the snapshot 2024-04-14T23:51:53.260 INFO:tasks.workunit.client.0.smithi060.stdout: create 133/32 file after the snapshot 2024-04-14T23:51:53.263 INFO:tasks.workunit.client.0.smithi060.stdout: create 133/33 file after the snapshot 2024-04-14T23:51:53.266 INFO:tasks.workunit.client.0.smithi060.stdout: create 133/34 file after the snapshot 2024-04-14T23:51:53.269 INFO:tasks.workunit.client.0.smithi060.stdout: create 133/35 file after the snapshot 2024-04-14T23:51:53.272 INFO:tasks.workunit.client.0.smithi060.stdout: create 133/36 file after the snapshot 2024-04-14T23:51:53.274 INFO:tasks.workunit.client.0.smithi060.stdout: create 133/37 file after the snapshot 2024-04-14T23:51:53.277 INFO:tasks.workunit.client.0.smithi060.stdout: create 133/38 file after the snapshot 2024-04-14T23:51:53.280 INFO:tasks.workunit.client.0.smithi060.stdout: create 133/39 file after the snapshot 2024-04-14T23:51:53.283 INFO:tasks.workunit.client.0.smithi060.stdout: create 134/.snap/snap-subdir-test 2024-04-14T23:51:53.355 INFO:tasks.workunit.client.0.smithi060.stdout: create 134/30 file after the snapshot 2024-04-14T23:51:53.357 INFO:tasks.workunit.client.0.smithi060.stdout: create 134/31 file after the snapshot 2024-04-14T23:51:53.360 INFO:tasks.workunit.client.0.smithi060.stdout: create 134/32 file after the snapshot 2024-04-14T23:51:53.363 INFO:tasks.workunit.client.0.smithi060.stdout: create 134/33 file after the snapshot 2024-04-14T23:51:53.366 INFO:tasks.workunit.client.0.smithi060.stdout: create 134/34 file after the snapshot 2024-04-14T23:51:53.368 INFO:tasks.workunit.client.0.smithi060.stdout: create 134/35 file after the snapshot 2024-04-14T23:51:53.371 INFO:tasks.workunit.client.0.smithi060.stdout: create 134/36 file after the snapshot 2024-04-14T23:51:53.860 INFO:tasks.workunit.client.0.smithi060.stdout: create 134/37 file after the snapshot 2024-04-14T23:51:54.178 INFO:tasks.workunit.client.0.smithi060.stdout: create 134/38 file after the snapshot 2024-04-14T23:51:54.180 INFO:tasks.workunit.client.0.smithi060.stdout: create 134/39 file after the snapshot 2024-04-14T23:51:54.183 INFO:tasks.workunit.client.0.smithi060.stdout: create 135/.snap/snap-subdir-test 2024-04-14T23:51:54.583 INFO:tasks.workunit.client.0.smithi060.stdout: create 135/30 file after the snapshot 2024-04-14T23:51:54.585 INFO:tasks.workunit.client.0.smithi060.stdout: create 135/31 file after the snapshot 2024-04-14T23:51:54.586 INFO:tasks.workunit.client.0.smithi060.stdout: create 135/32 file after the snapshot 2024-04-14T23:51:54.588 INFO:tasks.workunit.client.0.smithi060.stdout: create 135/33 file after the snapshot 2024-04-14T23:51:54.590 INFO:tasks.workunit.client.0.smithi060.stdout: create 135/34 file after the snapshot 2024-04-14T23:51:54.591 INFO:tasks.workunit.client.0.smithi060.stdout: create 135/35 file after the snapshot 2024-04-14T23:51:54.593 INFO:tasks.workunit.client.0.smithi060.stdout: create 135/36 file after the snapshot 2024-04-14T23:51:54.596 INFO:tasks.workunit.client.0.smithi060.stdout: create 135/37 file after the snapshot 2024-04-14T23:51:54.598 INFO:tasks.workunit.client.0.smithi060.stdout: create 135/38 file after the snapshot 2024-04-14T23:51:54.600 INFO:tasks.workunit.client.0.smithi060.stdout: create 135/39 file after the snapshot 2024-04-14T23:51:54.603 INFO:tasks.workunit.client.0.smithi060.stdout: create 136/.snap/snap-subdir-test 2024-04-14T23:51:54.654 INFO:tasks.workunit.client.0.smithi060.stdout: create 136/30 file after the snapshot 2024-04-14T23:51:54.657 INFO:tasks.workunit.client.0.smithi060.stdout: create 136/31 file after the snapshot 2024-04-14T23:51:54.660 INFO:tasks.workunit.client.0.smithi060.stdout: create 136/32 file after the snapshot 2024-04-14T23:51:54.663 INFO:tasks.workunit.client.0.smithi060.stdout: create 136/33 file after the snapshot 2024-04-14T23:51:54.666 INFO:tasks.workunit.client.0.smithi060.stdout: create 136/34 file after the snapshot 2024-04-14T23:51:54.669 INFO:tasks.workunit.client.0.smithi060.stdout: create 136/35 file after the snapshot 2024-04-14T23:51:54.671 INFO:tasks.workunit.client.0.smithi060.stdout: create 136/36 file after the snapshot 2024-04-14T23:51:54.674 INFO:tasks.workunit.client.0.smithi060.stdout: create 136/37 file after the snapshot 2024-04-14T23:51:54.677 INFO:tasks.workunit.client.0.smithi060.stdout: create 136/38 file after the snapshot 2024-04-14T23:51:54.680 INFO:tasks.workunit.client.0.smithi060.stdout: create 136/39 file after the snapshot 2024-04-14T23:51:54.683 INFO:tasks.workunit.client.0.smithi060.stdout: create 137/.snap/snap-subdir-test 2024-04-14T23:51:54.743 INFO:tasks.workunit.client.0.smithi060.stdout: create 137/30 file after the snapshot 2024-04-14T23:51:54.746 INFO:tasks.workunit.client.0.smithi060.stdout: create 137/31 file after the snapshot 2024-04-14T23:51:54.749 INFO:tasks.workunit.client.0.smithi060.stdout: create 137/32 file after the snapshot 2024-04-14T23:51:54.752 INFO:tasks.workunit.client.0.smithi060.stdout: create 137/33 file after the snapshot 2024-04-14T23:51:54.755 INFO:tasks.workunit.client.0.smithi060.stdout: create 137/34 file after the snapshot 2024-04-14T23:51:54.758 INFO:tasks.workunit.client.0.smithi060.stdout: create 137/35 file after the snapshot 2024-04-14T23:51:54.761 INFO:tasks.workunit.client.0.smithi060.stdout: create 137/36 file after the snapshot 2024-04-14T23:51:54.763 INFO:tasks.workunit.client.0.smithi060.stdout: create 137/37 file after the snapshot 2024-04-14T23:51:54.766 INFO:tasks.workunit.client.0.smithi060.stdout: create 137/38 file after the snapshot 2024-04-14T23:51:54.769 INFO:tasks.workunit.client.0.smithi060.stdout: create 137/39 file after the snapshot 2024-04-14T23:51:54.772 INFO:tasks.workunit.client.0.smithi060.stdout: create 138/.snap/snap-subdir-test 2024-04-14T23:51:54.823 INFO:tasks.workunit.client.0.smithi060.stdout: create 138/30 file after the snapshot 2024-04-14T23:51:54.825 INFO:tasks.workunit.client.0.smithi060.stdout: create 138/31 file after the snapshot 2024-04-14T23:51:54.827 INFO:tasks.workunit.client.0.smithi060.stdout: create 138/32 file after the snapshot 2024-04-14T23:51:54.830 INFO:tasks.workunit.client.0.smithi060.stdout: create 138/33 file after the snapshot 2024-04-14T23:51:54.832 INFO:tasks.workunit.client.0.smithi060.stdout: create 138/34 file after the snapshot 2024-04-14T23:51:54.835 INFO:tasks.workunit.client.0.smithi060.stdout: create 138/35 file after the snapshot 2024-04-14T23:51:54.868 INFO:tasks.workunit.client.0.smithi060.stdout: create 138/36 file after the snapshot 2024-04-14T23:51:54.872 INFO:tasks.workunit.client.0.smithi060.stdout: create 138/37 file after the snapshot 2024-04-14T23:51:54.876 INFO:tasks.workunit.client.0.smithi060.stdout: create 138/38 file after the snapshot 2024-04-14T23:51:54.881 INFO:tasks.workunit.client.0.smithi060.stdout: create 138/39 file after the snapshot 2024-04-14T23:51:54.884 INFO:tasks.workunit.client.0.smithi060.stdout: create 139/.snap/snap-subdir-test 2024-04-14T23:51:54.943 INFO:tasks.workunit.client.0.smithi060.stdout: create 139/30 file after the snapshot 2024-04-14T23:51:54.946 INFO:tasks.workunit.client.0.smithi060.stdout: create 139/31 file after the snapshot 2024-04-14T23:51:54.949 INFO:tasks.workunit.client.0.smithi060.stdout: create 139/32 file after the snapshot 2024-04-14T23:51:54.952 INFO:tasks.workunit.client.0.smithi060.stdout: create 139/33 file after the snapshot 2024-04-14T23:51:54.955 INFO:tasks.workunit.client.0.smithi060.stdout: create 139/34 file after the snapshot 2024-04-14T23:51:54.958 INFO:tasks.workunit.client.0.smithi060.stdout: create 139/35 file after the snapshot 2024-04-14T23:51:54.960 INFO:tasks.workunit.client.0.smithi060.stdout: create 139/36 file after the snapshot 2024-04-14T23:51:54.963 INFO:tasks.workunit.client.0.smithi060.stdout: create 139/37 file after the snapshot 2024-04-14T23:51:54.966 INFO:tasks.workunit.client.0.smithi060.stdout: create 139/38 file after the snapshot 2024-04-14T23:51:54.969 INFO:tasks.workunit.client.0.smithi060.stdout: create 139/39 file after the snapshot 2024-04-14T23:51:54.971 INFO:tasks.workunit.client.0.smithi060.stdout: create 140/.snap/snap-subdir-test 2024-04-14T23:51:55.028 INFO:tasks.workunit.client.0.smithi060.stdout: create 140/30 file after the snapshot 2024-04-14T23:51:55.031 INFO:tasks.workunit.client.0.smithi060.stdout: create 140/31 file after the snapshot 2024-04-14T23:51:55.034 INFO:tasks.workunit.client.0.smithi060.stdout: create 140/32 file after the snapshot 2024-04-14T23:51:55.037 INFO:tasks.workunit.client.0.smithi060.stdout: create 140/33 file after the snapshot 2024-04-14T23:51:55.039 INFO:tasks.workunit.client.0.smithi060.stdout: create 140/34 file after the snapshot 2024-04-14T23:51:55.042 INFO:tasks.workunit.client.0.smithi060.stdout: create 140/35 file after the snapshot 2024-04-14T23:51:55.045 INFO:tasks.workunit.client.0.smithi060.stdout: create 140/36 file after the snapshot 2024-04-14T23:51:55.048 INFO:tasks.workunit.client.0.smithi060.stdout: create 140/37 file after the snapshot 2024-04-14T23:51:55.051 INFO:tasks.workunit.client.0.smithi060.stdout: create 140/38 file after the snapshot 2024-04-14T23:51:55.054 INFO:tasks.workunit.client.0.smithi060.stdout: create 140/39 file after the snapshot 2024-04-14T23:51:55.056 INFO:tasks.workunit.client.0.smithi060.stdout: create 141/.snap/snap-subdir-test 2024-04-14T23:51:55.114 INFO:tasks.workunit.client.0.smithi060.stdout: create 141/30 file after the snapshot 2024-04-14T23:51:55.117 INFO:tasks.workunit.client.0.smithi060.stdout: create 141/31 file after the snapshot 2024-04-14T23:51:55.120 INFO:tasks.workunit.client.0.smithi060.stdout: create 141/32 file after the snapshot 2024-04-14T23:51:55.123 INFO:tasks.workunit.client.0.smithi060.stdout: create 141/33 file after the snapshot 2024-04-14T23:51:55.125 INFO:tasks.workunit.client.0.smithi060.stdout: create 141/34 file after the snapshot 2024-04-14T23:51:55.129 INFO:tasks.workunit.client.0.smithi060.stdout: create 141/35 file after the snapshot 2024-04-14T23:51:55.130 INFO:tasks.workunit.client.0.smithi060.stdout: create 141/36 file after the snapshot 2024-04-14T23:51:55.133 INFO:tasks.workunit.client.0.smithi060.stdout: create 141/37 file after the snapshot 2024-04-14T23:51:55.136 INFO:tasks.workunit.client.0.smithi060.stdout: create 141/38 file after the snapshot 2024-04-14T23:51:55.138 INFO:tasks.workunit.client.0.smithi060.stdout: create 141/39 file after the snapshot 2024-04-14T23:51:55.141 INFO:tasks.workunit.client.0.smithi060.stdout: create 142/.snap/snap-subdir-test 2024-04-14T23:51:55.199 INFO:tasks.workunit.client.0.smithi060.stdout: create 142/30 file after the snapshot 2024-04-14T23:51:55.202 INFO:tasks.workunit.client.0.smithi060.stdout: create 142/31 file after the snapshot 2024-04-14T23:51:55.203 INFO:tasks.workunit.client.0.smithi060.stdout: create 142/32 file after the snapshot 2024-04-14T23:51:55.205 INFO:tasks.workunit.client.0.smithi060.stdout: create 142/33 file after the snapshot 2024-04-14T23:51:55.206 INFO:tasks.workunit.client.0.smithi060.stdout: create 142/34 file after the snapshot 2024-04-14T23:51:55.207 INFO:tasks.workunit.client.0.smithi060.stdout: create 142/35 file after the snapshot 2024-04-14T23:51:55.209 INFO:tasks.workunit.client.0.smithi060.stdout: create 142/36 file after the snapshot 2024-04-14T23:51:55.210 INFO:tasks.workunit.client.0.smithi060.stdout: create 142/37 file after the snapshot 2024-04-14T23:51:55.211 INFO:tasks.workunit.client.0.smithi060.stdout: create 142/38 file after the snapshot 2024-04-14T23:51:55.213 INFO:tasks.workunit.client.0.smithi060.stdout: create 142/39 file after the snapshot 2024-04-14T23:51:55.214 INFO:tasks.workunit.client.0.smithi060.stdout: create 143/.snap/snap-subdir-test 2024-04-14T23:51:55.277 INFO:tasks.workunit.client.0.smithi060.stdout: create 143/30 file after the snapshot 2024-04-14T23:51:55.280 INFO:tasks.workunit.client.0.smithi060.stdout: create 143/31 file after the snapshot 2024-04-14T23:51:55.283 INFO:tasks.workunit.client.0.smithi060.stdout: create 143/32 file after the snapshot 2024-04-14T23:51:55.286 INFO:tasks.workunit.client.0.smithi060.stdout: create 143/33 file after the snapshot 2024-04-14T23:51:55.288 INFO:tasks.workunit.client.0.smithi060.stdout: create 143/34 file after the snapshot 2024-04-14T23:51:55.291 INFO:tasks.workunit.client.0.smithi060.stdout: create 143/35 file after the snapshot 2024-04-14T23:51:55.294 INFO:tasks.workunit.client.0.smithi060.stdout: create 143/36 file after the snapshot 2024-04-14T23:51:55.297 INFO:tasks.workunit.client.0.smithi060.stdout: create 143/37 file after the snapshot 2024-04-14T23:51:55.299 INFO:tasks.workunit.client.0.smithi060.stdout: create 143/38 file after the snapshot 2024-04-14T23:51:55.302 INFO:tasks.workunit.client.0.smithi060.stdout: create 143/39 file after the snapshot 2024-04-14T23:51:55.305 INFO:tasks.workunit.client.0.smithi060.stdout: create 144/.snap/snap-subdir-test 2024-04-14T23:51:55.364 INFO:tasks.workunit.client.0.smithi060.stdout: create 144/30 file after the snapshot 2024-04-14T23:51:55.367 INFO:tasks.workunit.client.0.smithi060.stdout: create 144/31 file after the snapshot 2024-04-14T23:51:55.370 INFO:tasks.workunit.client.0.smithi060.stdout: create 144/32 file after the snapshot 2024-04-14T23:51:55.372 INFO:tasks.workunit.client.0.smithi060.stdout: create 144/33 file after the snapshot 2024-04-14T23:51:55.375 INFO:tasks.workunit.client.0.smithi060.stdout: create 144/34 file after the snapshot 2024-04-14T23:51:55.378 INFO:tasks.workunit.client.0.smithi060.stdout: create 144/35 file after the snapshot 2024-04-14T23:51:55.381 INFO:tasks.workunit.client.0.smithi060.stdout: create 144/36 file after the snapshot 2024-04-14T23:51:55.384 INFO:tasks.workunit.client.0.smithi060.stdout: create 144/37 file after the snapshot 2024-04-14T23:51:55.387 INFO:tasks.workunit.client.0.smithi060.stdout: create 144/38 file after the snapshot 2024-04-14T23:51:55.390 INFO:tasks.workunit.client.0.smithi060.stdout: create 144/39 file after the snapshot 2024-04-14T23:51:55.393 INFO:tasks.workunit.client.0.smithi060.stdout: create 145/.snap/snap-subdir-test 2024-04-14T23:51:55.456 INFO:tasks.workunit.client.0.smithi060.stdout: create 145/30 file after the snapshot 2024-04-14T23:51:55.459 INFO:tasks.workunit.client.0.smithi060.stdout: create 145/31 file after the snapshot 2024-04-14T23:51:55.462 INFO:tasks.workunit.client.0.smithi060.stdout: create 145/32 file after the snapshot 2024-04-14T23:51:55.465 INFO:tasks.workunit.client.0.smithi060.stdout: create 145/33 file after the snapshot 2024-04-14T23:51:55.468 INFO:tasks.workunit.client.0.smithi060.stdout: create 145/34 file after the snapshot 2024-04-14T23:51:55.471 INFO:tasks.workunit.client.0.smithi060.stdout: create 145/35 file after the snapshot 2024-04-14T23:51:55.474 INFO:tasks.workunit.client.0.smithi060.stdout: create 145/36 file after the snapshot 2024-04-14T23:51:55.476 INFO:tasks.workunit.client.0.smithi060.stdout: create 145/37 file after the snapshot 2024-04-14T23:51:55.479 INFO:tasks.workunit.client.0.smithi060.stdout: create 145/38 file after the snapshot 2024-04-14T23:51:55.482 INFO:tasks.workunit.client.0.smithi060.stdout: create 145/39 file after the snapshot 2024-04-14T23:51:55.485 INFO:tasks.workunit.client.0.smithi060.stdout: create 146/.snap/snap-subdir-test 2024-04-14T23:51:55.537 INFO:tasks.workunit.client.0.smithi060.stdout: create 146/30 file after the snapshot 2024-04-14T23:51:55.539 INFO:tasks.workunit.client.0.smithi060.stdout: create 146/31 file after the snapshot 2024-04-14T23:51:55.542 INFO:tasks.workunit.client.0.smithi060.stdout: create 146/32 file after the snapshot 2024-04-14T23:51:55.547 INFO:tasks.workunit.client.0.smithi060.stdout: create 146/33 file after the snapshot 2024-04-14T23:51:55.550 INFO:tasks.workunit.client.0.smithi060.stdout: create 146/34 file after the snapshot 2024-04-14T23:51:55.553 INFO:tasks.workunit.client.0.smithi060.stdout: create 146/35 file after the snapshot 2024-04-14T23:51:55.556 INFO:tasks.workunit.client.0.smithi060.stdout: create 146/36 file after the snapshot 2024-04-14T23:51:55.559 INFO:tasks.workunit.client.0.smithi060.stdout: create 146/37 file after the snapshot 2024-04-14T23:51:55.563 INFO:tasks.workunit.client.0.smithi060.stdout: create 146/38 file after the snapshot 2024-04-14T23:51:55.566 INFO:tasks.workunit.client.0.smithi060.stdout: create 146/39 file after the snapshot 2024-04-14T23:51:55.568 INFO:tasks.workunit.client.0.smithi060.stdout: create 147/.snap/snap-subdir-test 2024-04-14T23:51:55.653 INFO:tasks.workunit.client.0.smithi060.stdout: create 147/30 file after the snapshot 2024-04-14T23:51:55.655 INFO:tasks.workunit.client.0.smithi060.stdout: create 147/31 file after the snapshot 2024-04-14T23:51:55.658 INFO:tasks.workunit.client.0.smithi060.stdout: create 147/32 file after the snapshot 2024-04-14T23:51:55.661 INFO:tasks.workunit.client.0.smithi060.stdout: create 147/33 file after the snapshot 2024-04-14T23:51:55.663 INFO:tasks.workunit.client.0.smithi060.stdout: create 147/34 file after the snapshot 2024-04-14T23:51:55.666 INFO:tasks.workunit.client.0.smithi060.stdout: create 147/35 file after the snapshot 2024-04-14T23:51:55.669 INFO:tasks.workunit.client.0.smithi060.stdout: create 147/36 file after the snapshot 2024-04-14T23:51:55.671 INFO:tasks.workunit.client.0.smithi060.stdout: create 147/37 file after the snapshot 2024-04-14T23:51:55.674 INFO:tasks.workunit.client.0.smithi060.stdout: create 147/38 file after the snapshot 2024-04-14T23:51:55.677 INFO:tasks.workunit.client.0.smithi060.stdout: create 147/39 file after the snapshot 2024-04-14T23:51:55.680 INFO:tasks.workunit.client.0.smithi060.stdout: create 148/.snap/snap-subdir-test 2024-04-14T23:51:55.757 INFO:tasks.workunit.client.0.smithi060.stdout: create 148/30 file after the snapshot 2024-04-14T23:51:55.761 INFO:tasks.workunit.client.0.smithi060.stdout: create 148/31 file after the snapshot 2024-04-14T23:51:55.765 INFO:tasks.workunit.client.0.smithi060.stdout: create 148/32 file after the snapshot 2024-04-14T23:51:55.769 INFO:tasks.workunit.client.0.smithi060.stdout: create 148/33 file after the snapshot 2024-04-14T23:51:55.773 INFO:tasks.workunit.client.0.smithi060.stdout: create 148/34 file after the snapshot 2024-04-14T23:51:55.777 INFO:tasks.workunit.client.0.smithi060.stdout: create 148/35 file after the snapshot 2024-04-14T23:51:55.782 INFO:tasks.workunit.client.0.smithi060.stdout: create 148/36 file after the snapshot 2024-04-14T23:51:55.786 INFO:tasks.workunit.client.0.smithi060.stdout: create 148/37 file after the snapshot 2024-04-14T23:51:55.790 INFO:tasks.workunit.client.0.smithi060.stdout: create 148/38 file after the snapshot 2024-04-14T23:51:55.794 INFO:tasks.workunit.client.0.smithi060.stdout: create 148/39 file after the snapshot 2024-04-14T23:51:55.799 INFO:tasks.workunit.client.0.smithi060.stdout: create 149/.snap/snap-subdir-test 2024-04-14T23:51:55.860 INFO:tasks.workunit.client.0.smithi060.stdout: create 149/30 file after the snapshot 2024-04-14T23:51:55.862 INFO:tasks.workunit.client.0.smithi060.stdout: create 149/31 file after the snapshot 2024-04-14T23:51:55.864 INFO:tasks.workunit.client.0.smithi060.stdout: create 149/32 file after the snapshot 2024-04-14T23:51:55.866 INFO:tasks.workunit.client.0.smithi060.stdout: create 149/33 file after the snapshot 2024-04-14T23:51:55.868 INFO:tasks.workunit.client.0.smithi060.stdout: create 149/34 file after the snapshot 2024-04-14T23:51:55.870 INFO:tasks.workunit.client.0.smithi060.stdout: create 149/35 file after the snapshot 2024-04-14T23:51:55.872 INFO:tasks.workunit.client.0.smithi060.stdout: create 149/36 file after the snapshot 2024-04-14T23:51:55.874 INFO:tasks.workunit.client.0.smithi060.stdout: create 149/37 file after the snapshot 2024-04-14T23:51:55.876 INFO:tasks.workunit.client.0.smithi060.stdout: create 149/38 file after the snapshot 2024-04-14T23:51:55.878 INFO:tasks.workunit.client.0.smithi060.stdout: create 149/39 file after the snapshot 2024-04-14T23:51:55.880 INFO:tasks.workunit.client.0.smithi060.stdout: create 150/.snap/snap-subdir-test 2024-04-14T23:51:55.925 INFO:tasks.workunit.client.0.smithi060.stdout: create 150/30 file after the snapshot 2024-04-14T23:51:55.928 INFO:tasks.workunit.client.0.smithi060.stdout: create 150/31 file after the snapshot 2024-04-14T23:51:55.930 INFO:tasks.workunit.client.0.smithi060.stdout: create 150/32 file after the snapshot 2024-04-14T23:51:55.933 INFO:tasks.workunit.client.0.smithi060.stdout: create 150/33 file after the snapshot 2024-04-14T23:51:55.935 INFO:tasks.workunit.client.0.smithi060.stdout: create 150/34 file after the snapshot 2024-04-14T23:51:55.938 INFO:tasks.workunit.client.0.smithi060.stdout: create 150/35 file after the snapshot 2024-04-14T23:51:55.940 INFO:tasks.workunit.client.0.smithi060.stdout: create 150/36 file after the snapshot 2024-04-14T23:51:55.943 INFO:tasks.workunit.client.0.smithi060.stdout: create 150/37 file after the snapshot 2024-04-14T23:51:56.663 INFO:tasks.workunit.client.0.smithi060.stdout: create 150/38 file after the snapshot 2024-04-14T23:51:56.666 INFO:tasks.workunit.client.0.smithi060.stdout: create 150/39 file after the snapshot 2024-04-14T23:51:56.669 INFO:tasks.workunit.client.0.smithi060.stdout: create 151/.snap/snap-subdir-test 2024-04-14T23:51:56.724 INFO:tasks.workunit.client.0.smithi060.stdout: create 151/30 file after the snapshot 2024-04-14T23:51:56.727 INFO:tasks.workunit.client.0.smithi060.stdout: create 151/31 file after the snapshot 2024-04-14T23:51:56.728 INFO:tasks.workunit.client.0.smithi060.stdout: create 151/32 file after the snapshot 2024-04-14T23:51:56.731 INFO:tasks.workunit.client.0.smithi060.stdout: create 151/33 file after the snapshot 2024-04-14T23:51:56.733 INFO:tasks.workunit.client.0.smithi060.stdout: create 151/34 file after the snapshot 2024-04-14T23:51:56.734 INFO:tasks.workunit.client.0.smithi060.stdout: create 151/35 file after the snapshot 2024-04-14T23:51:56.736 INFO:tasks.workunit.client.0.smithi060.stdout: create 151/36 file after the snapshot 2024-04-14T23:51:56.739 INFO:tasks.workunit.client.0.smithi060.stdout: create 151/37 file after the snapshot 2024-04-14T23:51:56.741 INFO:tasks.workunit.client.0.smithi060.stdout: create 151/38 file after the snapshot 2024-04-14T23:51:56.743 INFO:tasks.workunit.client.0.smithi060.stdout: create 151/39 file after the snapshot 2024-04-14T23:51:56.745 INFO:tasks.workunit.client.0.smithi060.stdout: create 152/.snap/snap-subdir-test 2024-04-14T23:51:56.804 INFO:tasks.workunit.client.0.smithi060.stdout: create 152/30 file after the snapshot 2024-04-14T23:51:56.807 INFO:tasks.workunit.client.0.smithi060.stdout: create 152/31 file after the snapshot 2024-04-14T23:51:56.810 INFO:tasks.workunit.client.0.smithi060.stdout: create 152/32 file after the snapshot 2024-04-14T23:51:56.812 INFO:tasks.workunit.client.0.smithi060.stdout: create 152/33 file after the snapshot 2024-04-14T23:51:56.815 INFO:tasks.workunit.client.0.smithi060.stdout: create 152/34 file after the snapshot 2024-04-14T23:51:56.818 INFO:tasks.workunit.client.0.smithi060.stdout: create 152/35 file after the snapshot 2024-04-14T23:51:56.821 INFO:tasks.workunit.client.0.smithi060.stdout: create 152/36 file after the snapshot 2024-04-14T23:51:56.824 INFO:tasks.workunit.client.0.smithi060.stdout: create 152/37 file after the snapshot 2024-04-14T23:51:56.870 INFO:tasks.workunit.client.0.smithi060.stdout: create 152/38 file after the snapshot 2024-04-14T23:51:57.270 INFO:tasks.workunit.client.0.smithi060.stdout: create 152/39 file after the snapshot 2024-04-14T23:51:57.274 INFO:tasks.workunit.client.0.smithi060.stdout: create 153/.snap/snap-subdir-test 2024-04-14T23:51:57.325 INFO:tasks.workunit.client.0.smithi060.stdout: create 153/30 file after the snapshot 2024-04-14T23:51:57.329 INFO:tasks.workunit.client.0.smithi060.stdout: create 153/31 file after the snapshot 2024-04-14T23:51:57.333 INFO:tasks.workunit.client.0.smithi060.stdout: create 153/32 file after the snapshot 2024-04-14T23:51:57.337 INFO:tasks.workunit.client.0.smithi060.stdout: create 153/33 file after the snapshot 2024-04-14T23:51:57.341 INFO:tasks.workunit.client.0.smithi060.stdout: create 153/34 file after the snapshot 2024-04-14T23:51:57.345 INFO:tasks.workunit.client.0.smithi060.stdout: create 153/35 file after the snapshot 2024-04-14T23:51:57.349 INFO:tasks.workunit.client.0.smithi060.stdout: create 153/36 file after the snapshot 2024-04-14T23:51:57.353 INFO:tasks.workunit.client.0.smithi060.stdout: create 153/37 file after the snapshot 2024-04-14T23:51:57.358 INFO:tasks.workunit.client.0.smithi060.stdout: create 153/38 file after the snapshot 2024-04-14T23:51:57.362 INFO:tasks.workunit.client.0.smithi060.stdout: create 153/39 file after the snapshot 2024-04-14T23:51:57.366 INFO:tasks.workunit.client.0.smithi060.stdout: create 154/.snap/snap-subdir-test 2024-04-14T23:51:57.445 INFO:tasks.workunit.client.0.smithi060.stdout: create 154/30 file after the snapshot 2024-04-14T23:51:57.449 INFO:tasks.workunit.client.0.smithi060.stdout: create 154/31 file after the snapshot 2024-04-14T23:51:57.453 INFO:tasks.workunit.client.0.smithi060.stdout: create 154/32 file after the snapshot 2024-04-14T23:51:57.458 INFO:tasks.workunit.client.0.smithi060.stdout: create 154/33 file after the snapshot 2024-04-14T23:51:57.462 INFO:tasks.workunit.client.0.smithi060.stdout: create 154/34 file after the snapshot 2024-04-14T23:51:57.466 INFO:tasks.workunit.client.0.smithi060.stdout: create 154/35 file after the snapshot 2024-04-14T23:51:57.470 INFO:tasks.workunit.client.0.smithi060.stdout: create 154/36 file after the snapshot 2024-04-14T23:51:57.474 INFO:tasks.workunit.client.0.smithi060.stdout: create 154/37 file after the snapshot 2024-04-14T23:51:57.478 INFO:tasks.workunit.client.0.smithi060.stdout: create 154/38 file after the snapshot 2024-04-14T23:51:57.483 INFO:tasks.workunit.client.0.smithi060.stdout: create 154/39 file after the snapshot 2024-04-14T23:51:57.487 INFO:tasks.workunit.client.0.smithi060.stdout: create 155/.snap/snap-subdir-test 2024-04-14T23:51:57.561 INFO:tasks.workunit.client.0.smithi060.stdout: create 155/30 file after the snapshot 2024-04-14T23:51:57.565 INFO:tasks.workunit.client.0.smithi060.stdout: create 155/31 file after the snapshot 2024-04-14T23:51:57.569 INFO:tasks.workunit.client.0.smithi060.stdout: create 155/32 file after the snapshot 2024-04-14T23:51:57.573 INFO:tasks.workunit.client.0.smithi060.stdout: create 155/33 file after the snapshot 2024-04-14T23:51:57.578 INFO:tasks.workunit.client.0.smithi060.stdout: create 155/34 file after the snapshot 2024-04-14T23:51:57.583 INFO:tasks.workunit.client.0.smithi060.stdout: create 155/35 file after the snapshot 2024-04-14T23:51:57.586 INFO:tasks.workunit.client.0.smithi060.stdout: create 155/36 file after the snapshot 2024-04-14T23:51:57.590 INFO:tasks.workunit.client.0.smithi060.stdout: create 155/37 file after the snapshot 2024-04-14T23:51:57.594 INFO:tasks.workunit.client.0.smithi060.stdout: create 155/38 file after the snapshot 2024-04-14T23:51:57.599 INFO:tasks.workunit.client.0.smithi060.stdout: create 155/39 file after the snapshot 2024-04-14T23:51:57.603 INFO:tasks.workunit.client.0.smithi060.stdout: create 156/.snap/snap-subdir-test 2024-04-14T23:51:57.674 INFO:tasks.workunit.client.0.smithi060.stdout: create 156/30 file after the snapshot 2024-04-14T23:51:57.678 INFO:tasks.workunit.client.0.smithi060.stdout: create 156/31 file after the snapshot 2024-04-14T23:51:57.682 INFO:tasks.workunit.client.0.smithi060.stdout: create 156/32 file after the snapshot 2024-04-14T23:51:57.687 INFO:tasks.workunit.client.0.smithi060.stdout: create 156/33 file after the snapshot 2024-04-14T23:51:57.691 INFO:tasks.workunit.client.0.smithi060.stdout: create 156/34 file after the snapshot 2024-04-14T23:51:57.695 INFO:tasks.workunit.client.0.smithi060.stdout: create 156/35 file after the snapshot 2024-04-14T23:51:57.699 INFO:tasks.workunit.client.0.smithi060.stdout: create 156/36 file after the snapshot 2024-04-14T23:51:57.703 INFO:tasks.workunit.client.0.smithi060.stdout: create 156/37 file after the snapshot 2024-04-14T23:51:57.708 INFO:tasks.workunit.client.0.smithi060.stdout: create 156/38 file after the snapshot 2024-04-14T23:51:57.712 INFO:tasks.workunit.client.0.smithi060.stdout: create 156/39 file after the snapshot 2024-04-14T23:51:57.716 INFO:tasks.workunit.client.0.smithi060.stdout: create 157/.snap/snap-subdir-test 2024-04-14T23:51:57.785 INFO:tasks.workunit.client.0.smithi060.stdout: create 157/30 file after the snapshot 2024-04-14T23:51:57.789 INFO:tasks.workunit.client.0.smithi060.stdout: create 157/31 file after the snapshot 2024-04-14T23:51:57.794 INFO:tasks.workunit.client.0.smithi060.stdout: create 157/32 file after the snapshot 2024-04-14T23:51:57.798 INFO:tasks.workunit.client.0.smithi060.stdout: create 157/33 file after the snapshot 2024-04-14T23:51:57.802 INFO:tasks.workunit.client.0.smithi060.stdout: create 157/34 file after the snapshot 2024-04-14T23:51:57.806 INFO:tasks.workunit.client.0.smithi060.stdout: create 157/35 file after the snapshot 2024-04-14T23:51:57.811 INFO:tasks.workunit.client.0.smithi060.stdout: create 157/36 file after the snapshot 2024-04-14T23:51:57.815 INFO:tasks.workunit.client.0.smithi060.stdout: create 157/37 file after the snapshot 2024-04-14T23:51:57.819 INFO:tasks.workunit.client.0.smithi060.stdout: create 157/38 file after the snapshot 2024-04-14T23:51:57.823 INFO:tasks.workunit.client.0.smithi060.stdout: create 157/39 file after the snapshot 2024-04-14T23:51:57.827 INFO:tasks.workunit.client.0.smithi060.stdout: create 158/.snap/snap-subdir-test 2024-04-14T23:51:57.919 INFO:tasks.workunit.client.0.smithi060.stdout: create 158/30 file after the snapshot 2024-04-14T23:51:57.921 INFO:tasks.workunit.client.0.smithi060.stdout: create 158/31 file after the snapshot 2024-04-14T23:51:57.923 INFO:tasks.workunit.client.0.smithi060.stdout: create 158/32 file after the snapshot 2024-04-14T23:51:57.925 INFO:tasks.workunit.client.0.smithi060.stdout: create 158/33 file after the snapshot 2024-04-14T23:51:57.927 INFO:tasks.workunit.client.0.smithi060.stdout: create 158/34 file after the snapshot 2024-04-14T23:51:57.930 INFO:tasks.workunit.client.0.smithi060.stdout: create 158/35 file after the snapshot 2024-04-14T23:51:57.933 INFO:tasks.workunit.client.0.smithi060.stdout: create 158/36 file after the snapshot 2024-04-14T23:51:57.936 INFO:tasks.workunit.client.0.smithi060.stdout: create 158/37 file after the snapshot 2024-04-14T23:51:57.938 INFO:tasks.workunit.client.0.smithi060.stdout: create 158/38 file after the snapshot 2024-04-14T23:51:57.941 INFO:tasks.workunit.client.0.smithi060.stdout: create 158/39 file after the snapshot 2024-04-14T23:51:57.944 INFO:tasks.workunit.client.0.smithi060.stdout: create 159/.snap/snap-subdir-test 2024-04-14T23:51:58.012 INFO:tasks.workunit.client.0.smithi060.stdout: create 159/30 file after the snapshot 2024-04-14T23:51:58.277 INFO:tasks.workunit.client.0.smithi060.stdout: create 159/31 file after the snapshot 2024-04-14T23:51:58.279 INFO:tasks.workunit.client.0.smithi060.stdout: create 159/32 file after the snapshot 2024-04-14T23:51:58.282 INFO:tasks.workunit.client.0.smithi060.stdout: create 159/33 file after the snapshot 2024-04-14T23:51:58.285 INFO:tasks.workunit.client.0.smithi060.stdout: create 159/34 file after the snapshot 2024-04-14T23:51:58.287 INFO:tasks.workunit.client.0.smithi060.stdout: create 159/35 file after the snapshot 2024-04-14T23:51:58.290 INFO:tasks.workunit.client.0.smithi060.stdout: create 159/36 file after the snapshot 2024-04-14T23:51:58.293 INFO:tasks.workunit.client.0.smithi060.stdout: create 159/37 file after the snapshot 2024-04-14T23:51:58.296 INFO:tasks.workunit.client.0.smithi060.stdout: create 159/38 file after the snapshot 2024-04-14T23:51:58.298 INFO:tasks.workunit.client.0.smithi060.stdout: create 159/39 file after the snapshot 2024-04-14T23:51:58.301 INFO:tasks.workunit.client.0.smithi060.stdout: create 160/.snap/snap-subdir-test 2024-04-14T23:51:58.368 INFO:tasks.workunit.client.0.smithi060.stdout: create 160/30 file after the snapshot 2024-04-14T23:51:58.371 INFO:tasks.workunit.client.0.smithi060.stdout: create 160/31 file after the snapshot 2024-04-14T23:51:58.374 INFO:tasks.workunit.client.0.smithi060.stdout: create 160/32 file after the snapshot 2024-04-14T23:51:58.376 INFO:tasks.workunit.client.0.smithi060.stdout: create 160/33 file after the snapshot 2024-04-14T23:51:58.379 INFO:tasks.workunit.client.0.smithi060.stdout: create 160/34 file after the snapshot 2024-04-14T23:51:58.382 INFO:tasks.workunit.client.0.smithi060.stdout: create 160/35 file after the snapshot 2024-04-14T23:51:58.385 INFO:tasks.workunit.client.0.smithi060.stdout: create 160/36 file after the snapshot 2024-04-14T23:51:58.387 INFO:tasks.workunit.client.0.smithi060.stdout: create 160/37 file after the snapshot 2024-04-14T23:51:58.390 INFO:tasks.workunit.client.0.smithi060.stdout: create 160/38 file after the snapshot 2024-04-14T23:51:58.393 INFO:tasks.workunit.client.0.smithi060.stdout: create 160/39 file after the snapshot 2024-04-14T23:51:58.395 INFO:tasks.workunit.client.0.smithi060.stdout: create 161/.snap/snap-subdir-test 2024-04-14T23:51:58.461 INFO:tasks.workunit.client.0.smithi060.stdout: create 161/30 file after the snapshot 2024-04-14T23:51:58.464 INFO:tasks.workunit.client.0.smithi060.stdout: create 161/31 file after the snapshot 2024-04-14T23:51:58.466 INFO:tasks.workunit.client.0.smithi060.stdout: create 161/32 file after the snapshot 2024-04-14T23:51:58.469 INFO:tasks.workunit.client.0.smithi060.stdout: create 161/33 file after the snapshot 2024-04-14T23:51:58.472 INFO:tasks.workunit.client.0.smithi060.stdout: create 161/34 file after the snapshot 2024-04-14T23:51:58.475 INFO:tasks.workunit.client.0.smithi060.stdout: create 161/35 file after the snapshot 2024-04-14T23:51:58.477 INFO:tasks.workunit.client.0.smithi060.stdout: create 161/36 file after the snapshot 2024-04-14T23:51:58.480 INFO:tasks.workunit.client.0.smithi060.stdout: create 161/37 file after the snapshot 2024-04-14T23:51:58.483 INFO:tasks.workunit.client.0.smithi060.stdout: create 161/38 file after the snapshot 2024-04-14T23:51:58.486 INFO:tasks.workunit.client.0.smithi060.stdout: create 161/39 file after the snapshot 2024-04-14T23:51:58.489 INFO:tasks.workunit.client.0.smithi060.stdout: create 162/.snap/snap-subdir-test 2024-04-14T23:51:58.672 INFO:tasks.workunit.client.0.smithi060.stdout: create 162/30 file after the snapshot 2024-04-14T23:51:58.674 INFO:tasks.workunit.client.0.smithi060.stdout: create 162/31 file after the snapshot 2024-04-14T23:51:58.675 INFO:tasks.workunit.client.0.smithi060.stdout: create 162/32 file after the snapshot 2024-04-14T23:51:58.677 INFO:tasks.workunit.client.0.smithi060.stdout: create 162/33 file after the snapshot 2024-04-14T23:51:58.679 INFO:tasks.workunit.client.0.smithi060.stdout: create 162/34 file after the snapshot 2024-04-14T23:51:58.681 INFO:tasks.workunit.client.0.smithi060.stdout: create 162/35 file after the snapshot 2024-04-14T23:51:58.682 INFO:tasks.workunit.client.0.smithi060.stdout: create 162/36 file after the snapshot 2024-04-14T23:51:58.684 INFO:tasks.workunit.client.0.smithi060.stdout: create 162/37 file after the snapshot 2024-04-14T23:51:58.852 INFO:tasks.workunit.client.0.smithi060.stdout: create 162/38 file after the snapshot 2024-04-14T23:51:59.415 INFO:tasks.workunit.client.0.smithi060.stdout: create 162/39 file after the snapshot 2024-04-14T23:51:59.418 INFO:tasks.workunit.client.0.smithi060.stdout: create 163/.snap/snap-subdir-test 2024-04-14T23:51:59.476 INFO:tasks.workunit.client.0.smithi060.stdout: create 163/30 file after the snapshot 2024-04-14T23:51:59.478 INFO:tasks.workunit.client.0.smithi060.stdout: create 163/31 file after the snapshot 2024-04-14T23:51:59.481 INFO:tasks.workunit.client.0.smithi060.stdout: create 163/32 file after the snapshot 2024-04-14T23:51:59.483 INFO:tasks.workunit.client.0.smithi060.stdout: create 163/33 file after the snapshot 2024-04-14T23:51:59.486 INFO:tasks.workunit.client.0.smithi060.stdout: create 163/34 file after the snapshot 2024-04-14T23:51:59.489 INFO:tasks.workunit.client.0.smithi060.stdout: create 163/35 file after the snapshot 2024-04-14T23:51:59.491 INFO:tasks.workunit.client.0.smithi060.stdout: create 163/36 file after the snapshot 2024-04-14T23:51:59.494 INFO:tasks.workunit.client.0.smithi060.stdout: create 163/37 file after the snapshot 2024-04-14T23:51:59.497 INFO:tasks.workunit.client.0.smithi060.stdout: create 163/38 file after the snapshot 2024-04-14T23:51:59.500 INFO:tasks.workunit.client.0.smithi060.stdout: create 163/39 file after the snapshot 2024-04-14T23:51:59.503 INFO:tasks.workunit.client.0.smithi060.stdout: create 164/.snap/snap-subdir-test 2024-04-14T23:51:59.558 INFO:tasks.workunit.client.0.smithi060.stdout: create 164/30 file after the snapshot 2024-04-14T23:51:59.560 INFO:tasks.workunit.client.0.smithi060.stdout: create 164/31 file after the snapshot 2024-04-14T23:51:59.562 INFO:tasks.workunit.client.0.smithi060.stdout: create 164/32 file after the snapshot 2024-04-14T23:51:59.565 INFO:tasks.workunit.client.0.smithi060.stdout: create 164/33 file after the snapshot 2024-04-14T23:51:59.568 INFO:tasks.workunit.client.0.smithi060.stdout: create 164/34 file after the snapshot 2024-04-14T23:51:59.571 INFO:tasks.workunit.client.0.smithi060.stdout: create 164/35 file after the snapshot 2024-04-14T23:51:59.574 INFO:tasks.workunit.client.0.smithi060.stdout: create 164/36 file after the snapshot 2024-04-14T23:51:59.577 INFO:tasks.workunit.client.0.smithi060.stdout: create 164/37 file after the snapshot 2024-04-14T23:51:59.579 INFO:tasks.workunit.client.0.smithi060.stdout: create 164/38 file after the snapshot 2024-04-14T23:51:59.582 INFO:tasks.workunit.client.0.smithi060.stdout: create 164/39 file after the snapshot 2024-04-14T23:51:59.585 INFO:tasks.workunit.client.0.smithi060.stdout: create 165/.snap/snap-subdir-test 2024-04-14T23:51:59.640 INFO:tasks.workunit.client.0.smithi060.stdout: create 165/30 file after the snapshot 2024-04-14T23:51:59.643 INFO:tasks.workunit.client.0.smithi060.stdout: create 165/31 file after the snapshot 2024-04-14T23:51:59.646 INFO:tasks.workunit.client.0.smithi060.stdout: create 165/32 file after the snapshot 2024-04-14T23:51:59.648 INFO:tasks.workunit.client.0.smithi060.stdout: create 165/33 file after the snapshot 2024-04-14T23:51:59.651 INFO:tasks.workunit.client.0.smithi060.stdout: create 165/34 file after the snapshot 2024-04-14T23:51:59.653 INFO:tasks.workunit.client.0.smithi060.stdout: create 165/35 file after the snapshot 2024-04-14T23:51:59.656 INFO:tasks.workunit.client.0.smithi060.stdout: create 165/36 file after the snapshot 2024-04-14T23:51:59.659 INFO:tasks.workunit.client.0.smithi060.stdout: create 165/37 file after the snapshot 2024-04-14T23:51:59.662 INFO:tasks.workunit.client.0.smithi060.stdout: create 165/38 file after the snapshot 2024-04-14T23:51:59.925 INFO:tasks.workunit.client.0.smithi060.stdout: create 165/39 file after the snapshot 2024-04-14T23:51:59.928 INFO:tasks.workunit.client.0.smithi060.stdout: create 166/.snap/snap-subdir-test 2024-04-14T23:51:59.971 INFO:tasks.workunit.client.0.smithi060.stdout: create 166/30 file after the snapshot 2024-04-14T23:51:59.973 INFO:tasks.workunit.client.0.smithi060.stdout: create 166/31 file after the snapshot 2024-04-14T23:51:59.976 INFO:tasks.workunit.client.0.smithi060.stdout: create 166/32 file after the snapshot 2024-04-14T23:51:59.979 INFO:tasks.workunit.client.0.smithi060.stdout: create 166/33 file after the snapshot 2024-04-14T23:51:59.981 INFO:tasks.workunit.client.0.smithi060.stdout: create 166/34 file after the snapshot 2024-04-14T23:51:59.984 INFO:tasks.workunit.client.0.smithi060.stdout: create 166/35 file after the snapshot 2024-04-14T23:51:59.986 INFO:tasks.workunit.client.0.smithi060.stdout: create 166/36 file after the snapshot 2024-04-14T23:51:59.989 INFO:tasks.workunit.client.0.smithi060.stdout: create 166/37 file after the snapshot 2024-04-14T23:51:59.992 INFO:tasks.workunit.client.0.smithi060.stdout: create 166/38 file after the snapshot 2024-04-14T23:51:59.995 INFO:tasks.workunit.client.0.smithi060.stdout: create 166/39 file after the snapshot 2024-04-14T23:51:59.997 INFO:tasks.workunit.client.0.smithi060.stdout: create 167/.snap/snap-subdir-test 2024-04-14T23:52:00.052 INFO:tasks.workunit.client.0.smithi060.stdout: create 167/30 file after the snapshot 2024-04-14T23:52:00.317 INFO:tasks.workunit.client.0.smithi060.stdout: create 167/31 file after the snapshot 2024-04-14T23:52:00.319 INFO:tasks.workunit.client.0.smithi060.stdout: create 167/32 file after the snapshot 2024-04-14T23:52:00.322 INFO:tasks.workunit.client.0.smithi060.stdout: create 167/33 file after the snapshot 2024-04-14T23:52:00.325 INFO:tasks.workunit.client.0.smithi060.stdout: create 167/34 file after the snapshot 2024-04-14T23:52:00.328 INFO:tasks.workunit.client.0.smithi060.stdout: create 167/35 file after the snapshot 2024-04-14T23:52:00.331 INFO:tasks.workunit.client.0.smithi060.stdout: create 167/36 file after the snapshot 2024-04-14T23:52:00.333 INFO:tasks.workunit.client.0.smithi060.stdout: create 167/37 file after the snapshot 2024-04-14T23:52:00.336 INFO:tasks.workunit.client.0.smithi060.stdout: create 167/38 file after the snapshot 2024-04-14T23:52:00.339 INFO:tasks.workunit.client.0.smithi060.stdout: create 167/39 file after the snapshot 2024-04-14T23:52:00.342 INFO:tasks.workunit.client.0.smithi060.stdout: create 168/.snap/snap-subdir-test 2024-04-14T23:52:00.389 INFO:tasks.workunit.client.0.smithi060.stdout: create 168/30 file after the snapshot 2024-04-14T23:52:00.392 INFO:tasks.workunit.client.0.smithi060.stdout: create 168/31 file after the snapshot 2024-04-14T23:52:00.394 INFO:tasks.workunit.client.0.smithi060.stdout: create 168/32 file after the snapshot 2024-04-14T23:52:00.397 INFO:tasks.workunit.client.0.smithi060.stdout: create 168/33 file after the snapshot 2024-04-14T23:52:00.400 INFO:tasks.workunit.client.0.smithi060.stdout: create 168/34 file after the snapshot 2024-04-14T23:52:00.403 INFO:tasks.workunit.client.0.smithi060.stdout: create 168/35 file after the snapshot 2024-04-14T23:52:00.405 INFO:tasks.workunit.client.0.smithi060.stdout: create 168/36 file after the snapshot 2024-04-14T23:52:00.408 INFO:tasks.workunit.client.0.smithi060.stdout: create 168/37 file after the snapshot 2024-04-14T23:52:00.411 INFO:tasks.workunit.client.0.smithi060.stdout: create 168/38 file after the snapshot 2024-04-14T23:52:00.414 INFO:tasks.workunit.client.0.smithi060.stdout: create 168/39 file after the snapshot 2024-04-14T23:52:00.418 INFO:tasks.workunit.client.0.smithi060.stdout: create 169/.snap/snap-subdir-test 2024-04-14T23:52:00.478 INFO:tasks.workunit.client.0.smithi060.stdout: create 169/30 file after the snapshot 2024-04-14T23:52:00.481 INFO:tasks.workunit.client.0.smithi060.stdout: create 169/31 file after the snapshot 2024-04-14T23:52:00.484 INFO:tasks.workunit.client.0.smithi060.stdout: create 169/32 file after the snapshot 2024-04-14T23:52:00.487 INFO:tasks.workunit.client.0.smithi060.stdout: create 169/33 file after the snapshot 2024-04-14T23:52:00.489 INFO:tasks.workunit.client.0.smithi060.stdout: create 169/34 file after the snapshot 2024-04-14T23:52:00.492 INFO:tasks.workunit.client.0.smithi060.stdout: create 169/35 file after the snapshot 2024-04-14T23:52:00.495 INFO:tasks.workunit.client.0.smithi060.stdout: create 169/36 file after the snapshot 2024-04-14T23:52:00.498 INFO:tasks.workunit.client.0.smithi060.stdout: create 169/37 file after the snapshot 2024-04-14T23:52:00.501 INFO:tasks.workunit.client.0.smithi060.stdout: create 169/38 file after the snapshot 2024-04-14T23:52:00.503 INFO:tasks.workunit.client.0.smithi060.stdout: create 169/39 file after the snapshot 2024-04-14T23:52:00.506 INFO:tasks.workunit.client.0.smithi060.stdout: create 170/.snap/snap-subdir-test 2024-04-14T23:52:00.554 INFO:tasks.workunit.client.0.smithi060.stdout: create 170/30 file after the snapshot 2024-04-14T23:52:00.555 INFO:tasks.workunit.client.0.smithi060.stdout: create 170/31 file after the snapshot 2024-04-14T23:52:00.557 INFO:tasks.workunit.client.0.smithi060.stdout: create 170/32 file after the snapshot 2024-04-14T23:52:00.559 INFO:tasks.workunit.client.0.smithi060.stdout: create 170/33 file after the snapshot 2024-04-14T23:52:00.560 INFO:tasks.workunit.client.0.smithi060.stdout: create 170/34 file after the snapshot 2024-04-14T23:52:00.562 INFO:tasks.workunit.client.0.smithi060.stdout: create 170/35 file after the snapshot 2024-04-14T23:52:00.565 INFO:tasks.workunit.client.0.smithi060.stdout: create 170/36 file after the snapshot 2024-04-14T23:52:00.567 INFO:tasks.workunit.client.0.smithi060.stdout: create 170/37 file after the snapshot 2024-04-14T23:52:00.569 INFO:tasks.workunit.client.0.smithi060.stdout: create 170/38 file after the snapshot 2024-04-14T23:52:00.571 INFO:tasks.workunit.client.0.smithi060.stdout: create 170/39 file after the snapshot 2024-04-14T23:52:00.573 INFO:tasks.workunit.client.0.smithi060.stdout: create 171/.snap/snap-subdir-test 2024-04-14T23:52:00.652 INFO:tasks.workunit.client.0.smithi060.stdout: create 171/30 file after the snapshot 2024-04-14T23:52:00.655 INFO:tasks.workunit.client.0.smithi060.stdout: create 171/31 file after the snapshot 2024-04-14T23:52:00.657 INFO:tasks.workunit.client.0.smithi060.stdout: create 171/32 file after the snapshot 2024-04-14T23:52:00.659 INFO:tasks.workunit.client.0.smithi060.stdout: create 171/33 file after the snapshot 2024-04-14T23:52:00.661 INFO:tasks.workunit.client.0.smithi060.stdout: create 171/34 file after the snapshot 2024-04-14T23:52:00.663 INFO:tasks.workunit.client.0.smithi060.stdout: create 171/35 file after the snapshot 2024-04-14T23:52:00.665 INFO:tasks.workunit.client.0.smithi060.stdout: create 171/36 file after the snapshot 2024-04-14T23:52:00.667 INFO:tasks.workunit.client.0.smithi060.stdout: create 171/37 file after the snapshot 2024-04-14T23:52:00.670 INFO:tasks.workunit.client.0.smithi060.stdout: create 171/38 file after the snapshot 2024-04-14T23:52:00.672 INFO:tasks.workunit.client.0.smithi060.stdout: create 171/39 file after the snapshot 2024-04-14T23:52:00.675 INFO:tasks.workunit.client.0.smithi060.stdout: create 172/.snap/snap-subdir-test 2024-04-14T23:52:00.729 INFO:tasks.workunit.client.0.smithi060.stdout: create 172/30 file after the snapshot 2024-04-14T23:52:00.731 INFO:tasks.workunit.client.0.smithi060.stdout: create 172/31 file after the snapshot 2024-04-14T23:52:00.734 INFO:tasks.workunit.client.0.smithi060.stdout: create 172/32 file after the snapshot 2024-04-14T23:52:00.737 INFO:tasks.workunit.client.0.smithi060.stdout: create 172/33 file after the snapshot 2024-04-14T23:52:00.740 INFO:tasks.workunit.client.0.smithi060.stdout: create 172/34 file after the snapshot 2024-04-14T23:52:00.743 INFO:tasks.workunit.client.0.smithi060.stdout: create 172/35 file after the snapshot 2024-04-14T23:52:00.745 INFO:tasks.workunit.client.0.smithi060.stdout: create 172/36 file after the snapshot 2024-04-14T23:52:00.748 INFO:tasks.workunit.client.0.smithi060.stdout: create 172/37 file after the snapshot 2024-04-14T23:52:00.751 INFO:tasks.workunit.client.0.smithi060.stdout: create 172/38 file after the snapshot 2024-04-14T23:52:00.753 INFO:tasks.workunit.client.0.smithi060.stdout: create 172/39 file after the snapshot 2024-04-14T23:52:00.756 INFO:tasks.workunit.client.0.smithi060.stdout: create 173/.snap/snap-subdir-test 2024-04-14T23:52:00.810 INFO:tasks.workunit.client.0.smithi060.stdout: create 173/30 file after the snapshot 2024-04-14T23:52:00.813 INFO:tasks.workunit.client.0.smithi060.stdout: create 173/31 file after the snapshot 2024-04-14T23:52:00.816 INFO:tasks.workunit.client.0.smithi060.stdout: create 173/32 file after the snapshot 2024-04-14T23:52:00.819 INFO:tasks.workunit.client.0.smithi060.stdout: create 173/33 file after the snapshot 2024-04-14T23:52:00.822 INFO:tasks.workunit.client.0.smithi060.stdout: create 173/34 file after the snapshot 2024-04-14T23:52:00.825 INFO:tasks.workunit.client.0.smithi060.stdout: create 173/35 file after the snapshot 2024-04-14T23:52:00.827 INFO:tasks.workunit.client.0.smithi060.stdout: create 173/36 file after the snapshot 2024-04-14T23:52:00.830 INFO:tasks.workunit.client.0.smithi060.stdout: create 173/37 file after the snapshot 2024-04-14T23:52:00.833 INFO:tasks.workunit.client.0.smithi060.stdout: create 173/38 file after the snapshot 2024-04-14T23:52:00.836 INFO:tasks.workunit.client.0.smithi060.stdout: create 173/39 file after the snapshot 2024-04-14T23:52:00.839 INFO:tasks.workunit.client.0.smithi060.stdout: create 174/.snap/snap-subdir-test 2024-04-14T23:52:00.881 INFO:tasks.workunit.client.0.smithi060.stdout: create 174/30 file after the snapshot 2024-04-14T23:52:00.883 INFO:tasks.workunit.client.0.smithi060.stdout: create 174/31 file after the snapshot 2024-04-14T23:52:00.884 INFO:tasks.workunit.client.0.smithi060.stdout: create 174/32 file after the snapshot 2024-04-14T23:52:00.886 INFO:tasks.workunit.client.0.smithi060.stdout: create 174/33 file after the snapshot 2024-04-14T23:52:00.888 INFO:tasks.workunit.client.0.smithi060.stdout: create 174/34 file after the snapshot 2024-04-14T23:52:00.890 INFO:tasks.workunit.client.0.smithi060.stdout: create 174/35 file after the snapshot 2024-04-14T23:52:00.891 INFO:tasks.workunit.client.0.smithi060.stdout: create 174/36 file after the snapshot 2024-04-14T23:52:00.893 INFO:tasks.workunit.client.0.smithi060.stdout: create 174/37 file after the snapshot 2024-04-14T23:52:00.895 INFO:tasks.workunit.client.0.smithi060.stdout: create 174/38 file after the snapshot 2024-04-14T23:52:00.897 INFO:tasks.workunit.client.0.smithi060.stdout: create 174/39 file after the snapshot 2024-04-14T23:52:00.898 INFO:tasks.workunit.client.0.smithi060.stdout: create 175/.snap/snap-subdir-test 2024-04-14T23:52:00.952 INFO:tasks.workunit.client.0.smithi060.stdout: create 175/30 file after the snapshot 2024-04-14T23:52:00.955 INFO:tasks.workunit.client.0.smithi060.stdout: create 175/31 file after the snapshot 2024-04-14T23:52:00.958 INFO:tasks.workunit.client.0.smithi060.stdout: create 175/32 file after the snapshot 2024-04-14T23:52:00.961 INFO:tasks.workunit.client.0.smithi060.stdout: create 175/33 file after the snapshot 2024-04-14T23:52:00.964 INFO:tasks.workunit.client.0.smithi060.stdout: create 175/34 file after the snapshot 2024-04-14T23:52:00.967 INFO:tasks.workunit.client.0.smithi060.stdout: create 175/35 file after the snapshot 2024-04-14T23:52:00.970 INFO:tasks.workunit.client.0.smithi060.stdout: create 175/36 file after the snapshot 2024-04-14T23:52:00.973 INFO:tasks.workunit.client.0.smithi060.stdout: create 175/37 file after the snapshot 2024-04-14T23:52:00.976 INFO:tasks.workunit.client.0.smithi060.stdout: create 175/38 file after the snapshot 2024-04-14T23:52:00.979 INFO:tasks.workunit.client.0.smithi060.stdout: create 175/39 file after the snapshot 2024-04-14T23:52:00.982 INFO:tasks.workunit.client.0.smithi060.stdout: create 176/.snap/snap-subdir-test 2024-04-14T23:52:01.026 INFO:tasks.workunit.client.0.smithi060.stdout: create 176/30 file after the snapshot 2024-04-14T23:52:01.028 INFO:tasks.workunit.client.0.smithi060.stdout: create 176/31 file after the snapshot 2024-04-14T23:52:01.030 INFO:tasks.workunit.client.0.smithi060.stdout: create 176/32 file after the snapshot 2024-04-14T23:52:01.031 INFO:tasks.workunit.client.0.smithi060.stdout: create 176/33 file after the snapshot 2024-04-14T23:52:01.033 INFO:tasks.workunit.client.0.smithi060.stdout: create 176/34 file after the snapshot 2024-04-14T23:52:01.034 INFO:tasks.workunit.client.0.smithi060.stdout: create 176/35 file after the snapshot 2024-04-14T23:52:01.036 INFO:tasks.workunit.client.0.smithi060.stdout: create 176/36 file after the snapshot 2024-04-14T23:52:01.038 INFO:tasks.workunit.client.0.smithi060.stdout: create 176/37 file after the snapshot 2024-04-14T23:52:01.039 INFO:tasks.workunit.client.0.smithi060.stdout: create 176/38 file after the snapshot 2024-04-14T23:52:01.041 INFO:tasks.workunit.client.0.smithi060.stdout: create 176/39 file after the snapshot 2024-04-14T23:52:01.044 INFO:tasks.workunit.client.0.smithi060.stdout: create 177/.snap/snap-subdir-test 2024-04-14T23:52:01.097 INFO:tasks.workunit.client.0.smithi060.stdout: create 177/30 file after the snapshot 2024-04-14T23:52:01.099 INFO:tasks.workunit.client.0.smithi060.stdout: create 177/31 file after the snapshot 2024-04-14T23:52:01.102 INFO:tasks.workunit.client.0.smithi060.stdout: create 177/32 file after the snapshot 2024-04-14T23:52:01.105 INFO:tasks.workunit.client.0.smithi060.stdout: create 177/33 file after the snapshot 2024-04-14T23:52:01.108 INFO:tasks.workunit.client.0.smithi060.stdout: create 177/34 file after the snapshot 2024-04-14T23:52:01.111 INFO:tasks.workunit.client.0.smithi060.stdout: create 177/35 file after the snapshot 2024-04-14T23:52:01.113 INFO:tasks.workunit.client.0.smithi060.stdout: create 177/36 file after the snapshot 2024-04-14T23:52:01.116 INFO:tasks.workunit.client.0.smithi060.stdout: create 177/37 file after the snapshot 2024-04-14T23:52:01.119 INFO:tasks.workunit.client.0.smithi060.stdout: create 177/38 file after the snapshot 2024-04-14T23:52:01.122 INFO:tasks.workunit.client.0.smithi060.stdout: create 177/39 file after the snapshot 2024-04-14T23:52:01.125 INFO:tasks.workunit.client.0.smithi060.stdout: create 178/.snap/snap-subdir-test 2024-04-14T23:52:01.179 INFO:tasks.workunit.client.0.smithi060.stdout: create 178/30 file after the snapshot 2024-04-14T23:52:01.181 INFO:tasks.workunit.client.0.smithi060.stdout: create 178/31 file after the snapshot 2024-04-14T23:52:01.183 INFO:tasks.workunit.client.0.smithi060.stdout: create 178/32 file after the snapshot 2024-04-14T23:52:01.185 INFO:tasks.workunit.client.0.smithi060.stdout: create 178/33 file after the snapshot 2024-04-14T23:52:01.187 INFO:tasks.workunit.client.0.smithi060.stdout: create 178/34 file after the snapshot 2024-04-14T23:52:01.189 INFO:tasks.workunit.client.0.smithi060.stdout: create 178/35 file after the snapshot 2024-04-14T23:52:01.191 INFO:tasks.workunit.client.0.smithi060.stdout: create 178/36 file after the snapshot 2024-04-14T23:52:01.192 INFO:tasks.workunit.client.0.smithi060.stdout: create 178/37 file after the snapshot 2024-04-14T23:52:01.194 INFO:tasks.workunit.client.0.smithi060.stdout: create 178/38 file after the snapshot 2024-04-14T23:52:01.196 INFO:tasks.workunit.client.0.smithi060.stdout: create 178/39 file after the snapshot 2024-04-14T23:52:01.199 INFO:tasks.workunit.client.0.smithi060.stdout: create 179/.snap/snap-subdir-test 2024-04-14T23:52:01.252 INFO:tasks.workunit.client.0.smithi060.stdout: create 179/30 file after the snapshot 2024-04-14T23:52:01.254 INFO:tasks.workunit.client.0.smithi060.stdout: create 179/31 file after the snapshot 2024-04-14T23:52:01.256 INFO:tasks.workunit.client.0.smithi060.stdout: create 179/32 file after the snapshot 2024-04-14T23:52:01.257 INFO:tasks.workunit.client.0.smithi060.stdout: create 179/33 file after the snapshot 2024-04-14T23:52:01.259 INFO:tasks.workunit.client.0.smithi060.stdout: create 179/34 file after the snapshot 2024-04-14T23:52:01.261 INFO:tasks.workunit.client.0.smithi060.stdout: create 179/35 file after the snapshot 2024-04-14T23:52:01.262 INFO:tasks.workunit.client.0.smithi060.stdout: create 179/36 file after the snapshot 2024-04-14T23:52:01.264 INFO:tasks.workunit.client.0.smithi060.stdout: create 179/37 file after the snapshot 2024-04-14T23:52:01.266 INFO:tasks.workunit.client.0.smithi060.stdout: create 179/38 file after the snapshot 2024-04-14T23:52:01.268 INFO:tasks.workunit.client.0.smithi060.stdout: create 179/39 file after the snapshot 2024-04-14T23:52:01.271 INFO:tasks.workunit.client.0.smithi060.stdout: create 180/.snap/snap-subdir-test 2024-04-14T23:52:01.324 INFO:tasks.workunit.client.0.smithi060.stdout: create 180/30 file after the snapshot 2024-04-14T23:52:01.325 INFO:tasks.workunit.client.0.smithi060.stdout: create 180/31 file after the snapshot 2024-04-14T23:52:01.328 INFO:tasks.workunit.client.0.smithi060.stdout: create 180/32 file after the snapshot 2024-04-14T23:52:01.330 INFO:tasks.workunit.client.0.smithi060.stdout: create 180/33 file after the snapshot 2024-04-14T23:52:01.331 INFO:tasks.workunit.client.0.smithi060.stdout: create 180/34 file after the snapshot 2024-04-14T23:52:01.333 INFO:tasks.workunit.client.0.smithi060.stdout: create 180/35 file after the snapshot 2024-04-14T23:52:01.335 INFO:tasks.workunit.client.0.smithi060.stdout: create 180/36 file after the snapshot 2024-04-14T23:52:01.339 INFO:tasks.workunit.client.0.smithi060.stdout: create 180/37 file after the snapshot 2024-04-14T23:52:01.342 INFO:tasks.workunit.client.0.smithi060.stdout: create 180/38 file after the snapshot 2024-04-14T23:52:01.345 INFO:tasks.workunit.client.0.smithi060.stdout: create 180/39 file after the snapshot 2024-04-14T23:52:01.348 INFO:tasks.workunit.client.0.smithi060.stdout: create 181/.snap/snap-subdir-test 2024-04-14T23:52:01.402 INFO:tasks.workunit.client.0.smithi060.stdout: create 181/30 file after the snapshot 2024-04-14T23:52:01.404 INFO:tasks.workunit.client.0.smithi060.stdout: create 181/31 file after the snapshot 2024-04-14T23:52:01.407 INFO:tasks.workunit.client.0.smithi060.stdout: create 181/32 file after the snapshot 2024-04-14T23:52:01.410 INFO:tasks.workunit.client.0.smithi060.stdout: create 181/33 file after the snapshot 2024-04-14T23:52:01.413 INFO:tasks.workunit.client.0.smithi060.stdout: create 181/34 file after the snapshot 2024-04-14T23:52:01.416 INFO:tasks.workunit.client.0.smithi060.stdout: create 181/35 file after the snapshot 2024-04-14T23:52:01.419 INFO:tasks.workunit.client.0.smithi060.stdout: create 181/36 file after the snapshot 2024-04-14T23:52:01.422 INFO:tasks.workunit.client.0.smithi060.stdout: create 181/37 file after the snapshot 2024-04-14T23:52:01.424 INFO:tasks.workunit.client.0.smithi060.stdout: create 181/38 file after the snapshot 2024-04-14T23:52:01.428 INFO:tasks.workunit.client.0.smithi060.stdout: create 181/39 file after the snapshot 2024-04-14T23:52:01.430 INFO:tasks.workunit.client.0.smithi060.stdout: create 182/.snap/snap-subdir-test 2024-04-14T23:52:01.480 INFO:tasks.workunit.client.0.smithi060.stdout: create 182/30 file after the snapshot 2024-04-14T23:52:01.482 INFO:tasks.workunit.client.0.smithi060.stdout: create 182/31 file after the snapshot 2024-04-14T23:52:01.485 INFO:tasks.workunit.client.0.smithi060.stdout: create 182/32 file after the snapshot 2024-04-14T23:52:01.486 INFO:tasks.workunit.client.0.smithi060.stdout: create 182/33 file after the snapshot 2024-04-14T23:52:01.488 INFO:tasks.workunit.client.0.smithi060.stdout: create 182/34 file after the snapshot 2024-04-14T23:52:01.490 INFO:tasks.workunit.client.0.smithi060.stdout: create 182/35 file after the snapshot 2024-04-14T23:52:01.492 INFO:tasks.workunit.client.0.smithi060.stdout: create 182/36 file after the snapshot 2024-04-14T23:52:01.494 INFO:tasks.workunit.client.0.smithi060.stdout: create 182/37 file after the snapshot 2024-04-14T23:52:01.497 INFO:tasks.workunit.client.0.smithi060.stdout: create 182/38 file after the snapshot 2024-04-14T23:52:01.499 INFO:tasks.workunit.client.0.smithi060.stdout: create 182/39 file after the snapshot 2024-04-14T23:52:01.501 INFO:tasks.workunit.client.0.smithi060.stdout: create 183/.snap/snap-subdir-test 2024-04-14T23:52:01.559 INFO:tasks.workunit.client.0.smithi060.stdout: create 183/30 file after the snapshot 2024-04-14T23:52:01.562 INFO:tasks.workunit.client.0.smithi060.stdout: create 183/31 file after the snapshot 2024-04-14T23:52:01.565 INFO:tasks.workunit.client.0.smithi060.stdout: create 183/32 file after the snapshot 2024-04-14T23:52:01.568 INFO:tasks.workunit.client.0.smithi060.stdout: create 183/33 file after the snapshot 2024-04-14T23:52:01.571 INFO:tasks.workunit.client.0.smithi060.stdout: create 183/34 file after the snapshot 2024-04-14T23:52:01.574 INFO:tasks.workunit.client.0.smithi060.stdout: create 183/35 file after the snapshot 2024-04-14T23:52:01.576 INFO:tasks.workunit.client.0.smithi060.stdout: create 183/36 file after the snapshot 2024-04-14T23:52:01.580 INFO:tasks.workunit.client.0.smithi060.stdout: create 183/37 file after the snapshot 2024-04-14T23:52:01.583 INFO:tasks.workunit.client.0.smithi060.stdout: create 183/38 file after the snapshot 2024-04-14T23:52:01.586 INFO:tasks.workunit.client.0.smithi060.stdout: create 183/39 file after the snapshot 2024-04-14T23:52:01.589 INFO:tasks.workunit.client.0.smithi060.stdout: create 184/.snap/snap-subdir-test 2024-04-14T23:52:01.642 INFO:tasks.workunit.client.0.smithi060.stdout: create 184/30 file after the snapshot 2024-04-14T23:52:01.645 INFO:tasks.workunit.client.0.smithi060.stdout: create 184/31 file after the snapshot 2024-04-14T23:52:01.648 INFO:tasks.workunit.client.0.smithi060.stdout: create 184/32 file after the snapshot 2024-04-14T23:52:01.651 INFO:tasks.workunit.client.0.smithi060.stdout: create 184/33 file after the snapshot 2024-04-14T23:52:01.654 INFO:tasks.workunit.client.0.smithi060.stdout: create 184/34 file after the snapshot 2024-04-14T23:52:01.656 INFO:tasks.workunit.client.0.smithi060.stdout: create 184/35 file after the snapshot 2024-04-14T23:52:01.659 INFO:tasks.workunit.client.0.smithi060.stdout: create 184/36 file after the snapshot 2024-04-14T23:52:01.662 INFO:tasks.workunit.client.0.smithi060.stdout: create 184/37 file after the snapshot 2024-04-14T23:52:01.666 INFO:tasks.workunit.client.0.smithi060.stdout: create 184/38 file after the snapshot 2024-04-14T23:52:01.669 INFO:tasks.workunit.client.0.smithi060.stdout: create 184/39 file after the snapshot 2024-04-14T23:52:01.672 INFO:tasks.workunit.client.0.smithi060.stdout: create 185/.snap/snap-subdir-test 2024-04-14T23:52:01.727 INFO:tasks.workunit.client.0.smithi060.stdout: create 185/30 file after the snapshot 2024-04-14T23:52:01.730 INFO:tasks.workunit.client.0.smithi060.stdout: create 185/31 file after the snapshot 2024-04-14T23:52:01.732 INFO:tasks.workunit.client.0.smithi060.stdout: create 185/32 file after the snapshot 2024-04-14T23:52:01.735 INFO:tasks.workunit.client.0.smithi060.stdout: create 185/33 file after the snapshot 2024-04-14T23:52:01.737 INFO:tasks.workunit.client.0.smithi060.stdout: create 185/34 file after the snapshot 2024-04-14T23:52:01.740 INFO:tasks.workunit.client.0.smithi060.stdout: create 185/35 file after the snapshot 2024-04-14T23:52:01.743 INFO:tasks.workunit.client.0.smithi060.stdout: create 185/36 file after the snapshot 2024-04-14T23:52:01.746 INFO:tasks.workunit.client.0.smithi060.stdout: create 185/37 file after the snapshot 2024-04-14T23:52:01.748 INFO:tasks.workunit.client.0.smithi060.stdout: create 185/38 file after the snapshot 2024-04-14T23:52:01.751 INFO:tasks.workunit.client.0.smithi060.stdout: create 185/39 file after the snapshot 2024-04-14T23:52:01.754 INFO:tasks.workunit.client.0.smithi060.stdout: create 186/.snap/snap-subdir-test 2024-04-14T23:52:01.799 INFO:tasks.workunit.client.0.smithi060.stdout: create 186/30 file after the snapshot 2024-04-14T23:52:01.802 INFO:tasks.workunit.client.0.smithi060.stdout: create 186/31 file after the snapshot 2024-04-14T23:52:01.804 INFO:tasks.workunit.client.0.smithi060.stdout: create 186/32 file after the snapshot 2024-04-14T23:52:01.807 INFO:tasks.workunit.client.0.smithi060.stdout: create 186/33 file after the snapshot 2024-04-14T23:52:01.811 INFO:tasks.workunit.client.0.smithi060.stdout: create 186/34 file after the snapshot 2024-04-14T23:52:01.815 INFO:tasks.workunit.client.0.smithi060.stdout: create 186/35 file after the snapshot 2024-04-14T23:52:01.819 INFO:tasks.workunit.client.0.smithi060.stdout: create 186/36 file after the snapshot 2024-04-14T23:52:01.823 INFO:tasks.workunit.client.0.smithi060.stdout: create 186/37 file after the snapshot 2024-04-14T23:52:01.827 INFO:tasks.workunit.client.0.smithi060.stdout: create 186/38 file after the snapshot 2024-04-14T23:52:01.832 INFO:tasks.workunit.client.0.smithi060.stdout: create 186/39 file after the snapshot 2024-04-14T23:52:01.836 INFO:tasks.workunit.client.0.smithi060.stdout: create 187/.snap/snap-subdir-test 2024-04-14T23:52:01.889 INFO:tasks.workunit.client.0.smithi060.stdout: create 187/30 file after the snapshot 2024-04-14T23:52:01.893 INFO:tasks.workunit.client.0.smithi060.stdout: create 187/31 file after the snapshot 2024-04-14T23:52:01.898 INFO:tasks.workunit.client.0.smithi060.stdout: create 187/32 file after the snapshot 2024-04-14T23:52:02.793 INFO:tasks.workunit.client.0.smithi060.stdout: create 187/33 file after the snapshot 2024-04-14T23:52:02.797 INFO:tasks.workunit.client.0.smithi060.stdout: create 187/34 file after the snapshot 2024-04-14T23:52:02.801 INFO:tasks.workunit.client.0.smithi060.stdout: create 187/35 file after the snapshot 2024-04-14T23:52:02.805 INFO:tasks.workunit.client.0.smithi060.stdout: create 187/36 file after the snapshot 2024-04-14T23:52:02.809 INFO:tasks.workunit.client.0.smithi060.stdout: create 187/37 file after the snapshot 2024-04-14T23:52:02.814 INFO:tasks.workunit.client.0.smithi060.stdout: create 187/38 file after the snapshot 2024-04-14T23:52:02.818 INFO:tasks.workunit.client.0.smithi060.stdout: create 187/39 file after the snapshot 2024-04-14T23:52:02.822 INFO:tasks.workunit.client.0.smithi060.stdout: create 188/.snap/snap-subdir-test 2024-04-14T23:52:02.894 INFO:tasks.workunit.client.0.smithi060.stdout: create 188/30 file after the snapshot 2024-04-14T23:52:02.898 INFO:tasks.workunit.client.0.smithi060.stdout: create 188/31 file after the snapshot 2024-04-14T23:52:02.903 INFO:tasks.workunit.client.0.smithi060.stdout: create 188/32 file after the snapshot 2024-04-14T23:52:02.907 INFO:tasks.workunit.client.0.smithi060.stdout: create 188/33 file after the snapshot 2024-04-14T23:52:02.912 INFO:tasks.workunit.client.0.smithi060.stdout: create 188/34 file after the snapshot 2024-04-14T23:52:02.916 INFO:tasks.workunit.client.0.smithi060.stdout: create 188/35 file after the snapshot 2024-04-14T23:52:02.920 INFO:tasks.workunit.client.0.smithi060.stdout: create 188/36 file after the snapshot 2024-04-14T23:52:02.924 INFO:tasks.workunit.client.0.smithi060.stdout: create 188/37 file after the snapshot 2024-04-14T23:52:02.929 INFO:tasks.workunit.client.0.smithi060.stdout: create 188/38 file after the snapshot 2024-04-14T23:52:02.933 INFO:tasks.workunit.client.0.smithi060.stdout: create 188/39 file after the snapshot 2024-04-14T23:52:02.938 INFO:tasks.workunit.client.0.smithi060.stdout: create 189/.snap/snap-subdir-test 2024-04-14T23:52:03.005 INFO:tasks.workunit.client.0.smithi060.stdout: create 189/30 file after the snapshot 2024-04-14T23:52:03.009 INFO:tasks.workunit.client.0.smithi060.stdout: create 189/31 file after the snapshot 2024-04-14T23:52:03.014 INFO:tasks.workunit.client.0.smithi060.stdout: create 189/32 file after the snapshot 2024-04-14T23:52:03.018 INFO:tasks.workunit.client.0.smithi060.stdout: create 189/33 file after the snapshot 2024-04-14T23:52:03.022 INFO:tasks.workunit.client.0.smithi060.stdout: create 189/34 file after the snapshot 2024-04-14T23:52:03.026 INFO:tasks.workunit.client.0.smithi060.stdout: create 189/35 file after the snapshot 2024-04-14T23:52:03.031 INFO:tasks.workunit.client.0.smithi060.stdout: create 189/36 file after the snapshot 2024-04-14T23:52:03.035 INFO:tasks.workunit.client.0.smithi060.stdout: create 189/37 file after the snapshot 2024-04-14T23:52:03.039 INFO:tasks.workunit.client.0.smithi060.stdout: create 189/38 file after the snapshot 2024-04-14T23:52:03.043 INFO:tasks.workunit.client.0.smithi060.stdout: create 189/39 file after the snapshot 2024-04-14T23:52:03.048 INFO:tasks.workunit.client.0.smithi060.stdout: create 190/.snap/snap-subdir-test 2024-04-14T23:52:03.113 INFO:tasks.workunit.client.0.smithi060.stdout: create 190/30 file after the snapshot 2024-04-14T23:52:03.117 INFO:tasks.workunit.client.0.smithi060.stdout: create 190/31 file after the snapshot 2024-04-14T23:52:03.121 INFO:tasks.workunit.client.0.smithi060.stdout: create 190/32 file after the snapshot 2024-04-14T23:52:03.126 INFO:tasks.workunit.client.0.smithi060.stdout: create 190/33 file after the snapshot 2024-04-14T23:52:03.130 INFO:tasks.workunit.client.0.smithi060.stdout: create 190/34 file after the snapshot 2024-04-14T23:52:03.134 INFO:tasks.workunit.client.0.smithi060.stdout: create 190/35 file after the snapshot 2024-04-14T23:52:03.138 INFO:tasks.workunit.client.0.smithi060.stdout: create 190/36 file after the snapshot 2024-04-14T23:52:03.142 INFO:tasks.workunit.client.0.smithi060.stdout: create 190/37 file after the snapshot 2024-04-14T23:52:03.147 INFO:tasks.workunit.client.0.smithi060.stdout: create 190/38 file after the snapshot 2024-04-14T23:52:03.151 INFO:tasks.workunit.client.0.smithi060.stdout: create 190/39 file after the snapshot 2024-04-14T23:52:03.155 INFO:tasks.workunit.client.0.smithi060.stdout: create 191/.snap/snap-subdir-test 2024-04-14T23:52:03.220 INFO:tasks.workunit.client.0.smithi060.stdout: create 191/30 file after the snapshot 2024-04-14T23:52:03.224 INFO:tasks.workunit.client.0.smithi060.stdout: create 191/31 file after the snapshot 2024-04-14T23:52:03.229 INFO:tasks.workunit.client.0.smithi060.stdout: create 191/32 file after the snapshot 2024-04-14T23:52:03.233 INFO:tasks.workunit.client.0.smithi060.stdout: create 191/33 file after the snapshot 2024-04-14T23:52:03.236 INFO:tasks.workunit.client.0.smithi060.stdout: create 191/34 file after the snapshot 2024-04-14T23:52:03.241 INFO:tasks.workunit.client.0.smithi060.stdout: create 191/35 file after the snapshot 2024-04-14T23:52:03.245 INFO:tasks.workunit.client.0.smithi060.stdout: create 191/36 file after the snapshot 2024-04-14T23:52:03.249 INFO:tasks.workunit.client.0.smithi060.stdout: create 191/37 file after the snapshot 2024-04-14T23:52:03.252 INFO:tasks.workunit.client.0.smithi060.stdout: create 191/38 file after the snapshot 2024-04-14T23:52:03.255 INFO:tasks.workunit.client.0.smithi060.stdout: create 191/39 file after the snapshot 2024-04-14T23:52:03.258 INFO:tasks.workunit.client.0.smithi060.stdout: create 192/.snap/snap-subdir-test 2024-04-14T23:52:03.662 INFO:tasks.workunit.client.0.smithi060.stdout: create 192/30 file after the snapshot 2024-04-14T23:52:03.665 INFO:tasks.workunit.client.0.smithi060.stdout: create 192/31 file after the snapshot 2024-04-14T23:52:03.667 INFO:tasks.workunit.client.0.smithi060.stdout: create 192/32 file after the snapshot 2024-04-14T23:52:03.670 INFO:tasks.workunit.client.0.smithi060.stdout: create 192/33 file after the snapshot 2024-04-14T23:52:03.673 INFO:tasks.workunit.client.0.smithi060.stdout: create 192/34 file after the snapshot 2024-04-14T23:52:03.676 INFO:tasks.workunit.client.0.smithi060.stdout: create 192/35 file after the snapshot 2024-04-14T23:52:03.678 INFO:tasks.workunit.client.0.smithi060.stdout: create 192/36 file after the snapshot 2024-04-14T23:52:03.681 INFO:tasks.workunit.client.0.smithi060.stdout: create 192/37 file after the snapshot 2024-04-14T23:52:03.684 INFO:tasks.workunit.client.0.smithi060.stdout: create 192/38 file after the snapshot 2024-04-14T23:52:03.686 INFO:tasks.workunit.client.0.smithi060.stdout: create 192/39 file after the snapshot 2024-04-14T23:52:03.689 INFO:tasks.workunit.client.0.smithi060.stdout: create 193/.snap/snap-subdir-test 2024-04-14T23:52:03.752 INFO:tasks.workunit.client.0.smithi060.stdout: create 193/30 file after the snapshot 2024-04-14T23:52:03.754 INFO:tasks.workunit.client.0.smithi060.stdout: create 193/31 file after the snapshot 2024-04-14T23:52:03.757 INFO:tasks.workunit.client.0.smithi060.stdout: create 193/32 file after the snapshot 2024-04-14T23:52:03.759 INFO:tasks.workunit.client.0.smithi060.stdout: create 193/33 file after the snapshot 2024-04-14T23:52:03.761 INFO:tasks.workunit.client.0.smithi060.stdout: create 193/34 file after the snapshot 2024-04-14T23:52:03.764 INFO:tasks.workunit.client.0.smithi060.stdout: create 193/35 file after the snapshot 2024-04-14T23:52:03.766 INFO:tasks.workunit.client.0.smithi060.stdout: create 193/36 file after the snapshot 2024-04-14T23:52:03.767 INFO:tasks.workunit.client.0.smithi060.stdout: create 193/37 file after the snapshot 2024-04-14T23:52:03.770 INFO:tasks.workunit.client.0.smithi060.stdout: create 193/38 file after the snapshot 2024-04-14T23:52:03.772 INFO:tasks.workunit.client.0.smithi060.stdout: create 193/39 file after the snapshot 2024-04-14T23:52:03.774 INFO:tasks.workunit.client.0.smithi060.stdout: create 194/.snap/snap-subdir-test 2024-04-14T23:52:03.834 INFO:tasks.workunit.client.0.smithi060.stdout: create 194/30 file after the snapshot 2024-04-14T23:52:03.835 INFO:tasks.workunit.client.0.smithi060.stdout: create 194/31 file after the snapshot 2024-04-14T23:52:03.838 INFO:tasks.workunit.client.0.smithi060.stdout: create 194/32 file after the snapshot 2024-04-14T23:52:03.839 INFO:tasks.workunit.client.0.smithi060.stdout: create 194/33 file after the snapshot 2024-04-14T23:52:03.842 INFO:tasks.workunit.client.0.smithi060.stdout: create 194/34 file after the snapshot 2024-04-14T23:52:03.844 INFO:tasks.workunit.client.0.smithi060.stdout: create 194/35 file after the snapshot 2024-04-14T23:52:03.848 INFO:tasks.workunit.client.0.smithi060.stdout: create 194/36 file after the snapshot 2024-04-14T23:52:03.852 INFO:tasks.workunit.client.0.smithi060.stdout: create 194/37 file after the snapshot 2024-04-14T23:52:03.856 INFO:tasks.workunit.client.0.smithi060.stdout: create 194/38 file after the snapshot 2024-04-14T23:52:03.861 INFO:tasks.workunit.client.0.smithi060.stdout: create 194/39 file after the snapshot 2024-04-14T23:52:03.865 INFO:tasks.workunit.client.0.smithi060.stdout: create 195/.snap/snap-subdir-test 2024-04-14T23:52:03.933 INFO:tasks.workunit.client.0.smithi060.stdout: create 195/30 file after the snapshot 2024-04-14T23:52:03.937 INFO:tasks.workunit.client.0.smithi060.stdout: create 195/31 file after the snapshot 2024-04-14T23:52:03.942 INFO:tasks.workunit.client.0.smithi060.stdout: create 195/32 file after the snapshot 2024-04-14T23:52:03.946 INFO:tasks.workunit.client.0.smithi060.stdout: create 195/33 file after the snapshot 2024-04-14T23:52:03.950 INFO:tasks.workunit.client.0.smithi060.stdout: create 195/34 file after the snapshot 2024-04-14T23:52:03.954 INFO:tasks.workunit.client.0.smithi060.stdout: create 195/35 file after the snapshot 2024-04-14T23:52:03.958 INFO:tasks.workunit.client.0.smithi060.stdout: create 195/36 file after the snapshot 2024-04-14T23:52:03.963 INFO:tasks.workunit.client.0.smithi060.stdout: create 195/37 file after the snapshot 2024-04-14T23:52:03.967 INFO:tasks.workunit.client.0.smithi060.stdout: create 195/38 file after the snapshot 2024-04-14T23:52:03.970 INFO:tasks.workunit.client.0.smithi060.stdout: create 195/39 file after the snapshot 2024-04-14T23:52:03.973 INFO:tasks.workunit.client.0.smithi060.stdout: create 196/.snap/snap-subdir-test 2024-04-14T23:52:04.043 INFO:tasks.workunit.client.0.smithi060.stdout: create 196/30 file after the snapshot 2024-04-14T23:52:04.046 INFO:tasks.workunit.client.0.smithi060.stdout: create 196/31 file after the snapshot 2024-04-14T23:52:04.049 INFO:tasks.workunit.client.0.smithi060.stdout: create 196/32 file after the snapshot 2024-04-14T23:52:04.051 INFO:tasks.workunit.client.0.smithi060.stdout: create 196/33 file after the snapshot 2024-04-14T23:52:04.054 INFO:tasks.workunit.client.0.smithi060.stdout: create 196/34 file after the snapshot 2024-04-14T23:52:04.057 INFO:tasks.workunit.client.0.smithi060.stdout: create 196/35 file after the snapshot 2024-04-14T23:52:04.060 INFO:tasks.workunit.client.0.smithi060.stdout: create 196/36 file after the snapshot 2024-04-14T23:52:04.063 INFO:tasks.workunit.client.0.smithi060.stdout: create 196/37 file after the snapshot 2024-04-14T23:52:04.066 INFO:tasks.workunit.client.0.smithi060.stdout: create 196/38 file after the snapshot 2024-04-14T23:52:04.068 INFO:tasks.workunit.client.0.smithi060.stdout: create 196/39 file after the snapshot 2024-04-14T23:52:04.071 INFO:tasks.workunit.client.0.smithi060.stdout: create 197/.snap/snap-subdir-test 2024-04-14T23:52:04.142 INFO:tasks.workunit.client.0.smithi060.stdout: create 197/30 file after the snapshot 2024-04-14T23:52:04.144 INFO:tasks.workunit.client.0.smithi060.stdout: create 197/31 file after the snapshot 2024-04-14T23:52:04.147 INFO:tasks.workunit.client.0.smithi060.stdout: create 197/32 file after the snapshot 2024-04-14T23:52:04.150 INFO:tasks.workunit.client.0.smithi060.stdout: create 197/33 file after the snapshot 2024-04-14T23:52:04.153 INFO:tasks.workunit.client.0.smithi060.stdout: create 197/34 file after the snapshot 2024-04-14T23:52:04.155 INFO:tasks.workunit.client.0.smithi060.stdout: create 197/35 file after the snapshot 2024-04-14T23:52:04.158 INFO:tasks.workunit.client.0.smithi060.stdout: create 197/36 file after the snapshot 2024-04-14T23:52:04.161 INFO:tasks.workunit.client.0.smithi060.stdout: create 197/37 file after the snapshot 2024-04-14T23:52:04.164 INFO:tasks.workunit.client.0.smithi060.stdout: create 197/38 file after the snapshot 2024-04-14T23:52:04.167 INFO:tasks.workunit.client.0.smithi060.stdout: create 197/39 file after the snapshot 2024-04-14T23:52:04.170 INFO:tasks.workunit.client.0.smithi060.stdout: create 198/.snap/snap-subdir-test 2024-04-14T23:52:04.228 INFO:tasks.workunit.client.0.smithi060.stdout: create 198/30 file after the snapshot 2024-04-14T23:52:04.230 INFO:tasks.workunit.client.0.smithi060.stdout: create 198/31 file after the snapshot 2024-04-14T23:52:04.232 INFO:tasks.workunit.client.0.smithi060.stdout: create 198/32 file after the snapshot 2024-04-14T23:52:04.233 INFO:tasks.workunit.client.0.smithi060.stdout: create 198/33 file after the snapshot 2024-04-14T23:52:04.236 INFO:tasks.workunit.client.0.smithi060.stdout: create 198/34 file after the snapshot 2024-04-14T23:52:04.239 INFO:tasks.workunit.client.0.smithi060.stdout: create 198/35 file after the snapshot 2024-04-14T23:52:04.243 INFO:tasks.workunit.client.0.smithi060.stdout: create 198/36 file after the snapshot 2024-04-14T23:52:04.246 INFO:tasks.workunit.client.0.smithi060.stdout: create 198/37 file after the snapshot 2024-04-14T23:52:04.248 INFO:tasks.workunit.client.0.smithi060.stdout: create 198/38 file after the snapshot 2024-04-14T23:52:04.251 INFO:tasks.workunit.client.0.smithi060.stdout: create 198/39 file after the snapshot 2024-04-14T23:52:04.254 INFO:tasks.workunit.client.0.smithi060.stdout: create 199/.snap/snap-subdir-test 2024-04-14T23:52:04.309 INFO:tasks.workunit.client.0.smithi060.stdout: create 199/30 file after the snapshot 2024-04-14T23:52:04.313 INFO:tasks.workunit.client.0.smithi060.stdout: create 199/31 file after the snapshot 2024-04-14T23:52:04.317 INFO:tasks.workunit.client.0.smithi060.stdout: create 199/32 file after the snapshot 2024-04-14T23:52:04.321 INFO:tasks.workunit.client.0.smithi060.stdout: create 199/33 file after the snapshot 2024-04-14T23:52:04.326 INFO:tasks.workunit.client.0.smithi060.stdout: create 199/34 file after the snapshot 2024-04-14T23:52:04.330 INFO:tasks.workunit.client.0.smithi060.stdout: create 199/35 file after the snapshot 2024-04-14T23:52:04.334 INFO:tasks.workunit.client.0.smithi060.stdout: create 199/36 file after the snapshot 2024-04-14T23:52:04.338 INFO:tasks.workunit.client.0.smithi060.stdout: create 199/37 file after the snapshot 2024-04-14T23:52:04.343 INFO:tasks.workunit.client.0.smithi060.stdout: create 199/38 file after the snapshot 2024-04-14T23:52:04.347 INFO:tasks.workunit.client.0.smithi060.stdout: create 199/39 file after the snapshot 2024-04-14T23:52:04.351 INFO:tasks.workunit.client.0.smithi060.stdout: create 200/.snap/snap-subdir-test 2024-04-14T23:52:04.405 INFO:tasks.workunit.client.0.smithi060.stdout: create 200/30 file after the snapshot 2024-04-14T23:52:04.408 INFO:tasks.workunit.client.0.smithi060.stdout: create 200/31 file after the snapshot 2024-04-14T23:52:04.410 INFO:tasks.workunit.client.0.smithi060.stdout: create 200/32 file after the snapshot 2024-04-14T23:52:04.413 INFO:tasks.workunit.client.0.smithi060.stdout: create 200/33 file after the snapshot 2024-04-14T23:52:04.416 INFO:tasks.workunit.client.0.smithi060.stdout: create 200/34 file after the snapshot 2024-04-14T23:52:04.419 INFO:tasks.workunit.client.0.smithi060.stdout: create 200/35 file after the snapshot 2024-04-14T23:52:04.421 INFO:tasks.workunit.client.0.smithi060.stdout: create 200/36 file after the snapshot 2024-04-14T23:52:04.424 INFO:tasks.workunit.client.0.smithi060.stdout: create 200/37 file after the snapshot 2024-04-14T23:52:04.428 INFO:tasks.workunit.client.0.smithi060.stdout: create 200/38 file after the snapshot 2024-04-14T23:52:04.433 INFO:tasks.workunit.client.0.smithi060.stdout: create 200/39 file after the snapshot 2024-04-14T23:52:04.435 INFO:tasks.workunit.client.0.smithi060.stdout: create 201/.snap/snap-subdir-test 2024-04-14T23:52:04.492 INFO:tasks.workunit.client.0.smithi060.stdout: create 201/30 file after the snapshot 2024-04-14T23:52:04.495 INFO:tasks.workunit.client.0.smithi060.stdout: create 201/31 file after the snapshot 2024-04-14T23:52:04.497 INFO:tasks.workunit.client.0.smithi060.stdout: create 201/32 file after the snapshot 2024-04-14T23:52:04.501 INFO:tasks.workunit.client.0.smithi060.stdout: create 201/33 file after the snapshot 2024-04-14T23:52:04.503 INFO:tasks.workunit.client.0.smithi060.stdout: create 201/34 file after the snapshot 2024-04-14T23:52:04.506 INFO:tasks.workunit.client.0.smithi060.stdout: create 201/35 file after the snapshot 2024-04-14T23:52:04.509 INFO:tasks.workunit.client.0.smithi060.stdout: create 201/36 file after the snapshot 2024-04-14T23:52:04.512 INFO:tasks.workunit.client.0.smithi060.stdout: create 201/37 file after the snapshot 2024-04-14T23:52:04.514 INFO:tasks.workunit.client.0.smithi060.stdout: create 201/38 file after the snapshot 2024-04-14T23:52:04.517 INFO:tasks.workunit.client.0.smithi060.stdout: create 201/39 file after the snapshot 2024-04-14T23:52:04.520 INFO:tasks.workunit.client.0.smithi060.stdout: create 202/.snap/snap-subdir-test 2024-04-14T23:52:04.580 INFO:tasks.workunit.client.0.smithi060.stdout: create 202/30 file after the snapshot 2024-04-14T23:52:04.583 INFO:tasks.workunit.client.0.smithi060.stdout: create 202/31 file after the snapshot 2024-04-14T23:52:04.586 INFO:tasks.workunit.client.0.smithi060.stdout: create 202/32 file after the snapshot 2024-04-14T23:52:04.589 INFO:tasks.workunit.client.0.smithi060.stdout: create 202/33 file after the snapshot 2024-04-14T23:52:04.592 INFO:tasks.workunit.client.0.smithi060.stdout: create 202/34 file after the snapshot 2024-04-14T23:52:04.594 INFO:tasks.workunit.client.0.smithi060.stdout: create 202/35 file after the snapshot 2024-04-14T23:52:04.597 INFO:tasks.workunit.client.0.smithi060.stdout: create 202/36 file after the snapshot 2024-04-14T23:52:04.600 INFO:tasks.workunit.client.0.smithi060.stdout: create 202/37 file after the snapshot 2024-04-14T23:52:04.603 INFO:tasks.workunit.client.0.smithi060.stdout: create 202/38 file after the snapshot 2024-04-14T23:52:04.606 INFO:tasks.workunit.client.0.smithi060.stdout: create 202/39 file after the snapshot 2024-04-14T23:52:04.608 INFO:tasks.workunit.client.0.smithi060.stdout: create 203/.snap/snap-subdir-test 2024-04-14T23:52:04.672 INFO:tasks.workunit.client.0.smithi060.stdout: create 203/30 file after the snapshot 2024-04-14T23:52:04.675 INFO:tasks.workunit.client.0.smithi060.stdout: create 203/31 file after the snapshot 2024-04-14T23:52:04.677 INFO:tasks.workunit.client.0.smithi060.stdout: create 203/32 file after the snapshot 2024-04-14T23:52:04.680 INFO:tasks.workunit.client.0.smithi060.stdout: create 203/33 file after the snapshot 2024-04-14T23:52:04.683 INFO:tasks.workunit.client.0.smithi060.stdout: create 203/34 file after the snapshot 2024-04-14T23:52:04.685 INFO:tasks.workunit.client.0.smithi060.stdout: create 203/35 file after the snapshot 2024-04-14T23:52:04.688 INFO:tasks.workunit.client.0.smithi060.stdout: create 203/36 file after the snapshot 2024-04-14T23:52:04.690 INFO:tasks.workunit.client.0.smithi060.stdout: create 203/37 file after the snapshot 2024-04-14T23:52:04.693 INFO:tasks.workunit.client.0.smithi060.stdout: create 203/38 file after the snapshot 2024-04-14T23:52:04.695 INFO:tasks.workunit.client.0.smithi060.stdout: create 203/39 file after the snapshot 2024-04-14T23:52:04.698 INFO:tasks.workunit.client.0.smithi060.stdout: create 204/.snap/snap-subdir-test 2024-04-14T23:52:04.760 INFO:tasks.workunit.client.0.smithi060.stdout: create 204/30 file after the snapshot 2024-04-14T23:52:04.763 INFO:tasks.workunit.client.0.smithi060.stdout: create 204/31 file after the snapshot 2024-04-14T23:52:04.765 INFO:tasks.workunit.client.0.smithi060.stdout: create 204/32 file after the snapshot 2024-04-14T23:52:04.768 INFO:tasks.workunit.client.0.smithi060.stdout: create 204/33 file after the snapshot 2024-04-14T23:52:04.771 INFO:tasks.workunit.client.0.smithi060.stdout: create 204/34 file after the snapshot 2024-04-14T23:52:04.774 INFO:tasks.workunit.client.0.smithi060.stdout: create 204/35 file after the snapshot 2024-04-14T23:52:04.777 INFO:tasks.workunit.client.0.smithi060.stdout: create 204/36 file after the snapshot 2024-04-14T23:52:04.780 INFO:tasks.workunit.client.0.smithi060.stdout: create 204/37 file after the snapshot 2024-04-14T23:52:04.783 INFO:tasks.workunit.client.0.smithi060.stdout: create 204/38 file after the snapshot 2024-04-14T23:52:04.785 INFO:tasks.workunit.client.0.smithi060.stdout: create 204/39 file after the snapshot 2024-04-14T23:52:04.788 INFO:tasks.workunit.client.0.smithi060.stdout: create 205/.snap/snap-subdir-test 2024-04-14T23:52:04.848 INFO:tasks.workunit.client.0.smithi060.stdout: create 205/30 file after the snapshot 2024-04-14T23:52:04.851 INFO:tasks.workunit.client.0.smithi060.stdout: create 205/31 file after the snapshot 2024-04-14T23:52:04.854 INFO:tasks.workunit.client.0.smithi060.stdout: create 205/32 file after the snapshot 2024-04-14T23:52:04.857 INFO:tasks.workunit.client.0.smithi060.stdout: create 205/33 file after the snapshot 2024-04-14T23:52:04.859 INFO:tasks.workunit.client.0.smithi060.stdout: create 205/34 file after the snapshot 2024-04-14T23:52:04.862 INFO:tasks.workunit.client.0.smithi060.stdout: create 205/35 file after the snapshot 2024-04-14T23:52:04.865 INFO:tasks.workunit.client.0.smithi060.stdout: create 205/36 file after the snapshot 2024-04-14T23:52:04.868 INFO:tasks.workunit.client.0.smithi060.stdout: create 205/37 file after the snapshot 2024-04-14T23:52:04.871 INFO:tasks.workunit.client.0.smithi060.stdout: create 205/38 file after the snapshot 2024-04-14T23:52:04.874 INFO:tasks.workunit.client.0.smithi060.stdout: create 205/39 file after the snapshot 2024-04-14T23:52:04.876 INFO:tasks.workunit.client.0.smithi060.stdout: create 206/.snap/snap-subdir-test 2024-04-14T23:52:04.942 INFO:tasks.workunit.client.0.smithi060.stdout: create 206/30 file after the snapshot 2024-04-14T23:52:04.943 INFO:tasks.workunit.client.0.smithi060.stdout: create 206/31 file after the snapshot 2024-04-14T23:52:04.945 INFO:tasks.workunit.client.0.smithi060.stdout: create 206/32 file after the snapshot 2024-04-14T23:52:04.948 INFO:tasks.workunit.client.0.smithi060.stdout: create 206/33 file after the snapshot 2024-04-14T23:52:04.951 INFO:tasks.workunit.client.0.smithi060.stdout: create 206/34 file after the snapshot 2024-04-14T23:52:04.953 INFO:tasks.workunit.client.0.smithi060.stdout: create 206/35 file after the snapshot 2024-04-14T23:52:04.956 INFO:tasks.workunit.client.0.smithi060.stdout: create 206/36 file after the snapshot 2024-04-14T23:52:04.959 INFO:tasks.workunit.client.0.smithi060.stdout: create 206/37 file after the snapshot 2024-04-14T23:52:04.963 INFO:tasks.workunit.client.0.smithi060.stdout: create 206/38 file after the snapshot 2024-04-14T23:52:04.965 INFO:tasks.workunit.client.0.smithi060.stdout: create 206/39 file after the snapshot 2024-04-14T23:52:04.968 INFO:tasks.workunit.client.0.smithi060.stdout: create 207/.snap/snap-subdir-test 2024-04-14T23:52:05.029 INFO:tasks.workunit.client.0.smithi060.stdout: create 207/30 file after the snapshot 2024-04-14T23:52:05.032 INFO:tasks.workunit.client.0.smithi060.stdout: create 207/31 file after the snapshot 2024-04-14T23:52:05.035 INFO:tasks.workunit.client.0.smithi060.stdout: create 207/32 file after the snapshot 2024-04-14T23:52:05.038 INFO:tasks.workunit.client.0.smithi060.stdout: create 207/33 file after the snapshot 2024-04-14T23:52:05.040 INFO:tasks.workunit.client.0.smithi060.stdout: create 207/34 file after the snapshot 2024-04-14T23:52:05.043 INFO:tasks.workunit.client.0.smithi060.stdout: create 207/35 file after the snapshot 2024-04-14T23:52:05.046 INFO:tasks.workunit.client.0.smithi060.stdout: create 207/36 file after the snapshot 2024-04-14T23:52:05.049 INFO:tasks.workunit.client.0.smithi060.stdout: create 207/37 file after the snapshot 2024-04-14T23:52:05.052 INFO:tasks.workunit.client.0.smithi060.stdout: create 207/38 file after the snapshot 2024-04-14T23:52:05.055 INFO:tasks.workunit.client.0.smithi060.stdout: create 207/39 file after the snapshot 2024-04-14T23:52:05.058 INFO:tasks.workunit.client.0.smithi060.stdout: create 208/.snap/snap-subdir-test 2024-04-14T23:52:05.118 INFO:tasks.workunit.client.0.smithi060.stdout: create 208/30 file after the snapshot 2024-04-14T23:52:05.120 INFO:tasks.workunit.client.0.smithi060.stdout: create 208/31 file after the snapshot 2024-04-14T23:52:05.123 INFO:tasks.workunit.client.0.smithi060.stdout: create 208/32 file after the snapshot 2024-04-14T23:52:05.126 INFO:tasks.workunit.client.0.smithi060.stdout: create 208/33 file after the snapshot 2024-04-14T23:52:05.129 INFO:tasks.workunit.client.0.smithi060.stdout: create 208/34 file after the snapshot 2024-04-14T23:52:05.131 INFO:tasks.workunit.client.0.smithi060.stdout: create 208/35 file after the snapshot 2024-04-14T23:52:05.134 INFO:tasks.workunit.client.0.smithi060.stdout: create 208/36 file after the snapshot 2024-04-14T23:52:05.137 INFO:tasks.workunit.client.0.smithi060.stdout: create 208/37 file after the snapshot 2024-04-14T23:52:05.140 INFO:tasks.workunit.client.0.smithi060.stdout: create 208/38 file after the snapshot 2024-04-14T23:52:05.142 INFO:tasks.workunit.client.0.smithi060.stdout: create 208/39 file after the snapshot 2024-04-14T23:52:05.145 INFO:tasks.workunit.client.0.smithi060.stdout: create 209/.snap/snap-subdir-test 2024-04-14T23:52:05.204 INFO:tasks.workunit.client.0.smithi060.stdout: create 209/30 file after the snapshot 2024-04-14T23:52:05.207 INFO:tasks.workunit.client.0.smithi060.stdout: create 209/31 file after the snapshot 2024-04-14T23:52:05.210 INFO:tasks.workunit.client.0.smithi060.stdout: create 209/32 file after the snapshot 2024-04-14T23:52:05.212 INFO:tasks.workunit.client.0.smithi060.stdout: create 209/33 file after the snapshot 2024-04-14T23:52:05.215 INFO:tasks.workunit.client.0.smithi060.stdout: create 209/34 file after the snapshot 2024-04-14T23:52:05.218 INFO:tasks.workunit.client.0.smithi060.stdout: create 209/35 file after the snapshot 2024-04-14T23:52:05.220 INFO:tasks.workunit.client.0.smithi060.stdout: create 209/36 file after the snapshot 2024-04-14T23:52:05.223 INFO:tasks.workunit.client.0.smithi060.stdout: create 209/37 file after the snapshot 2024-04-14T23:52:05.226 INFO:tasks.workunit.client.0.smithi060.stdout: create 209/38 file after the snapshot 2024-04-14T23:52:05.229 INFO:tasks.workunit.client.0.smithi060.stdout: create 209/39 file after the snapshot 2024-04-14T23:52:05.231 INFO:tasks.workunit.client.0.smithi060.stdout: create 210/.snap/snap-subdir-test 2024-04-14T23:52:05.287 INFO:tasks.workunit.client.0.smithi060.stdout: create 210/30 file after the snapshot 2024-04-14T23:52:05.289 INFO:tasks.workunit.client.0.smithi060.stdout: create 210/31 file after the snapshot 2024-04-14T23:52:05.292 INFO:tasks.workunit.client.0.smithi060.stdout: create 210/32 file after the snapshot 2024-04-14T23:52:05.294 INFO:tasks.workunit.client.0.smithi060.stdout: create 210/33 file after the snapshot 2024-04-14T23:52:05.298 INFO:tasks.workunit.client.0.smithi060.stdout: create 210/34 file after the snapshot 2024-04-14T23:52:05.300 INFO:tasks.workunit.client.0.smithi060.stdout: create 210/35 file after the snapshot 2024-04-14T23:52:05.303 INFO:tasks.workunit.client.0.smithi060.stdout: create 210/36 file after the snapshot 2024-04-14T23:52:05.306 INFO:tasks.workunit.client.0.smithi060.stdout: create 210/37 file after the snapshot 2024-04-14T23:52:05.309 INFO:tasks.workunit.client.0.smithi060.stdout: create 210/38 file after the snapshot 2024-04-14T23:52:05.312 INFO:tasks.workunit.client.0.smithi060.stdout: create 210/39 file after the snapshot 2024-04-14T23:52:05.315 INFO:tasks.workunit.client.0.smithi060.stdout: create 211/.snap/snap-subdir-test 2024-04-14T23:52:05.372 INFO:tasks.workunit.client.0.smithi060.stdout: create 211/30 file after the snapshot 2024-04-14T23:52:05.375 INFO:tasks.workunit.client.0.smithi060.stdout: create 211/31 file after the snapshot 2024-04-14T23:52:05.377 INFO:tasks.workunit.client.0.smithi060.stdout: create 211/32 file after the snapshot 2024-04-14T23:52:05.380 INFO:tasks.workunit.client.0.smithi060.stdout: create 211/33 file after the snapshot 2024-04-14T23:52:05.383 INFO:tasks.workunit.client.0.smithi060.stdout: create 211/34 file after the snapshot 2024-04-14T23:52:05.386 INFO:tasks.workunit.client.0.smithi060.stdout: create 211/35 file after the snapshot 2024-04-14T23:52:05.389 INFO:tasks.workunit.client.0.smithi060.stdout: create 211/36 file after the snapshot 2024-04-14T23:52:05.392 INFO:tasks.workunit.client.0.smithi060.stdout: create 211/37 file after the snapshot 2024-04-14T23:52:05.394 INFO:tasks.workunit.client.0.smithi060.stdout: create 211/38 file after the snapshot 2024-04-14T23:52:05.397 INFO:tasks.workunit.client.0.smithi060.stdout: create 211/39 file after the snapshot 2024-04-14T23:52:05.400 INFO:tasks.workunit.client.0.smithi060.stdout: create 212/.snap/snap-subdir-test 2024-04-14T23:52:05.456 INFO:tasks.workunit.client.0.smithi060.stdout: create 212/30 file after the snapshot 2024-04-14T23:52:05.459 INFO:tasks.workunit.client.0.smithi060.stdout: create 212/31 file after the snapshot 2024-04-14T23:52:05.461 INFO:tasks.workunit.client.0.smithi060.stdout: create 212/32 file after the snapshot 2024-04-14T23:52:05.464 INFO:tasks.workunit.client.0.smithi060.stdout: create 212/33 file after the snapshot 2024-04-14T23:52:05.467 INFO:tasks.workunit.client.0.smithi060.stdout: create 212/34 file after the snapshot 2024-04-14T23:52:05.470 INFO:tasks.workunit.client.0.smithi060.stdout: create 212/35 file after the snapshot 2024-04-14T23:52:05.472 INFO:tasks.workunit.client.0.smithi060.stdout: create 212/36 file after the snapshot 2024-04-14T23:52:05.475 INFO:tasks.workunit.client.0.smithi060.stdout: create 212/37 file after the snapshot 2024-04-14T23:52:05.478 INFO:tasks.workunit.client.0.smithi060.stdout: create 212/38 file after the snapshot 2024-04-14T23:52:05.481 INFO:tasks.workunit.client.0.smithi060.stdout: create 212/39 file after the snapshot 2024-04-14T23:52:05.484 INFO:tasks.workunit.client.0.smithi060.stdout: create 213/.snap/snap-subdir-test 2024-04-14T23:52:05.542 INFO:tasks.workunit.client.0.smithi060.stdout: create 213/30 file after the snapshot 2024-04-14T23:52:05.544 INFO:tasks.workunit.client.0.smithi060.stdout: create 213/31 file after the snapshot 2024-04-14T23:52:05.547 INFO:tasks.workunit.client.0.smithi060.stdout: create 213/32 file after the snapshot 2024-04-14T23:52:05.550 INFO:tasks.workunit.client.0.smithi060.stdout: create 213/33 file after the snapshot 2024-04-14T23:52:05.553 INFO:tasks.workunit.client.0.smithi060.stdout: create 213/34 file after the snapshot 2024-04-14T23:52:05.555 INFO:tasks.workunit.client.0.smithi060.stdout: create 213/35 file after the snapshot 2024-04-14T23:52:05.558 INFO:tasks.workunit.client.0.smithi060.stdout: create 213/36 file after the snapshot 2024-04-14T23:52:05.561 INFO:tasks.workunit.client.0.smithi060.stdout: create 213/37 file after the snapshot 2024-04-14T23:52:05.564 INFO:tasks.workunit.client.0.smithi060.stdout: create 213/38 file after the snapshot 2024-04-14T23:52:05.567 INFO:tasks.workunit.client.0.smithi060.stdout: create 213/39 file after the snapshot 2024-04-14T23:52:05.571 INFO:tasks.workunit.client.0.smithi060.stdout: create 214/.snap/snap-subdir-test 2024-04-14T23:52:05.626 INFO:tasks.workunit.client.0.smithi060.stdout: create 214/30 file after the snapshot 2024-04-14T23:52:05.629 INFO:tasks.workunit.client.0.smithi060.stdout: create 214/31 file after the snapshot 2024-04-14T23:52:05.632 INFO:tasks.workunit.client.0.smithi060.stdout: create 214/32 file after the snapshot 2024-04-14T23:52:05.635 INFO:tasks.workunit.client.0.smithi060.stdout: create 214/33 file after the snapshot 2024-04-14T23:52:05.638 INFO:tasks.workunit.client.0.smithi060.stdout: create 214/34 file after the snapshot 2024-04-14T23:52:05.640 INFO:tasks.workunit.client.0.smithi060.stdout: create 214/35 file after the snapshot 2024-04-14T23:52:05.643 INFO:tasks.workunit.client.0.smithi060.stdout: create 214/36 file after the snapshot 2024-04-14T23:52:05.646 INFO:tasks.workunit.client.0.smithi060.stdout: create 214/37 file after the snapshot 2024-04-14T23:52:05.649 INFO:tasks.workunit.client.0.smithi060.stdout: create 214/38 file after the snapshot 2024-04-14T23:52:05.652 INFO:tasks.workunit.client.0.smithi060.stdout: create 214/39 file after the snapshot 2024-04-14T23:52:05.655 INFO:tasks.workunit.client.0.smithi060.stdout: create 215/.snap/snap-subdir-test 2024-04-14T23:52:05.710 INFO:tasks.workunit.client.0.smithi060.stdout: create 215/30 file after the snapshot 2024-04-14T23:52:05.713 INFO:tasks.workunit.client.0.smithi060.stdout: create 215/31 file after the snapshot 2024-04-14T23:52:05.716 INFO:tasks.workunit.client.0.smithi060.stdout: create 215/32 file after the snapshot 2024-04-14T23:52:05.718 INFO:tasks.workunit.client.0.smithi060.stdout: create 215/33 file after the snapshot 2024-04-14T23:52:05.721 INFO:tasks.workunit.client.0.smithi060.stdout: create 215/34 file after the snapshot 2024-04-14T23:52:05.724 INFO:tasks.workunit.client.0.smithi060.stdout: create 215/35 file after the snapshot 2024-04-14T23:52:05.727 INFO:tasks.workunit.client.0.smithi060.stdout: create 215/36 file after the snapshot 2024-04-14T23:52:05.828 INFO:tasks.workunit.client.0.smithi060.stdout: create 215/37 file after the snapshot 2024-04-14T23:52:06.226 INFO:tasks.workunit.client.0.smithi060.stdout: create 215/38 file after the snapshot 2024-04-14T23:52:06.229 INFO:tasks.workunit.client.0.smithi060.stdout: create 215/39 file after the snapshot 2024-04-14T23:52:06.231 INFO:tasks.workunit.client.0.smithi060.stdout: create 216/.snap/snap-subdir-test 2024-04-14T23:52:06.289 INFO:tasks.workunit.client.0.smithi060.stdout: create 216/30 file after the snapshot 2024-04-14T23:52:06.291 INFO:tasks.workunit.client.0.smithi060.stdout: create 216/31 file after the snapshot 2024-04-14T23:52:06.294 INFO:tasks.workunit.client.0.smithi060.stdout: create 216/32 file after the snapshot 2024-04-14T23:52:06.297 INFO:tasks.workunit.client.0.smithi060.stdout: create 216/33 file after the snapshot 2024-04-14T23:52:06.299 INFO:tasks.workunit.client.0.smithi060.stdout: create 216/34 file after the snapshot 2024-04-14T23:52:06.302 INFO:tasks.workunit.client.0.smithi060.stdout: create 216/35 file after the snapshot 2024-04-14T23:52:06.304 INFO:tasks.workunit.client.0.smithi060.stdout: create 216/36 file after the snapshot 2024-04-14T23:52:06.307 INFO:tasks.workunit.client.0.smithi060.stdout: create 216/37 file after the snapshot 2024-04-14T23:52:06.310 INFO:tasks.workunit.client.0.smithi060.stdout: create 216/38 file after the snapshot 2024-04-14T23:52:06.313 INFO:tasks.workunit.client.0.smithi060.stdout: create 216/39 file after the snapshot 2024-04-14T23:52:06.316 INFO:tasks.workunit.client.0.smithi060.stdout: create 217/.snap/snap-subdir-test 2024-04-14T23:52:06.366 INFO:tasks.workunit.client.0.smithi060.stdout: create 217/30 file after the snapshot 2024-04-14T23:52:06.369 INFO:tasks.workunit.client.0.smithi060.stdout: create 217/31 file after the snapshot 2024-04-14T23:52:06.372 INFO:tasks.workunit.client.0.smithi060.stdout: create 217/32 file after the snapshot 2024-04-14T23:52:06.374 INFO:tasks.workunit.client.0.smithi060.stdout: create 217/33 file after the snapshot 2024-04-14T23:52:06.377 INFO:tasks.workunit.client.0.smithi060.stdout: create 217/34 file after the snapshot 2024-04-14T23:52:06.380 INFO:tasks.workunit.client.0.smithi060.stdout: create 217/35 file after the snapshot 2024-04-14T23:52:06.383 INFO:tasks.workunit.client.0.smithi060.stdout: create 217/36 file after the snapshot 2024-04-14T23:52:06.385 INFO:tasks.workunit.client.0.smithi060.stdout: create 217/37 file after the snapshot 2024-04-14T23:52:06.389 INFO:tasks.workunit.client.0.smithi060.stdout: create 217/38 file after the snapshot 2024-04-14T23:52:06.391 INFO:tasks.workunit.client.0.smithi060.stdout: create 217/39 file after the snapshot 2024-04-14T23:52:06.394 INFO:tasks.workunit.client.0.smithi060.stdout: create 218/.snap/snap-subdir-test 2024-04-14T23:52:06.450 INFO:tasks.workunit.client.0.smithi060.stdout: create 218/30 file after the snapshot 2024-04-14T23:52:06.453 INFO:tasks.workunit.client.0.smithi060.stdout: create 218/31 file after the snapshot 2024-04-14T23:52:06.455 INFO:tasks.workunit.client.0.smithi060.stdout: create 218/32 file after the snapshot 2024-04-14T23:52:06.458 INFO:tasks.workunit.client.0.smithi060.stdout: create 218/33 file after the snapshot 2024-04-14T23:52:06.461 INFO:tasks.workunit.client.0.smithi060.stdout: create 218/34 file after the snapshot 2024-04-14T23:52:06.463 INFO:tasks.workunit.client.0.smithi060.stdout: create 218/35 file after the snapshot 2024-04-14T23:52:06.467 INFO:tasks.workunit.client.0.smithi060.stdout: create 218/36 file after the snapshot 2024-04-14T23:52:06.470 INFO:tasks.workunit.client.0.smithi060.stdout: create 218/37 file after the snapshot 2024-04-14T23:52:06.473 INFO:tasks.workunit.client.0.smithi060.stdout: create 218/38 file after the snapshot 2024-04-14T23:52:06.476 INFO:tasks.workunit.client.0.smithi060.stdout: create 218/39 file after the snapshot 2024-04-14T23:52:06.479 INFO:tasks.workunit.client.0.smithi060.stdout: create 219/.snap/snap-subdir-test 2024-04-14T23:52:06.526 INFO:tasks.workunit.client.0.smithi060.stdout: create 219/30 file after the snapshot 2024-04-14T23:52:06.529 INFO:tasks.workunit.client.0.smithi060.stdout: create 219/31 file after the snapshot 2024-04-14T23:52:06.532 INFO:tasks.workunit.client.0.smithi060.stdout: create 219/32 file after the snapshot 2024-04-14T23:52:06.744 INFO:tasks.workunit.client.0.smithi060.stdout: create 219/33 file after the snapshot 2024-04-14T23:52:06.782 INFO:tasks.workunit.client.0.smithi060.stdout: create 219/34 file after the snapshot 2024-04-14T23:52:07.364 INFO:tasks.workunit.client.0.smithi060.stdout: create 219/35 file after the snapshot 2024-04-14T23:52:07.563 INFO:tasks.workunit.client.0.smithi060.stdout: create 219/36 file after the snapshot 2024-04-14T23:52:07.724 INFO:tasks.workunit.client.0.smithi060.stdout: create 219/37 file after the snapshot 2024-04-14T23:52:07.735 INFO:tasks.workunit.client.0.smithi060.stdout: create 219/38 file after the snapshot 2024-04-14T23:52:07.738 INFO:tasks.workunit.client.0.smithi060.stdout: create 219/39 file after the snapshot 2024-04-14T23:52:07.740 INFO:tasks.workunit.client.0.smithi060.stdout: create 220/.snap/snap-subdir-test 2024-04-14T23:52:07.845 INFO:tasks.workunit.client.0.smithi060.stdout: create 220/30 file after the snapshot 2024-04-14T23:52:07.846 INFO:tasks.workunit.client.0.smithi060.stdout: create 220/31 file after the snapshot 2024-04-14T23:52:07.848 INFO:tasks.workunit.client.0.smithi060.stdout: create 220/32 file after the snapshot 2024-04-14T23:52:07.849 INFO:tasks.workunit.client.0.smithi060.stdout: create 220/33 file after the snapshot 2024-04-14T23:52:07.850 INFO:tasks.workunit.client.0.smithi060.stdout: create 220/34 file after the snapshot 2024-04-14T23:52:07.852 INFO:tasks.workunit.client.0.smithi060.stdout: create 220/35 file after the snapshot 2024-04-14T23:52:07.854 INFO:tasks.workunit.client.0.smithi060.stdout: create 220/36 file after the snapshot 2024-04-14T23:52:07.855 INFO:tasks.workunit.client.0.smithi060.stdout: create 220/37 file after the snapshot 2024-04-14T23:52:07.857 INFO:tasks.workunit.client.0.smithi060.stdout: create 220/38 file after the snapshot 2024-04-14T23:52:07.859 INFO:tasks.workunit.client.0.smithi060.stdout: create 220/39 file after the snapshot 2024-04-14T23:52:07.860 INFO:tasks.workunit.client.0.smithi060.stdout: create 221/.snap/snap-subdir-test 2024-04-14T23:52:07.913 INFO:tasks.workunit.client.0.smithi060.stdout: create 221/30 file after the snapshot 2024-04-14T23:52:07.915 INFO:tasks.workunit.client.0.smithi060.stdout: create 221/31 file after the snapshot 2024-04-14T23:52:07.918 INFO:tasks.workunit.client.0.smithi060.stdout: create 221/32 file after the snapshot 2024-04-14T23:52:07.921 INFO:tasks.workunit.client.0.smithi060.stdout: create 221/33 file after the snapshot 2024-04-14T23:52:07.924 INFO:tasks.workunit.client.0.smithi060.stdout: create 221/34 file after the snapshot 2024-04-14T23:52:07.927 INFO:tasks.workunit.client.0.smithi060.stdout: create 221/35 file after the snapshot 2024-04-14T23:52:07.930 INFO:tasks.workunit.client.0.smithi060.stdout: create 221/36 file after the snapshot 2024-04-14T23:52:07.933 INFO:tasks.workunit.client.0.smithi060.stdout: create 221/37 file after the snapshot 2024-04-14T23:52:07.936 INFO:tasks.workunit.client.0.smithi060.stdout: create 221/38 file after the snapshot 2024-04-14T23:52:07.939 INFO:tasks.workunit.client.0.smithi060.stdout: create 221/39 file after the snapshot 2024-04-14T23:52:07.941 INFO:tasks.workunit.client.0.smithi060.stdout: create 222/.snap/snap-subdir-test 2024-04-14T23:52:07.995 INFO:tasks.workunit.client.0.smithi060.stdout: create 222/30 file after the snapshot 2024-04-14T23:52:07.997 INFO:tasks.workunit.client.0.smithi060.stdout: create 222/31 file after the snapshot 2024-04-14T23:52:08.000 INFO:tasks.workunit.client.0.smithi060.stdout: create 222/32 file after the snapshot 2024-04-14T23:52:08.003 INFO:tasks.workunit.client.0.smithi060.stdout: create 222/33 file after the snapshot 2024-04-14T23:52:08.006 INFO:tasks.workunit.client.0.smithi060.stdout: create 222/34 file after the snapshot 2024-04-14T23:52:08.009 INFO:tasks.workunit.client.0.smithi060.stdout: create 222/35 file after the snapshot 2024-04-14T23:52:08.011 INFO:tasks.workunit.client.0.smithi060.stdout: create 222/36 file after the snapshot 2024-04-14T23:52:08.014 INFO:tasks.workunit.client.0.smithi060.stdout: create 222/37 file after the snapshot 2024-04-14T23:52:08.017 INFO:tasks.workunit.client.0.smithi060.stdout: create 222/38 file after the snapshot 2024-04-14T23:52:08.020 INFO:tasks.workunit.client.0.smithi060.stdout: create 222/39 file after the snapshot 2024-04-14T23:52:08.023 INFO:tasks.workunit.client.0.smithi060.stdout: create 223/.snap/snap-subdir-test 2024-04-14T23:52:08.085 INFO:tasks.workunit.client.0.smithi060.stdout: create 223/30 file after the snapshot 2024-04-14T23:52:08.088 INFO:tasks.workunit.client.0.smithi060.stdout: create 223/31 file after the snapshot 2024-04-14T23:52:08.091 INFO:tasks.workunit.client.0.smithi060.stdout: create 223/32 file after the snapshot 2024-04-14T23:52:08.094 INFO:tasks.workunit.client.0.smithi060.stdout: create 223/33 file after the snapshot 2024-04-14T23:52:08.097 INFO:tasks.workunit.client.0.smithi060.stdout: create 223/34 file after the snapshot 2024-04-14T23:52:08.099 INFO:tasks.workunit.client.0.smithi060.stdout: create 223/35 file after the snapshot 2024-04-14T23:52:08.102 INFO:tasks.workunit.client.0.smithi060.stdout: create 223/36 file after the snapshot 2024-04-14T23:52:08.105 INFO:tasks.workunit.client.0.smithi060.stdout: create 223/37 file after the snapshot 2024-04-14T23:52:08.108 INFO:tasks.workunit.client.0.smithi060.stdout: create 223/38 file after the snapshot 2024-04-14T23:52:08.111 INFO:tasks.workunit.client.0.smithi060.stdout: create 223/39 file after the snapshot 2024-04-14T23:52:08.113 INFO:tasks.workunit.client.0.smithi060.stdout: create 224/.snap/snap-subdir-test 2024-04-14T23:52:08.181 INFO:tasks.workunit.client.0.smithi060.stdout: create 224/30 file after the snapshot 2024-04-14T23:52:08.184 INFO:tasks.workunit.client.0.smithi060.stdout: create 224/31 file after the snapshot 2024-04-14T23:52:08.186 INFO:tasks.workunit.client.0.smithi060.stdout: create 224/32 file after the snapshot 2024-04-14T23:52:08.189 INFO:tasks.workunit.client.0.smithi060.stdout: create 224/33 file after the snapshot 2024-04-14T23:52:08.192 INFO:tasks.workunit.client.0.smithi060.stdout: create 224/34 file after the snapshot 2024-04-14T23:52:08.194 INFO:tasks.workunit.client.0.smithi060.stdout: create 224/35 file after the snapshot 2024-04-14T23:52:08.197 INFO:tasks.workunit.client.0.smithi060.stdout: create 224/36 file after the snapshot 2024-04-14T23:52:08.200 INFO:tasks.workunit.client.0.smithi060.stdout: create 224/37 file after the snapshot 2024-04-14T23:52:08.203 INFO:tasks.workunit.client.0.smithi060.stdout: create 224/38 file after the snapshot 2024-04-14T23:52:08.205 INFO:tasks.workunit.client.0.smithi060.stdout: create 224/39 file after the snapshot 2024-04-14T23:52:08.208 INFO:tasks.workunit.client.0.smithi060.stdout: create 225/.snap/snap-subdir-test 2024-04-14T23:52:08.258 INFO:tasks.workunit.client.0.smithi060.stdout: create 225/30 file after the snapshot 2024-04-14T23:52:08.260 INFO:tasks.workunit.client.0.smithi060.stdout: create 225/31 file after the snapshot 2024-04-14T23:52:08.263 INFO:tasks.workunit.client.0.smithi060.stdout: create 225/32 file after the snapshot 2024-04-14T23:52:08.265 INFO:tasks.workunit.client.0.smithi060.stdout: create 225/33 file after the snapshot 2024-04-14T23:52:08.268 INFO:tasks.workunit.client.0.smithi060.stdout: create 225/34 file after the snapshot 2024-04-14T23:52:08.271 INFO:tasks.workunit.client.0.smithi060.stdout: create 225/35 file after the snapshot 2024-04-14T23:52:08.273 INFO:tasks.workunit.client.0.smithi060.stdout: create 225/36 file after the snapshot 2024-04-14T23:52:08.276 INFO:tasks.workunit.client.0.smithi060.stdout: create 225/37 file after the snapshot 2024-04-14T23:52:08.279 INFO:tasks.workunit.client.0.smithi060.stdout: create 225/38 file after the snapshot 2024-04-14T23:52:08.282 INFO:tasks.workunit.client.0.smithi060.stdout: create 225/39 file after the snapshot 2024-04-14T23:52:08.285 INFO:tasks.workunit.client.0.smithi060.stdout: create 226/.snap/snap-subdir-test 2024-04-14T23:52:08.343 INFO:tasks.workunit.client.0.smithi060.stdout: create 226/30 file after the snapshot 2024-04-14T23:52:08.346 INFO:tasks.workunit.client.0.smithi060.stdout: create 226/31 file after the snapshot 2024-04-14T23:52:08.349 INFO:tasks.workunit.client.0.smithi060.stdout: create 226/32 file after the snapshot 2024-04-14T23:52:08.352 INFO:tasks.workunit.client.0.smithi060.stdout: create 226/33 file after the snapshot 2024-04-14T23:52:08.355 INFO:tasks.workunit.client.0.smithi060.stdout: create 226/34 file after the snapshot 2024-04-14T23:52:08.358 INFO:tasks.workunit.client.0.smithi060.stdout: create 226/35 file after the snapshot 2024-04-14T23:52:08.362 INFO:tasks.workunit.client.0.smithi060.stdout: create 226/36 file after the snapshot 2024-04-14T23:52:08.364 INFO:tasks.workunit.client.0.smithi060.stdout: create 226/37 file after the snapshot 2024-04-14T23:52:08.367 INFO:tasks.workunit.client.0.smithi060.stdout: create 226/38 file after the snapshot 2024-04-14T23:52:08.370 INFO:tasks.workunit.client.0.smithi060.stdout: create 226/39 file after the snapshot 2024-04-14T23:52:08.373 INFO:tasks.workunit.client.0.smithi060.stdout: create 227/.snap/snap-subdir-test 2024-04-14T23:52:08.432 INFO:tasks.workunit.client.0.smithi060.stdout: create 227/30 file after the snapshot 2024-04-14T23:52:08.435 INFO:tasks.workunit.client.0.smithi060.stdout: create 227/31 file after the snapshot 2024-04-14T23:52:08.438 INFO:tasks.workunit.client.0.smithi060.stdout: create 227/32 file after the snapshot 2024-04-14T23:52:08.440 INFO:tasks.workunit.client.0.smithi060.stdout: create 227/33 file after the snapshot 2024-04-14T23:52:08.443 INFO:tasks.workunit.client.0.smithi060.stdout: create 227/34 file after the snapshot 2024-04-14T23:52:08.446 INFO:tasks.workunit.client.0.smithi060.stdout: create 227/35 file after the snapshot 2024-04-14T23:52:08.449 INFO:tasks.workunit.client.0.smithi060.stdout: create 227/36 file after the snapshot 2024-04-14T23:52:08.452 INFO:tasks.workunit.client.0.smithi060.stdout: create 227/37 file after the snapshot 2024-04-14T23:52:08.454 INFO:tasks.workunit.client.0.smithi060.stdout: create 227/38 file after the snapshot 2024-04-14T23:52:08.457 INFO:tasks.workunit.client.0.smithi060.stdout: create 227/39 file after the snapshot 2024-04-14T23:52:08.460 INFO:tasks.workunit.client.0.smithi060.stdout: create 228/.snap/snap-subdir-test 2024-04-14T23:52:08.514 INFO:tasks.workunit.client.0.smithi060.stdout: create 228/30 file after the snapshot 2024-04-14T23:52:08.516 INFO:tasks.workunit.client.0.smithi060.stdout: create 228/31 file after the snapshot 2024-04-14T23:52:08.518 INFO:tasks.workunit.client.0.smithi060.stdout: create 228/32 file after the snapshot 2024-04-14T23:52:08.520 INFO:tasks.workunit.client.0.smithi060.stdout: create 228/33 file after the snapshot 2024-04-14T23:52:08.522 INFO:tasks.workunit.client.0.smithi060.stdout: create 228/34 file after the snapshot 2024-04-14T23:52:08.523 INFO:tasks.workunit.client.0.smithi060.stdout: create 228/35 file after the snapshot 2024-04-14T23:52:08.525 INFO:tasks.workunit.client.0.smithi060.stdout: create 228/36 file after the snapshot 2024-04-14T23:52:08.527 INFO:tasks.workunit.client.0.smithi060.stdout: create 228/37 file after the snapshot 2024-04-14T23:52:08.529 INFO:tasks.workunit.client.0.smithi060.stdout: create 228/38 file after the snapshot 2024-04-14T23:52:08.531 INFO:tasks.workunit.client.0.smithi060.stdout: create 228/39 file after the snapshot 2024-04-14T23:52:08.533 INFO:tasks.workunit.client.0.smithi060.stdout: create 229/.snap/snap-subdir-test 2024-04-14T23:52:08.585 INFO:tasks.workunit.client.0.smithi060.stdout: create 229/30 file after the snapshot 2024-04-14T23:52:08.588 INFO:tasks.workunit.client.0.smithi060.stdout: create 229/31 file after the snapshot 2024-04-14T23:52:08.590 INFO:tasks.workunit.client.0.smithi060.stdout: create 229/32 file after the snapshot 2024-04-14T23:52:08.593 INFO:tasks.workunit.client.0.smithi060.stdout: create 229/33 file after the snapshot 2024-04-14T23:52:08.596 INFO:tasks.workunit.client.0.smithi060.stdout: create 229/34 file after the snapshot 2024-04-14T23:52:08.599 INFO:tasks.workunit.client.0.smithi060.stdout: create 229/35 file after the snapshot 2024-04-14T23:52:08.601 INFO:tasks.workunit.client.0.smithi060.stdout: create 229/36 file after the snapshot 2024-04-14T23:52:08.604 INFO:tasks.workunit.client.0.smithi060.stdout: create 229/37 file after the snapshot 2024-04-14T23:52:08.607 INFO:tasks.workunit.client.0.smithi060.stdout: create 229/38 file after the snapshot 2024-04-14T23:52:08.610 INFO:tasks.workunit.client.0.smithi060.stdout: create 229/39 file after the snapshot 2024-04-14T23:52:08.612 INFO:tasks.workunit.client.0.smithi060.stdout: create 230/.snap/snap-subdir-test 2024-04-14T23:52:08.675 INFO:tasks.workunit.client.0.smithi060.stdout: create 230/30 file after the snapshot 2024-04-14T23:52:08.678 INFO:tasks.workunit.client.0.smithi060.stdout: create 230/31 file after the snapshot 2024-04-14T23:52:08.681 INFO:tasks.workunit.client.0.smithi060.stdout: create 230/32 file after the snapshot 2024-04-14T23:52:08.684 INFO:tasks.workunit.client.0.smithi060.stdout: create 230/33 file after the snapshot 2024-04-14T23:52:08.687 INFO:tasks.workunit.client.0.smithi060.stdout: create 230/34 file after the snapshot 2024-04-14T23:52:08.837 INFO:tasks.workunit.client.0.smithi060.stdout: create 230/35 file after the snapshot 2024-04-14T23:52:09.540 INFO:tasks.workunit.client.0.smithi060.stdout: create 230/36 file after the snapshot 2024-04-14T23:52:09.542 INFO:tasks.workunit.client.0.smithi060.stdout: create 230/37 file after the snapshot 2024-04-14T23:52:09.545 INFO:tasks.workunit.client.0.smithi060.stdout: create 230/38 file after the snapshot 2024-04-14T23:52:09.548 INFO:tasks.workunit.client.0.smithi060.stdout: create 230/39 file after the snapshot 2024-04-14T23:52:09.551 INFO:tasks.workunit.client.0.smithi060.stdout: create 231/.snap/snap-subdir-test 2024-04-14T23:52:09.599 INFO:tasks.workunit.client.0.smithi060.stdout: create 231/30 file after the snapshot 2024-04-14T23:52:09.602 INFO:tasks.workunit.client.0.smithi060.stdout: create 231/31 file after the snapshot 2024-04-14T23:52:09.605 INFO:tasks.workunit.client.0.smithi060.stdout: create 231/32 file after the snapshot 2024-04-14T23:52:09.609 INFO:tasks.workunit.client.0.smithi060.stdout: create 231/33 file after the snapshot 2024-04-14T23:52:09.613 INFO:tasks.workunit.client.0.smithi060.stdout: create 231/34 file after the snapshot 2024-04-14T23:52:09.618 INFO:tasks.workunit.client.0.smithi060.stdout: create 231/35 file after the snapshot 2024-04-14T23:52:09.620 INFO:tasks.workunit.client.0.smithi060.stdout: create 231/36 file after the snapshot 2024-04-14T23:52:09.623 INFO:tasks.workunit.client.0.smithi060.stdout: create 231/37 file after the snapshot 2024-04-14T23:52:09.626 INFO:tasks.workunit.client.0.smithi060.stdout: create 231/38 file after the snapshot 2024-04-14T23:52:09.629 INFO:tasks.workunit.client.0.smithi060.stdout: create 231/39 file after the snapshot 2024-04-14T23:52:09.631 INFO:tasks.workunit.client.0.smithi060.stdout: create 232/.snap/snap-subdir-test 2024-04-14T23:52:09.680 INFO:tasks.workunit.client.0.smithi060.stdout: create 232/30 file after the snapshot 2024-04-14T23:52:09.682 INFO:tasks.workunit.client.0.smithi060.stdout: create 232/31 file after the snapshot 2024-04-14T23:52:09.684 INFO:tasks.workunit.client.0.smithi060.stdout: create 232/32 file after the snapshot 2024-04-14T23:52:09.687 INFO:tasks.workunit.client.0.smithi060.stdout: create 232/33 file after the snapshot 2024-04-14T23:52:09.688 INFO:tasks.workunit.client.0.smithi060.stdout: create 232/34 file after the snapshot 2024-04-14T23:52:09.691 INFO:tasks.workunit.client.0.smithi060.stdout: create 232/35 file after the snapshot 2024-04-14T23:52:09.693 INFO:tasks.workunit.client.0.smithi060.stdout: create 232/36 file after the snapshot 2024-04-14T23:52:09.695 INFO:tasks.workunit.client.0.smithi060.stdout: create 232/37 file after the snapshot 2024-04-14T23:52:09.697 INFO:tasks.workunit.client.0.smithi060.stdout: create 232/38 file after the snapshot 2024-04-14T23:52:09.699 INFO:tasks.workunit.client.0.smithi060.stdout: create 232/39 file after the snapshot 2024-04-14T23:52:09.701 INFO:tasks.workunit.client.0.smithi060.stdout: create 233/.snap/snap-subdir-test 2024-04-14T23:52:09.744 INFO:tasks.workunit.client.0.smithi060.stdout: create 233/30 file after the snapshot 2024-04-14T23:52:09.746 INFO:tasks.workunit.client.0.smithi060.stdout: create 233/31 file after the snapshot 2024-04-14T23:52:09.747 INFO:tasks.workunit.client.0.smithi060.stdout: create 233/32 file after the snapshot 2024-04-14T23:52:09.749 INFO:tasks.workunit.client.0.smithi060.stdout: create 233/33 file after the snapshot 2024-04-14T23:52:09.751 INFO:tasks.workunit.client.0.smithi060.stdout: create 233/34 file after the snapshot 2024-04-14T23:52:09.752 INFO:tasks.workunit.client.0.smithi060.stdout: create 233/35 file after the snapshot 2024-04-14T23:52:09.754 INFO:tasks.workunit.client.0.smithi060.stdout: create 233/36 file after the snapshot 2024-04-14T23:52:09.755 INFO:tasks.workunit.client.0.smithi060.stdout: create 233/37 file after the snapshot 2024-04-14T23:52:09.757 INFO:tasks.workunit.client.0.smithi060.stdout: create 233/38 file after the snapshot 2024-04-14T23:52:09.758 INFO:tasks.workunit.client.0.smithi060.stdout: create 233/39 file after the snapshot 2024-04-14T23:52:09.760 INFO:tasks.workunit.client.0.smithi060.stdout: create 234/.snap/snap-subdir-test 2024-04-14T23:52:09.876 INFO:tasks.workunit.client.0.smithi060.stdout: create 234/30 file after the snapshot 2024-04-14T23:52:09.879 INFO:tasks.workunit.client.0.smithi060.stdout: create 234/31 file after the snapshot 2024-04-14T23:52:09.881 INFO:tasks.workunit.client.0.smithi060.stdout: create 234/32 file after the snapshot 2024-04-14T23:52:09.884 INFO:tasks.workunit.client.0.smithi060.stdout: create 234/33 file after the snapshot 2024-04-14T23:52:09.887 INFO:tasks.workunit.client.0.smithi060.stdout: create 234/34 file after the snapshot 2024-04-14T23:52:09.889 INFO:tasks.workunit.client.0.smithi060.stdout: create 234/35 file after the snapshot 2024-04-14T23:52:09.892 INFO:tasks.workunit.client.0.smithi060.stdout: create 234/36 file after the snapshot 2024-04-14T23:52:09.894 INFO:tasks.workunit.client.0.smithi060.stdout: create 234/37 file after the snapshot 2024-04-14T23:52:09.897 INFO:tasks.workunit.client.0.smithi060.stdout: create 234/38 file after the snapshot 2024-04-14T23:52:09.899 INFO:tasks.workunit.client.0.smithi060.stdout: create 234/39 file after the snapshot 2024-04-14T23:52:09.902 INFO:tasks.workunit.client.0.smithi060.stdout: create 235/.snap/snap-subdir-test 2024-04-14T23:52:09.951 INFO:tasks.workunit.client.0.smithi060.stdout: create 235/30 file after the snapshot 2024-04-14T23:52:09.954 INFO:tasks.workunit.client.0.smithi060.stdout: create 235/31 file after the snapshot 2024-04-14T23:52:09.957 INFO:tasks.workunit.client.0.smithi060.stdout: create 235/32 file after the snapshot 2024-04-14T23:52:09.959 INFO:tasks.workunit.client.0.smithi060.stdout: create 235/33 file after the snapshot 2024-04-14T23:52:09.962 INFO:tasks.workunit.client.0.smithi060.stdout: create 235/34 file after the snapshot 2024-04-14T23:52:09.965 INFO:tasks.workunit.client.0.smithi060.stdout: create 235/35 file after the snapshot 2024-04-14T23:52:09.968 INFO:tasks.workunit.client.0.smithi060.stdout: create 235/36 file after the snapshot 2024-04-14T23:52:09.971 INFO:tasks.workunit.client.0.smithi060.stdout: create 235/37 file after the snapshot 2024-04-14T23:52:09.973 INFO:tasks.workunit.client.0.smithi060.stdout: create 235/38 file after the snapshot 2024-04-14T23:52:09.976 INFO:tasks.workunit.client.0.smithi060.stdout: create 235/39 file after the snapshot 2024-04-14T23:52:09.979 INFO:tasks.workunit.client.0.smithi060.stdout: create 236/.snap/snap-subdir-test 2024-04-14T23:52:10.017 INFO:tasks.workunit.client.0.smithi060.stdout: create 236/30 file after the snapshot 2024-04-14T23:52:10.020 INFO:tasks.workunit.client.0.smithi060.stdout: create 236/31 file after the snapshot 2024-04-14T23:52:10.023 INFO:tasks.workunit.client.0.smithi060.stdout: create 236/32 file after the snapshot 2024-04-14T23:52:10.073 INFO:tasks.workunit.client.0.smithi060.stdout: create 236/33 file after the snapshot 2024-04-14T23:52:10.076 INFO:tasks.workunit.client.0.smithi060.stdout: create 236/34 file after the snapshot 2024-04-14T23:52:10.078 INFO:tasks.workunit.client.0.smithi060.stdout: create 236/35 file after the snapshot 2024-04-14T23:52:10.081 INFO:tasks.workunit.client.0.smithi060.stdout: create 236/36 file after the snapshot 2024-04-14T23:52:10.084 INFO:tasks.workunit.client.0.smithi060.stdout: create 236/37 file after the snapshot 2024-04-14T23:52:10.087 INFO:tasks.workunit.client.0.smithi060.stdout: create 236/38 file after the snapshot 2024-04-14T23:52:10.090 INFO:tasks.workunit.client.0.smithi060.stdout: create 236/39 file after the snapshot 2024-04-14T23:52:10.093 INFO:tasks.workunit.client.0.smithi060.stdout: create 237/.snap/snap-subdir-test 2024-04-14T23:52:10.140 INFO:tasks.workunit.client.0.smithi060.stdout: create 237/30 file after the snapshot 2024-04-14T23:52:10.143 INFO:tasks.workunit.client.0.smithi060.stdout: create 237/31 file after the snapshot 2024-04-14T23:52:10.146 INFO:tasks.workunit.client.0.smithi060.stdout: create 237/32 file after the snapshot 2024-04-14T23:52:10.148 INFO:tasks.workunit.client.0.smithi060.stdout: create 237/33 file after the snapshot 2024-04-14T23:52:10.151 INFO:tasks.workunit.client.0.smithi060.stdout: create 237/34 file after the snapshot 2024-04-14T23:52:10.154 INFO:tasks.workunit.client.0.smithi060.stdout: create 237/35 file after the snapshot 2024-04-14T23:52:10.157 INFO:tasks.workunit.client.0.smithi060.stdout: create 237/36 file after the snapshot 2024-04-14T23:52:10.159 INFO:tasks.workunit.client.0.smithi060.stdout: create 237/37 file after the snapshot 2024-04-14T23:52:10.162 INFO:tasks.workunit.client.0.smithi060.stdout: create 237/38 file after the snapshot 2024-04-14T23:52:10.165 INFO:tasks.workunit.client.0.smithi060.stdout: create 237/39 file after the snapshot 2024-04-14T23:52:10.168 INFO:tasks.workunit.client.0.smithi060.stdout: create 238/.snap/snap-subdir-test 2024-04-14T23:52:10.208 INFO:tasks.workunit.client.0.smithi060.stdout: create 238/30 file after the snapshot 2024-04-14T23:52:10.211 INFO:tasks.workunit.client.0.smithi060.stdout: create 238/31 file after the snapshot 2024-04-14T23:52:10.213 INFO:tasks.workunit.client.0.smithi060.stdout: create 238/32 file after the snapshot 2024-04-14T23:52:10.216 INFO:tasks.workunit.client.0.smithi060.stdout: create 238/33 file after the snapshot 2024-04-14T23:52:10.218 INFO:tasks.workunit.client.0.smithi060.stdout: create 238/34 file after the snapshot 2024-04-14T23:52:10.221 INFO:tasks.workunit.client.0.smithi060.stdout: create 238/35 file after the snapshot 2024-04-14T23:52:10.224 INFO:tasks.workunit.client.0.smithi060.stdout: create 238/36 file after the snapshot 2024-04-14T23:52:10.227 INFO:tasks.workunit.client.0.smithi060.stdout: create 238/37 file after the snapshot 2024-04-14T23:52:10.230 INFO:tasks.workunit.client.0.smithi060.stdout: create 238/38 file after the snapshot 2024-04-14T23:52:10.232 INFO:tasks.workunit.client.0.smithi060.stdout: create 238/39 file after the snapshot 2024-04-14T23:52:10.235 INFO:tasks.workunit.client.0.smithi060.stdout: create 239/.snap/snap-subdir-test 2024-04-14T23:52:10.627 INFO:tasks.workunit.client.0.smithi060.stdout: create 239/30 file after the snapshot 2024-04-14T23:52:10.630 INFO:tasks.workunit.client.0.smithi060.stdout: create 239/31 file after the snapshot 2024-04-14T23:52:10.633 INFO:tasks.workunit.client.0.smithi060.stdout: create 239/32 file after the snapshot 2024-04-14T23:52:10.635 INFO:tasks.workunit.client.0.smithi060.stdout: create 239/33 file after the snapshot 2024-04-14T23:52:10.638 INFO:tasks.workunit.client.0.smithi060.stdout: create 239/34 file after the snapshot 2024-04-14T23:52:10.641 INFO:tasks.workunit.client.0.smithi060.stdout: create 239/35 file after the snapshot 2024-04-14T23:52:10.643 INFO:tasks.workunit.client.0.smithi060.stdout: create 239/36 file after the snapshot 2024-04-14T23:52:10.646 INFO:tasks.workunit.client.0.smithi060.stdout: create 239/37 file after the snapshot 2024-04-14T23:52:10.649 INFO:tasks.workunit.client.0.smithi060.stdout: create 239/38 file after the snapshot 2024-04-14T23:52:10.652 INFO:tasks.workunit.client.0.smithi060.stdout: create 239/39 file after the snapshot 2024-04-14T23:52:10.655 INFO:tasks.workunit.client.0.smithi060.stdout: create 240/.snap/snap-subdir-test 2024-04-14T23:52:10.706 INFO:tasks.workunit.client.0.smithi060.stdout: create 240/30 file after the snapshot 2024-04-14T23:52:10.708 INFO:tasks.workunit.client.0.smithi060.stdout: create 240/31 file after the snapshot 2024-04-14T23:52:10.710 INFO:tasks.workunit.client.0.smithi060.stdout: create 240/32 file after the snapshot 2024-04-14T23:52:10.712 INFO:tasks.workunit.client.0.smithi060.stdout: create 240/33 file after the snapshot 2024-04-14T23:52:10.714 INFO:tasks.workunit.client.0.smithi060.stdout: create 240/34 file after the snapshot 2024-04-14T23:52:10.716 INFO:tasks.workunit.client.0.smithi060.stdout: create 240/35 file after the snapshot 2024-04-14T23:52:10.717 INFO:tasks.workunit.client.0.smithi060.stdout: create 240/36 file after the snapshot 2024-04-14T23:52:10.719 INFO:tasks.workunit.client.0.smithi060.stdout: create 240/37 file after the snapshot 2024-04-14T23:52:10.722 INFO:tasks.workunit.client.0.smithi060.stdout: create 240/38 file after the snapshot 2024-04-14T23:52:10.725 INFO:tasks.workunit.client.0.smithi060.stdout: create 240/39 file after the snapshot 2024-04-14T23:52:10.727 INFO:tasks.workunit.client.0.smithi060.stdout: create 241/.snap/snap-subdir-test 2024-04-14T23:52:10.779 INFO:tasks.workunit.client.0.smithi060.stdout: create 241/30 file after the snapshot 2024-04-14T23:52:10.781 INFO:tasks.workunit.client.0.smithi060.stdout: create 241/31 file after the snapshot 2024-04-14T23:52:10.783 INFO:tasks.workunit.client.0.smithi060.stdout: create 241/32 file after the snapshot 2024-04-14T23:52:10.785 INFO:tasks.workunit.client.0.smithi060.stdout: create 241/33 file after the snapshot 2024-04-14T23:52:10.788 INFO:tasks.workunit.client.0.smithi060.stdout: create 241/34 file after the snapshot 2024-04-14T23:52:10.790 INFO:tasks.workunit.client.0.smithi060.stdout: create 241/35 file after the snapshot 2024-04-14T23:52:10.793 INFO:tasks.workunit.client.0.smithi060.stdout: create 241/36 file after the snapshot 2024-04-14T23:52:10.796 INFO:tasks.workunit.client.0.smithi060.stdout: create 241/37 file after the snapshot 2024-04-14T23:52:10.798 INFO:tasks.workunit.client.0.smithi060.stdout: create 241/38 file after the snapshot 2024-04-14T23:52:10.801 INFO:tasks.workunit.client.0.smithi060.stdout: create 241/39 file after the snapshot 2024-04-14T23:52:10.804 INFO:tasks.workunit.client.0.smithi060.stdout: create 242/.snap/snap-subdir-test 2024-04-14T23:52:10.843 INFO:tasks.workunit.client.0.smithi060.stdout: create 242/30 file after the snapshot 2024-04-14T23:52:10.846 INFO:tasks.workunit.client.0.smithi060.stdout: create 242/31 file after the snapshot 2024-04-14T23:52:10.848 INFO:tasks.workunit.client.0.smithi060.stdout: create 242/32 file after the snapshot 2024-04-14T23:52:10.851 INFO:tasks.workunit.client.0.smithi060.stdout: create 242/33 file after the snapshot 2024-04-14T23:52:10.853 INFO:tasks.workunit.client.0.smithi060.stdout: create 242/34 file after the snapshot 2024-04-14T23:52:10.856 INFO:tasks.workunit.client.0.smithi060.stdout: create 242/35 file after the snapshot 2024-04-14T23:52:10.859 INFO:tasks.workunit.client.0.smithi060.stdout: create 242/36 file after the snapshot 2024-04-14T23:52:10.862 INFO:tasks.workunit.client.0.smithi060.stdout: create 242/37 file after the snapshot 2024-04-14T23:52:10.865 INFO:tasks.workunit.client.0.smithi060.stdout: create 242/38 file after the snapshot 2024-04-14T23:52:10.868 INFO:tasks.workunit.client.0.smithi060.stdout: create 242/39 file after the snapshot 2024-04-14T23:52:10.871 INFO:tasks.workunit.client.0.smithi060.stdout: create 243/.snap/snap-subdir-test 2024-04-14T23:52:10.919 INFO:tasks.workunit.client.0.smithi060.stdout: create 243/30 file after the snapshot 2024-04-14T23:52:10.921 INFO:tasks.workunit.client.0.smithi060.stdout: create 243/31 file after the snapshot 2024-04-14T23:52:10.923 INFO:tasks.workunit.client.0.smithi060.stdout: create 243/32 file after the snapshot 2024-04-14T23:52:10.925 INFO:tasks.workunit.client.0.smithi060.stdout: create 243/33 file after the snapshot 2024-04-14T23:52:10.928 INFO:tasks.workunit.client.0.smithi060.stdout: create 243/34 file after the snapshot 2024-04-14T23:52:10.931 INFO:tasks.workunit.client.0.smithi060.stdout: create 243/35 file after the snapshot 2024-04-14T23:52:10.934 INFO:tasks.workunit.client.0.smithi060.stdout: create 243/36 file after the snapshot 2024-04-14T23:52:10.937 INFO:tasks.workunit.client.0.smithi060.stdout: create 243/37 file after the snapshot 2024-04-14T23:52:10.940 INFO:tasks.workunit.client.0.smithi060.stdout: create 243/38 file after the snapshot 2024-04-14T23:52:10.943 INFO:tasks.workunit.client.0.smithi060.stdout: create 243/39 file after the snapshot 2024-04-14T23:52:10.945 INFO:tasks.workunit.client.0.smithi060.stdout: create 244/.snap/snap-subdir-test 2024-04-14T23:52:10.998 INFO:tasks.workunit.client.0.smithi060.stdout: create 244/30 file after the snapshot 2024-04-14T23:52:11.000 INFO:tasks.workunit.client.0.smithi060.stdout: create 244/31 file after the snapshot 2024-04-14T23:52:11.002 INFO:tasks.workunit.client.0.smithi060.stdout: create 244/32 file after the snapshot 2024-04-14T23:52:11.004 INFO:tasks.workunit.client.0.smithi060.stdout: create 244/33 file after the snapshot 2024-04-14T23:52:11.006 INFO:tasks.workunit.client.0.smithi060.stdout: create 244/34 file after the snapshot 2024-04-14T23:52:11.010 INFO:tasks.workunit.client.0.smithi060.stdout: create 244/35 file after the snapshot 2024-04-14T23:52:11.010 INFO:tasks.workunit.client.0.smithi060.stdout: create 244/36 file after the snapshot 2024-04-14T23:52:11.012 INFO:tasks.workunit.client.0.smithi060.stdout: create 244/37 file after the snapshot 2024-04-14T23:52:11.015 INFO:tasks.workunit.client.0.smithi060.stdout: create 244/38 file after the snapshot 2024-04-14T23:52:11.017 INFO:tasks.workunit.client.0.smithi060.stdout: create 244/39 file after the snapshot 2024-04-14T23:52:11.019 INFO:tasks.workunit.client.0.smithi060.stdout: create 245/.snap/snap-subdir-test 2024-04-14T23:52:11.070 INFO:tasks.workunit.client.0.smithi060.stdout: create 245/30 file after the snapshot 2024-04-14T23:52:11.072 INFO:tasks.workunit.client.0.smithi060.stdout: create 245/31 file after the snapshot 2024-04-14T23:52:11.075 INFO:tasks.workunit.client.0.smithi060.stdout: create 245/32 file after the snapshot 2024-04-14T23:52:11.077 INFO:tasks.workunit.client.0.smithi060.stdout: create 245/33 file after the snapshot 2024-04-14T23:52:11.080 INFO:tasks.workunit.client.0.smithi060.stdout: create 245/34 file after the snapshot 2024-04-14T23:52:11.083 INFO:tasks.workunit.client.0.smithi060.stdout: create 245/35 file after the snapshot 2024-04-14T23:52:11.086 INFO:tasks.workunit.client.0.smithi060.stdout: create 245/36 file after the snapshot 2024-04-14T23:52:11.089 INFO:tasks.workunit.client.0.smithi060.stdout: create 245/37 file after the snapshot 2024-04-14T23:52:11.092 INFO:tasks.workunit.client.0.smithi060.stdout: create 245/38 file after the snapshot 2024-04-14T23:52:11.094 INFO:tasks.workunit.client.0.smithi060.stdout: create 245/39 file after the snapshot 2024-04-14T23:52:11.097 INFO:tasks.workunit.client.0.smithi060.stdout: create 246/.snap/snap-subdir-test 2024-04-14T23:52:11.152 INFO:tasks.workunit.client.0.smithi060.stdout: create 246/30 file after the snapshot 2024-04-14T23:52:11.154 INFO:tasks.workunit.client.0.smithi060.stdout: create 246/31 file after the snapshot 2024-04-14T23:52:11.157 INFO:tasks.workunit.client.0.smithi060.stdout: create 246/32 file after the snapshot 2024-04-14T23:52:11.160 INFO:tasks.workunit.client.0.smithi060.stdout: create 246/33 file after the snapshot 2024-04-14T23:52:11.162 INFO:tasks.workunit.client.0.smithi060.stdout: create 246/34 file after the snapshot 2024-04-14T23:52:11.165 INFO:tasks.workunit.client.0.smithi060.stdout: create 246/35 file after the snapshot 2024-04-14T23:52:11.168 INFO:tasks.workunit.client.0.smithi060.stdout: create 246/36 file after the snapshot 2024-04-14T23:52:11.171 INFO:tasks.workunit.client.0.smithi060.stdout: create 246/37 file after the snapshot 2024-04-14T23:52:11.174 INFO:tasks.workunit.client.0.smithi060.stdout: create 246/38 file after the snapshot 2024-04-14T23:52:11.176 INFO:tasks.workunit.client.0.smithi060.stdout: create 246/39 file after the snapshot 2024-04-14T23:52:11.179 INFO:tasks.workunit.client.0.smithi060.stdout: create 247/.snap/snap-subdir-test 2024-04-14T23:52:11.229 INFO:tasks.workunit.client.0.smithi060.stdout: create 247/30 file after the snapshot 2024-04-14T23:52:11.232 INFO:tasks.workunit.client.0.smithi060.stdout: create 247/31 file after the snapshot 2024-04-14T23:52:11.235 INFO:tasks.workunit.client.0.smithi060.stdout: create 247/32 file after the snapshot 2024-04-14T23:52:11.237 INFO:tasks.workunit.client.0.smithi060.stdout: create 247/33 file after the snapshot 2024-04-14T23:52:11.242 INFO:tasks.workunit.client.0.smithi060.stdout: create 247/34 file after the snapshot 2024-04-14T23:52:11.245 INFO:tasks.workunit.client.0.smithi060.stdout: create 247/35 file after the snapshot 2024-04-14T23:52:11.249 INFO:tasks.workunit.client.0.smithi060.stdout: create 247/36 file after the snapshot 2024-04-14T23:52:11.253 INFO:tasks.workunit.client.0.smithi060.stdout: create 247/37 file after the snapshot 2024-04-14T23:52:11.255 INFO:tasks.workunit.client.0.smithi060.stdout: create 247/38 file after the snapshot 2024-04-14T23:52:11.258 INFO:tasks.workunit.client.0.smithi060.stdout: create 247/39 file after the snapshot 2024-04-14T23:52:11.261 INFO:tasks.workunit.client.0.smithi060.stdout: create 248/.snap/snap-subdir-test 2024-04-14T23:52:11.309 INFO:tasks.workunit.client.0.smithi060.stdout: create 248/30 file after the snapshot 2024-04-14T23:52:11.311 INFO:tasks.workunit.client.0.smithi060.stdout: create 248/31 file after the snapshot 2024-04-14T23:52:11.314 INFO:tasks.workunit.client.0.smithi060.stdout: create 248/32 file after the snapshot 2024-04-14T23:52:11.316 INFO:tasks.workunit.client.0.smithi060.stdout: create 248/33 file after the snapshot 2024-04-14T23:52:11.319 INFO:tasks.workunit.client.0.smithi060.stdout: create 248/34 file after the snapshot 2024-04-14T23:52:11.321 INFO:tasks.workunit.client.0.smithi060.stdout: create 248/35 file after the snapshot 2024-04-14T23:52:11.324 INFO:tasks.workunit.client.0.smithi060.stdout: create 248/36 file after the snapshot 2024-04-14T23:52:11.327 INFO:tasks.workunit.client.0.smithi060.stdout: create 248/37 file after the snapshot 2024-04-14T23:52:11.329 INFO:tasks.workunit.client.0.smithi060.stdout: create 248/38 file after the snapshot 2024-04-14T23:52:11.332 INFO:tasks.workunit.client.0.smithi060.stdout: create 248/39 file after the snapshot 2024-04-14T23:52:11.335 INFO:tasks.workunit.client.0.smithi060.stdout: create 249/.snap/snap-subdir-test 2024-04-14T23:52:11.387 INFO:tasks.workunit.client.0.smithi060.stdout: create 249/30 file after the snapshot 2024-04-14T23:52:11.389 INFO:tasks.workunit.client.0.smithi060.stdout: create 249/31 file after the snapshot 2024-04-14T23:52:11.391 INFO:tasks.workunit.client.0.smithi060.stdout: create 249/32 file after the snapshot 2024-04-14T23:52:11.394 INFO:tasks.workunit.client.0.smithi060.stdout: create 249/33 file after the snapshot 2024-04-14T23:52:11.397 INFO:tasks.workunit.client.0.smithi060.stdout: create 249/34 file after the snapshot 2024-04-14T23:52:11.400 INFO:tasks.workunit.client.0.smithi060.stdout: create 249/35 file after the snapshot 2024-04-14T23:52:11.402 INFO:tasks.workunit.client.0.smithi060.stdout: create 249/36 file after the snapshot 2024-04-14T23:52:11.405 INFO:tasks.workunit.client.0.smithi060.stdout: create 249/37 file after the snapshot 2024-04-14T23:52:11.408 INFO:tasks.workunit.client.0.smithi060.stdout: create 249/38 file after the snapshot 2024-04-14T23:52:11.411 INFO:tasks.workunit.client.0.smithi060.stdout: create 249/39 file after the snapshot 2024-04-14T23:52:11.849 INFO:tasks.workunit.client.0.smithi060.stdout: create 250/.snap/snap-subdir-test 2024-04-14T23:52:12.338 INFO:tasks.workunit.client.0.smithi060.stdout: create 250/30 file after the snapshot 2024-04-14T23:52:12.342 INFO:tasks.workunit.client.0.smithi060.stdout: create 250/31 file after the snapshot 2024-04-14T23:52:12.346 INFO:tasks.workunit.client.0.smithi060.stdout: create 250/32 file after the snapshot 2024-04-14T23:52:12.350 INFO:tasks.workunit.client.0.smithi060.stdout: create 250/33 file after the snapshot 2024-04-14T23:52:12.354 INFO:tasks.workunit.client.0.smithi060.stdout: create 250/34 file after the snapshot 2024-04-14T23:52:12.358 INFO:tasks.workunit.client.0.smithi060.stdout: create 250/35 file after the snapshot 2024-04-14T23:52:12.362 INFO:tasks.workunit.client.0.smithi060.stdout: create 250/36 file after the snapshot 2024-04-14T23:52:12.366 INFO:tasks.workunit.client.0.smithi060.stdout: create 250/37 file after the snapshot 2024-04-14T23:52:12.370 INFO:tasks.workunit.client.0.smithi060.stdout: create 250/38 file after the snapshot 2024-04-14T23:52:12.374 INFO:tasks.workunit.client.0.smithi060.stdout: create 250/39 file after the snapshot 2024-04-14T23:52:12.379 INFO:tasks.workunit.client.0.smithi060.stdout: create 251/.snap/snap-subdir-test 2024-04-14T23:52:12.422 INFO:tasks.workunit.client.0.smithi060.stdout: create 251/30 file after the snapshot 2024-04-14T23:52:12.425 INFO:tasks.workunit.client.0.smithi060.stdout: create 251/31 file after the snapshot 2024-04-14T23:52:12.430 INFO:tasks.workunit.client.0.smithi060.stdout: create 251/32 file after the snapshot 2024-04-14T23:52:12.434 INFO:tasks.workunit.client.0.smithi060.stdout: create 251/33 file after the snapshot 2024-04-14T23:52:12.438 INFO:tasks.workunit.client.0.smithi060.stdout: create 251/34 file after the snapshot 2024-04-14T23:52:12.442 INFO:tasks.workunit.client.0.smithi060.stdout: create 251/35 file after the snapshot 2024-04-14T23:52:12.446 INFO:tasks.workunit.client.0.smithi060.stdout: create 251/36 file after the snapshot 2024-04-14T23:52:12.451 INFO:tasks.workunit.client.0.smithi060.stdout: create 251/37 file after the snapshot 2024-04-14T23:52:12.454 INFO:tasks.workunit.client.0.smithi060.stdout: create 251/38 file after the snapshot 2024-04-14T23:52:12.456 INFO:tasks.workunit.client.0.smithi060.stdout: create 251/39 file after the snapshot 2024-04-14T23:52:12.459 INFO:tasks.workunit.client.0.smithi060.stdout: create 252/.snap/snap-subdir-test 2024-04-14T23:52:12.506 INFO:tasks.workunit.client.0.smithi060.stdout: create 252/30 file after the snapshot 2024-04-14T23:52:12.509 INFO:tasks.workunit.client.0.smithi060.stdout: create 252/31 file after the snapshot 2024-04-14T23:52:12.513 INFO:tasks.workunit.client.0.smithi060.stdout: create 252/32 file after the snapshot 2024-04-14T23:52:12.514 INFO:tasks.workunit.client.0.smithi060.stdout: create 252/33 file after the snapshot 2024-04-14T23:52:12.517 INFO:tasks.workunit.client.0.smithi060.stdout: create 252/34 file after the snapshot 2024-04-14T23:52:12.520 INFO:tasks.workunit.client.0.smithi060.stdout: create 252/35 file after the snapshot 2024-04-14T23:52:12.522 INFO:tasks.workunit.client.0.smithi060.stdout: create 252/36 file after the snapshot 2024-04-14T23:52:12.525 INFO:tasks.workunit.client.0.smithi060.stdout: create 252/37 file after the snapshot 2024-04-14T23:52:12.528 INFO:tasks.workunit.client.0.smithi060.stdout: create 252/38 file after the snapshot 2024-04-14T23:52:12.531 INFO:tasks.workunit.client.0.smithi060.stdout: create 252/39 file after the snapshot 2024-04-14T23:52:12.533 INFO:tasks.workunit.client.0.smithi060.stdout: create 253/.snap/snap-subdir-test 2024-04-14T23:52:12.581 INFO:tasks.workunit.client.0.smithi060.stdout: create 253/30 file after the snapshot 2024-04-14T23:52:12.584 INFO:tasks.workunit.client.0.smithi060.stdout: create 253/31 file after the snapshot 2024-04-14T23:52:12.586 INFO:tasks.workunit.client.0.smithi060.stdout: create 253/32 file after the snapshot 2024-04-14T23:52:12.589 INFO:tasks.workunit.client.0.smithi060.stdout: create 253/33 file after the snapshot 2024-04-14T23:52:12.592 INFO:tasks.workunit.client.0.smithi060.stdout: create 253/34 file after the snapshot 2024-04-14T23:52:12.594 INFO:tasks.workunit.client.0.smithi060.stdout: create 253/35 file after the snapshot 2024-04-14T23:52:12.597 INFO:tasks.workunit.client.0.smithi060.stdout: create 253/36 file after the snapshot 2024-04-14T23:52:12.600 INFO:tasks.workunit.client.0.smithi060.stdout: create 253/37 file after the snapshot 2024-04-14T23:52:12.603 INFO:tasks.workunit.client.0.smithi060.stdout: create 253/38 file after the snapshot 2024-04-14T23:52:12.607 INFO:tasks.workunit.client.0.smithi060.stdout: create 253/39 file after the snapshot 2024-04-14T23:52:12.609 INFO:tasks.workunit.client.0.smithi060.stdout: create 254/.snap/snap-subdir-test 2024-04-14T23:52:12.656 INFO:tasks.workunit.client.0.smithi060.stdout: create 254/30 file after the snapshot 2024-04-14T23:52:12.658 INFO:tasks.workunit.client.0.smithi060.stdout: create 254/31 file after the snapshot 2024-04-14T23:52:12.661 INFO:tasks.workunit.client.0.smithi060.stdout: create 254/32 file after the snapshot 2024-04-14T23:52:12.663 INFO:tasks.workunit.client.0.smithi060.stdout: create 254/33 file after the snapshot 2024-04-14T23:52:12.666 INFO:tasks.workunit.client.0.smithi060.stdout: create 254/34 file after the snapshot 2024-04-14T23:52:12.669 INFO:tasks.workunit.client.0.smithi060.stdout: create 254/35 file after the snapshot 2024-04-14T23:52:12.672 INFO:tasks.workunit.client.0.smithi060.stdout: create 254/36 file after the snapshot 2024-04-14T23:52:12.674 INFO:tasks.workunit.client.0.smithi060.stdout: create 254/37 file after the snapshot 2024-04-14T23:52:12.678 INFO:tasks.workunit.client.0.smithi060.stdout: create 254/38 file after the snapshot 2024-04-14T23:52:12.680 INFO:tasks.workunit.client.0.smithi060.stdout: create 254/39 file after the snapshot 2024-04-14T23:52:12.683 INFO:tasks.workunit.client.0.smithi060.stdout: create 255/.snap/snap-subdir-test 2024-04-14T23:52:12.720 INFO:tasks.workunit.client.0.smithi060.stdout: create 255/30 file after the snapshot 2024-04-14T23:52:12.721 INFO:tasks.workunit.client.0.smithi060.stdout: create 255/31 file after the snapshot 2024-04-14T23:52:12.724 INFO:tasks.workunit.client.0.smithi060.stdout: create 255/32 file after the snapshot 2024-04-14T23:52:12.726 INFO:tasks.workunit.client.0.smithi060.stdout: create 255/33 file after the snapshot 2024-04-14T23:52:12.727 INFO:tasks.workunit.client.0.smithi060.stdout: create 255/34 file after the snapshot 2024-04-14T23:52:12.729 INFO:tasks.workunit.client.0.smithi060.stdout: create 255/35 file after the snapshot 2024-04-14T23:52:12.731 INFO:tasks.workunit.client.0.smithi060.stdout: create 255/36 file after the snapshot 2024-04-14T23:52:12.733 INFO:tasks.workunit.client.0.smithi060.stdout: create 255/37 file after the snapshot 2024-04-14T23:52:12.735 INFO:tasks.workunit.client.0.smithi060.stdout: create 255/38 file after the snapshot 2024-04-14T23:52:12.738 INFO:tasks.workunit.client.0.smithi060.stdout: create 255/39 file after the snapshot 2024-04-14T23:52:12.740 INFO:tasks.workunit.client.0.smithi060.stdout: create 256/.snap/snap-subdir-test 2024-04-14T23:52:12.785 INFO:tasks.workunit.client.0.smithi060.stdout: create 256/30 file after the snapshot 2024-04-14T23:52:12.788 INFO:tasks.workunit.client.0.smithi060.stdout: create 256/31 file after the snapshot 2024-04-14T23:52:12.790 INFO:tasks.workunit.client.0.smithi060.stdout: create 256/32 file after the snapshot 2024-04-14T23:52:12.793 INFO:tasks.workunit.client.0.smithi060.stdout: create 256/33 file after the snapshot 2024-04-14T23:52:12.796 INFO:tasks.workunit.client.0.smithi060.stdout: create 256/34 file after the snapshot 2024-04-14T23:52:12.799 INFO:tasks.workunit.client.0.smithi060.stdout: create 256/35 file after the snapshot 2024-04-14T23:52:12.802 INFO:tasks.workunit.client.0.smithi060.stdout: create 256/36 file after the snapshot 2024-04-14T23:52:12.804 INFO:tasks.workunit.client.0.smithi060.stdout: create 256/37 file after the snapshot 2024-04-14T23:52:12.807 INFO:tasks.workunit.client.0.smithi060.stdout: create 256/38 file after the snapshot 2024-04-14T23:52:12.810 INFO:tasks.workunit.client.0.smithi060.stdout: create 256/39 file after the snapshot 2024-04-14T23:52:12.813 INFO:tasks.workunit.client.0.smithi060.stdout: create 257/.snap/snap-subdir-test 2024-04-14T23:52:12.858 INFO:tasks.workunit.client.0.smithi060.stdout: create 257/30 file after the snapshot 2024-04-14T23:52:12.861 INFO:tasks.workunit.client.0.smithi060.stdout: create 257/31 file after the snapshot 2024-04-14T23:52:12.864 INFO:tasks.workunit.client.0.smithi060.stdout: create 257/32 file after the snapshot 2024-04-14T23:52:12.867 INFO:tasks.workunit.client.0.smithi060.stdout: create 257/33 file after the snapshot 2024-04-14T23:52:12.870 INFO:tasks.workunit.client.0.smithi060.stdout: create 257/34 file after the snapshot 2024-04-14T23:52:12.873 INFO:tasks.workunit.client.0.smithi060.stdout: create 257/35 file after the snapshot 2024-04-14T23:52:12.876 INFO:tasks.workunit.client.0.smithi060.stdout: create 257/36 file after the snapshot 2024-04-14T23:52:12.878 INFO:tasks.workunit.client.0.smithi060.stdout: create 257/37 file after the snapshot 2024-04-14T23:52:12.881 INFO:tasks.workunit.client.0.smithi060.stdout: create 257/38 file after the snapshot 2024-04-14T23:52:12.884 INFO:tasks.workunit.client.0.smithi060.stdout: create 257/39 file after the snapshot 2024-04-14T23:52:12.886 INFO:tasks.workunit.client.0.smithi060.stdout: create 258/.snap/snap-subdir-test 2024-04-14T23:52:12.945 INFO:tasks.workunit.client.0.smithi060.stdout: create 258/30 file after the snapshot 2024-04-14T23:52:12.947 INFO:tasks.workunit.client.0.smithi060.stdout: create 258/31 file after the snapshot 2024-04-14T23:52:12.950 INFO:tasks.workunit.client.0.smithi060.stdout: create 258/32 file after the snapshot 2024-04-14T23:52:12.952 INFO:tasks.workunit.client.0.smithi060.stdout: create 258/33 file after the snapshot 2024-04-14T23:52:12.955 INFO:tasks.workunit.client.0.smithi060.stdout: create 258/34 file after the snapshot 2024-04-14T23:52:12.958 INFO:tasks.workunit.client.0.smithi060.stdout: create 258/35 file after the snapshot 2024-04-14T23:52:12.960 INFO:tasks.workunit.client.0.smithi060.stdout: create 258/36 file after the snapshot 2024-04-14T23:52:12.963 INFO:tasks.workunit.client.0.smithi060.stdout: create 258/37 file after the snapshot 2024-04-14T23:52:12.966 INFO:tasks.workunit.client.0.smithi060.stdout: create 258/38 file after the snapshot 2024-04-14T23:52:12.968 INFO:tasks.workunit.client.0.smithi060.stdout: create 258/39 file after the snapshot 2024-04-14T23:52:12.971 INFO:tasks.workunit.client.0.smithi060.stdout: create 259/.snap/snap-subdir-test 2024-04-14T23:52:13.029 INFO:tasks.workunit.client.0.smithi060.stdout: create 259/30 file after the snapshot 2024-04-14T23:52:13.030 INFO:tasks.workunit.client.0.smithi060.stdout: create 259/31 file after the snapshot 2024-04-14T23:52:13.033 INFO:tasks.workunit.client.0.smithi060.stdout: create 259/32 file after the snapshot 2024-04-14T23:52:13.035 INFO:tasks.workunit.client.0.smithi060.stdout: create 259/33 file after the snapshot 2024-04-14T23:52:13.038 INFO:tasks.workunit.client.0.smithi060.stdout: create 259/34 file after the snapshot 2024-04-14T23:52:13.042 INFO:tasks.workunit.client.0.smithi060.stdout: create 259/35 file after the snapshot 2024-04-14T23:52:13.044 INFO:tasks.workunit.client.0.smithi060.stdout: create 259/36 file after the snapshot 2024-04-14T23:52:13.046 INFO:tasks.workunit.client.0.smithi060.stdout: create 259/37 file after the snapshot 2024-04-14T23:52:13.049 INFO:tasks.workunit.client.0.smithi060.stdout: create 259/38 file after the snapshot 2024-04-14T23:52:13.052 INFO:tasks.workunit.client.0.smithi060.stdout: create 259/39 file after the snapshot 2024-04-14T23:52:13.055 INFO:tasks.workunit.client.0.smithi060.stdout: create 260/.snap/snap-subdir-test 2024-04-14T23:52:13.112 INFO:tasks.workunit.client.0.smithi060.stdout: create 260/30 file after the snapshot 2024-04-14T23:52:13.115 INFO:tasks.workunit.client.0.smithi060.stdout: create 260/31 file after the snapshot 2024-04-14T23:52:13.117 INFO:tasks.workunit.client.0.smithi060.stdout: create 260/32 file after the snapshot 2024-04-14T23:52:13.120 INFO:tasks.workunit.client.0.smithi060.stdout: create 260/33 file after the snapshot 2024-04-14T23:52:13.123 INFO:tasks.workunit.client.0.smithi060.stdout: create 260/34 file after the snapshot 2024-04-14T23:52:13.126 INFO:tasks.workunit.client.0.smithi060.stdout: create 260/35 file after the snapshot 2024-04-14T23:52:13.128 INFO:tasks.workunit.client.0.smithi060.stdout: create 260/36 file after the snapshot 2024-04-14T23:52:13.131 INFO:tasks.workunit.client.0.smithi060.stdout: create 260/37 file after the snapshot 2024-04-14T23:52:13.134 INFO:tasks.workunit.client.0.smithi060.stdout: create 260/38 file after the snapshot 2024-04-14T23:52:13.137 INFO:tasks.workunit.client.0.smithi060.stdout: create 260/39 file after the snapshot 2024-04-14T23:52:13.140 INFO:tasks.workunit.client.0.smithi060.stdout: create 261/.snap/snap-subdir-test 2024-04-14T23:52:13.197 INFO:tasks.workunit.client.0.smithi060.stdout: create 261/30 file after the snapshot 2024-04-14T23:52:13.199 INFO:tasks.workunit.client.0.smithi060.stdout: create 261/31 file after the snapshot 2024-04-14T23:52:13.202 INFO:tasks.workunit.client.0.smithi060.stdout: create 261/32 file after the snapshot 2024-04-14T23:52:13.205 INFO:tasks.workunit.client.0.smithi060.stdout: create 261/33 file after the snapshot 2024-04-14T23:52:13.208 INFO:tasks.workunit.client.0.smithi060.stdout: create 261/34 file after the snapshot 2024-04-14T23:52:13.210 INFO:tasks.workunit.client.0.smithi060.stdout: create 261/35 file after the snapshot 2024-04-14T23:52:13.213 INFO:tasks.workunit.client.0.smithi060.stdout: create 261/36 file after the snapshot 2024-04-14T23:52:13.216 INFO:tasks.workunit.client.0.smithi060.stdout: create 261/37 file after the snapshot 2024-04-14T23:52:13.219 INFO:tasks.workunit.client.0.smithi060.stdout: create 261/38 file after the snapshot 2024-04-14T23:52:13.222 INFO:tasks.workunit.client.0.smithi060.stdout: create 261/39 file after the snapshot 2024-04-14T23:52:13.224 INFO:tasks.workunit.client.0.smithi060.stdout: create 262/.snap/snap-subdir-test 2024-04-14T23:52:13.281 INFO:tasks.workunit.client.0.smithi060.stdout: create 262/30 file after the snapshot 2024-04-14T23:52:13.284 INFO:tasks.workunit.client.0.smithi060.stdout: create 262/31 file after the snapshot 2024-04-14T23:52:13.287 INFO:tasks.workunit.client.0.smithi060.stdout: create 262/32 file after the snapshot 2024-04-14T23:52:13.290 INFO:tasks.workunit.client.0.smithi060.stdout: create 262/33 file after the snapshot 2024-04-14T23:52:13.292 INFO:tasks.workunit.client.0.smithi060.stdout: create 262/34 file after the snapshot 2024-04-14T23:52:13.295 INFO:tasks.workunit.client.0.smithi060.stdout: create 262/35 file after the snapshot 2024-04-14T23:52:13.299 INFO:tasks.workunit.client.0.smithi060.stdout: create 262/36 file after the snapshot 2024-04-14T23:52:13.302 INFO:tasks.workunit.client.0.smithi060.stdout: create 262/37 file after the snapshot 2024-04-14T23:52:13.305 INFO:tasks.workunit.client.0.smithi060.stdout: create 262/38 file after the snapshot 2024-04-14T23:52:13.308 INFO:tasks.workunit.client.0.smithi060.stdout: create 262/39 file after the snapshot 2024-04-14T23:52:13.311 INFO:tasks.workunit.client.0.smithi060.stdout: create 263/.snap/snap-subdir-test 2024-04-14T23:52:13.362 INFO:tasks.workunit.client.0.smithi060.stdout: create 263/30 file after the snapshot 2024-04-14T23:52:13.365 INFO:tasks.workunit.client.0.smithi060.stdout: create 263/31 file after the snapshot 2024-04-14T23:52:13.368 INFO:tasks.workunit.client.0.smithi060.stdout: create 263/32 file after the snapshot 2024-04-14T23:52:13.371 INFO:tasks.workunit.client.0.smithi060.stdout: create 263/33 file after the snapshot 2024-04-14T23:52:13.374 INFO:tasks.workunit.client.0.smithi060.stdout: create 263/34 file after the snapshot 2024-04-14T23:52:13.376 INFO:tasks.workunit.client.0.smithi060.stdout: create 263/35 file after the snapshot 2024-04-14T23:52:13.379 INFO:tasks.workunit.client.0.smithi060.stdout: create 263/36 file after the snapshot 2024-04-14T23:52:13.382 INFO:tasks.workunit.client.0.smithi060.stdout: create 263/37 file after the snapshot 2024-04-14T23:52:13.385 INFO:tasks.workunit.client.0.smithi060.stdout: create 263/38 file after the snapshot 2024-04-14T23:52:13.388 INFO:tasks.workunit.client.0.smithi060.stdout: create 263/39 file after the snapshot 2024-04-14T23:52:13.391 INFO:tasks.workunit.client.0.smithi060.stdout: create 264/.snap/snap-subdir-test 2024-04-14T23:52:13.439 INFO:tasks.workunit.client.0.smithi060.stdout: create 264/30 file after the snapshot 2024-04-14T23:52:13.441 INFO:tasks.workunit.client.0.smithi060.stdout: create 264/31 file after the snapshot 2024-04-14T23:52:13.444 INFO:tasks.workunit.client.0.smithi060.stdout: create 264/32 file after the snapshot 2024-04-14T23:52:13.446 INFO:tasks.workunit.client.0.smithi060.stdout: create 264/33 file after the snapshot 2024-04-14T23:52:13.447 INFO:tasks.workunit.client.0.smithi060.stdout: create 264/34 file after the snapshot 2024-04-14T23:52:13.449 INFO:tasks.workunit.client.0.smithi060.stdout: create 264/35 file after the snapshot 2024-04-14T23:52:13.451 INFO:tasks.workunit.client.0.smithi060.stdout: create 264/36 file after the snapshot 2024-04-14T23:52:13.453 INFO:tasks.workunit.client.0.smithi060.stdout: create 264/37 file after the snapshot 2024-04-14T23:52:13.456 INFO:tasks.workunit.client.0.smithi060.stdout: create 264/38 file after the snapshot 2024-04-14T23:52:13.459 INFO:tasks.workunit.client.0.smithi060.stdout: create 264/39 file after the snapshot 2024-04-14T23:52:13.462 INFO:tasks.workunit.client.0.smithi060.stdout: create 265/.snap/snap-subdir-test 2024-04-14T23:52:13.518 INFO:tasks.workunit.client.0.smithi060.stdout: create 265/30 file after the snapshot 2024-04-14T23:52:13.520 INFO:tasks.workunit.client.0.smithi060.stdout: create 265/31 file after the snapshot 2024-04-14T23:52:13.523 INFO:tasks.workunit.client.0.smithi060.stdout: create 265/32 file after the snapshot 2024-04-14T23:52:13.526 INFO:tasks.workunit.client.0.smithi060.stdout: create 265/33 file after the snapshot 2024-04-14T23:52:13.529 INFO:tasks.workunit.client.0.smithi060.stdout: create 265/34 file after the snapshot 2024-04-14T23:52:13.531 INFO:tasks.workunit.client.0.smithi060.stdout: create 265/35 file after the snapshot 2024-04-14T23:52:13.534 INFO:tasks.workunit.client.0.smithi060.stdout: create 265/36 file after the snapshot 2024-04-14T23:52:13.537 INFO:tasks.workunit.client.0.smithi060.stdout: create 265/37 file after the snapshot 2024-04-14T23:52:13.540 INFO:tasks.workunit.client.0.smithi060.stdout: create 265/38 file after the snapshot 2024-04-14T23:52:13.542 INFO:tasks.workunit.client.0.smithi060.stdout: create 265/39 file after the snapshot 2024-04-14T23:52:13.545 INFO:tasks.workunit.client.0.smithi060.stdout: create 266/.snap/snap-subdir-test 2024-04-14T23:52:13.597 INFO:tasks.workunit.client.0.smithi060.stdout: create 266/30 file after the snapshot 2024-04-14T23:52:13.600 INFO:tasks.workunit.client.0.smithi060.stdout: create 266/31 file after the snapshot 2024-04-14T23:52:13.603 INFO:tasks.workunit.client.0.smithi060.stdout: create 266/32 file after the snapshot 2024-04-14T23:52:13.605 INFO:tasks.workunit.client.0.smithi060.stdout: create 266/33 file after the snapshot 2024-04-14T23:52:13.608 INFO:tasks.workunit.client.0.smithi060.stdout: create 266/34 file after the snapshot 2024-04-14T23:52:13.611 INFO:tasks.workunit.client.0.smithi060.stdout: create 266/35 file after the snapshot 2024-04-14T23:52:13.614 INFO:tasks.workunit.client.0.smithi060.stdout: create 266/36 file after the snapshot 2024-04-14T23:52:13.616 INFO:tasks.workunit.client.0.smithi060.stdout: create 266/37 file after the snapshot 2024-04-14T23:52:13.619 INFO:tasks.workunit.client.0.smithi060.stdout: create 266/38 file after the snapshot 2024-04-14T23:52:13.622 INFO:tasks.workunit.client.0.smithi060.stdout: create 266/39 file after the snapshot 2024-04-14T23:52:13.828 INFO:tasks.workunit.client.0.smithi060.stdout: create 267/.snap/snap-subdir-test 2024-04-14T23:52:14.628 INFO:tasks.workunit.client.0.smithi060.stdout: create 267/30 file after the snapshot 2024-04-14T23:52:14.632 INFO:tasks.workunit.client.0.smithi060.stdout: create 267/31 file after the snapshot 2024-04-14T23:52:14.636 INFO:tasks.workunit.client.0.smithi060.stdout: create 267/32 file after the snapshot 2024-04-14T23:52:14.640 INFO:tasks.workunit.client.0.smithi060.stdout: create 267/33 file after the snapshot 2024-04-14T23:52:14.644 INFO:tasks.workunit.client.0.smithi060.stdout: create 267/34 file after the snapshot 2024-04-14T23:52:14.648 INFO:tasks.workunit.client.0.smithi060.stdout: create 267/35 file after the snapshot 2024-04-14T23:52:14.653 INFO:tasks.workunit.client.0.smithi060.stdout: create 267/36 file after the snapshot 2024-04-14T23:52:14.657 INFO:tasks.workunit.client.0.smithi060.stdout: create 267/37 file after the snapshot 2024-04-14T23:52:14.661 INFO:tasks.workunit.client.0.smithi060.stdout: create 267/38 file after the snapshot 2024-04-14T23:52:14.665 INFO:tasks.workunit.client.0.smithi060.stdout: create 267/39 file after the snapshot 2024-04-14T23:52:14.670 INFO:tasks.workunit.client.0.smithi060.stdout: create 268/.snap/snap-subdir-test 2024-04-14T23:52:14.711 INFO:tasks.workunit.client.0.smithi060.stdout: create 268/30 file after the snapshot 2024-04-14T23:52:14.713 INFO:tasks.workunit.client.0.smithi060.stdout: create 268/31 file after the snapshot 2024-04-14T23:52:14.715 INFO:tasks.workunit.client.0.smithi060.stdout: create 268/32 file after the snapshot 2024-04-14T23:52:14.718 INFO:tasks.workunit.client.0.smithi060.stdout: create 268/33 file after the snapshot 2024-04-14T23:52:14.720 INFO:tasks.workunit.client.0.smithi060.stdout: create 268/34 file after the snapshot 2024-04-14T23:52:14.722 INFO:tasks.workunit.client.0.smithi060.stdout: create 268/35 file after the snapshot 2024-04-14T23:52:14.724 INFO:tasks.workunit.client.0.smithi060.stdout: create 268/36 file after the snapshot 2024-04-14T23:52:14.726 INFO:tasks.workunit.client.0.smithi060.stdout: create 268/37 file after the snapshot 2024-04-14T23:52:14.728 INFO:tasks.workunit.client.0.smithi060.stdout: create 268/38 file after the snapshot 2024-04-14T23:52:14.730 INFO:tasks.workunit.client.0.smithi060.stdout: create 268/39 file after the snapshot 2024-04-14T23:52:14.732 INFO:tasks.workunit.client.0.smithi060.stdout: create 269/.snap/snap-subdir-test 2024-04-14T23:52:14.772 INFO:tasks.workunit.client.0.smithi060.stdout: create 269/30 file after the snapshot 2024-04-14T23:52:14.774 INFO:tasks.workunit.client.0.smithi060.stdout: create 269/31 file after the snapshot 2024-04-14T23:52:14.777 INFO:tasks.workunit.client.0.smithi060.stdout: create 269/32 file after the snapshot 2024-04-14T23:52:14.780 INFO:tasks.workunit.client.0.smithi060.stdout: create 269/33 file after the snapshot 2024-04-14T23:52:14.783 INFO:tasks.workunit.client.0.smithi060.stdout: create 269/34 file after the snapshot 2024-04-14T23:52:14.785 INFO:tasks.workunit.client.0.smithi060.stdout: create 269/35 file after the snapshot 2024-04-14T23:52:14.788 INFO:tasks.workunit.client.0.smithi060.stdout: create 269/36 file after the snapshot 2024-04-14T23:52:14.791 INFO:tasks.workunit.client.0.smithi060.stdout: create 269/37 file after the snapshot 2024-04-14T23:52:14.794 INFO:tasks.workunit.client.0.smithi060.stdout: create 269/38 file after the snapshot 2024-04-14T23:52:14.796 INFO:tasks.workunit.client.0.smithi060.stdout: create 269/39 file after the snapshot 2024-04-14T23:52:14.799 INFO:tasks.workunit.client.0.smithi060.stdout: create 270/.snap/snap-subdir-test 2024-04-14T23:52:14.815 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-14T23:52:14.826 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-14T23:52:14.841 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:52:14.845 INFO:tasks.workunit.client.0.smithi060.stdout: create 270/30 file after the snapshot 2024-04-14T23:52:14.850 INFO:tasks.workunit.client.0.smithi060.stdout: create 270/31 file after the snapshot 2024-04-14T23:52:14.850 INFO:tasks.workunit.client.0.smithi060.stdout: create 270/32 file after the snapshot 2024-04-14T23:52:14.850 INFO:tasks.workunit.client.0.smithi060.stdout: create 270/33 file after the snapshot 2024-04-14T23:52:14.850 INFO:tasks.workunit.client.0.smithi060.stdout: create 270/34 file after the snapshot 2024-04-14T23:52:14.851 INFO:tasks.workunit.client.0.smithi060.stdout: create 270/35 file after the snapshot 2024-04-14T23:52:14.853 INFO:tasks.workunit.client.0.smithi060.stdout: create 270/36 file after the snapshot 2024-04-14T23:52:14.855 INFO:tasks.workunit.client.0.smithi060.stdout: create 270/37 file after the snapshot 2024-04-14T23:52:14.856 INFO:tasks.workunit.client.0.smithi060.stdout: create 270/38 file after the snapshot 2024-04-14T23:52:14.858 INFO:tasks.workunit.client.0.smithi060.stdout: create 270/39 file after the snapshot 2024-04-14T23:52:14.860 INFO:tasks.workunit.client.0.smithi060.stdout: create 271/.snap/snap-subdir-test 2024-04-14T23:52:14.893 INFO:tasks.workunit.client.0.smithi060.stdout: create 271/30 file after the snapshot 2024-04-14T23:52:14.895 INFO:tasks.workunit.client.0.smithi060.stdout: create 271/31 file after the snapshot 2024-04-14T23:52:14.896 INFO:tasks.workunit.client.0.smithi060.stdout: create 271/32 file after the snapshot 2024-04-14T23:52:14.898 INFO:tasks.workunit.client.0.smithi060.stdout: create 271/33 file after the snapshot 2024-04-14T23:52:14.899 INFO:tasks.workunit.client.0.smithi060.stdout: create 271/34 file after the snapshot 2024-04-14T23:52:14.900 INFO:tasks.workunit.client.0.smithi060.stdout: create 271/35 file after the snapshot 2024-04-14T23:52:14.901 INFO:tasks.workunit.client.0.smithi060.stdout: create 271/36 file after the snapshot 2024-04-14T23:52:14.903 INFO:tasks.workunit.client.0.smithi060.stdout: create 271/37 file after the snapshot 2024-04-14T23:52:14.904 INFO:tasks.workunit.client.0.smithi060.stdout: create 271/38 file after the snapshot 2024-04-14T23:52:15.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.037+0000 7f477ce00640 1 Processor -- start 2024-04-14T23:52:15.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.037+0000 7f477ce00640 1 -- start start 2024-04-14T23:52:15.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.039+0000 7f477ce00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f47780731e0 0x7f47781664b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:15.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.039+0000 7f477ce00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f477815b0d0 con 0x7f477806f5d0 2024-04-14T23:52:15.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.039+0000 7f477ce00640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f477815b370 con 0x7f477806f1f0 2024-04-14T23:52:15.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.039+0000 7f477ce00640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f47780c6e10 con 0x7f47780731e0 2024-04-14T23:52:15.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.039+0000 7f4775400640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f477806f5d0 0x7f4778154980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:40122/0 (socket says 172.21.15.60:40122) 2024-04-14T23:52:15.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.040+0000 7f4775400640 1 -- 172.21.15.60:0/2758375512 learned_addr learned my addr 172.21.15.60:0/2758375512 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:52:15.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.040+0000 7f4776800640 1 --2- 172.21.15.60:0/2758375512 >> v2:172.21.15.60:3300/0 conn(0x7f47780731e0 0x7f47781664b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:15.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.040+0000 7f4774a00640 1 -- 172.21.15.60:0/2758375512 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3029676067 0 0) 0x7f477815b370 con 0x7f477806f1f0 2024-04-14T23:52:15.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.040+0000 7f4774a00640 1 -- 172.21.15.60:0/2758375512 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f475c003520 con 0x7f477806f1f0 2024-04-14T23:52:15.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.040+0000 7f4774a00640 1 -- 172.21.15.60:0/2758375512 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1016353728 0 0) 0x7f477815b0d0 con 0x7f477806f5d0 2024-04-14T23:52:15.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.040+0000 7f4774a00640 1 -- 172.21.15.60:0/2758375512 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f477815b370 con 0x7f477806f5d0 2024-04-14T23:52:15.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.040+0000 7f4776800640 1 -- 172.21.15.60:0/2758375512 >> v1:172.21.15.186:6789/0 conn(0x7f477806f1f0 legacy=0x7f47780c72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:15.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.040+0000 7f4776800640 1 -- 172.21.15.60:0/2758375512 >> v1:172.21.15.60:6789/0 conn(0x7f477806f5d0 legacy=0x7f4778154980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:15.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.040+0000 7f4776800640 1 -- 172.21.15.60:0/2758375512 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f477815b650 con 0x7f47780731e0 2024-04-14T23:52:15.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.041+0000 7f4776800640 1 --2- 172.21.15.60:0/2758375512 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f47780731e0 0x7f47781664b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f4770009d90 tx=0x7f477002bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de6f46db3d70429a server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:52:15.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.042+0000 7f4774a00640 1 -- 172.21.15.60:0/2758375512 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4770039070 con 0x7f47780731e0 2024-04-14T23:52:15.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.042+0000 7f4774a00640 1 -- 172.21.15.60:0/2758375512 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f477002cd80 con 0x7f47780731e0 2024-04-14T23:52:15.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.042+0000 7f4774a00640 1 -- 172.21.15.60:0/2758375512 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4770034660 con 0x7f47780731e0 2024-04-14T23:52:15.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.042+0000 7f477ce00640 1 -- 172.21.15.60:0/2758375512 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f47780731e0 msgr2=0x7f47781664b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:15.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.042+0000 7f477ce00640 1 --2- 172.21.15.60:0/2758375512 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f47780731e0 0x7f47781664b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f4770009d90 tx=0x7f477002bfd0 comp rx=0 tx=0).stop 2024-04-14T23:52:15.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.042+0000 7f477ce00640 1 -- 172.21.15.60:0/2758375512 shutdown_connections 2024-04-14T23:52:15.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.042+0000 7f477ce00640 1 --2- 172.21.15.60:0/2758375512 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f47780731e0 0x7f47781664b0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:15.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.042+0000 7f477ce00640 1 -- 172.21.15.60:0/2758375512 >> 172.21.15.60:0/2758375512 conn(0x7f47780cc170 msgr2=0x7f47780cc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:52:15.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.043+0000 7f477ce00640 1 -- 172.21.15.60:0/2758375512 shutdown_connections 2024-04-14T23:52:15.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.043+0000 7f477ce00640 1 -- 172.21.15.60:0/2758375512 wait complete. 2024-04-14T23:52:15.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.044+0000 7f477ce00640 1 Processor -- start 2024-04-14T23:52:15.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.045+0000 7f477ce00640 1 -- start start 2024-04-14T23:52:15.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.045+0000 7f477ce00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f477806f1f0 0x7f47780cb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:15.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.045+0000 7f477ce00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f477806f5d0 0x7f477814efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:15.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.046+0000 7f477ce00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f47780731e0 0x7f47780c57e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:15.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.046+0000 7f477ce00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f47780c79b0 con 0x7f47780731e0 2024-04-14T23:52:15.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.046+0000 7f477ce00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4778071180 con 0x7f477806f1f0 2024-04-14T23:52:15.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.046+0000 7f477ce00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f47780c7b30 con 0x7f477806f5d0 2024-04-14T23:52:15.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.046+0000 7f4776800640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f47780731e0 0x7f47780c57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:15.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.046+0000 7f4776800640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f47780731e0 0x7f47780c57e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53166/0 (socket says 172.21.15.60:53166) 2024-04-14T23:52:15.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.046+0000 7f4776800640 1 -- 172.21.15.60:0/408907714 learned_addr learned my addr 172.21.15.60:0/408907714 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:52:15.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.046+0000 7f4775e00640 1 --2- 172.21.15.60:0/408907714 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f477806f1f0 0x7f47780cb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:15.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.046+0000 7f4775400640 1 --2- 172.21.15.60:0/408907714 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f477806f5d0 0x7f477814efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:15.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.046+0000 7f4776800640 1 -- 172.21.15.60:0/408907714 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f477806f5d0 msgr2=0x7f477814efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:15.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.046+0000 7f4776800640 1 --2- 172.21.15.60:0/408907714 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f477806f5d0 0x7f477814efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:15.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.046+0000 7f4776800640 1 -- 172.21.15.60:0/408907714 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f477806f1f0 msgr2=0x7f47780cb4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:15.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.046+0000 7f4776800640 1 --2- 172.21.15.60:0/408907714 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f477806f1f0 0x7f47780cb4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:15.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.046+0000 7f4776800640 1 -- 172.21.15.60:0/408907714 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47781f44f0 con 0x7f47780731e0 2024-04-14T23:52:15.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.046+0000 7f4775e00640 1 --2- 172.21.15.60:0/408907714 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f477806f1f0 0x7f47780cb4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:52:15.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.047+0000 7f4776800640 1 --2- 172.21.15.60:0/408907714 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f47780731e0 0x7f47780c57e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f4770009eb0 tx=0x7f4770004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:52:15.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.047+0000 7f476e000640 1 -- 172.21.15.60:0/408907714 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4770039050 con 0x7f47780731e0 2024-04-14T23:52:15.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.047+0000 7f476e000640 1 -- 172.21.15.60:0/408907714 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4770045070 con 0x7f47780731e0 2024-04-14T23:52:15.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.047+0000 7f476e000640 1 -- 172.21.15.60:0/408907714 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4770009450 con 0x7f47780731e0 2024-04-14T23:52:15.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.047+0000 7f477ce00640 1 -- 172.21.15.60:0/408907714 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47780c42d0 con 0x7f47780731e0 2024-04-14T23:52:15.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.047+0000 7f477ce00640 1 -- 172.21.15.60:0/408907714 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47781f0780 con 0x7f47780731e0 2024-04-14T23:52:15.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.049+0000 7f477ce00640 1 -- 172.21.15.60:0/408907714 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4778076970 con 0x7f47780731e0 2024-04-14T23:52:15.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.050+0000 7f476e000640 1 -- 172.21.15.60:0/408907714 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f477003e830 con 0x7f47780731e0 2024-04-14T23:52:15.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.050+0000 7f476e000640 1 --2- 172.21.15.60:0/408907714 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f475806b990 0x7f475806de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:15.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.051+0000 7f476e000640 1 -- 172.21.15.60:0/408907714 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 5911+0+0 (secure 0 0 0) 0x7f47700ad710 con 0x7f47780731e0 2024-04-14T23:52:15.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.051+0000 7f4775e00640 1 --2- 172.21.15.60:0/408907714 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f475806b990 0x7f475806de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:15.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.052+0000 7f4775e00640 1 --2- 172.21.15.60:0/408907714 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f475806b990 0x7f475806de50 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f47680011e0 tx=0x7f4768009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:52:15.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.056+0000 7f476e000640 1 -- 172.21.15.60:0/408907714 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f477007bcb0 con 0x7f47780731e0 2024-04-14T23:52:15.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.294+0000 7f477ce00640 1 -- 172.21.15.60:0/408907714 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f47781474f0 con 0x7f47780731e0 2024-04-14T23:52:15.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.295+0000 7f476e000640 1 -- 172.21.15.60:0/408907714 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6832 (secure 0 0 0) 0x7f477007fd50 con 0x7f47780731e0 2024-04-14T23:52:15.301 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:52:15.301 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4839,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/1714133698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1714133698},{"type":"v1","addr":"172.21.15.60:6835","nonce":1714133698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":14}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:51:24.313553+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4276},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4276":{"gid":4276,"name":"a","rank":0,"incarnation":9,"state":"up:active","state_seq":45,"addr":"172.21.15.60:6833/78303888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":78303888},{"type":"v1","addr":"172.21.15.60:6833","nonce":78303888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4276,"qdb_cluster":[4276]},"id":1}]} 2024-04-14T23:52:15.301 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 14 2024-04-14T23:52:15.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.298+0000 7f477ce00640 1 -- 172.21.15.60:0/408907714 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f475806b990 msgr2=0x7f475806de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:15.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.298+0000 7f477ce00640 1 --2- 172.21.15.60:0/408907714 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f475806b990 0x7f475806de50 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f47680011e0 tx=0x7f4768009040 comp rx=0 tx=0).stop 2024-04-14T23:52:15.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.298+0000 7f477ce00640 1 -- 172.21.15.60:0/408907714 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f47780731e0 msgr2=0x7f47780c57e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:15.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.298+0000 7f477ce00640 1 --2- 172.21.15.60:0/408907714 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f47780731e0 0x7f47780c57e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f4770009eb0 tx=0x7f4770004a40 comp rx=0 tx=0).stop 2024-04-14T23:52:15.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.298+0000 7f477ce00640 1 -- 172.21.15.60:0/408907714 shutdown_connections 2024-04-14T23:52:15.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.298+0000 7f477ce00640 1 --2- 172.21.15.60:0/408907714 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f475806b990 0x7f475806de50 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:15.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.298+0000 7f477ce00640 1 --2- 172.21.15.60:0/408907714 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f47780731e0 0x7f47780c57e0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:15.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.298+0000 7f477ce00640 1 --2- 172.21.15.60:0/408907714 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f477806f5d0 0x7f477814efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:15.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.298+0000 7f477ce00640 1 --2- 172.21.15.60:0/408907714 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f477806f1f0 0x7f47780cb4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:15.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.298+0000 7f477ce00640 1 -- 172.21.15.60:0/408907714 >> 172.21.15.60:0/408907714 conn(0x7f47780cc170 msgr2=0x7f477814a3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:52:15.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.298+0000 7f477ce00640 1 -- 172.21.15.60:0/408907714 shutdown_connections 2024-04-14T23:52:15.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.298+0000 7f477ce00640 1 -- 172.21.15.60:0/408907714 wait complete. 2024-04-14T23:52:15.321 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2024-04-14T23:52:15.321 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-14T23:52:15.321 INFO:teuthology.orchestra.run:waiting for 300 2024-04-14T23:52:15.347 INFO:tasks.ceph.mds.a:Stopped 2024-04-14T23:52:15.347 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:52:15.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.613+0000 7ff898e00640 1 Processor -- start 2024-04-14T23:52:15.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.613+0000 7ff898e00640 1 -- start start 2024-04-14T23:52:15.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.615+0000 7ff898e00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff8940c7080 0x7ff89406b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:15.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.615+0000 7ff898e00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff8940c6950 0x7ff89415b360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:15.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.615+0000 7ff898e00640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff89406b850 con 0x7ff8940c7460 2024-04-14T23:52:15.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.615+0000 7ff898e00640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7ff89406c740 con 0x7ff8940c7080 2024-04-14T23:52:15.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.615+0000 7ff898e00640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7ff89406bb30 con 0x7ff8940c6950 2024-04-14T23:52:15.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.615+0000 7ff893400640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff8940c7080 0x7ff89406b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:15.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.615+0000 7ff893400640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff8940c7080 0x7ff89406b1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53168/0 (socket says 172.21.15.60:53168) 2024-04-14T23:52:15.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.615+0000 7ff893400640 1 -- 172.21.15.60:0/192018177 learned_addr learned my addr 172.21.15.60:0/192018177 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:52:15.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.615+0000 7ff892a00640 1 --2- 172.21.15.60:0/192018177 >> v2:172.21.15.186:3300/0 conn(0x7ff8940c6950 0x7ff89415b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:15.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.615+0000 7ff893400640 1 -- 172.21.15.60:0/192018177 >> v2:172.21.15.186:3300/0 conn(0x7ff8940c6950 msgr2=0x7ff89415b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:52:15.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.615+0000 7ff893400640 1 --2- 172.21.15.60:0/192018177 >> v2:172.21.15.186:3300/0 conn(0x7ff8940c6950 0x7ff89415b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:15.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.615+0000 7ff893400640 1 -- 172.21.15.60:0/192018177 >> v1:172.21.15.186:6789/0 conn(0x7ff8940c7460 legacy=0x7ff894070d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:15.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.615+0000 7ff893400640 1 -- 172.21.15.60:0/192018177 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff89406be10 con 0x7ff8940c7080 2024-04-14T23:52:15.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.617+0000 7ff893400640 1 --2- 172.21.15.60:0/192018177 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8940c7080 0x7ff89406b1c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff8940733b0 tx=0x7ff88402ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e5a4d737bbeebdc server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:52:15.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.617+0000 7ff892000640 1 -- 172.21.15.60:0/192018177 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff88402f850 con 0x7ff8940c7080 2024-04-14T23:52:15.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.617+0000 7ff892000640 1 -- 172.21.15.60:0/192018177 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff88402f9f0 con 0x7ff8940c7080 2024-04-14T23:52:15.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.617+0000 7ff892000640 1 -- 172.21.15.60:0/192018177 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff88402fdc0 con 0x7ff8940c7080 2024-04-14T23:52:15.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.617+0000 7ff898e00640 1 -- 172.21.15.60:0/192018177 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8940c7080 msgr2=0x7ff89406b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:15.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.617+0000 7ff898e00640 1 --2- 172.21.15.60:0/192018177 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8940c7080 0x7ff89406b1c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff8940733b0 tx=0x7ff88402ba30 comp rx=0 tx=0).stop 2024-04-14T23:52:15.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.618+0000 7ff898e00640 1 -- 172.21.15.60:0/192018177 shutdown_connections 2024-04-14T23:52:15.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.618+0000 7ff898e00640 1 --2- 172.21.15.60:0/192018177 >> v2:172.21.15.186:3300/0 conn(0x7ff8940c6950 0x7ff89415b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:15.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.618+0000 7ff898e00640 1 --2- 172.21.15.60:0/192018177 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8940c7080 0x7ff89406b1c0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:15.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.618+0000 7ff898e00640 1 -- 172.21.15.60:0/192018177 >> 172.21.15.60:0/192018177 conn(0x7ff8940c8b50 msgr2=0x7ff8940c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:52:15.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.618+0000 7ff898e00640 1 -- 172.21.15.60:0/192018177 shutdown_connections 2024-04-14T23:52:15.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.618+0000 7ff898e00640 1 -- 172.21.15.60:0/192018177 wait complete. 2024-04-14T23:52:15.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.619+0000 7ff898e00640 1 Processor -- start 2024-04-14T23:52:15.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.620+0000 7ff898e00640 1 -- start start 2024-04-14T23:52:15.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.620+0000 7ff898e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8940c6950 0x7ff8941e68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:15.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.620+0000 7ff898e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff8940c7080 0x7ff8940c4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:15.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.620+0000 7ff898e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff8940c7460 0x7ff8941566b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:15.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.620+0000 7ff898e00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff894071460 con 0x7ff8940c6950 2024-04-14T23:52:15.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.621+0000 7ff898e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff89406c3d0 con 0x7ff8940c7080 2024-04-14T23:52:15.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.621+0000 7ff898e00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7ff8940715e0 con 0x7ff8940c7460 2024-04-14T23:52:15.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.621+0000 7ff893400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8940c6950 0x7ff8941e68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:15.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.621+0000 7ff893400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8940c6950 0x7ff8941e68d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53172/0 (socket says 172.21.15.60:53172) 2024-04-14T23:52:15.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.621+0000 7ff893400640 1 -- 172.21.15.60:0/2025795841 learned_addr learned my addr 172.21.15.60:0/2025795841 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:52:15.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.621+0000 7ff892a00640 1 --2- 172.21.15.60:0/2025795841 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff8940c7080 0x7ff8940c4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:15.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.621+0000 7ff893e00640 1 --2- 172.21.15.60:0/2025795841 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff8940c7460 0x7ff8941566b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:15.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.621+0000 7ff892a00640 1 -- 172.21.15.60:0/2025795841 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff8940c7460 msgr2=0x7ff8941566b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:15.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.621+0000 7ff892a00640 1 --2- 172.21.15.60:0/2025795841 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff8940c7460 0x7ff8941566b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:15.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.621+0000 7ff892a00640 1 -- 172.21.15.60:0/2025795841 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8940c6950 msgr2=0x7ff8941e68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:15.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.621+0000 7ff892a00640 1 --2- 172.21.15.60:0/2025795841 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8940c6950 0x7ff8941e68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:15.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.621+0000 7ff892a00640 1 -- 172.21.15.60:0/2025795841 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8941e6e10 con 0x7ff8940c7080 2024-04-14T23:52:15.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.621+0000 7ff893e00640 1 --2- 172.21.15.60:0/2025795841 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff8940c7460 0x7ff8941566b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:52:15.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.621+0000 7ff893400640 1 --2- 172.21.15.60:0/2025795841 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8940c6950 0x7ff8941e68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:52:15.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.622+0000 7ff892a00640 1 --2- 172.21.15.60:0/2025795841 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff8940c7080 0x7ff8940c4180 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ff88000e880 tx=0x7ff88000ed40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:52:15.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.622+0000 7ff88b400640 1 -- 172.21.15.60:0/2025795841 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff880017070 con 0x7ff8940c7080 2024-04-14T23:52:15.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.622+0000 7ff88b400640 1 -- 172.21.15.60:0/2025795841 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff88000c940 con 0x7ff8940c7080 2024-04-14T23:52:15.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.622+0000 7ff88b400640 1 -- 172.21.15.60:0/2025795841 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff88000cd30 con 0x7ff8940c7080 2024-04-14T23:52:15.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.622+0000 7ff898e00640 1 -- 172.21.15.60:0/2025795841 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8941e4e00 con 0x7ff8940c7080 2024-04-14T23:52:15.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.622+0000 7ff898e00640 1 -- 172.21.15.60:0/2025795841 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8941e6080 con 0x7ff8940c7080 2024-04-14T23:52:15.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.623+0000 7ff898e00640 1 -- 172.21.15.60:0/2025795841 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff89404f540 con 0x7ff8940c7080 2024-04-14T23:52:15.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.624+0000 7ff88b400640 1 -- 172.21.15.60:0/2025795841 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7ff88001a070 con 0x7ff8940c7080 2024-04-14T23:52:15.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.625+0000 7ff88b400640 1 --2- 172.21.15.60:0/2025795841 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff87406b990 0x7ff87406de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:15.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.625+0000 7ff88b400640 1 -- 172.21.15.60:0/2025795841 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 5911+0+0 (secure 0 0 0) 0x7ff88008a920 con 0x7ff8940c7080 2024-04-14T23:52:15.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.625+0000 7ff893400640 1 --2- 172.21.15.60:0/2025795841 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff87406b990 0x7ff87406de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:15.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.626+0000 7ff893400640 1 --2- 172.21.15.60:0/2025795841 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff87406b990 0x7ff87406de50 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff884009630 tx=0x7ff884008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:52:15.635 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.629+0000 7ff88b400640 1 -- 172.21.15.60:0/2025795841 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff8941e6e10 con 0x7ff8940c7080 2024-04-14T23:52:15.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.875+0000 7ff898e00640 1 -- 172.21.15.60:0/2025795841 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff894159780 con 0x7ff8940c7080 2024-04-14T23:52:15.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.876+0000 7ff88b400640 1 -- 172.21.15.60:0/2025795841 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6832 (secure 0 0 0) 0x7ff880058ce0 con 0x7ff8940c7080 2024-04-14T23:52:15.882 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:52:15.882 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4839,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/1714133698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1714133698},{"type":"v1","addr":"172.21.15.60:6835","nonce":1714133698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":14}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:51:24.313553+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4276},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4276":{"gid":4276,"name":"a","rank":0,"incarnation":9,"state":"up:active","state_seq":45,"addr":"172.21.15.60:6833/78303888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":78303888},{"type":"v1","addr":"172.21.15.60:6833","nonce":78303888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4276,"qdb_cluster":[4276]},"id":1}]} 2024-04-14T23:52:15.883 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 14 2024-04-14T23:52:15.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.879+0000 7ff898e00640 1 -- 172.21.15.60:0/2025795841 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff87406b990 msgr2=0x7ff87406de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:15.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.879+0000 7ff898e00640 1 --2- 172.21.15.60:0/2025795841 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff87406b990 0x7ff87406de50 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff884009630 tx=0x7ff884008030 comp rx=0 tx=0).stop 2024-04-14T23:52:15.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.879+0000 7ff898e00640 1 -- 172.21.15.60:0/2025795841 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff8940c7080 msgr2=0x7ff8940c4180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:15.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.879+0000 7ff898e00640 1 --2- 172.21.15.60:0/2025795841 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff8940c7080 0x7ff8940c4180 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ff88000e880 tx=0x7ff88000ed40 comp rx=0 tx=0).stop 2024-04-14T23:52:15.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.879+0000 7ff898e00640 1 -- 172.21.15.60:0/2025795841 shutdown_connections 2024-04-14T23:52:15.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.879+0000 7ff898e00640 1 --2- 172.21.15.60:0/2025795841 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff87406b990 0x7ff87406de50 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:15.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.879+0000 7ff898e00640 1 --2- 172.21.15.60:0/2025795841 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff8940c7460 0x7ff8941566b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:15.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.879+0000 7ff898e00640 1 --2- 172.21.15.60:0/2025795841 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff8940c7080 0x7ff8940c4180 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:15.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.879+0000 7ff898e00640 1 --2- 172.21.15.60:0/2025795841 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8940c6950 0x7ff8941e68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:15.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.879+0000 7ff898e00640 1 -- 172.21.15.60:0/2025795841 >> 172.21.15.60:0/2025795841 conn(0x7ff8940c8b50 msgr2=0x7ff89415b660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:52:15.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.879+0000 7ff898e00640 1 -- 172.21.15.60:0/2025795841 shutdown_connections 2024-04-14T23:52:15.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:15.879+0000 7ff898e00640 1 -- 172.21.15.60:0/2025795841 wait complete. 2024-04-14T23:52:15.904 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-14T23:52:17.905 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:52:18.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.132+0000 7f3734000640 1 Processor -- start 2024-04-14T23:52:18.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.132+0000 7f3734000640 1 -- start start 2024-04-14T23:52:18.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.134+0000 7f3734000640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f372c06f1f0 0x7f372c0c72a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:18.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.134+0000 7f3734000640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f372c15b0d0 con 0x7f372c06f5d0 2024-04-14T23:52:18.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.134+0000 7f3734000640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f372c15b370 con 0x7f372c0731e0 2024-04-14T23:52:18.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.134+0000 7f3734000640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f372c0c6e10 con 0x7f372c06f1f0 2024-04-14T23:52:18.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.134+0000 7f3731600640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f372c06f1f0 0x7f372c0c72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:18.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.134+0000 7f3731600640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f372c06f1f0 0x7f372c0c72a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53184/0 (socket says 172.21.15.60:53184) 2024-04-14T23:52:18.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.134+0000 7f3731600640 1 -- 172.21.15.60:0/1480322452 learned_addr learned my addr 172.21.15.60:0/1480322452 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:52:18.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.135+0000 7f3731600640 1 -- 172.21.15.60:0/1480322452 >> v1:172.21.15.186:6789/0 conn(0x7f372c0731e0 legacy=0x7f372c1654a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:18.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.135+0000 7f3731600640 1 -- 172.21.15.60:0/1480322452 >> v1:172.21.15.60:6789/0 conn(0x7f372c06f5d0 legacy=0x7f372c154980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:18.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.135+0000 7f3731600640 1 -- 172.21.15.60:0/1480322452 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f372c15b650 con 0x7f372c06f1f0 2024-04-14T23:52:18.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.135+0000 7f372be00640 1 -- 172.21.15.60:0/1480322452 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2095835762 0 0) 0x7f372c15b0d0 con 0x7f372c06f5d0 2024-04-14T23:52:18.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.136+0000 7f3731600640 1 --2- 172.21.15.60:0/1480322452 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f372c06f1f0 0x7f372c0c72a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3720009870 tx=0x7f372002bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fc1372acd16beb8 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:52:18.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.136+0000 7f372be00640 1 -- 172.21.15.60:0/1480322452 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f372002d870 con 0x7f372c06f1f0 2024-04-14T23:52:18.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.136+0000 7f372be00640 1 -- 172.21.15.60:0/1480322452 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f372002da10 con 0x7f372c06f1f0 2024-04-14T23:52:18.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.136+0000 7f372be00640 1 -- 172.21.15.60:0/1480322452 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f372002dde0 con 0x7f372c06f1f0 2024-04-14T23:52:18.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.137+0000 7f3734000640 1 -- 172.21.15.60:0/1480322452 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f372c06f1f0 msgr2=0x7f372c0c72a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:18.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.137+0000 7f3734000640 1 --2- 172.21.15.60:0/1480322452 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f372c06f1f0 0x7f372c0c72a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3720009870 tx=0x7f372002bab0 comp rx=0 tx=0).stop 2024-04-14T23:52:18.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.137+0000 7f3734000640 1 -- 172.21.15.60:0/1480322452 shutdown_connections 2024-04-14T23:52:18.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.137+0000 7f3734000640 1 --2- 172.21.15.60:0/1480322452 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f372c06f1f0 0x7f372c0c72a0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:18.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.137+0000 7f3734000640 1 -- 172.21.15.60:0/1480322452 >> 172.21.15.60:0/1480322452 conn(0x7f372c0cc170 msgr2=0x7f372c0cc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:52:18.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.137+0000 7f3734000640 1 -- 172.21.15.60:0/1480322452 shutdown_connections 2024-04-14T23:52:18.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.137+0000 7f3734000640 1 -- 172.21.15.60:0/1480322452 wait complete. 2024-04-14T23:52:18.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.139+0000 7f3734000640 1 Processor -- start 2024-04-14T23:52:18.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.139+0000 7f3734000640 1 -- start start 2024-04-14T23:52:18.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.140+0000 7f3734000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f372c06f1f0 0x7f372c15a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:18.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.140+0000 7f3734000640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f372c06f5d0 0x7f372c1572a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:18.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.140+0000 7f3734000640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f372c0731e0 0x7f372c1577e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:18.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.140+0000 7f3734000640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f372c155090 con 0x7f372c06f5d0 2024-04-14T23:52:18.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.140+0000 7f3734000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f372c071180 con 0x7f372c06f1f0 2024-04-14T23:52:18.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.140+0000 7f3734000640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f372c155210 con 0x7f372c0731e0 2024-04-14T23:52:18.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.140+0000 7f3730c00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f372c06f5d0 0x7f372c1572a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:18.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.140+0000 7f3730c00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f372c06f5d0 0x7f372c1572a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53198/0 (socket says 172.21.15.60:53198) 2024-04-14T23:52:18.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.140+0000 7f3730c00640 1 -- 172.21.15.60:0/1064193498 learned_addr learned my addr 172.21.15.60:0/1064193498 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:52:18.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.140+0000 7f3731600640 1 --2- 172.21.15.60:0/1064193498 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f372c06f1f0 0x7f372c15a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:18.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.141+0000 7f3732000640 1 --2- 172.21.15.60:0/1064193498 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f372c0731e0 0x7f372c1577e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:18.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.141+0000 7f3730c00640 1 -- 172.21.15.60:0/1064193498 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f372c0731e0 msgr2=0x7f372c1577e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:18.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.141+0000 7f3730c00640 1 --2- 172.21.15.60:0/1064193498 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f372c0731e0 0x7f372c1577e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:18.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.141+0000 7f3730c00640 1 -- 172.21.15.60:0/1064193498 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f372c06f1f0 msgr2=0x7f372c15a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:18.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.141+0000 7f3730c00640 1 --2- 172.21.15.60:0/1064193498 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f372c06f1f0 0x7f372c15a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:18.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.141+0000 7f3730c00640 1 -- 172.21.15.60:0/1064193498 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f372c1f0cc0 con 0x7f372c06f5d0 2024-04-14T23:52:18.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.141+0000 7f3730c00640 1 --2- 172.21.15.60:0/1064193498 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f372c06f5d0 0x7f372c1572a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f371800c3f0 tx=0x7f371800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:52:18.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.141+0000 7f3729600640 1 -- 172.21.15.60:0/1064193498 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3718017070 con 0x7f372c06f5d0 2024-04-14T23:52:18.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.142+0000 7f3729600640 1 -- 172.21.15.60:0/1064193498 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37180035f0 con 0x7f372c06f5d0 2024-04-14T23:52:18.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.142+0000 7f3734000640 1 -- 172.21.15.60:0/1064193498 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f372c1f0fa0 con 0x7f372c06f5d0 2024-04-14T23:52:18.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.142+0000 7f3729600640 1 -- 172.21.15.60:0/1064193498 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37180039e0 con 0x7f372c06f5d0 2024-04-14T23:52:18.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.142+0000 7f3734000640 1 -- 172.21.15.60:0/1064193498 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f372c1f1560 con 0x7f372c06f5d0 2024-04-14T23:52:18.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.143+0000 7f3734000640 1 -- 172.21.15.60:0/1064193498 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f372c076970 con 0x7f372c06f5d0 2024-04-14T23:52:18.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.144+0000 7f3729600640 1 -- 172.21.15.60:0/1064193498 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f371801a070 con 0x7f372c06f5d0 2024-04-14T23:52:18.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.145+0000 7f3729600640 1 --2- 172.21.15.60:0/1064193498 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f370c06b990 0x7f370c06de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:18.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.145+0000 7f3729600640 1 -- 172.21.15.60:0/1064193498 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 5911+0+0 (secure 0 0 0) 0x7f3718089ce0 con 0x7f372c06f5d0 2024-04-14T23:52:18.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.145+0000 7f3731600640 1 --2- 172.21.15.60:0/1064193498 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f370c06b990 0x7f370c06de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:18.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.146+0000 7f3731600640 1 --2- 172.21.15.60:0/1064193498 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f370c06b990 0x7f370c06de50 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3720036000 tx=0x7f3720008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:52:18.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.149+0000 7f3729600640 1 -- 172.21.15.60:0/1064193498 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3718058130 con 0x7f372c06f5d0 2024-04-14T23:52:18.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.392+0000 7f3734000640 1 -- 172.21.15.60:0/1064193498 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f372c0ccb30 con 0x7f372c06f5d0 2024-04-14T23:52:18.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.393+0000 7f3729600640 1 -- 172.21.15.60:0/1064193498 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6832 (secure 0 0 0) 0x7f371805c1d0 con 0x7f372c06f5d0 2024-04-14T23:52:18.399 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:52:18.399 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4839,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/1714133698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1714133698},{"type":"v1","addr":"172.21.15.60:6835","nonce":1714133698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":14}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:51:24.313553+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4276},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4276":{"gid":4276,"name":"a","rank":0,"incarnation":9,"state":"up:active","state_seq":45,"addr":"172.21.15.60:6833/78303888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":78303888},{"type":"v1","addr":"172.21.15.60:6833","nonce":78303888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4276,"qdb_cluster":[4276]},"id":1}]} 2024-04-14T23:52:18.399 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 14 2024-04-14T23:52:18.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.395+0000 7f3734000640 1 -- 172.21.15.60:0/1064193498 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f370c06b990 msgr2=0x7f370c06de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:18.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.395+0000 7f3734000640 1 --2- 172.21.15.60:0/1064193498 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f370c06b990 0x7f370c06de50 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3720036000 tx=0x7f3720008030 comp rx=0 tx=0).stop 2024-04-14T23:52:18.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.395+0000 7f3734000640 1 -- 172.21.15.60:0/1064193498 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f372c06f5d0 msgr2=0x7f372c1572a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:18.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.395+0000 7f3734000640 1 --2- 172.21.15.60:0/1064193498 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f372c06f5d0 0x7f372c1572a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f371800c3f0 tx=0x7f371800c8b0 comp rx=0 tx=0).stop 2024-04-14T23:52:18.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.395+0000 7f3734000640 1 -- 172.21.15.60:0/1064193498 shutdown_connections 2024-04-14T23:52:18.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.396+0000 7f3734000640 1 --2- 172.21.15.60:0/1064193498 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f370c06b990 0x7f370c06de50 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:18.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.396+0000 7f3734000640 1 --2- 172.21.15.60:0/1064193498 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f372c0731e0 0x7f372c1577e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:18.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.396+0000 7f3734000640 1 --2- 172.21.15.60:0/1064193498 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f372c06f5d0 0x7f372c1572a0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:18.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.396+0000 7f3734000640 1 --2- 172.21.15.60:0/1064193498 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f372c06f1f0 0x7f372c15a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:18.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.396+0000 7f3734000640 1 -- 172.21.15.60:0/1064193498 >> 172.21.15.60:0/1064193498 conn(0x7f372c0cc170 msgr2=0x7f372c0c72a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:52:18.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.396+0000 7f3734000640 1 -- 172.21.15.60:0/1064193498 shutdown_connections 2024-04-14T23:52:18.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:18.396+0000 7f3734000640 1 -- 172.21.15.60:0/1064193498 wait complete. 2024-04-14T23:52:18.420 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-14T23:52:20.421 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:52:20.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.642+0000 7f8bc9800640 1 Processor -- start 2024-04-14T23:52:20.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.642+0000 7f8bc9800640 1 -- start start 2024-04-14T23:52:20.650 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.644+0000 7f8bc9800640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8bc4154cf0 0x7f8bc415a940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:20.650 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.644+0000 7f8bc9800640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f8bc406c310 0x7f8bc406c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:20.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.644+0000 7f8bc9800640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8bc40c5520 con 0x7f8bc40cb120 2024-04-14T23:52:20.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.644+0000 7f8bc9800640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f8bc41550d0 con 0x7f8bc4154cf0 2024-04-14T23:52:20.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.644+0000 7f8bc9800640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f8bc415b040 con 0x7f8bc406c310 2024-04-14T23:52:20.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.645+0000 7f8bc3e00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8bc4154cf0 0x7f8bc415a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:20.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.645+0000 7f8bc8e00640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f8bc40cb120 0x7f8bc40cbd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:40142/0 (socket says 172.21.15.60:40142) 2024-04-14T23:52:20.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.645+0000 7f8bc8e00640 1 -- 172.21.15.60:0/657953635 learned_addr learned my addr 172.21.15.60:0/657953635 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:52:20.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.645+0000 7f8bc3400640 1 --2- 172.21.15.60:0/657953635 >> v2:172.21.15.186:3300/0 conn(0x7f8bc406c310 0x7f8bc406c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:20.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.645+0000 7f8bc3400640 1 -- 172.21.15.60:0/657953635 >> v2:172.21.15.60:3300/0 conn(0x7f8bc4154cf0 msgr2=0x7f8bc415a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:52:20.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.645+0000 7f8bc3400640 1 --2- 172.21.15.60:0/657953635 >> v2:172.21.15.60:3300/0 conn(0x7f8bc4154cf0 0x7f8bc415a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:20.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.645+0000 7f8bc2a00640 1 -- 172.21.15.60:0/657953635 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1488055455 0 0) 0x7f8bc40c5520 con 0x7f8bc40cb120 2024-04-14T23:52:20.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.645+0000 7f8bc3400640 1 -- 172.21.15.60:0/657953635 >> v1:172.21.15.60:6789/0 conn(0x7f8bc40cb120 legacy=0x7f8bc40cbd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:20.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.645+0000 7f8bc3400640 1 -- 172.21.15.60:0/657953635 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bc415b300 con 0x7f8bc406c310 2024-04-14T23:52:20.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.646+0000 7f8bc3400640 1 --2- 172.21.15.60:0/657953635 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8bc406c310 0x7f8bc406c710 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8bc40d1340 tx=0x7f8bac02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c521e35c5327274a server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:52:20.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.647+0000 7f8bc2a00640 1 -- 172.21.15.60:0/657953635 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bac034070 con 0x7f8bc406c310 2024-04-14T23:52:20.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.647+0000 7f8bc2a00640 1 -- 172.21.15.60:0/657953635 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8bac030610 con 0x7f8bc406c310 2024-04-14T23:52:20.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.647+0000 7f8bc2a00640 1 -- 172.21.15.60:0/657953635 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bac0309e0 con 0x7f8bc406c310 2024-04-14T23:52:20.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.647+0000 7f8bc9800640 1 -- 172.21.15.60:0/657953635 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8bc406c310 msgr2=0x7f8bc406c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:20.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.647+0000 7f8bc9800640 1 --2- 172.21.15.60:0/657953635 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8bc406c310 0x7f8bc406c710 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8bc40d1340 tx=0x7f8bac02b960 comp rx=0 tx=0).stop 2024-04-14T23:52:20.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.647+0000 7f8bc9800640 1 -- 172.21.15.60:0/657953635 shutdown_connections 2024-04-14T23:52:20.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.647+0000 7f8bc9800640 1 --2- 172.21.15.60:0/657953635 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8bc406c310 0x7f8bc406c710 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:20.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.647+0000 7f8bc9800640 1 --2- 172.21.15.60:0/657953635 >> v2:172.21.15.60:3300/0 conn(0x7f8bc4154cf0 0x7f8bc415a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:20.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.647+0000 7f8bc9800640 1 -- 172.21.15.60:0/657953635 >> 172.21.15.60:0/657953635 conn(0x7f8bc40670f0 msgr2=0x7f8bc4067500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:52:20.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.647+0000 7f8bc9800640 1 -- 172.21.15.60:0/657953635 shutdown_connections 2024-04-14T23:52:20.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.647+0000 7f8bc9800640 1 -- 172.21.15.60:0/657953635 wait complete. 2024-04-14T23:52:20.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.649+0000 7f8bc9800640 1 Processor -- start 2024-04-14T23:52:20.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.649+0000 7f8bc9800640 1 -- start start 2024-04-14T23:52:20.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.650+0000 7f8bc9800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8bc406c310 0x7f8bc414ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:20.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.650+0000 7f8bc9800640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8bc40cb120 0x7f8bc41e7220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:20.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.650+0000 7f8bc9800640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8bc4154cf0 0x7f8bc415e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:20.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.650+0000 7f8bc9800640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f8bc406bb10 con 0x7f8bc4154cf0 2024-04-14T23:52:20.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.650+0000 7f8bc9800640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8bc415b8c0 con 0x7f8bc406c310 2024-04-14T23:52:20.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.650+0000 7f8bc9800640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f8bc406bdf0 con 0x7f8bc40cb120 2024-04-14T23:52:20.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.651+0000 7f8bc8e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8bc4154cf0 0x7f8bc415e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:20.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.651+0000 7f8bc3e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8bc406c310 0x7f8bc414ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:20.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.651+0000 7f8bc3400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8bc40cb120 0x7f8bc41e7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:20.658 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.651+0000 7f8bc8e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8bc4154cf0 0x7f8bc415e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53226/0 (socket says 172.21.15.60:53226) 2024-04-14T23:52:20.658 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.651+0000 7f8bc3e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8bc406c310 0x7f8bc414ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:45908/0 (socket says 172.21.15.60:45908) 2024-04-14T23:52:20.658 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.651+0000 7f8bc8e00640 1 -- 172.21.15.60:0/1782274758 learned_addr learned my addr 172.21.15.60:0/1782274758 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:52:20.658 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.651+0000 7f8bc3e00640 1 -- 172.21.15.60:0/1782274758 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8bc40cb120 msgr2=0x7f8bc41e7220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:20.658 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.651+0000 7f8bc3e00640 1 --2- 172.21.15.60:0/1782274758 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8bc40cb120 0x7f8bc41e7220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:20.658 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.651+0000 7f8bc3e00640 1 -- 172.21.15.60:0/1782274758 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8bc4154cf0 msgr2=0x7f8bc415e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:20.658 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.651+0000 7f8bc3e00640 1 --2- 172.21.15.60:0/1782274758 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8bc4154cf0 0x7f8bc415e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:20.658 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.651+0000 7f8bc3e00640 1 -- 172.21.15.60:0/1782274758 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bc41e7cc0 con 0x7f8bc406c310 2024-04-14T23:52:20.658 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.651+0000 7f8bc8e00640 1 --2- 172.21.15.60:0/1782274758 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8bc4154cf0 0x7f8bc415e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:52:20.658 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.651+0000 7f8bc3e00640 1 --2- 172.21.15.60:0/1782274758 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8bc406c310 0x7f8bc414ef90 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f8bb400ebc0 tx=0x7f8bb400c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:52:20.659 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.652+0000 7f8bbbe00640 1 -- 172.21.15.60:0/1782274758 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bb4018020 con 0x7f8bc406c310 2024-04-14T23:52:20.659 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.652+0000 7f8bbbe00640 1 -- 172.21.15.60:0/1782274758 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8bb400cda0 con 0x7f8bc406c310 2024-04-14T23:52:20.659 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.652+0000 7f8bbbe00640 1 -- 172.21.15.60:0/1782274758 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bb40146c0 con 0x7f8bc406c310 2024-04-14T23:52:20.659 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.652+0000 7f8bc9800640 1 -- 172.21.15.60:0/1782274758 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bc41f4790 con 0x7f8bc406c310 2024-04-14T23:52:20.659 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.652+0000 7f8bc9800640 1 -- 172.21.15.60:0/1782274758 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bc41f39d0 con 0x7f8bc406c310 2024-04-14T23:52:20.660 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.653+0000 7f8bc9800640 1 -- 172.21.15.60:0/1782274758 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bc404f540 con 0x7f8bc406c310 2024-04-14T23:52:20.660 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.654+0000 7f8bbbe00640 1 -- 172.21.15.60:0/1782274758 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f8bb4005d10 con 0x7f8bc406c310 2024-04-14T23:52:20.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.655+0000 7f8bbbe00640 1 --2- 172.21.15.60:0/1782274758 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f8ba006b990 0x7f8ba006de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:20.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.655+0000 7f8bbbe00640 1 -- 172.21.15.60:0/1782274758 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 5911+0+0 (secure 0 0 0) 0x7f8bb408b4d0 con 0x7f8bc406c310 2024-04-14T23:52:20.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.655+0000 7f8bc3400640 1 --2- 172.21.15.60:0/1782274758 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f8ba006b990 0x7f8ba006de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:20.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.656+0000 7f8bc3400640 1 --2- 172.21.15.60:0/1782274758 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f8ba006b990 0x7f8ba006de50 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f8bac036000 tx=0x7f8bac03b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:52:20.665 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.660+0000 7f8bbbe00640 1 -- 172.21.15.60:0/1782274758 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8bb4022030 con 0x7f8bc406c310 2024-04-14T23:52:20.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.903+0000 7f8bc9800640 1 -- 172.21.15.60:0/1782274758 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8bc40cb8a0 con 0x7f8bc406c310 2024-04-14T23:52:20.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.904+0000 7f8bbbe00640 1 -- 172.21.15.60:0/1782274758 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6832 (secure 0 0 0) 0x7f8bb4010040 con 0x7f8bc406c310 2024-04-14T23:52:20.910 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:52:20.910 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4839,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/1714133698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1714133698},{"type":"v1","addr":"172.21.15.60:6835","nonce":1714133698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":14}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:51:24.313553+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4276},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4276":{"gid":4276,"name":"a","rank":0,"incarnation":9,"state":"up:active","state_seq":45,"addr":"172.21.15.60:6833/78303888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":78303888},{"type":"v1","addr":"172.21.15.60:6833","nonce":78303888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4276,"qdb_cluster":[4276]},"id":1}]} 2024-04-14T23:52:20.910 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 14 2024-04-14T23:52:20.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.907+0000 7f8bc9800640 1 -- 172.21.15.60:0/1782274758 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f8ba006b990 msgr2=0x7f8ba006de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:20.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.907+0000 7f8bc9800640 1 --2- 172.21.15.60:0/1782274758 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f8ba006b990 0x7f8ba006de50 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f8bac036000 tx=0x7f8bac03b040 comp rx=0 tx=0).stop 2024-04-14T23:52:20.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.907+0000 7f8bc9800640 1 -- 172.21.15.60:0/1782274758 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8bc406c310 msgr2=0x7f8bc414ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:20.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.907+0000 7f8bc9800640 1 --2- 172.21.15.60:0/1782274758 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8bc406c310 0x7f8bc414ef90 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f8bb400ebc0 tx=0x7f8bb400c490 comp rx=0 tx=0).stop 2024-04-14T23:52:20.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.907+0000 7f8bc9800640 1 -- 172.21.15.60:0/1782274758 shutdown_connections 2024-04-14T23:52:20.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.907+0000 7f8bc9800640 1 --2- 172.21.15.60:0/1782274758 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f8ba006b990 0x7f8ba006de50 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:20.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.907+0000 7f8bc9800640 1 --2- 172.21.15.60:0/1782274758 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8bc4154cf0 0x7f8bc415e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:20.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.907+0000 7f8bc9800640 1 --2- 172.21.15.60:0/1782274758 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8bc40cb120 0x7f8bc41e7220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:20.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.907+0000 7f8bc9800640 1 --2- 172.21.15.60:0/1782274758 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8bc406c310 0x7f8bc414ef90 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:20.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.907+0000 7f8bc9800640 1 -- 172.21.15.60:0/1782274758 >> 172.21.15.60:0/1782274758 conn(0x7f8bc40670f0 msgr2=0x7f8bc40cf960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:52:20.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.907+0000 7f8bc9800640 1 -- 172.21.15.60:0/1782274758 shutdown_connections 2024-04-14T23:52:20.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:20.907+0000 7f8bc9800640 1 -- 172.21.15.60:0/1782274758 wait complete. 2024-04-14T23:52:20.933 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-14T23:52:22.934 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:52:23.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.161+0000 7f6dca200640 1 Processor -- start 2024-04-14T23:52:23.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.161+0000 7f6dca200640 1 -- start start 2024-04-14T23:52:23.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.163+0000 7f6dca200640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6dc4152cf0 0x7f6dc415c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:23.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.163+0000 7f6dca200640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6dc40c63c0 0x7f6dc415c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:23.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.163+0000 7f6dca200640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6dc41530d0 con 0x7f6dc4159450 2024-04-14T23:52:23.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.163+0000 7f6dca200640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f6dc40751f0 con 0x7f6dc40c63c0 2024-04-14T23:52:23.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.163+0000 7f6dca200640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f6dc40c8370 con 0x7f6dc4152cf0 2024-04-14T23:52:23.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.163+0000 7f6dc2a00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6dc40c63c0 0x7f6dc415c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:23.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.163+0000 7f6dc3400640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6dc4152cf0 0x7f6dc415c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:23.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.163+0000 7f6dc3e00640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f6dc4159450 0x7f6dc40c3f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56882/0 (socket says 172.21.15.60:56882) 2024-04-14T23:52:23.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.163+0000 7f6dc2a00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6dc40c63c0 0x7f6dc415c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45000/0 (socket says 172.21.15.60:45000) 2024-04-14T23:52:23.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.163+0000 7f6dc3e00640 1 -- 172.21.15.60:0/45702805 learned_addr learned my addr 172.21.15.60:0/45702805 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:52:23.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.163+0000 7f6dc3400640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6dc4152cf0 0x7f6dc415c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:53596/0 (socket says 172.21.15.60:53596) 2024-04-14T23:52:23.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.164+0000 7f6dc3400640 1 -- 172.21.15.60:0/45702805 >> v2:172.21.15.60:3300/0 conn(0x7f6dc40c63c0 msgr2=0x7f6dc415c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:52:23.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.164+0000 7f6dc3400640 1 --2- 172.21.15.60:0/45702805 >> v2:172.21.15.60:3300/0 conn(0x7f6dc40c63c0 0x7f6dc415c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:23.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.164+0000 7f6dc3400640 1 -- 172.21.15.60:0/45702805 >> v1:172.21.15.60:6789/0 conn(0x7f6dc4159450 legacy=0x7f6dc40c3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:23.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.164+0000 7f6dc3400640 1 -- 172.21.15.60:0/45702805 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dc40c3810 con 0x7f6dc4152cf0 2024-04-14T23:52:23.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.164+0000 7f6dc2a00640 1 --2- 172.21.15.60:0/45702805 >> v2:172.21.15.60:3300/0 conn(0x7f6dc40c63c0 0x7f6dc415c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:52:23.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.165+0000 7f6dc3400640 1 --2- 172.21.15.60:0/45702805 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6dc4152cf0 0x7f6dc415c2a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6dc406fbe0 tx=0x7f6db402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=10329434907606f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:52:23.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.165+0000 7f6dc2000640 1 -- 172.21.15.60:0/45702805 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6db4034070 con 0x7f6dc4152cf0 2024-04-14T23:52:23.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.166+0000 7f6dc2000640 1 -- 172.21.15.60:0/45702805 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6db4030610 con 0x7f6dc4152cf0 2024-04-14T23:52:23.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.166+0000 7f6dc2000640 1 -- 172.21.15.60:0/45702805 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6db40309e0 con 0x7f6dc4152cf0 2024-04-14T23:52:23.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.166+0000 7f6dca200640 1 -- 172.21.15.60:0/45702805 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6dc4152cf0 msgr2=0x7f6dc415c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:23.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.166+0000 7f6dca200640 1 --2- 172.21.15.60:0/45702805 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6dc4152cf0 0x7f6dc415c2a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6dc406fbe0 tx=0x7f6db402b960 comp rx=0 tx=0).stop 2024-04-14T23:52:23.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.166+0000 7f6dca200640 1 -- 172.21.15.60:0/45702805 shutdown_connections 2024-04-14T23:52:23.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.166+0000 7f6dca200640 1 --2- 172.21.15.60:0/45702805 >> v2:172.21.15.60:3300/0 conn(0x7f6dc40c63c0 0x7f6dc415c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:23.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.166+0000 7f6dca200640 1 --2- 172.21.15.60:0/45702805 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6dc4152cf0 0x7f6dc415c2a0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:23.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.166+0000 7f6dca200640 1 -- 172.21.15.60:0/45702805 >> 172.21.15.60:0/45702805 conn(0x7f6dc40cc170 msgr2=0x7f6dc40cc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:52:23.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.166+0000 7f6dca200640 1 -- 172.21.15.60:0/45702805 shutdown_connections 2024-04-14T23:52:23.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.166+0000 7f6dca200640 1 -- 172.21.15.60:0/45702805 wait complete. 2024-04-14T23:52:23.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.168+0000 7f6dca200640 1 Processor -- start 2024-04-14T23:52:23.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.168+0000 7f6dca200640 1 -- start start 2024-04-14T23:52:23.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.169+0000 7f6dca200640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6dc40c63c0 0x7f6dc41f1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:23.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.169+0000 7f6dca200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6dc4152cf0 0x7f6dc4071030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:23.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.169+0000 7f6dca200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6dc4159450 0x7f6dc415a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:23.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.169+0000 7f6dca200640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f6dc40c88f0 con 0x7f6dc40c63c0 2024-04-14T23:52:23.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.169+0000 7f6dca200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6dc40c8610 con 0x7f6dc4159450 2024-04-14T23:52:23.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.169+0000 7f6dca200640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f6dc40c4690 con 0x7f6dc4152cf0 2024-04-14T23:52:23.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.169+0000 7f6dc3400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6dc40c63c0 0x7f6dc41f1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:23.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.169+0000 7f6dc3400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6dc40c63c0 0x7f6dc41f1ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45006/0 (socket says 172.21.15.60:45006) 2024-04-14T23:52:23.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.169+0000 7f6dc3400640 1 -- 172.21.15.60:0/1973999544 learned_addr learned my addr 172.21.15.60:0/1973999544 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:52:23.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.169+0000 7f6dc3e00640 1 --2- 172.21.15.60:0/1973999544 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6dc4159450 0x7f6dc415a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:23.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.170+0000 7f6dc2a00640 1 --2- 172.21.15.60:0/1973999544 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6dc4152cf0 0x7f6dc4071030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:23.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.170+0000 7f6dc3400640 1 -- 172.21.15.60:0/1973999544 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6dc4152cf0 msgr2=0x7f6dc4071030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:23.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.170+0000 7f6dc3400640 1 --2- 172.21.15.60:0/1973999544 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6dc4152cf0 0x7f6dc4071030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:23.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.170+0000 7f6dc3400640 1 -- 172.21.15.60:0/1973999544 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6dc4159450 msgr2=0x7f6dc415a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:23.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.170+0000 7f6dc3400640 1 --2- 172.21.15.60:0/1973999544 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6dc4159450 0x7f6dc415a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:23.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.170+0000 7f6dc3400640 1 -- 172.21.15.60:0/1973999544 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dc4073020 con 0x7f6dc40c63c0 2024-04-14T23:52:23.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.170+0000 7f6dc3e00640 1 --2- 172.21.15.60:0/1973999544 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6dc4159450 0x7f6dc415a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:52:23.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.170+0000 7f6dc3400640 1 --2- 172.21.15.60:0/1973999544 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6dc40c63c0 0x7f6dc41f1ee0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f6db4009890 tx=0x7f6db40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:52:23.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.171+0000 7f6dbb400640 1 -- 172.21.15.60:0/1973999544 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6db4034040 con 0x7f6dc40c63c0 2024-04-14T23:52:23.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.171+0000 7f6dbb400640 1 -- 172.21.15.60:0/1973999544 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6db4030410 con 0x7f6dc40c63c0 2024-04-14T23:52:23.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.171+0000 7f6dbb400640 1 -- 172.21.15.60:0/1973999544 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6db4039c50 con 0x7f6dc40c63c0 2024-04-14T23:52:23.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.171+0000 7f6dca200640 1 -- 172.21.15.60:0/1973999544 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6dc4072490 con 0x7f6dc40c63c0 2024-04-14T23:52:23.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.171+0000 7f6dca200640 1 -- 172.21.15.60:0/1973999544 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6dc41f16e0 con 0x7f6dc40c63c0 2024-04-14T23:52:23.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.172+0000 7f6dca200640 1 -- 172.21.15.60:0/1973999544 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6dc415c800 con 0x7f6dc40c63c0 2024-04-14T23:52:23.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.173+0000 7f6dbb400640 1 -- 172.21.15.60:0/1973999544 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f6db4037070 con 0x7f6dc40c63c0 2024-04-14T23:52:23.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.174+0000 7f6dbb400640 1 --2- 172.21.15.60:0/1973999544 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f6da406b990 0x7f6da406de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:23.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.174+0000 7f6dbb400640 1 -- 172.21.15.60:0/1973999544 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 5911+0+0 (secure 0 0 0) 0x7f6db40ac7e0 con 0x7f6dc40c63c0 2024-04-14T23:52:23.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.175+0000 7f6dc2a00640 1 --2- 172.21.15.60:0/1973999544 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f6da406b990 0x7f6da406de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:23.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.175+0000 7f6dc2a00640 1 --2- 172.21.15.60:0/1973999544 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f6da406b990 0x7f6da406de50 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6dc41494e0 tx=0x7f6db0009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:52:23.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.179+0000 7f6dbb400640 1 -- 172.21.15.60:0/1973999544 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6dc415c800 con 0x7f6dc40c63c0 2024-04-14T23:52:23.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.420+0000 7f6dca200640 1 -- 172.21.15.60:0/1973999544 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6dc4076970 con 0x7f6dc40c63c0 2024-04-14T23:52:23.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.421+0000 7f6dbb400640 1 -- 172.21.15.60:0/1973999544 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6832 (secure 0 0 0) 0x7f6db407ad50 con 0x7f6dc40c63c0 2024-04-14T23:52:23.427 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:52:23.427 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4839,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/1714133698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1714133698},{"type":"v1","addr":"172.21.15.60:6835","nonce":1714133698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":14}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:51:24.313553+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4276},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4276":{"gid":4276,"name":"a","rank":0,"incarnation":9,"state":"up:active","state_seq":45,"addr":"172.21.15.60:6833/78303888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":78303888},{"type":"v1","addr":"172.21.15.60:6833","nonce":78303888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4276,"qdb_cluster":[4276]},"id":1}]} 2024-04-14T23:52:23.427 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 14 2024-04-14T23:52:23.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.424+0000 7f6dca200640 1 -- 172.21.15.60:0/1973999544 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f6da406b990 msgr2=0x7f6da406de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:23.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.424+0000 7f6dca200640 1 --2- 172.21.15.60:0/1973999544 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f6da406b990 0x7f6da406de50 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6dc41494e0 tx=0x7f6db0009190 comp rx=0 tx=0).stop 2024-04-14T23:52:23.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.424+0000 7f6dca200640 1 -- 172.21.15.60:0/1973999544 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6dc40c63c0 msgr2=0x7f6dc41f1ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:23.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.424+0000 7f6dca200640 1 --2- 172.21.15.60:0/1973999544 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6dc40c63c0 0x7f6dc41f1ee0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f6db4009890 tx=0x7f6db40046c0 comp rx=0 tx=0).stop 2024-04-14T23:52:23.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.424+0000 7f6dca200640 1 -- 172.21.15.60:0/1973999544 shutdown_connections 2024-04-14T23:52:23.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.424+0000 7f6dca200640 1 --2- 172.21.15.60:0/1973999544 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f6da406b990 0x7f6da406de50 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:23.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.424+0000 7f6dca200640 1 --2- 172.21.15.60:0/1973999544 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6dc4159450 0x7f6dc415a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:23.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.424+0000 7f6dca200640 1 --2- 172.21.15.60:0/1973999544 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6dc4152cf0 0x7f6dc4071030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:23.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.424+0000 7f6dca200640 1 --2- 172.21.15.60:0/1973999544 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6dc40c63c0 0x7f6dc41f1ee0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:23.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.424+0000 7f6dca200640 1 -- 172.21.15.60:0/1973999544 >> 172.21.15.60:0/1973999544 conn(0x7f6dc40cc170 msgr2=0x7f6dc415c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:52:23.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.424+0000 7f6dca200640 1 -- 172.21.15.60:0/1973999544 shutdown_connections 2024-04-14T23:52:23.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:23.424+0000 7f6dca200640 1 -- 172.21.15.60:0/1973999544 wait complete. 2024-04-14T23:52:23.449 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-14T23:52:25.451 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:52:25.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.682+0000 7facc4e00640 1 Processor -- start 2024-04-14T23:52:25.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.682+0000 7facc4e00640 1 -- start start 2024-04-14T23:52:25.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.684+0000 7facc4e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7facc0154cf0 0x7facc015a940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:25.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.684+0000 7facc4e00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7facc00c5520 con 0x7facc006c710 2024-04-14T23:52:25.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.684+0000 7facc4e00640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7facc015b040 con 0x7facc006c310 2024-04-14T23:52:25.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.684+0000 7facc4e00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7facc01550d0 con 0x7facc0154cf0 2024-04-14T23:52:25.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.684+0000 7facbfe00640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7facc006c710 0x7facc00cbd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56888/0 (socket says 172.21.15.60:56888) 2024-04-14T23:52:25.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.684+0000 7facbfe00640 1 -- 172.21.15.60:0/4025729869 learned_addr learned my addr 172.21.15.60:0/4025729869 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:52:25.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.684+0000 7facbf400640 1 --2- 172.21.15.60:0/4025729869 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7facc0154cf0 0x7facc015a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:25.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.685+0000 7facbea00640 1 -- 172.21.15.60:0/4025729869 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3236657658 0 0) 0x7facc00c5520 con 0x7facc006c710 2024-04-14T23:52:25.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.685+0000 7facbea00640 1 -- 172.21.15.60:0/4025729869 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faca4003520 con 0x7facc006c710 2024-04-14T23:52:25.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.685+0000 7facbf400640 1 -- 172.21.15.60:0/4025729869 >> v1:172.21.15.186:6789/0 conn(0x7facc006c310 legacy=0x7facc00cb120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:25.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.685+0000 7facbf400640 1 -- 172.21.15.60:0/4025729869 >> v1:172.21.15.60:6789/0 conn(0x7facc006c710 legacy=0x7facc00cbd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:25.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.685+0000 7facbf400640 1 -- 172.21.15.60:0/4025729869 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facc015b300 con 0x7facc0154cf0 2024-04-14T23:52:25.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.686+0000 7facbf400640 1 --2- 172.21.15.60:0/4025729869 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7facc0154cf0 0x7facc015a940 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7facb00097d0 tx=0x7facb002b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=89a4c9ce9ce959ec server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:52:25.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.687+0000 7facbea00640 1 -- 172.21.15.60:0/4025729869 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facb0034070 con 0x7facc0154cf0 2024-04-14T23:52:25.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.687+0000 7facbea00640 1 -- 172.21.15.60:0/4025729869 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facb0030880 con 0x7facc0154cf0 2024-04-14T23:52:25.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.687+0000 7facbea00640 1 -- 172.21.15.60:0/4025729869 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facb0030c50 con 0x7facc0154cf0 2024-04-14T23:52:25.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.687+0000 7facc4e00640 1 -- 172.21.15.60:0/4025729869 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7facc0154cf0 msgr2=0x7facc015a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:25.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.687+0000 7facc4e00640 1 --2- 172.21.15.60:0/4025729869 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7facc0154cf0 0x7facc015a940 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7facb00097d0 tx=0x7facb002b9c0 comp rx=0 tx=0).stop 2024-04-14T23:52:25.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.687+0000 7facc4e00640 1 -- 172.21.15.60:0/4025729869 shutdown_connections 2024-04-14T23:52:25.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.687+0000 7facc4e00640 1 --2- 172.21.15.60:0/4025729869 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7facc0154cf0 0x7facc015a940 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:25.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.687+0000 7facc4e00640 1 -- 172.21.15.60:0/4025729869 >> 172.21.15.60:0/4025729869 conn(0x7facc00670f0 msgr2=0x7facc0067500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:52:25.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.687+0000 7facc4e00640 1 -- 172.21.15.60:0/4025729869 shutdown_connections 2024-04-14T23:52:25.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.687+0000 7facc4e00640 1 -- 172.21.15.60:0/4025729869 wait complete. 2024-04-14T23:52:25.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.689+0000 7facc4e00640 1 Processor -- start 2024-04-14T23:52:25.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.689+0000 7facc4e00640 1 -- start start 2024-04-14T23:52:25.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.690+0000 7facc4e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7facc006c310 0x7facc01f0f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:25.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.690+0000 7facc4e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7facc006c710 0x7facc0146390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:25.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.690+0000 7facc4e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facc0154cf0 0x7facc015d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:25.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.690+0000 7facc4e00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7facc006bb10 con 0x7facc0154cf0 2024-04-14T23:52:25.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.690+0000 7facc4e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7facc015b8c0 con 0x7facc006c710 2024-04-14T23:52:25.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.690+0000 7facc4e00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7facc006bdf0 con 0x7facc006c310 2024-04-14T23:52:25.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.691+0000 7facbfe00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facc0154cf0 0x7facc015d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:25.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.691+0000 7facbf400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7facc006c310 0x7facc01f0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:25.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.691+0000 7facbfe00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facc0154cf0 0x7facc015d470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45022/0 (socket says 172.21.15.60:45022) 2024-04-14T23:52:25.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.691+0000 7facbfe00640 1 -- 172.21.15.60:0/3583398685 learned_addr learned my addr 172.21.15.60:0/3583398685 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:52:25.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.691+0000 7facb7e00640 1 --2- 172.21.15.60:0/3583398685 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7facc006c710 0x7facc0146390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:25.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.691+0000 7facbfe00640 1 -- 172.21.15.60:0/3583398685 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7facc006c310 msgr2=0x7facc01f0f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:25.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.691+0000 7facbfe00640 1 --2- 172.21.15.60:0/3583398685 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7facc006c310 0x7facc01f0f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:25.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.691+0000 7facbfe00640 1 -- 172.21.15.60:0/3583398685 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7facc006c710 msgr2=0x7facc0146390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:25.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.691+0000 7facbfe00640 1 --2- 172.21.15.60:0/3583398685 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7facc006c710 0x7facc0146390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:25.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.691+0000 7facbfe00640 1 -- 172.21.15.60:0/3583398685 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facc014f030 con 0x7facc0154cf0 2024-04-14T23:52:25.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.691+0000 7facbfe00640 1 --2- 172.21.15.60:0/3583398685 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facc0154cf0 0x7facc015d470 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7facb800cb40 tx=0x7facb800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:52:25.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.692+0000 7facb7400640 1 -- 172.21.15.60:0/3583398685 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facb8017070 con 0x7facc0154cf0 2024-04-14T23:52:25.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.692+0000 7facb7400640 1 -- 172.21.15.60:0/3583398685 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facb800aca0 con 0x7facc0154cf0 2024-04-14T23:52:25.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.692+0000 7facb7400640 1 -- 172.21.15.60:0/3583398685 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facb8012620 con 0x7facc0154cf0 2024-04-14T23:52:25.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.692+0000 7facc4e00640 1 -- 172.21.15.60:0/3583398685 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facc014f310 con 0x7facc0154cf0 2024-04-14T23:52:25.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.692+0000 7facc4e00640 1 -- 172.21.15.60:0/3583398685 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facc014f8d0 con 0x7facc0154cf0 2024-04-14T23:52:25.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.693+0000 7facc4e00640 1 -- 172.21.15.60:0/3583398685 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facc004f540 con 0x7facc0154cf0 2024-04-14T23:52:25.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.694+0000 7facb7400640 1 -- 172.21.15.60:0/3583398685 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7facb801a070 con 0x7facc0154cf0 2024-04-14T23:52:25.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.695+0000 7facb7400640 1 --2- 172.21.15.60:0/3583398685 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fac9c06b990 0x7fac9c06de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:25.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.695+0000 7facb7400640 1 -- 172.21.15.60:0/3583398685 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 5911+0+0 (secure 0 0 0) 0x7facb808a5b0 con 0x7facc0154cf0 2024-04-14T23:52:25.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.695+0000 7facbf400640 1 --2- 172.21.15.60:0/3583398685 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fac9c06b990 0x7fac9c06de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:25.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.696+0000 7facbf400640 1 --2- 172.21.15.60:0/3583398685 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fac9c06b990 0x7fac9c06de50 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7facb0036000 tx=0x7facb003b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:52:25.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.700+0000 7facb7400640 1 -- 172.21.15.60:0/3583398685 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7facb8058b30 con 0x7facc0154cf0 2024-04-14T23:52:25.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.940+0000 7facc4e00640 1 -- 172.21.15.60:0/3583398685 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7facc00cbb10 con 0x7facc0154cf0 2024-04-14T23:52:25.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.941+0000 7facb7400640 1 -- 172.21.15.60:0/3583398685 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6832 (secure 0 0 0) 0x7facb805cbd0 con 0x7facc0154cf0 2024-04-14T23:52:25.947 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:52:25.947 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4839,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/1714133698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1714133698},{"type":"v1","addr":"172.21.15.60:6835","nonce":1714133698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":14}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:51:24.313553+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4276},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4276":{"gid":4276,"name":"a","rank":0,"incarnation":9,"state":"up:active","state_seq":45,"addr":"172.21.15.60:6833/78303888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":78303888},{"type":"v1","addr":"172.21.15.60:6833","nonce":78303888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4276,"qdb_cluster":[4276]},"id":1}]} 2024-04-14T23:52:25.947 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 14 2024-04-14T23:52:25.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.943+0000 7facc4e00640 1 -- 172.21.15.60:0/3583398685 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fac9c06b990 msgr2=0x7fac9c06de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:25.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.943+0000 7facc4e00640 1 --2- 172.21.15.60:0/3583398685 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fac9c06b990 0x7fac9c06de50 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7facb0036000 tx=0x7facb003b040 comp rx=0 tx=0).stop 2024-04-14T23:52:25.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.943+0000 7facc4e00640 1 -- 172.21.15.60:0/3583398685 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facc0154cf0 msgr2=0x7facc015d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:25.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.943+0000 7facc4e00640 1 --2- 172.21.15.60:0/3583398685 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facc0154cf0 0x7facc015d470 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7facb800cb40 tx=0x7facb800a060 comp rx=0 tx=0).stop 2024-04-14T23:52:25.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.944+0000 7facc4e00640 1 -- 172.21.15.60:0/3583398685 shutdown_connections 2024-04-14T23:52:25.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.944+0000 7facc4e00640 1 --2- 172.21.15.60:0/3583398685 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fac9c06b990 0x7fac9c06de50 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:25.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.944+0000 7facc4e00640 1 --2- 172.21.15.60:0/3583398685 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facc0154cf0 0x7facc015d470 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:25.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.944+0000 7facc4e00640 1 --2- 172.21.15.60:0/3583398685 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7facc006c710 0x7facc0146390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:25.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.944+0000 7facc4e00640 1 --2- 172.21.15.60:0/3583398685 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7facc006c310 0x7facc01f0f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:25.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.944+0000 7facc4e00640 1 -- 172.21.15.60:0/3583398685 >> 172.21.15.60:0/3583398685 conn(0x7facc00670f0 msgr2=0x7facc0158790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:52:25.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.944+0000 7facc4e00640 1 -- 172.21.15.60:0/3583398685 shutdown_connections 2024-04-14T23:52:25.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:25.944+0000 7facc4e00640 1 -- 172.21.15.60:0/3583398685 wait complete. 2024-04-14T23:52:25.968 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-14T23:52:27.970 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:52:28.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.196+0000 7f9cefe00640 1 Processor -- start 2024-04-14T23:52:28.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.197+0000 7f9cefe00640 1 -- start start 2024-04-14T23:52:28.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.198+0000 7f9cefe00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9cf00cbd30 0x7f9cf00cc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:28.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.198+0000 7f9cefe00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9cf00c5520 con 0x7f9cf015a940 2024-04-14T23:52:28.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.198+0000 7f9cefe00640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9cf015b040 con 0x7f9cf0154cf0 2024-04-14T23:52:28.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.198+0000 7f9cefe00640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f9cf01550d0 con 0x7f9cf00cbd30 2024-04-14T23:52:28.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.199+0000 7f9cee000640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f9cf015a940 0x7f9cf00cb120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56902/0 (socket says 172.21.15.60:56902) 2024-04-14T23:52:28.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.199+0000 7f9cee000640 1 -- 172.21.15.60:0/3269431931 learned_addr learned my addr 172.21.15.60:0/3269431931 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:52:28.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.199+0000 7f9cef400640 1 --2- 172.21.15.60:0/3269431931 >> v2:172.21.15.60:3300/0 conn(0x7f9cf00cbd30 0x7f9cf00cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:28.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.199+0000 7f9ced600640 1 -- 172.21.15.60:0/3269431931 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 144954404 0 0) 0x7f9cf00c5520 con 0x7f9cf015a940 2024-04-14T23:52:28.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.199+0000 7f9ced600640 1 -- 172.21.15.60:0/3269431931 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9cd4003520 con 0x7f9cf015a940 2024-04-14T23:52:28.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.199+0000 7f9ced600640 1 -- 172.21.15.60:0/3269431931 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1213928880 0 0) 0x7f9cf015b040 con 0x7f9cf0154cf0 2024-04-14T23:52:28.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.199+0000 7f9cef400640 1 -- 172.21.15.60:0/3269431931 >> v1:172.21.15.186:6789/0 conn(0x7f9cf0154cf0 legacy=0x7f9cf006c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:28.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.199+0000 7f9cef400640 1 -- 172.21.15.60:0/3269431931 >> v1:172.21.15.60:6789/0 conn(0x7f9cf015a940 legacy=0x7f9cf00cb120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:28.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.199+0000 7f9cef400640 1 -- 172.21.15.60:0/3269431931 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cf015b300 con 0x7f9cf00cbd30 2024-04-14T23:52:28.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.200+0000 7f9cef400640 1 --2- 172.21.15.60:0/3269431931 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9cf00cbd30 0x7f9cf00cc160 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9ce8009d90 tx=0x7f9ce802bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f836de91d3d0672 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:52:28.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.201+0000 7f9ced600640 1 -- 172.21.15.60:0/3269431931 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ce8039070 con 0x7f9cf00cbd30 2024-04-14T23:52:28.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.201+0000 7f9ced600640 1 -- 172.21.15.60:0/3269431931 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ce802cd80 con 0x7f9cf00cbd30 2024-04-14T23:52:28.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.201+0000 7f9ced600640 1 -- 172.21.15.60:0/3269431931 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ce8034660 con 0x7f9cf00cbd30 2024-04-14T23:52:28.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.201+0000 7f9cefe00640 1 -- 172.21.15.60:0/3269431931 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9cf00cbd30 msgr2=0x7f9cf00cc160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:28.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.201+0000 7f9cefe00640 1 --2- 172.21.15.60:0/3269431931 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9cf00cbd30 0x7f9cf00cc160 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9ce8009d90 tx=0x7f9ce802bfd0 comp rx=0 tx=0).stop 2024-04-14T23:52:28.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.202+0000 7f9cefe00640 1 -- 172.21.15.60:0/3269431931 shutdown_connections 2024-04-14T23:52:28.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.202+0000 7f9cefe00640 1 --2- 172.21.15.60:0/3269431931 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9cf00cbd30 0x7f9cf00cc160 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:28.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.202+0000 7f9cefe00640 1 -- 172.21.15.60:0/3269431931 >> 172.21.15.60:0/3269431931 conn(0x7f9cf00670f0 msgr2=0x7f9cf0067500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:52:28.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.202+0000 7f9cefe00640 1 -- 172.21.15.60:0/3269431931 shutdown_connections 2024-04-14T23:52:28.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.202+0000 7f9cefe00640 1 -- 172.21.15.60:0/3269431931 wait complete. 2024-04-14T23:52:28.209 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.203+0000 7f9cefe00640 1 Processor -- start 2024-04-14T23:52:28.209 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.204+0000 7f9cefe00640 1 -- start start 2024-04-14T23:52:28.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.204+0000 7f9cefe00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9cf00cbd30 0x7f9cf00b56f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:28.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.204+0000 7f9cefe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9cf0154cf0 0x7f9cf00b5c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:28.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.204+0000 7f9cefe00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9cf015a940 0x7f9cf01f3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:28.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.204+0000 7f9cefe00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f9cf006bb10 con 0x7f9cf015a940 2024-04-14T23:52:28.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.204+0000 7f9cefe00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9cf015b8c0 con 0x7f9cf0154cf0 2024-04-14T23:52:28.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.204+0000 7f9cefe00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f9cf006bdf0 con 0x7f9cf00cbd30 2024-04-14T23:52:28.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.205+0000 7f9cee000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9cf0154cf0 0x7f9cf00b5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:28.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.205+0000 7f9ceea00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9cf00cbd30 0x7f9cf00b56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:28.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.205+0000 7f9cee000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9cf0154cf0 0x7f9cf00b5c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:53626/0 (socket says 172.21.15.60:53626) 2024-04-14T23:52:28.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.205+0000 7f9cee000640 1 -- 172.21.15.60:0/1134514516 learned_addr learned my addr 172.21.15.60:0/1134514516 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:52:28.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.205+0000 7f9cef400640 1 --2- 172.21.15.60:0/1134514516 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9cf015a940 0x7f9cf01f3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:28.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.205+0000 7f9ceea00640 1 -- 172.21.15.60:0/1134514516 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9cf0154cf0 msgr2=0x7f9cf00b5c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:28.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.205+0000 7f9ceea00640 1 --2- 172.21.15.60:0/1134514516 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9cf0154cf0 0x7f9cf00b5c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:28.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.205+0000 7f9ceea00640 1 -- 172.21.15.60:0/1134514516 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9cf015a940 msgr2=0x7f9cf01f3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:28.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.205+0000 7f9ceea00640 1 --2- 172.21.15.60:0/1134514516 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9cf015a940 0x7f9cf01f3f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:28.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.205+0000 7f9ceea00640 1 -- 172.21.15.60:0/1134514516 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cf014bb70 con 0x7f9cf00cbd30 2024-04-14T23:52:28.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.205+0000 7f9cee000640 1 --2- 172.21.15.60:0/1134514516 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9cf0154cf0 0x7f9cf00b5c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:52:28.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.205+0000 7f9cef400640 1 --2- 172.21.15.60:0/1134514516 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9cf015a940 0x7f9cf01f3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:52:28.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.206+0000 7f9ceea00640 1 --2- 172.21.15.60:0/1134514516 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9cf00cbd30 0x7f9cf00b56f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9ce000c410 tx=0x7f9ce000c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:52:28.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.206+0000 7f9ce7400640 1 -- 172.21.15.60:0/1134514516 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ce0017070 con 0x7f9cf00cbd30 2024-04-14T23:52:28.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.206+0000 7f9ce7400640 1 -- 172.21.15.60:0/1134514516 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ce00035f0 con 0x7f9cf00cbd30 2024-04-14T23:52:28.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.206+0000 7f9ce7400640 1 -- 172.21.15.60:0/1134514516 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ce00039e0 con 0x7f9cf00cbd30 2024-04-14T23:52:28.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.206+0000 7f9cefe00640 1 -- 172.21.15.60:0/1134514516 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cf014be10 con 0x7f9cf00cbd30 2024-04-14T23:52:28.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.206+0000 7f9cefe00640 1 -- 172.21.15.60:0/1134514516 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cf014c3d0 con 0x7f9cf00cbd30 2024-04-14T23:52:28.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.207+0000 7f9cefe00640 1 -- 172.21.15.60:0/1134514516 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9cf00685b0 con 0x7f9cf00cbd30 2024-04-14T23:52:28.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.208+0000 7f9ce7400640 1 -- 172.21.15.60:0/1134514516 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f9ce001a070 con 0x7f9cf00cbd30 2024-04-14T23:52:28.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.209+0000 7f9ce7400640 1 --2- 172.21.15.60:0/1134514516 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f9cd006b990 0x7f9cd006de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:28.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.209+0000 7f9ce7400640 1 -- 172.21.15.60:0/1134514516 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 5911+0+0 (secure 0 0 0) 0x7f9ce0089c50 con 0x7f9cf00cbd30 2024-04-14T23:52:28.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.210+0000 7f9cee000640 1 --2- 172.21.15.60:0/1134514516 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f9cd006b990 0x7f9cd006de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:28.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.210+0000 7f9cee000640 1 --2- 172.21.15.60:0/1134514516 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f9cd006b990 0x7f9cd006de50 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9cf00b2d70 tx=0x7f9cdc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:52:28.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.214+0000 7f9ce7400640 1 -- 172.21.15.60:0/1134514516 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f9cf00685b0 con 0x7f9cf00cbd30 2024-04-14T23:52:28.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.454+0000 7f9cefe00640 1 -- 172.21.15.60:0/1134514516 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9cf004f540 con 0x7f9cf00cbd30 2024-04-14T23:52:28.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.456+0000 7f9ce7400640 1 -- 172.21.15.60:0/1134514516 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6832 (secure 0 0 0) 0x7f9ce0058130 con 0x7f9cf00cbd30 2024-04-14T23:52:28.462 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:52:28.462 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4839,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/1714133698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1714133698},{"type":"v1","addr":"172.21.15.60:6835","nonce":1714133698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":14}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:51:24.313553+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4276},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4276":{"gid":4276,"name":"a","rank":0,"incarnation":9,"state":"up:active","state_seq":45,"addr":"172.21.15.60:6833/78303888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":78303888},{"type":"v1","addr":"172.21.15.60:6833","nonce":78303888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4276,"qdb_cluster":[4276]},"id":1}]} 2024-04-14T23:52:28.462 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 14 2024-04-14T23:52:28.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.458+0000 7f9cefe00640 1 -- 172.21.15.60:0/1134514516 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f9cd006b990 msgr2=0x7f9cd006de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:28.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.458+0000 7f9cefe00640 1 --2- 172.21.15.60:0/1134514516 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f9cd006b990 0x7f9cd006de50 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9cf00b2d70 tx=0x7f9cdc009040 comp rx=0 tx=0).stop 2024-04-14T23:52:28.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.458+0000 7f9cefe00640 1 -- 172.21.15.60:0/1134514516 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9cf00cbd30 msgr2=0x7f9cf00b56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:28.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.458+0000 7f9cefe00640 1 --2- 172.21.15.60:0/1134514516 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9cf00cbd30 0x7f9cf00b56f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9ce000c410 tx=0x7f9ce000c8d0 comp rx=0 tx=0).stop 2024-04-14T23:52:28.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.459+0000 7f9cefe00640 1 -- 172.21.15.60:0/1134514516 shutdown_connections 2024-04-14T23:52:28.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.459+0000 7f9cefe00640 1 --2- 172.21.15.60:0/1134514516 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f9cd006b990 0x7f9cd006de50 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:28.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.459+0000 7f9cefe00640 1 --2- 172.21.15.60:0/1134514516 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9cf015a940 0x7f9cf01f3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:28.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.459+0000 7f9cefe00640 1 --2- 172.21.15.60:0/1134514516 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9cf0154cf0 0x7f9cf00b5c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:28.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.459+0000 7f9cefe00640 1 --2- 172.21.15.60:0/1134514516 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9cf00cbd30 0x7f9cf00b56f0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:28.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.459+0000 7f9cefe00640 1 -- 172.21.15.60:0/1134514516 >> 172.21.15.60:0/1134514516 conn(0x7f9cf00670f0 msgr2=0x7f9cf0159fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:52:28.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.459+0000 7f9cefe00640 1 -- 172.21.15.60:0/1134514516 shutdown_connections 2024-04-14T23:52:28.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:28.459+0000 7f9cefe00640 1 -- 172.21.15.60:0/1134514516 wait complete. 2024-04-14T23:52:28.484 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-14T23:52:30.485 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:52:30.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.714+0000 7f3cb6200640 1 Processor -- start 2024-04-14T23:52:30.721 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.715+0000 7f3cb6200640 1 -- start start 2024-04-14T23:52:30.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.716+0000 7f3cb6200640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f3cb015a940 0x7f3cb00cb120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:30.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.716+0000 7f3cb6200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3cb00cb660 0x7f3cb00cbd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:30.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.716+0000 7f3cb6200640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cb00c5520 con 0x7f3cb0154cf0 2024-04-14T23:52:30.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.716+0000 7f3cb6200640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f3cb01550d0 con 0x7f3cb015a940 2024-04-14T23:52:30.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.716+0000 7f3cb6200640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f3cb015b040 con 0x7f3cb00cb660 2024-04-14T23:52:30.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.717+0000 7f3cb5800640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3cb00cb660 0x7f3cb00cbd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:30.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.717+0000 7f3cafe00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f3cb015a940 0x7f3cb00cb120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:30.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.717+0000 7f3cb4e00640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f3cb0154cf0 0x7f3cb006c310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56914/0 (socket says 172.21.15.60:56914) 2024-04-14T23:52:30.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.717+0000 7f3cb4e00640 1 -- 172.21.15.60:0/421379191 learned_addr learned my addr 172.21.15.60:0/421379191 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:52:30.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.717+0000 7f3cafe00640 1 -- 172.21.15.60:0/421379191 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3cb00cb660 msgr2=0x7f3cb00cbd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:52:30.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.717+0000 7f3cafe00640 1 --2- 172.21.15.60:0/421379191 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3cb00cb660 0x7f3cb00cbd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:30.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.717+0000 7f3cafe00640 1 -- 172.21.15.60:0/421379191 >> v1:172.21.15.60:6789/0 conn(0x7f3cb0154cf0 legacy=0x7f3cb006c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:30.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.717+0000 7f3cafe00640 1 -- 172.21.15.60:0/421379191 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cb015b300 con 0x7f3cb015a940 2024-04-14T23:52:30.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.717+0000 7f3cb5800640 1 --2- 172.21.15.60:0/421379191 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3cb00cb660 0x7f3cb00cbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:52:30.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.718+0000 7f3cafe00640 1 --2- 172.21.15.60:0/421379191 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3cb015a940 0x7f3cb00cb120 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f3ca0009d90 tx=0x7f3ca002bf40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b702c86a535f3733 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:52:30.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.719+0000 7f3caf400640 1 -- 172.21.15.60:0/421379191 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ca0039070 con 0x7f3cb015a940 2024-04-14T23:52:30.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.719+0000 7f3caf400640 1 -- 172.21.15.60:0/421379191 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ca002ccf0 con 0x7f3cb015a940 2024-04-14T23:52:30.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.719+0000 7f3caf400640 1 -- 172.21.15.60:0/421379191 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ca0034660 con 0x7f3cb015a940 2024-04-14T23:52:30.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.719+0000 7f3cb6200640 1 -- 172.21.15.60:0/421379191 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3cb015a940 msgr2=0x7f3cb00cb120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:30.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.719+0000 7f3cb6200640 1 --2- 172.21.15.60:0/421379191 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3cb015a940 0x7f3cb00cb120 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f3ca0009d90 tx=0x7f3ca002bf40 comp rx=0 tx=0).stop 2024-04-14T23:52:30.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.720+0000 7f3cb6200640 1 -- 172.21.15.60:0/421379191 shutdown_connections 2024-04-14T23:52:30.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.720+0000 7f3cb6200640 1 --2- 172.21.15.60:0/421379191 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3cb015a940 0x7f3cb00cb120 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:30.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.720+0000 7f3cb6200640 1 --2- 172.21.15.60:0/421379191 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3cb00cb660 0x7f3cb00cbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:30.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.720+0000 7f3cb6200640 1 -- 172.21.15.60:0/421379191 >> 172.21.15.60:0/421379191 conn(0x7f3cb00670f0 msgr2=0x7f3cb0067500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:52:30.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.720+0000 7f3cb6200640 1 -- 172.21.15.60:0/421379191 shutdown_connections 2024-04-14T23:52:30.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.720+0000 7f3cb6200640 1 -- 172.21.15.60:0/421379191 wait complete. 2024-04-14T23:52:30.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.722+0000 7f3cb6200640 1 Processor -- start 2024-04-14T23:52:30.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.722+0000 7f3cb6200640 1 -- start start 2024-04-14T23:52:30.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.723+0000 7f3cb6200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3cb00cb660 0x7f3cb00b4150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:30.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.723+0000 7f3cb6200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3cb0154cf0 0x7f3cb0150250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:30.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.723+0000 7f3cb6200640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3cb015a940 0x7f3cb0143c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:30.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.723+0000 7f3cb6200640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f3cb006bb10 con 0x7f3cb015a940 2024-04-14T23:52:30.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.723+0000 7f3cb6200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3cb015b8c0 con 0x7f3cb0154cf0 2024-04-14T23:52:30.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.723+0000 7f3cb6200640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f3cb006bdf0 con 0x7f3cb00cb660 2024-04-14T23:52:30.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.724+0000 7f3cafe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3cb0154cf0 0x7f3cb0150250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:30.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.724+0000 7f3cb4e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3cb00cb660 0x7f3cb00b4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:30.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.724+0000 7f3cafe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3cb0154cf0 0x7f3cb0150250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:53632/0 (socket says 172.21.15.60:53632) 2024-04-14T23:52:30.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.724+0000 7f3cafe00640 1 -- 172.21.15.60:0/1821507481 learned_addr learned my addr 172.21.15.60:0/1821507481 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:52:30.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.724+0000 7f3cb5800640 1 --2- 172.21.15.60:0/1821507481 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3cb015a940 0x7f3cb0143c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:30.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.724+0000 7f3cafe00640 1 -- 172.21.15.60:0/1821507481 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3cb00cb660 msgr2=0x7f3cb00b4150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:30.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.724+0000 7f3cafe00640 1 --2- 172.21.15.60:0/1821507481 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3cb00cb660 0x7f3cb00b4150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:30.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.724+0000 7f3cafe00640 1 -- 172.21.15.60:0/1821507481 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3cb015a940 msgr2=0x7f3cb0143c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:30.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.724+0000 7f3cafe00640 1 --2- 172.21.15.60:0/1821507481 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3cb015a940 0x7f3cb0143c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:30.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.724+0000 7f3cafe00640 1 -- 172.21.15.60:0/1821507481 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cb006cc40 con 0x7f3cb0154cf0 2024-04-14T23:52:30.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.725+0000 7f3cafe00640 1 --2- 172.21.15.60:0/1821507481 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3cb0154cf0 0x7f3cb0150250 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f3ca0037040 tx=0x7f3ca0004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:52:30.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.725+0000 7f3cacc00640 1 -- 172.21.15.60:0/1821507481 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ca0039050 con 0x7f3cb0154cf0 2024-04-14T23:52:30.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.725+0000 7f3cacc00640 1 -- 172.21.15.60:0/1821507481 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ca0045070 con 0x7f3cb0154cf0 2024-04-14T23:52:30.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.725+0000 7f3cb6200640 1 -- 172.21.15.60:0/1821507481 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cb01e69c0 con 0x7f3cb0154cf0 2024-04-14T23:52:30.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.725+0000 7f3cacc00640 1 -- 172.21.15.60:0/1821507481 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ca003e850 con 0x7f3cb0154cf0 2024-04-14T23:52:30.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.725+0000 7f3cb6200640 1 -- 172.21.15.60:0/1821507481 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cb00b6070 con 0x7f3cb0154cf0 2024-04-14T23:52:30.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.726+0000 7f3cb6200640 1 -- 172.21.15.60:0/1821507481 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cb004f540 con 0x7f3cb0154cf0 2024-04-14T23:52:30.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.728+0000 7f3cacc00640 1 -- 172.21.15.60:0/1821507481 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f3ca003e9f0 con 0x7f3cb0154cf0 2024-04-14T23:52:30.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.728+0000 7f3cacc00640 1 --2- 172.21.15.60:0/1821507481 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f3c8c06b990 0x7f3c8c06de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:30.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.728+0000 7f3cacc00640 1 -- 172.21.15.60:0/1821507481 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5997+0+0 (secure 0 0 0) 0x7f3ca00ad7e0 con 0x7f3cb0154cf0 2024-04-14T23:52:30.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.729+0000 7f3cb4e00640 1 --2- 172.21.15.60:0/1821507481 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f3c8c06b990 0x7f3c8c06de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:30.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.729+0000 7f3cb4e00640 1 --2- 172.21.15.60:0/1821507481 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f3c8c06b990 0x7f3c8c06de50 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3c98007fd0 tx=0x7f3c98007e80 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:52:30.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.733+0000 7f3cacc00640 1 -- 172.21.15.60:0/1821507481 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3ca007c3e0 con 0x7f3cb0154cf0 2024-04-14T23:52:30.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.976+0000 7f3cb6200640 1 -- 172.21.15.60:0/1821507481 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3cb00cbb10 con 0x7f3cb0154cf0 2024-04-14T23:52:30.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.977+0000 7f3cacc00640 1 -- 172.21.15.60:0/1821507481 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6006 (secure 0 0 0) 0x7f3ca007bce0 con 0x7f3cb0154cf0 2024-04-14T23:52:30.984 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:52:30.984 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:52:28.906420+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4839},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4839":{"gid":4839,"name":"c","rank":0,"incarnation":15,"state":"up:replay","state_seq":1,"addr":"172.21.15.60:6835/1714133698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1714133698},{"type":"v1","addr":"172.21.15.60:6835","nonce":1714133698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-14T23:52:30.985 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 15 2024-04-14T23:52:30.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.980+0000 7f3cb6200640 1 -- 172.21.15.60:0/1821507481 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f3c8c06b990 msgr2=0x7f3c8c06de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:30.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.980+0000 7f3cb6200640 1 --2- 172.21.15.60:0/1821507481 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f3c8c06b990 0x7f3c8c06de50 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3c98007fd0 tx=0x7f3c98007e80 comp rx=0 tx=0).stop 2024-04-14T23:52:30.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.980+0000 7f3cb6200640 1 -- 172.21.15.60:0/1821507481 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3cb0154cf0 msgr2=0x7f3cb0150250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:30.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.980+0000 7f3cb6200640 1 --2- 172.21.15.60:0/1821507481 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3cb0154cf0 0x7f3cb0150250 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f3ca0037040 tx=0x7f3ca0004a40 comp rx=0 tx=0).stop 2024-04-14T23:52:30.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.980+0000 7f3cb6200640 1 -- 172.21.15.60:0/1821507481 shutdown_connections 2024-04-14T23:52:30.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.980+0000 7f3cb6200640 1 --2- 172.21.15.60:0/1821507481 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f3c8c06b990 0x7f3c8c06de50 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:30.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.980+0000 7f3cb6200640 1 --2- 172.21.15.60:0/1821507481 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3cb015a940 0x7f3cb0143c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:30.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.980+0000 7f3cb6200640 1 --2- 172.21.15.60:0/1821507481 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3cb0154cf0 0x7f3cb0150250 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:30.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.980+0000 7f3cb6200640 1 --2- 172.21.15.60:0/1821507481 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3cb00cb660 0x7f3cb00b4150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:30.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.980+0000 7f3cb6200640 1 -- 172.21.15.60:0/1821507481 >> 172.21.15.60:0/1821507481 conn(0x7f3cb00670f0 msgr2=0x7f3cb00cbd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:52:30.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.980+0000 7f3cb6200640 1 -- 172.21.15.60:0/1821507481 shutdown_connections 2024-04-14T23:52:30.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:30.980+0000 7f3cb6200640 1 -- 172.21.15.60:0/1821507481 wait complete. 2024-04-14T23:52:31.007 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2024-04-14T23:52:31.008 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-04-14T23:52:31.008 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:52:31.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.237+0000 7f1107000640 1 Processor -- start 2024-04-14T23:52:31.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.238+0000 7f1107000640 1 -- start start 2024-04-14T23:52:31.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.239+0000 7f1107000640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f11000c7080 0x7f110006b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:31.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.239+0000 7f1107000640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f110015ba70 0x7f1100070d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:31.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.239+0000 7f1107000640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f110006b850 con 0x7f11000c6950 2024-04-14T23:52:31.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.239+0000 7f1107000640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f110006c740 con 0x7f110015ba70 2024-04-14T23:52:31.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.239+0000 7f1107000640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f110006bb30 con 0x7f11000c7080 2024-04-14T23:52:31.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.240+0000 7f1106600640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f110015ba70 0x7f1100070d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:31.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.240+0000 7f1105200640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f11000c6950 0x7f110015b360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56916/0 (socket says 172.21.15.60:56916) 2024-04-14T23:52:31.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.240+0000 7f1105200640 1 -- 172.21.15.60:0/847242883 learned_addr learned my addr 172.21.15.60:0/847242883 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:52:31.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.240+0000 7f1106600640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f110015ba70 0x7f1100070d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45062/0 (socket says 172.21.15.60:45062) 2024-04-14T23:52:31.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.240+0000 7f1105c00640 1 --2- 172.21.15.60:0/847242883 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f11000c7080 0x7f110006b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:31.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.240+0000 7f1106600640 1 -- 172.21.15.60:0/847242883 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f11000c7080 msgr2=0x7f110006b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:52:31.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.240+0000 7f1106600640 1 --2- 172.21.15.60:0/847242883 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f11000c7080 0x7f110006b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:31.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.240+0000 7f1106600640 1 -- 172.21.15.60:0/847242883 >> v1:172.21.15.60:6789/0 conn(0x7f11000c6950 legacy=0x7f110015b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:31.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.240+0000 7f10ffe00640 1 -- 172.21.15.60:0/847242883 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3957104420 0 0) 0x7f110006b850 con 0x7f11000c6950 2024-04-14T23:52:31.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.240+0000 7f1106600640 1 -- 172.21.15.60:0/847242883 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f110006be10 con 0x7f110015ba70 2024-04-14T23:52:31.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.241+0000 7f1106600640 1 --2- 172.21.15.60:0/847242883 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f110015ba70 0x7f1100070d50 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f10f80097a0 tx=0x7f10f802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4ee4de4bec552c server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:52:31.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.242+0000 7f10ffe00640 1 -- 172.21.15.60:0/847242883 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f8034070 con 0x7f110015ba70 2024-04-14T23:52:31.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.242+0000 7f10ffe00640 1 -- 172.21.15.60:0/847242883 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10f8030610 con 0x7f110015ba70 2024-04-14T23:52:31.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.242+0000 7f10ffe00640 1 -- 172.21.15.60:0/847242883 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f80309e0 con 0x7f110015ba70 2024-04-14T23:52:31.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.242+0000 7f1107000640 1 -- 172.21.15.60:0/847242883 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f110015ba70 msgr2=0x7f1100070d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:31.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.242+0000 7f1107000640 1 --2- 172.21.15.60:0/847242883 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f110015ba70 0x7f1100070d50 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f10f80097a0 tx=0x7f10f802b960 comp rx=0 tx=0).stop 2024-04-14T23:52:31.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.242+0000 7f1107000640 1 -- 172.21.15.60:0/847242883 shutdown_connections 2024-04-14T23:52:31.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.242+0000 7f1107000640 1 --2- 172.21.15.60:0/847242883 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f110015ba70 0x7f1100070d50 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:31.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.242+0000 7f1107000640 1 --2- 172.21.15.60:0/847242883 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f11000c7080 0x7f110006b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:31.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.242+0000 7f1107000640 1 -- 172.21.15.60:0/847242883 >> 172.21.15.60:0/847242883 conn(0x7f11000c8b50 msgr2=0x7f11000c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:52:31.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.243+0000 7f1107000640 1 -- 172.21.15.60:0/847242883 shutdown_connections 2024-04-14T23:52:31.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.243+0000 7f1107000640 1 -- 172.21.15.60:0/847242883 wait complete. 2024-04-14T23:52:31.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.244+0000 7f1107000640 1 Processor -- start 2024-04-14T23:52:31.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.244+0000 7f1107000640 1 -- start start 2024-04-14T23:52:31.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.245+0000 7f1107000640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f11000c6950 0x7f11000b1470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:31.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.245+0000 7f1107000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11000c7080 0x7f11001f1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:31.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.245+0000 7f1107000640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f110015ba70 0x7f11000cafd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:31.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.245+0000 7f1107000640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f11000c7460 con 0x7f110015ba70 2024-04-14T23:52:31.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.245+0000 7f1107000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f110006c3d0 con 0x7f11000c7080 2024-04-14T23:52:31.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.245+0000 7f1107000640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f11000c75e0 con 0x7f11000c6950 2024-04-14T23:52:31.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.246+0000 7f1106600640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f110015ba70 0x7f11000cafd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:31.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.246+0000 7f1105200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11000c7080 0x7f11001f1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:31.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.246+0000 7f1105c00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f11000c6950 0x7f11000b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:31.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.246+0000 7f1105200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11000c7080 0x7f11001f1700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:53642/0 (socket says 172.21.15.60:53642) 2024-04-14T23:52:31.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.246+0000 7f1105200640 1 -- 172.21.15.60:0/162034513 learned_addr learned my addr 172.21.15.60:0/162034513 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:52:31.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.246+0000 7f1105200640 1 -- 172.21.15.60:0/162034513 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f11000c6950 msgr2=0x7f11000b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:31.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.246+0000 7f1105200640 1 --2- 172.21.15.60:0/162034513 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f11000c6950 0x7f11000b1470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:31.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.246+0000 7f1105200640 1 -- 172.21.15.60:0/162034513 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f110015ba70 msgr2=0x7f11000cafd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:31.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.246+0000 7f1105200640 1 --2- 172.21.15.60:0/162034513 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f110015ba70 0x7f11000cafd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:31.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.246+0000 7f1105200640 1 -- 172.21.15.60:0/162034513 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11001f1c40 con 0x7f11000c7080 2024-04-14T23:52:31.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.247+0000 7f1105200640 1 --2- 172.21.15.60:0/162034513 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11000c7080 0x7f11001f1700 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f10ec00c4b0 tx=0x7f10ec00c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:52:31.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.247+0000 7f10fd600640 1 -- 172.21.15.60:0/162034513 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10ec018070 con 0x7f11000c7080 2024-04-14T23:52:31.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.247+0000 7f10fd600640 1 -- 172.21.15.60:0/162034513 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10ec007500 con 0x7f11000c7080 2024-04-14T23:52:31.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.247+0000 7f10fd600640 1 -- 172.21.15.60:0/162034513 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10ec004a20 con 0x7f11000c7080 2024-04-14T23:52:31.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.247+0000 7f1107000640 1 -- 172.21.15.60:0/162034513 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11001f10c0 con 0x7f11000c7080 2024-04-14T23:52:31.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.247+0000 7f1107000640 1 -- 172.21.15.60:0/162034513 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11001e1510 con 0x7f11000c7080 2024-04-14T23:52:31.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.248+0000 7f1107000640 1 -- 172.21.15.60:0/162034513 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1100068bc0 con 0x7f11000c7080 2024-04-14T23:52:31.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.249+0000 7f10fd600640 1 -- 172.21.15.60:0/162034513 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f10ec004bc0 con 0x7f11000c7080 2024-04-14T23:52:31.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.250+0000 7f10fd600640 1 --2- 172.21.15.60:0/162034513 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f10e006b990 0x7f10e006de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:31.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.250+0000 7f10fd600640 1 -- 172.21.15.60:0/162034513 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5997+0+0 (secure 0 0 0) 0x7f10ec089cf0 con 0x7f11000c7080 2024-04-14T23:52:31.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.251+0000 7f1105c00640 1 --2- 172.21.15.60:0/162034513 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f10e006b990 0x7f10e006de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:31.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.251+0000 7f1105c00640 1 --2- 172.21.15.60:0/162034513 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f10e006b990 0x7f10e006de50 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f10f40113e0 tx=0x7f10f400b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:52:31.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.255+0000 7f10fd600640 1 -- 172.21.15.60:0/162034513 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1100068bc0 con 0x7f11000c7080 2024-04-14T23:52:31.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.496+0000 7f1107000640 1 -- 172.21.15.60:0/162034513 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f110004f540 con 0x7f11000c7080 2024-04-14T23:52:31.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.498+0000 7f10fd600640 1 -- 172.21.15.60:0/162034513 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6006 (secure 0 0 0) 0x7f10ec058120 con 0x7f11000c7080 2024-04-14T23:52:31.504 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:52:31.504 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:52:28.906420+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4839},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4839":{"gid":4839,"name":"c","rank":0,"incarnation":15,"state":"up:replay","state_seq":1,"addr":"172.21.15.60:6835/1714133698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1714133698},{"type":"v1","addr":"172.21.15.60:6835","nonce":1714133698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-14T23:52:31.504 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 15 2024-04-14T23:52:31.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.500+0000 7f1107000640 1 -- 172.21.15.60:0/162034513 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f10e006b990 msgr2=0x7f10e006de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:31.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.500+0000 7f1107000640 1 --2- 172.21.15.60:0/162034513 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f10e006b990 0x7f10e006de50 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f10f40113e0 tx=0x7f10f400b040 comp rx=0 tx=0).stop 2024-04-14T23:52:31.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.500+0000 7f1107000640 1 -- 172.21.15.60:0/162034513 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11000c7080 msgr2=0x7f11001f1700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:31.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.500+0000 7f1107000640 1 --2- 172.21.15.60:0/162034513 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11000c7080 0x7f11001f1700 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f10ec00c4b0 tx=0x7f10ec00c970 comp rx=0 tx=0).stop 2024-04-14T23:52:31.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.500+0000 7f1107000640 1 -- 172.21.15.60:0/162034513 shutdown_connections 2024-04-14T23:52:31.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.500+0000 7f1107000640 1 --2- 172.21.15.60:0/162034513 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f10e006b990 0x7f10e006de50 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:31.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.500+0000 7f1107000640 1 --2- 172.21.15.60:0/162034513 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f110015ba70 0x7f11000cafd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:31.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.500+0000 7f1107000640 1 --2- 172.21.15.60:0/162034513 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11000c7080 0x7f11001f1700 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:31.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.500+0000 7f1107000640 1 --2- 172.21.15.60:0/162034513 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f11000c6950 0x7f11000b1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:31.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.500+0000 7f1107000640 1 -- 172.21.15.60:0/162034513 >> 172.21.15.60:0/162034513 conn(0x7f11000c8b50 msgr2=0x7f1100159810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:52:31.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.500+0000 7f1107000640 1 -- 172.21.15.60:0/162034513 shutdown_connections 2024-04-14T23:52:31.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:31.500+0000 7f1107000640 1 -- 172.21.15.60:0/162034513 wait complete. 2024-04-14T23:52:31.524 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 15, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-14T23:48:42.700188+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 15, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4839": { "addr": "172.21.15.60:6835/1714133698", "addrs": { "addrvec": [ { "addr": "172.21.15.60:6834", "nonce": 1714133698, "type": "v2" }, { "addr": "172.21.15.60:6835", "nonce": 1714133698, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4839, "incarnation": 15, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 27, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-14T23:52:28.906420+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4839 } } } ], "standbys": [ { "addr": "172.21.15.60:6837/1527815634", "addrs": { "addrvec": [ { "addr": "172.21.15.60:6836", "nonce": 1527815634, "type": "v2" }, { "addr": "172.21.15.60:6837", "nonce": 1527815634, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4197, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6838/3958319486", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6835", "nonce": 3958319486, "type": "v2" }, { "addr": "172.21.15.186:6838", "nonce": 3958319486, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4220, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6837/3910951185", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6834", "nonce": 3910951185, "type": "v2" }, { "addr": "172.21.15.186:6837", "nonce": 3910951185, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4264, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6839/2155020404", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6836", "nonce": 2155020404, "type": "v2" }, { "addr": "172.21.15.186:6839", "nonce": 2155020404, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4267, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-14T23:52:33.525 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:52:33.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:33.750+0000 7f6b4fe00640 1 Processor -- start 2024-04-14T23:52:33.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:33.751+0000 7f6b4fe00640 1 -- start start 2024-04-14T23:52:33.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:33.752+0000 7f6b4fe00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6b50154d60 0x7f6b500c5160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:33.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:33.752+0000 7f6b4fe00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6b5006c5f0 0x7f6b500cb370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:33.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:33.752+0000 7f6b4fe00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b5006ca10 con 0x7f6b5015a8d0 2024-04-14T23:52:33.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:33.752+0000 7f6b4fe00640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f6b5006c160 con 0x7f6b5006c5f0 2024-04-14T23:52:33.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:33.752+0000 7f6b4fe00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f6b5015b820 con 0x7f6b50154d60 2024-04-14T23:52:33.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:33.753+0000 7f6b4f400640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f6b5015a8d0 0x7f6b500cc140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:32976/0 (socket says 172.21.15.60:32976) 2024-04-14T23:52:33.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:33.753+0000 7f6b4f400640 1 -- 172.21.15.60:0/2979831974 learned_addr learned my addr 172.21.15.60:0/2979831974 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:52:33.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:33.753+0000 7f6b4e000640 1 --2- 172.21.15.60:0/2979831974 >> v2:172.21.15.60:3300/0 conn(0x7f6b5006c5f0 0x7f6b500cb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:33.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:33.753+0000 7f6b4ea00640 1 --2- 172.21.15.60:0/2979831974 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6b50154d60 0x7f6b500c5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:33.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:33.753+0000 7f6b4e000640 1 -- 172.21.15.60:0/2979831974 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6b50154d60 msgr2=0x7f6b500c5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:52:33.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:33.753+0000 7f6b4d600640 1 -- 172.21.15.60:0/2979831974 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1355722428 0 0) 0x7f6b5006ca10 con 0x7f6b5015a8d0 2024-04-14T23:52:33.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:33.753+0000 7f6b4e000640 1 --2- 172.21.15.60:0/2979831974 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6b50154d60 0x7f6b500c5160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:33.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:33.753+0000 7f6b4e000640 1 -- 172.21.15.60:0/2979831974 >> v1:172.21.15.60:6789/0 conn(0x7f6b5015a8d0 legacy=0x7f6b500cc140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:33.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:33.754+0000 7f6b4e000640 1 -- 172.21.15.60:0/2979831974 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b5015b260 con 0x7f6b5006c5f0 2024-04-14T23:52:33.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:33.755+0000 7f6b4e000640 1 --2- 172.21.15.60:0/2979831974 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b5006c5f0 0x7f6b500cb370 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c0097a0 tx=0x7f6b3c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16698a5cfc4fee86 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:52:33.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:33.756+0000 7f6b4d600640 1 -- 172.21.15.60:0/2979831974 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b3c034070 con 0x7f6b5006c5f0 2024-04-14T23:52:33.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:33.756+0000 7f6b4d600640 1 -- 172.21.15.60:0/2979831974 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b3c030610 con 0x7f6b5006c5f0 2024-04-14T23:52:33.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:33.756+0000 7f6b4d600640 1 -- 172.21.15.60:0/2979831974 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b3c0309e0 con 0x7f6b5006c5f0 2024-04-14T23:52:33.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:33.756+0000 7f6b4fe00640 1 -- 172.21.15.60:0/2979831974 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b5006c5f0 msgr2=0x7f6b500cb370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:33.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:33.756+0000 7f6b4fe00640 1 --2- 172.21.15.60:0/2979831974 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b5006c5f0 0x7f6b500cb370 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c0097a0 tx=0x7f6b3c02b960 comp rx=0 tx=0).stop 2024-04-14T23:52:33.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:33.756+0000 7f6b4fe00640 1 -- 172.21.15.60:0/2979831974 shutdown_connections 2024-04-14T23:52:33.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:33.756+0000 7f6b4fe00640 1 --2- 172.21.15.60:0/2979831974 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b5006c5f0 0x7f6b500cb370 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:33.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:33.756+0000 7f6b4fe00640 1 --2- 172.21.15.60:0/2979831974 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6b50154d60 0x7f6b500c5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:33.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:33.756+0000 7f6b4fe00640 1 -- 172.21.15.60:0/2979831974 >> 172.21.15.60:0/2979831974 conn(0x7f6b500670f0 msgr2=0x7f6b50067500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:52:33.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:33.756+0000 7f6b4fe00640 1 -- 172.21.15.60:0/2979831974 shutdown_connections 2024-04-14T23:52:33.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:33.757+0000 7f6b4fe00640 1 -- 172.21.15.60:0/2979831974 wait complete. 2024-04-14T23:52:33.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:33.759+0000 7f6b4fe00640 1 Processor -- start 2024-04-14T23:52:33.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:33.759+0000 7f6b4fe00640 1 -- start start 2024-04-14T23:52:33.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:33.760+0000 7f6b4fe00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b5006c5f0 0x7f6b50144fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:33.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:33.760+0000 7f6b4fe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b50154d60 0x7f6b501522e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:33.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:33.760+0000 7f6b4fe00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6b5015a8d0 0x7f6b500cf2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:33.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:33.760+0000 7f6b4fe00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f6b5006bdf0 con 0x7f6b5006c5f0 2024-04-14T23:52:33.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:33.760+0000 7f6b4fe00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6b5006bb10 con 0x7f6b50154d60 2024-04-14T23:52:33.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:33.760+0000 7f6b4fe00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f6b500cb8b0 con 0x7f6b5015a8d0 2024-04-14T23:52:33.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:33.760+0000 7f6b4ea00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b5006c5f0 0x7f6b50144fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:33.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:33.760+0000 7f6b4ea00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b5006c5f0 0x7f6b50144fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49354/0 (socket says 172.21.15.60:49354) 2024-04-14T23:52:33.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:33.760+0000 7f6b4ea00640 1 -- 172.21.15.60:0/3150899767 learned_addr learned my addr 172.21.15.60:0/3150899767 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:52:33.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:33.760+0000 7f6b4e000640 1 --2- 172.21.15.60:0/3150899767 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b50154d60 0x7f6b501522e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:33.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:33.760+0000 7f6b4f400640 1 --2- 172.21.15.60:0/3150899767 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6b5015a8d0 0x7f6b500cf2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:33.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:33.761+0000 7f6b4ea00640 1 -- 172.21.15.60:0/3150899767 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6b5015a8d0 msgr2=0x7f6b500cf2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:33.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:33.761+0000 7f6b4ea00640 1 --2- 172.21.15.60:0/3150899767 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6b5015a8d0 0x7f6b500cf2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:33.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:33.761+0000 7f6b4ea00640 1 -- 172.21.15.60:0/3150899767 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b50154d60 msgr2=0x7f6b501522e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:33.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:33.761+0000 7f6b4ea00640 1 --2- 172.21.15.60:0/3150899767 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b50154d60 0x7f6b501522e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:33.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:33.761+0000 7f6b4ea00640 1 -- 172.21.15.60:0/3150899767 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b501544e0 con 0x7f6b5006c5f0 2024-04-14T23:52:33.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:33.761+0000 7f6b4e000640 1 --2- 172.21.15.60:0/3150899767 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b50154d60 0x7f6b501522e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:52:33.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:33.761+0000 7f6b4ea00640 1 --2- 172.21.15.60:0/3150899767 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b5006c5f0 0x7f6b50144fb0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6b4000e9b0 tx=0x7f6b4000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:52:33.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:33.762+0000 7f6b46a00640 1 -- 172.21.15.60:0/3150899767 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b40017070 con 0x7f6b5006c5f0 2024-04-14T23:52:33.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:33.762+0000 7f6b46a00640 1 -- 172.21.15.60:0/3150899767 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b4000c9a0 con 0x7f6b5006c5f0 2024-04-14T23:52:33.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:33.762+0000 7f6b46a00640 1 -- 172.21.15.60:0/3150899767 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b4000cd90 con 0x7f6b5006c5f0 2024-04-14T23:52:33.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:33.762+0000 7f6b4fe00640 1 -- 172.21.15.60:0/3150899767 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b501f1600 con 0x7f6b5006c5f0 2024-04-14T23:52:33.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:33.762+0000 7f6b4fe00640 1 -- 172.21.15.60:0/3150899767 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b50154200 con 0x7f6b5006c5f0 2024-04-14T23:52:33.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:33.763+0000 7f6b4fe00640 1 -- 172.21.15.60:0/3150899767 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b5004f540 con 0x7f6b5006c5f0 2024-04-14T23:52:33.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:33.764+0000 7f6b46a00640 1 -- 172.21.15.60:0/3150899767 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f6b4001a070 con 0x7f6b5006c5f0 2024-04-14T23:52:33.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:33.765+0000 7f6b46a00640 1 --2- 172.21.15.60:0/3150899767 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f6b3006b990 0x7f6b3006de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:33.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:33.765+0000 7f6b46a00640 1 -- 172.21.15.60:0/3150899767 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5997+0+0 (secure 0 0 0) 0x7f6b4008ad40 con 0x7f6b5006c5f0 2024-04-14T23:52:33.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:33.766+0000 7f6b4e000640 1 --2- 172.21.15.60:0/3150899767 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f6b3006b990 0x7f6b3006de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:33.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:33.766+0000 7f6b4e000640 1 --2- 172.21.15.60:0/3150899767 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f6b3006b990 0x7f6b3006de50 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c036000 tx=0x7f6b3c03b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:52:33.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:33.771+0000 7f6b46a00640 1 -- 172.21.15.60:0/3150899767 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6b40059140 con 0x7f6b5006c5f0 2024-04-14T23:52:34.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:34.066+0000 7f6b4fe00640 1 -- 172.21.15.60:0/3150899767 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6b500cbb60 con 0x7f6b5006c5f0 2024-04-14T23:52:34.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:34.067+0000 7f6b46a00640 1 -- 172.21.15.60:0/3150899767 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6006 (secure 0 0 0) 0x7f6b4005d1e0 con 0x7f6b5006c5f0 2024-04-14T23:52:34.073 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:52:34.073 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:52:28.906420+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4839},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4839":{"gid":4839,"name":"c","rank":0,"incarnation":15,"state":"up:replay","state_seq":1,"addr":"172.21.15.60:6835/1714133698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1714133698},{"type":"v1","addr":"172.21.15.60:6835","nonce":1714133698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-14T23:52:34.074 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 15 2024-04-14T23:52:34.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:34.069+0000 7f6b4fe00640 1 -- 172.21.15.60:0/3150899767 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f6b3006b990 msgr2=0x7f6b3006de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:34.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:34.070+0000 7f6b4fe00640 1 --2- 172.21.15.60:0/3150899767 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f6b3006b990 0x7f6b3006de50 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c036000 tx=0x7f6b3c03b040 comp rx=0 tx=0).stop 2024-04-14T23:52:34.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:34.070+0000 7f6b4fe00640 1 -- 172.21.15.60:0/3150899767 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b5006c5f0 msgr2=0x7f6b50144fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:34.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:34.070+0000 7f6b4fe00640 1 --2- 172.21.15.60:0/3150899767 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b5006c5f0 0x7f6b50144fb0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6b4000e9b0 tx=0x7f6b4000ee70 comp rx=0 tx=0).stop 2024-04-14T23:52:34.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:34.070+0000 7f6b4fe00640 1 -- 172.21.15.60:0/3150899767 shutdown_connections 2024-04-14T23:52:34.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:34.070+0000 7f6b4fe00640 1 --2- 172.21.15.60:0/3150899767 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f6b3006b990 0x7f6b3006de50 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:34.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:34.070+0000 7f6b4fe00640 1 --2- 172.21.15.60:0/3150899767 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6b5015a8d0 0x7f6b500cf2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:34.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:34.070+0000 7f6b4fe00640 1 --2- 172.21.15.60:0/3150899767 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b50154d60 0x7f6b501522e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:34.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:34.070+0000 7f6b4fe00640 1 --2- 172.21.15.60:0/3150899767 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b5006c5f0 0x7f6b50144fb0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:34.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:34.070+0000 7f6b4fe00640 1 -- 172.21.15.60:0/3150899767 >> 172.21.15.60:0/3150899767 conn(0x7f6b500670f0 msgr2=0x7f6b501590a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:52:34.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:34.070+0000 7f6b4fe00640 1 -- 172.21.15.60:0/3150899767 shutdown_connections 2024-04-14T23:52:34.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:34.070+0000 7f6b4fe00640 1 -- 172.21.15.60:0/3150899767 wait complete. 2024-04-14T23:52:34.093 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-14T23:52:36.094 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:52:36.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.321+0000 7f9c9f200640 1 Processor -- start 2024-04-14T23:52:36.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.321+0000 7f9c9f200640 1 -- start start 2024-04-14T23:52:36.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.323+0000 7f9c9f200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ca015c8a0 0x7f9ca0072390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:36.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.323+0000 7f9c9f200640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f9ca00c6950 0x7f9ca01698f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:36.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.323+0000 7f9c9f200640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ca0071bf0 con 0x7f9ca00c86d0 2024-04-14T23:52:36.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.323+0000 7f9c9f200640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f9ca015cc80 con 0x7f9ca00c6950 2024-04-14T23:52:36.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.323+0000 7f9c9f200640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f9ca01632a0 con 0x7f9ca015c8a0 2024-04-14T23:52:36.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.323+0000 7f9c9e800640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f9ca00c6950 0x7f9ca01698f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:36.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.323+0000 7f9c9de00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ca015c8a0 0x7f9ca0072390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:36.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.323+0000 7f9c9e800640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f9ca00c6950 0x7f9ca01698f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:38076/0 (socket says 172.21.15.60:38076) 2024-04-14T23:52:36.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.323+0000 7f9c9e800640 1 -- 172.21.15.60:0/3756774171 learned_addr learned my addr 172.21.15.60:0/3756774171 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:52:36.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.324+0000 7f9c9de00640 1 -- 172.21.15.60:0/3756774171 >> v2:172.21.15.186:3300/0 conn(0x7f9ca00c6950 msgr2=0x7f9ca01698f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:52:36.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.324+0000 7f9c9de00640 1 --2- 172.21.15.60:0/3756774171 >> v2:172.21.15.186:3300/0 conn(0x7f9ca00c6950 0x7f9ca01698f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:36.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.324+0000 7f9c9de00640 1 -- 172.21.15.60:0/3756774171 >> v1:172.21.15.186:6789/0 conn(0x7f9ca00c86d0 legacy=0x7f9ca01625c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:36.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.324+0000 7f9c9de00640 1 -- 172.21.15.60:0/3756774171 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ca00c6fb0 con 0x7f9ca015c8a0 2024-04-14T23:52:36.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.324+0000 7f9c9e800640 1 --2- 172.21.15.60:0/3756774171 >> v2:172.21.15.186:3300/0 conn(0x7f9ca00c6950 0x7f9ca01698f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:52:36.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.325+0000 7f9c9de00640 1 --2- 172.21.15.60:0/3756774171 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ca015c8a0 0x7f9ca0072390 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f9c900097a0 tx=0x7f9c9002b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11f6ac276b664dff server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:52:36.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.325+0000 7f9c9d400640 1 -- 172.21.15.60:0/3756774171 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c90035020 con 0x7f9ca015c8a0 2024-04-14T23:52:36.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.325+0000 7f9c9d400640 1 -- 172.21.15.60:0/3756774171 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c90031810 con 0x7f9ca015c8a0 2024-04-14T23:52:36.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.325+0000 7f9c9d400640 1 -- 172.21.15.60:0/3756774171 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c90031be0 con 0x7f9ca015c8a0 2024-04-14T23:52:36.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.326+0000 7f9c9f200640 1 -- 172.21.15.60:0/3756774171 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ca015c8a0 msgr2=0x7f9ca0072390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:36.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.326+0000 7f9c9f200640 1 --2- 172.21.15.60:0/3756774171 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ca015c8a0 0x7f9ca0072390 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f9c900097a0 tx=0x7f9c9002b8b0 comp rx=0 tx=0).stop 2024-04-14T23:52:36.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.326+0000 7f9c9f200640 1 -- 172.21.15.60:0/3756774171 shutdown_connections 2024-04-14T23:52:36.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.326+0000 7f9c9f200640 1 --2- 172.21.15.60:0/3756774171 >> v2:172.21.15.186:3300/0 conn(0x7f9ca00c6950 0x7f9ca01698f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:36.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.326+0000 7f9c9f200640 1 --2- 172.21.15.60:0/3756774171 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ca015c8a0 0x7f9ca0072390 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:36.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.326+0000 7f9c9f200640 1 -- 172.21.15.60:0/3756774171 >> 172.21.15.60:0/3756774171 conn(0x7f9ca00c8b50 msgr2=0x7f9ca00c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:52:36.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.326+0000 7f9c9f200640 1 -- 172.21.15.60:0/3756774171 shutdown_connections 2024-04-14T23:52:36.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.326+0000 7f9c9f200640 1 -- 172.21.15.60:0/3756774171 wait complete. 2024-04-14T23:52:36.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.329+0000 7f9c9f200640 1 Processor -- start 2024-04-14T23:52:36.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.329+0000 7f9c9f200640 1 -- start start 2024-04-14T23:52:36.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.330+0000 7f9c9f200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ca00c6950 0x7f9ca0155480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:36.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.330+0000 7f9c9f200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ca00c86d0 0x7f9ca0158bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:36.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.330+0000 7f9c9f200640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9ca015c8a0 0x7f9ca00caf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:36.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.330+0000 7f9c9f200640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca00728d0 con 0x7f9ca015c8a0 2024-04-14T23:52:36.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.330+0000 7f9c9f200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca0071e70 con 0x7f9ca00c86d0 2024-04-14T23:52:36.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.330+0000 7f9c9f200640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f9ca006b1c0 con 0x7f9ca00c6950 2024-04-14T23:52:36.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.330+0000 7f9c9de00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ca00c6950 0x7f9ca0155480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:36.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.330+0000 7f9c9de00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ca00c6950 0x7f9ca0155480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.60:56948/0 (socket says 172.21.15.60:56948) 2024-04-14T23:52:36.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.330+0000 7f9c9de00640 1 -- 172.21.15.60:0/1247205480 learned_addr learned my addr 172.21.15.60:0/1247205480 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:52:36.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.330+0000 7f9c95400640 1 --2- 172.21.15.60:0/1247205480 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ca00c86d0 0x7f9ca0158bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:36.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.330+0000 7f9c9e800640 1 --2- 172.21.15.60:0/1247205480 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9ca015c8a0 0x7f9ca00caf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:36.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.331+0000 7f9c9de00640 1 -- 172.21.15.60:0/1247205480 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ca00c86d0 msgr2=0x7f9ca0158bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:36.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.331+0000 7f9c9de00640 1 --2- 172.21.15.60:0/1247205480 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ca00c86d0 0x7f9ca0158bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:36.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.331+0000 7f9c9de00640 1 -- 172.21.15.60:0/1247205480 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9ca015c8a0 msgr2=0x7f9ca00caf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:36.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.331+0000 7f9c9de00640 1 --2- 172.21.15.60:0/1247205480 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9ca015c8a0 0x7f9ca00caf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:36.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.331+0000 7f9c9de00640 1 -- 172.21.15.60:0/1247205480 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ca01559c0 con 0x7f9ca00c6950 2024-04-14T23:52:36.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.331+0000 7f9c9de00640 1 --2- 172.21.15.60:0/1247205480 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ca00c6950 0x7f9ca0155480 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f9c9002bec0 tx=0x7f9c900046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:52:36.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.332+0000 7f9c96a00640 1 -- 172.21.15.60:0/1247205480 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c90035040 con 0x7f9ca00c6950 2024-04-14T23:52:36.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.332+0000 7f9c96a00640 1 -- 172.21.15.60:0/1247205480 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c9003ad00 con 0x7f9ca00c6950 2024-04-14T23:52:36.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.332+0000 7f9c96a00640 1 -- 172.21.15.60:0/1247205480 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c900466c0 con 0x7f9ca00c6950 2024-04-14T23:52:36.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.332+0000 7f9c9f200640 1 -- 172.21.15.60:0/1247205480 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ca015cec0 con 0x7f9ca00c6950 2024-04-14T23:52:36.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.332+0000 7f9c9f200640 1 -- 172.21.15.60:0/1247205480 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ca01f47e0 con 0x7f9ca00c6950 2024-04-14T23:52:36.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.333+0000 7f9c9f200640 1 -- 172.21.15.60:0/1247205480 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ca004f540 con 0x7f9ca00c6950 2024-04-14T23:52:36.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.334+0000 7f9c96a00640 1 -- 172.21.15.60:0/1247205480 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f9c90038020 con 0x7f9ca00c6950 2024-04-14T23:52:36.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.335+0000 7f9c96a00640 1 --2- 172.21.15.60:0/1247205480 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f9c7c06b990 0x7f9c7c06de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:36.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.335+0000 7f9c96a00640 1 -- 172.21.15.60:0/1247205480 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(27..27 src has 1..27) ==== 5997+0+0 (secure 0 0 0) 0x7f9c900aca10 con 0x7f9ca00c6950 2024-04-14T23:52:36.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.335+0000 7f9c95400640 1 --2- 172.21.15.60:0/1247205480 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f9c7c06b990 0x7f9c7c06de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:36.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.336+0000 7f9c95400640 1 --2- 172.21.15.60:0/1247205480 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f9c7c06b990 0x7f9c7c06de50 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f9ca015d110 tx=0x7f9c88009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:52:36.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.340+0000 7f9c96a00640 1 -- 172.21.15.60:0/1247205480 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f9c9007aea0 con 0x7f9ca00c6950 2024-04-14T23:52:36.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.580+0000 7f9c9f200640 1 -- 172.21.15.60:0/1247205480 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9ca00c7970 con 0x7f9ca00c6950 2024-04-14T23:52:36.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.582+0000 7f9c96a00640 1 -- 172.21.15.60:0/1247205480 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+6010 (secure 0 0 0) 0x7f9c9007ef40 con 0x7f9ca00c6950 2024-04-14T23:52:36.587 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:52:36.587 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:52:34.615770+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4839},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4839":{"gid":4839,"name":"c","rank":0,"incarnation":15,"state":"up:reconnect","state_seq":18,"addr":"172.21.15.60:6835/1714133698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1714133698},{"type":"v1","addr":"172.21.15.60:6835","nonce":1714133698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-14T23:52:36.587 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 16 2024-04-14T23:52:36.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.584+0000 7f9c9f200640 1 -- 172.21.15.60:0/1247205480 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f9c7c06b990 msgr2=0x7f9c7c06de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:36.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.584+0000 7f9c9f200640 1 --2- 172.21.15.60:0/1247205480 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f9c7c06b990 0x7f9c7c06de50 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f9ca015d110 tx=0x7f9c88009040 comp rx=0 tx=0).stop 2024-04-14T23:52:36.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.584+0000 7f9c9f200640 1 -- 172.21.15.60:0/1247205480 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ca00c6950 msgr2=0x7f9ca0155480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:36.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.584+0000 7f9c9f200640 1 --2- 172.21.15.60:0/1247205480 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ca00c6950 0x7f9ca0155480 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f9c9002bec0 tx=0x7f9c900046c0 comp rx=0 tx=0).stop 2024-04-14T23:52:36.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.584+0000 7f9c9f200640 1 -- 172.21.15.60:0/1247205480 shutdown_connections 2024-04-14T23:52:36.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.584+0000 7f9c9f200640 1 --2- 172.21.15.60:0/1247205480 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f9c7c06b990 0x7f9c7c06de50 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:36.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.584+0000 7f9c9f200640 1 --2- 172.21.15.60:0/1247205480 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9ca015c8a0 0x7f9ca00caf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:36.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.584+0000 7f9c9f200640 1 --2- 172.21.15.60:0/1247205480 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ca00c86d0 0x7f9ca0158bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:36.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.584+0000 7f9c9f200640 1 --2- 172.21.15.60:0/1247205480 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ca00c6950 0x7f9ca0155480 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:36.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.584+0000 7f9c9f200640 1 -- 172.21.15.60:0/1247205480 >> 172.21.15.60:0/1247205480 conn(0x7f9ca00c8b50 msgr2=0x7f9ca006bdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:52:36.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.584+0000 7f9c9f200640 1 -- 172.21.15.60:0/1247205480 shutdown_connections 2024-04-14T23:52:36.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:36.584+0000 7f9c9f200640 1 -- 172.21.15.60:0/1247205480 wait complete. 2024-04-14T23:52:36.608 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-14T23:52:38.609 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:52:38.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:38.834+0000 7f6e31200640 1 Processor -- start 2024-04-14T23:52:38.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:38.834+0000 7f6e31200640 1 -- start start 2024-04-14T23:52:38.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:38.836+0000 7f6e31200640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6e2c0c86d0 0x7f6e2c1625c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:38.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:38.836+0000 7f6e31200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6e2c0c6950 0x7f6e2c162b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:38.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:38.836+0000 7f6e31200640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e2c071bf0 con 0x7f6e2c15c8a0 2024-04-14T23:52:38.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:38.836+0000 7f6e31200640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f6e2c163040 con 0x7f6e2c0c86d0 2024-04-14T23:52:38.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:38.836+0000 7f6e31200640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f6e2c15cc80 con 0x7f6e2c0c6950 2024-04-14T23:52:38.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:38.836+0000 7f6e2aa00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6e2c0c86d0 0x7f6e2c1625c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:38.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:38.836+0000 7f6e2b400640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f6e2c15c8a0 0x7f6e2c072390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.60:38998/0 (socket says 172.21.15.60:38998) 2024-04-14T23:52:38.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:38.836+0000 7f6e2b400640 1 -- 172.21.15.60:0/27359559 learned_addr learned my addr 172.21.15.60:0/27359559 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:52:38.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:38.836+0000 7f6e2be00640 1 --2- 172.21.15.60:0/27359559 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6e2c0c6950 0x7f6e2c162b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:38.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:38.836+0000 7f6e2a000640 1 -- 172.21.15.60:0/27359559 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3900370984 0 0) 0x7f6e1c001110 con 0x7f6e2c15c8a0 2024-04-14T23:52:38.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:38.836+0000 7f6e2a000640 1 -- 172.21.15.60:0/27359559 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6e10003520 con 0x7f6e2c15c8a0 2024-04-14T23:52:38.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:38.836+0000 7f6e2be00640 1 -- 172.21.15.60:0/27359559 >> v2:172.21.15.60:3300/0 conn(0x7f6e2c0c86d0 msgr2=0x7f6e2c1625c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:52:38.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:38.836+0000 7f6e2be00640 1 --2- 172.21.15.60:0/27359559 >> v2:172.21.15.60:3300/0 conn(0x7f6e2c0c86d0 0x7f6e2c1625c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:38.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:38.836+0000 7f6e2be00640 1 -- 172.21.15.60:0/27359559 >> v1:172.21.15.186:6789/0 conn(0x7f6e2c15c8a0 legacy=0x7f6e2c072390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:38.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:38.836+0000 7f6e2be00640 1 -- 172.21.15.60:0/27359559 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e2c1632a0 con 0x7f6e2c0c6950 2024-04-14T23:52:38.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:38.836+0000 7f6e2aa00640 1 --2- 172.21.15.60:0/27359559 >> v2:172.21.15.60:3300/0 conn(0x7f6e2c0c86d0 0x7f6e2c1625c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:52:38.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:38.837+0000 7f6e2be00640 1 --2- 172.21.15.60:0/27359559 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6e2c0c6950 0x7f6e2c162b00 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6e24009d90 tx=0x7f6e2402bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e1ff5092fa228ce server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:52:38.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:38.837+0000 7f6e2a000640 1 -- 172.21.15.60:0/27359559 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e24039070 con 0x7f6e2c0c6950 2024-04-14T23:52:38.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:38.838+0000 7f6e2a000640 1 -- 172.21.15.60:0/27359559 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e2402cc50 con 0x7f6e2c0c6950 2024-04-14T23:52:38.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:38.838+0000 7f6e2a000640 1 -- 172.21.15.60:0/27359559 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e24034440 con 0x7f6e2c0c6950 2024-04-14T23:52:38.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:38.838+0000 7f6e31200640 1 -- 172.21.15.60:0/27359559 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6e2c0c6950 msgr2=0x7f6e2c162b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:38.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:38.838+0000 7f6e31200640 1 --2- 172.21.15.60:0/27359559 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6e2c0c6950 0x7f6e2c162b00 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6e24009d90 tx=0x7f6e2402bea0 comp rx=0 tx=0).stop 2024-04-14T23:52:38.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:38.838+0000 7f6e31200640 1 -- 172.21.15.60:0/27359559 shutdown_connections 2024-04-14T23:52:38.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:38.838+0000 7f6e31200640 1 --2- 172.21.15.60:0/27359559 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6e2c0c6950 0x7f6e2c162b00 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:38.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:38.838+0000 7f6e31200640 1 --2- 172.21.15.60:0/27359559 >> v2:172.21.15.60:3300/0 conn(0x7f6e2c0c86d0 0x7f6e2c1625c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:38.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:38.838+0000 7f6e31200640 1 -- 172.21.15.60:0/27359559 >> 172.21.15.60:0/27359559 conn(0x7f6e2c0c8b50 msgr2=0x7f6e2c0c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:52:38.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:38.839+0000 7f6e31200640 1 -- 172.21.15.60:0/27359559 shutdown_connections 2024-04-14T23:52:38.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:38.839+0000 7f6e31200640 1 -- 172.21.15.60:0/27359559 wait complete. 2024-04-14T23:52:38.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:38.840+0000 7f6e31200640 1 Processor -- start 2024-04-14T23:52:38.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:38.840+0000 7f6e31200640 1 -- start start 2024-04-14T23:52:38.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:38.841+0000 7f6e31200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6e2c0c6950 0x7f6e2c1619b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:38.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:38.841+0000 7f6e31200640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e2c0c86d0 0x7f6e2c0cb1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:38.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:38.841+0000 7f6e31200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6e2c15c8a0 0x7f6e2c1e94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:38.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:38.841+0000 7f6e31200640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f6e2c071e70 con 0x7f6e2c0c86d0 2024-04-14T23:52:38.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:38.841+0000 7f6e31200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6e2c0c7290 con 0x7f6e2c15c8a0 2024-04-14T23:52:38.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:38.841+0000 7f6e31200640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f6e2c06b1c0 con 0x7f6e2c0c6950 2024-04-14T23:52:38.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:38.842+0000 7f6e2aa00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e2c0c86d0 0x7f6e2c0cb1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:38.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:38.842+0000 7f6e2b400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6e2c0c6950 0x7f6e2c1619b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:38.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:38.842+0000 7f6e2aa00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e2c0c86d0 0x7f6e2c0cb1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49362/0 (socket says 172.21.15.60:49362) 2024-04-14T23:52:38.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:38.842+0000 7f6e2aa00640 1 -- 172.21.15.60:0/1369458271 learned_addr learned my addr 172.21.15.60:0/1369458271 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:52:38.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:38.842+0000 7f6e2be00640 1 --2- 172.21.15.60:0/1369458271 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6e2c15c8a0 0x7f6e2c1e94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:38.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:38.842+0000 7f6e2aa00640 1 -- 172.21.15.60:0/1369458271 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6e2c0c6950 msgr2=0x7f6e2c1619b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:38.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:38.842+0000 7f6e2aa00640 1 --2- 172.21.15.60:0/1369458271 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6e2c0c6950 0x7f6e2c1619b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:38.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:38.842+0000 7f6e2aa00640 1 -- 172.21.15.60:0/1369458271 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6e2c15c8a0 msgr2=0x7f6e2c1e94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:38.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:38.842+0000 7f6e2aa00640 1 --2- 172.21.15.60:0/1369458271 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6e2c15c8a0 0x7f6e2c1e94c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:38.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:38.842+0000 7f6e2aa00640 1 -- 172.21.15.60:0/1369458271 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e2c161fe0 con 0x7f6e2c0c86d0 2024-04-14T23:52:38.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:38.843+0000 7f6e2aa00640 1 --2- 172.21.15.60:0/1369458271 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e2c0c86d0 0x7f6e2c0cb1f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f6e1400e9b0 tx=0x7f6e1400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:52:38.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:38.843+0000 7f6e23400640 1 -- 172.21.15.60:0/1369458271 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e14017070 con 0x7f6e2c0c86d0 2024-04-14T23:52:38.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:38.843+0000 7f6e23400640 1 -- 172.21.15.60:0/1369458271 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e1400ca00 con 0x7f6e2c0c86d0 2024-04-14T23:52:38.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:38.843+0000 7f6e23400640 1 -- 172.21.15.60:0/1369458271 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e1400cdf0 con 0x7f6e2c0c86d0 2024-04-14T23:52:38.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:38.843+0000 7f6e31200640 1 -- 172.21.15.60:0/1369458271 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e2c1f4ea0 con 0x7f6e2c0c86d0 2024-04-14T23:52:38.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:38.843+0000 7f6e31200640 1 -- 172.21.15.60:0/1369458271 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e2c1f5460 con 0x7f6e2c0c86d0 2024-04-14T23:52:38.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:38.844+0000 7f6e31200640 1 -- 172.21.15.60:0/1369458271 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e2c0c6d30 con 0x7f6e2c0c86d0 2024-04-14T23:52:38.851 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:38.845+0000 7f6e23400640 1 -- 172.21.15.60:0/1369458271 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f6e1401a070 con 0x7f6e2c0c86d0 2024-04-14T23:52:38.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:38.846+0000 7f6e23400640 1 --2- 172.21.15.60:0/1369458271 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f6e0806b990 0x7f6e0806de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:38.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:38.846+0000 7f6e23400640 1 -- 172.21.15.60:0/1369458271 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5997+0+0 (secure 0 0 0) 0x7f6e1408a9a0 con 0x7f6e2c0c86d0 2024-04-14T23:52:38.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:38.847+0000 7f6e2b400640 1 --2- 172.21.15.60:0/1369458271 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f6e0806b990 0x7f6e0806de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:38.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:38.847+0000 7f6e2b400640 1 --2- 172.21.15.60:0/1369458271 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f6e0806b990 0x7f6e0806de50 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c005880 tx=0x7f6e1c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:52:38.856 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:38.852+0000 7f6e23400640 1 -- 172.21.15.60:0/1369458271 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6e2c0c6d30 con 0x7f6e2c0c86d0 2024-04-14T23:52:39.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:39.088+0000 7f6e31200640 1 -- 172.21.15.60:0/1369458271 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6e2c0c9830 con 0x7f6e2c0c86d0 2024-04-14T23:52:39.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:39.089+0000 7f6e23400640 1 -- 172.21.15.60:0/1369458271 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+6010 (secure 0 0 0) 0x7f6e2c0c9830 con 0x7f6e2c0c86d0 2024-04-14T23:52:39.096 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:52:39.096 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:52:34.615770+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4839},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4839":{"gid":4839,"name":"c","rank":0,"incarnation":15,"state":"up:reconnect","state_seq":18,"addr":"172.21.15.60:6835/1714133698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1714133698},{"type":"v1","addr":"172.21.15.60:6835","nonce":1714133698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-14T23:52:39.096 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 16 2024-04-14T23:52:39.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:39.092+0000 7f6e31200640 1 -- 172.21.15.60:0/1369458271 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f6e0806b990 msgr2=0x7f6e0806de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:39.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:39.092+0000 7f6e31200640 1 --2- 172.21.15.60:0/1369458271 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f6e0806b990 0x7f6e0806de50 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c005880 tx=0x7f6e1c009040 comp rx=0 tx=0).stop 2024-04-14T23:52:39.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:39.092+0000 7f6e31200640 1 -- 172.21.15.60:0/1369458271 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e2c0c86d0 msgr2=0x7f6e2c0cb1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:39.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:39.092+0000 7f6e31200640 1 --2- 172.21.15.60:0/1369458271 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e2c0c86d0 0x7f6e2c0cb1f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f6e1400e9b0 tx=0x7f6e1400ee70 comp rx=0 tx=0).stop 2024-04-14T23:52:39.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:39.092+0000 7f6e31200640 1 -- 172.21.15.60:0/1369458271 shutdown_connections 2024-04-14T23:52:39.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:39.092+0000 7f6e31200640 1 --2- 172.21.15.60:0/1369458271 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f6e0806b990 0x7f6e0806de50 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:39.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:39.092+0000 7f6e31200640 1 --2- 172.21.15.60:0/1369458271 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6e2c15c8a0 0x7f6e2c1e94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:39.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:39.092+0000 7f6e31200640 1 --2- 172.21.15.60:0/1369458271 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e2c0c86d0 0x7f6e2c0cb1f0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:39.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:39.092+0000 7f6e31200640 1 --2- 172.21.15.60:0/1369458271 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6e2c0c6950 0x7f6e2c1619b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:39.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:39.092+0000 7f6e31200640 1 -- 172.21.15.60:0/1369458271 >> 172.21.15.60:0/1369458271 conn(0x7f6e2c0c8b50 msgr2=0x7f6e2c06b560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:52:39.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:39.092+0000 7f6e31200640 1 -- 172.21.15.60:0/1369458271 shutdown_connections 2024-04-14T23:52:39.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:39.092+0000 7f6e31200640 1 -- 172.21.15.60:0/1369458271 wait complete. 2024-04-14T23:52:39.118 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-14T23:52:41.120 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:52:41.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.346+0000 7fad74c00640 1 Processor -- start 2024-04-14T23:52:41.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.347+0000 7fad74c00640 1 -- start start 2024-04-14T23:52:41.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.348+0000 7fad74c00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fad700c3410 0x7fad70152c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:41.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.349+0000 7fad74c00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fad70071220 0x7fad700ce380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:41.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.349+0000 7fad74c00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad700c9100 con 0x7fad70152820 2024-04-14T23:52:41.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.349+0000 7fad74c00640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fad700c93a0 con 0x7fad700c3410 2024-04-14T23:52:41.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.349+0000 7fad74c00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fad700c9680 con 0x7fad70071220 2024-04-14T23:52:41.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.349+0000 7fad6f400640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fad70152820 0x7fad70159280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:32986/0 (socket says 172.21.15.60:32986) 2024-04-14T23:52:41.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.349+0000 7fad6f400640 1 -- 172.21.15.60:0/788427619 learned_addr learned my addr 172.21.15.60:0/788427619 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:52:41.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.349+0000 7fad6ea00640 1 --2- 172.21.15.60:0/788427619 >> v2:172.21.15.60:3300/0 conn(0x7fad700c3410 0x7fad70152c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:41.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.349+0000 7fad6fe00640 1 --2- 172.21.15.60:0/788427619 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fad70071220 0x7fad700ce380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:41.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.349+0000 7fad6ea00640 1 -- 172.21.15.60:0/788427619 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fad70071220 msgr2=0x7fad700ce380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:52:41.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.350+0000 7fad6ea00640 1 --2- 172.21.15.60:0/788427619 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fad70071220 0x7fad700ce380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:41.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.350+0000 7fad6ea00640 1 -- 172.21.15.60:0/788427619 >> v1:172.21.15.60:6789/0 conn(0x7fad70152820 legacy=0x7fad70159280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:41.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.350+0000 7fad6ea00640 1 -- 172.21.15.60:0/788427619 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad700c9960 con 0x7fad700c3410 2024-04-14T23:52:41.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.350+0000 7fad6e000640 1 -- 172.21.15.60:0/788427619 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1129347621 0 0) 0x7fad700c9100 con 0x7fad70152820 2024-04-14T23:52:41.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.351+0000 7fad6ea00640 1 --2- 172.21.15.60:0/788427619 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad700c3410 0x7fad70152c00 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fad60009d90 tx=0x7fad6002bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=107c8bb97f80a2af server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:52:41.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.351+0000 7fad6e000640 1 -- 172.21.15.60:0/788427619 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad60039070 con 0x7fad700c3410 2024-04-14T23:52:41.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.351+0000 7fad6e000640 1 -- 172.21.15.60:0/788427619 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad6002cce0 con 0x7fad700c3410 2024-04-14T23:52:41.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.351+0000 7fad6e000640 1 -- 172.21.15.60:0/788427619 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad60034660 con 0x7fad700c3410 2024-04-14T23:52:41.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.352+0000 7fad74c00640 1 -- 172.21.15.60:0/788427619 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad700c3410 msgr2=0x7fad70152c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:41.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.352+0000 7fad74c00640 1 --2- 172.21.15.60:0/788427619 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad700c3410 0x7fad70152c00 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fad60009d90 tx=0x7fad6002bf30 comp rx=0 tx=0).stop 2024-04-14T23:52:41.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.352+0000 7fad74c00640 1 -- 172.21.15.60:0/788427619 shutdown_connections 2024-04-14T23:52:41.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.352+0000 7fad74c00640 1 --2- 172.21.15.60:0/788427619 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad700c3410 0x7fad70152c00 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:41.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.352+0000 7fad74c00640 1 --2- 172.21.15.60:0/788427619 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fad70071220 0x7fad700ce380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:41.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.352+0000 7fad74c00640 1 -- 172.21.15.60:0/788427619 >> 172.21.15.60:0/788427619 conn(0x7fad70066f10 msgr2=0x7fad70067320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:52:41.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.352+0000 7fad74c00640 1 -- 172.21.15.60:0/788427619 shutdown_connections 2024-04-14T23:52:41.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.352+0000 7fad74c00640 1 -- 172.21.15.60:0/788427619 wait complete. 2024-04-14T23:52:41.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.354+0000 7fad74c00640 1 Processor -- start 2024-04-14T23:52:41.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.354+0000 7fad74c00640 1 -- start start 2024-04-14T23:52:41.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.355+0000 7fad74c00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad70071220 0x7fad701f1960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:41.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.355+0000 7fad74c00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fad700c3410 0x7fad701f1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:41.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.355+0000 7fad74c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fad70152820 0x7fad700c5f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:41.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.355+0000 7fad74c00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fad700c8df0 con 0x7fad70071220 2024-04-14T23:52:41.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.355+0000 7fad74c00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fad70158d60 con 0x7fad70152820 2024-04-14T23:52:41.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.355+0000 7fad74c00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fad700b0fa0 con 0x7fad700c3410 2024-04-14T23:52:41.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.355+0000 7fad6f400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad70071220 0x7fad701f1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:41.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.355+0000 7fad6f400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad70071220 0x7fad701f1960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49378/0 (socket says 172.21.15.60:49378) 2024-04-14T23:52:41.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.355+0000 7fad6f400640 1 -- 172.21.15.60:0/3373154059 learned_addr learned my addr 172.21.15.60:0/3373154059 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:52:41.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.355+0000 7fad6ea00640 1 --2- 172.21.15.60:0/3373154059 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fad700c3410 0x7fad701f1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:41.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.355+0000 7fad6fe00640 1 --2- 172.21.15.60:0/3373154059 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fad70152820 0x7fad700c5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:41.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.356+0000 7fad6f400640 1 -- 172.21.15.60:0/3373154059 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fad700c3410 msgr2=0x7fad701f1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:41.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.356+0000 7fad6f400640 1 --2- 172.21.15.60:0/3373154059 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fad700c3410 0x7fad701f1ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:41.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.356+0000 7fad6f400640 1 -- 172.21.15.60:0/3373154059 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fad70152820 msgr2=0x7fad700c5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:41.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.356+0000 7fad6f400640 1 --2- 172.21.15.60:0/3373154059 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fad70152820 0x7fad700c5f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:41.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.356+0000 7fad6f400640 1 -- 172.21.15.60:0/3373154059 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad7015ba10 con 0x7fad70071220 2024-04-14T23:52:41.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.356+0000 7fad6f400640 1 --2- 172.21.15.60:0/3373154059 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad70071220 0x7fad701f1960 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fad5800c4d0 tx=0x7fad5800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:52:41.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.356+0000 7fad67400640 1 -- 172.21.15.60:0/3373154059 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad58018070 con 0x7fad70071220 2024-04-14T23:52:41.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.356+0000 7fad67400640 1 -- 172.21.15.60:0/3373154059 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad58007500 con 0x7fad70071220 2024-04-14T23:52:41.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.356+0000 7fad67400640 1 -- 172.21.15.60:0/3373154059 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad58004a20 con 0x7fad70071220 2024-04-14T23:52:41.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.357+0000 7fad74c00640 1 -- 172.21.15.60:0/3373154059 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad701f24c0 con 0x7fad70071220 2024-04-14T23:52:41.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.357+0000 7fad74c00640 1 -- 172.21.15.60:0/3373154059 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad700c5780 con 0x7fad70071220 2024-04-14T23:52:41.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.358+0000 7fad74c00640 1 -- 172.21.15.60:0/3373154059 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad7004f540 con 0x7fad70071220 2024-04-14T23:52:41.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.359+0000 7fad67400640 1 -- 172.21.15.60:0/3373154059 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7fad58004bc0 con 0x7fad70071220 2024-04-14T23:52:41.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.360+0000 7fad67400640 1 --2- 172.21.15.60:0/3373154059 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fad4c06b9c0 0x7fad4c06de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:41.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.360+0000 7fad67400640 1 -- 172.21.15.60:0/3373154059 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5997+0+0 (secure 0 0 0) 0x7fad58089be0 con 0x7fad70071220 2024-04-14T23:52:41.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.360+0000 7fad6ea00640 1 --2- 172.21.15.60:0/3373154059 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fad4c06b9c0 0x7fad4c06de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:41.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.361+0000 7fad6ea00640 1 --2- 172.21.15.60:0/3373154059 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fad4c06b9c0 0x7fad4c06de80 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fad60036000 tx=0x7fad60030000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:52:41.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.365+0000 7fad67400640 1 -- 172.21.15.60:0/3373154059 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fad58057fe0 con 0x7fad70071220 2024-04-14T23:52:41.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.607+0000 7fad74c00640 1 -- 172.21.15.60:0/3373154059 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fad70067470 con 0x7fad70071220 2024-04-14T23:52:41.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.608+0000 7fad67400640 1 -- 172.21.15.60:0/3373154059 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+6007 (secure 0 0 0) 0x7fad5805c080 con 0x7fad70071220 2024-04-14T23:52:41.613 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:52:41.613 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:52:40.560878+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4839},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4839":{"gid":4839,"name":"c","rank":0,"incarnation":15,"state":"up:rejoin","state_seq":20,"addr":"172.21.15.60:6835/1714133698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1714133698},{"type":"v1","addr":"172.21.15.60:6835","nonce":1714133698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-14T23:52:41.614 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 17 2024-04-14T23:52:41.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.611+0000 7fad74c00640 1 -- 172.21.15.60:0/3373154059 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fad4c06b9c0 msgr2=0x7fad4c06de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:41.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.611+0000 7fad74c00640 1 --2- 172.21.15.60:0/3373154059 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fad4c06b9c0 0x7fad4c06de80 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fad60036000 tx=0x7fad60030000 comp rx=0 tx=0).stop 2024-04-14T23:52:41.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.611+0000 7fad74c00640 1 -- 172.21.15.60:0/3373154059 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad70071220 msgr2=0x7fad701f1960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:41.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.611+0000 7fad74c00640 1 --2- 172.21.15.60:0/3373154059 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad70071220 0x7fad701f1960 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fad5800c4d0 tx=0x7fad5800c990 comp rx=0 tx=0).stop 2024-04-14T23:52:41.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.611+0000 7fad74c00640 1 -- 172.21.15.60:0/3373154059 shutdown_connections 2024-04-14T23:52:41.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.611+0000 7fad74c00640 1 --2- 172.21.15.60:0/3373154059 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fad4c06b9c0 0x7fad4c06de80 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:41.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.611+0000 7fad74c00640 1 --2- 172.21.15.60:0/3373154059 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fad70152820 0x7fad700c5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:41.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.611+0000 7fad74c00640 1 --2- 172.21.15.60:0/3373154059 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fad700c3410 0x7fad701f1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:41.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.611+0000 7fad74c00640 1 --2- 172.21.15.60:0/3373154059 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad70071220 0x7fad701f1960 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:41.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.612+0000 7fad74c00640 1 -- 172.21.15.60:0/3373154059 >> 172.21.15.60:0/3373154059 conn(0x7fad70066f10 msgr2=0x7fad700ca6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:52:41.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.612+0000 7fad74c00640 1 -- 172.21.15.60:0/3373154059 shutdown_connections 2024-04-14T23:52:41.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:41.612+0000 7fad74c00640 1 -- 172.21.15.60:0/3373154059 wait complete. 2024-04-14T23:52:41.635 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-14T23:52:43.637 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:52:43.868 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:43.863+0000 7eff53600640 1 Processor -- start 2024-04-14T23:52:43.868 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:43.863+0000 7eff53600640 1 -- start start 2024-04-14T23:52:43.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:43.865+0000 7eff53600640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7eff4c152cf0 0x7eff4c15c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:43.870 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:43.865+0000 7eff53600640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7eff4c159450 0x7eff4c15c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:43.870 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:43.865+0000 7eff53600640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff4c1530d0 con 0x7eff4c0c63c0 2024-04-14T23:52:43.870 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:43.865+0000 7eff53600640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7eff4c0751f0 con 0x7eff4c159450 2024-04-14T23:52:43.870 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:43.865+0000 7eff53600640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7eff4c0c8370 con 0x7eff4c152cf0 2024-04-14T23:52:43.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:43.865+0000 7eff51600640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7eff4c159450 0x7eff4c15c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:43.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:43.865+0000 7eff50c00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7eff4c152cf0 0x7eff4c15c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:43.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:43.865+0000 7eff50c00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7eff4c152cf0 0x7eff4c15c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.60:56652/0 (socket says 172.21.15.60:56652) 2024-04-14T23:52:43.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:43.865+0000 7eff50c00640 1 -- 172.21.15.60:0/2138601422 learned_addr learned my addr 172.21.15.60:0/2138601422 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:52:43.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:43.866+0000 7eff4b400640 1 -- 172.21.15.60:0/2138601422 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 939385173 0 0) 0x7eff4c1530d0 con 0x7eff4c0c63c0 2024-04-14T23:52:43.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:43.866+0000 7eff4b400640 1 -- 172.21.15.60:0/2138601422 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff34003520 con 0x7eff4c0c63c0 2024-04-14T23:52:43.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:43.866+0000 7eff50c00640 1 -- 172.21.15.60:0/2138601422 >> v2:172.21.15.186:3300/0 conn(0x7eff4c159450 msgr2=0x7eff4c15c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:52:43.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:43.866+0000 7eff50c00640 1 --2- 172.21.15.60:0/2138601422 >> v2:172.21.15.186:3300/0 conn(0x7eff4c159450 0x7eff4c15c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:43.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:43.866+0000 7eff50c00640 1 -- 172.21.15.60:0/2138601422 >> v1:172.21.15.186:6789/0 conn(0x7eff4c0c63c0 legacy=0x7eff4c0c3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:43.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:43.866+0000 7eff50c00640 1 -- 172.21.15.60:0/2138601422 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff4c0c3810 con 0x7eff4c152cf0 2024-04-14T23:52:43.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:43.866+0000 7eff51600640 1 --2- 172.21.15.60:0/2138601422 >> v2:172.21.15.186:3300/0 conn(0x7eff4c159450 0x7eff4c15c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:52:43.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:43.867+0000 7eff50c00640 1 --2- 172.21.15.60:0/2138601422 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7eff4c152cf0 0x7eff4c15c2a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7eff40009870 tx=0x7eff4002ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6163c2fc33d61f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:52:43.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:43.867+0000 7eff4b400640 1 -- 172.21.15.60:0/2138601422 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff4002f830 con 0x7eff4c152cf0 2024-04-14T23:52:43.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:43.867+0000 7eff4b400640 1 -- 172.21.15.60:0/2138601422 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff4002f9d0 con 0x7eff4c152cf0 2024-04-14T23:52:43.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:43.867+0000 7eff4b400640 1 -- 172.21.15.60:0/2138601422 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff4002fda0 con 0x7eff4c152cf0 2024-04-14T23:52:43.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:43.868+0000 7eff53600640 1 -- 172.21.15.60:0/2138601422 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7eff4c152cf0 msgr2=0x7eff4c15c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:43.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:43.868+0000 7eff53600640 1 --2- 172.21.15.60:0/2138601422 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7eff4c152cf0 0x7eff4c15c2a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7eff40009870 tx=0x7eff4002ba10 comp rx=0 tx=0).stop 2024-04-14T23:52:43.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:43.868+0000 7eff53600640 1 -- 172.21.15.60:0/2138601422 shutdown_connections 2024-04-14T23:52:43.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:43.868+0000 7eff53600640 1 --2- 172.21.15.60:0/2138601422 >> v2:172.21.15.186:3300/0 conn(0x7eff4c159450 0x7eff4c15c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:43.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:43.868+0000 7eff53600640 1 --2- 172.21.15.60:0/2138601422 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7eff4c152cf0 0x7eff4c15c2a0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:43.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:43.868+0000 7eff53600640 1 -- 172.21.15.60:0/2138601422 >> 172.21.15.60:0/2138601422 conn(0x7eff4c0cc170 msgr2=0x7eff4c0cc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:52:43.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:43.868+0000 7eff53600640 1 -- 172.21.15.60:0/2138601422 shutdown_connections 2024-04-14T23:52:43.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:43.868+0000 7eff53600640 1 -- 172.21.15.60:0/2138601422 wait complete. 2024-04-14T23:52:43.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:43.870+0000 7eff53600640 1 Processor -- start 2024-04-14T23:52:43.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:43.870+0000 7eff53600640 1 -- start start 2024-04-14T23:52:43.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:43.871+0000 7eff53600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7eff4c0c63c0 0x7eff4c157320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:43.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:43.871+0000 7eff53600640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7eff4c152cf0 0x7eff4c1f46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:43.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:43.871+0000 7eff53600640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7eff4c159450 0x7eff4c1f4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:43.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:43.871+0000 7eff53600640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7eff4c0c88f0 con 0x7eff4c159450 2024-04-14T23:52:43.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:43.871+0000 7eff53600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7eff4c0c8610 con 0x7eff4c0c63c0 2024-04-14T23:52:43.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:43.871+0000 7eff53600640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7eff4c0c4690 con 0x7eff4c152cf0 2024-04-14T23:52:43.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:43.871+0000 7eff51600640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7eff4c159450 0x7eff4c1f4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:43.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:43.871+0000 7eff50c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7eff4c0c63c0 0x7eff4c157320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:43.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:43.871+0000 7eff4be00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7eff4c152cf0 0x7eff4c1f46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:43.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:43.871+0000 7eff51600640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7eff4c159450 0x7eff4c1f4bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41842/0 (socket says 172.21.15.60:41842) 2024-04-14T23:52:43.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:43.871+0000 7eff51600640 1 -- 172.21.15.60:0/12574757 learned_addr learned my addr 172.21.15.60:0/12574757 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:52:43.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:43.872+0000 7eff51600640 1 -- 172.21.15.60:0/12574757 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7eff4c152cf0 msgr2=0x7eff4c1f46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:43.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:43.872+0000 7eff51600640 1 --2- 172.21.15.60:0/12574757 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7eff4c152cf0 0x7eff4c1f46b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:43.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:43.872+0000 7eff51600640 1 -- 172.21.15.60:0/12574757 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7eff4c0c63c0 msgr2=0x7eff4c157320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:43.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:43.872+0000 7eff51600640 1 --2- 172.21.15.60:0/12574757 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7eff4c0c63c0 0x7eff4c157320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:43.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:43.872+0000 7eff51600640 1 -- 172.21.15.60:0/12574757 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff4c1455e0 con 0x7eff4c159450 2024-04-14T23:52:43.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:43.872+0000 7eff51600640 1 --2- 172.21.15.60:0/12574757 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7eff4c159450 0x7eff4c1f4bf0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7eff4400efb0 tx=0x7eff4400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:52:43.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:43.873+0000 7eff48c00640 1 -- 172.21.15.60:0/12574757 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff44018020 con 0x7eff4c159450 2024-04-14T23:52:43.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:43.873+0000 7eff48c00640 1 -- 172.21.15.60:0/12574757 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff44014480 con 0x7eff4c159450 2024-04-14T23:52:43.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:43.873+0000 7eff53600640 1 -- 172.21.15.60:0/12574757 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff4c159d30 con 0x7eff4c159450 2024-04-14T23:52:43.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:43.873+0000 7eff48c00640 1 -- 172.21.15.60:0/12574757 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff44014850 con 0x7eff4c159450 2024-04-14T23:52:43.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:43.873+0000 7eff53600640 1 -- 172.21.15.60:0/12574757 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff4c151f40 con 0x7eff4c159450 2024-04-14T23:52:43.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:43.874+0000 7eff53600640 1 -- 172.21.15.60:0/12574757 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff4c0736e0 con 0x7eff4c159450 2024-04-14T23:52:43.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:43.875+0000 7eff48c00640 1 -- 172.21.15.60:0/12574757 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7eff44005d10 con 0x7eff4c159450 2024-04-14T23:52:43.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:43.876+0000 7eff48c00640 1 --2- 172.21.15.60:0/12574757 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7eff2c06b990 0x7eff2c06de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:43.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:43.876+0000 7eff48c00640 1 -- 172.21.15.60:0/12574757 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5997+0+0 (secure 0 0 0) 0x7eff4408b670 con 0x7eff4c159450 2024-04-14T23:52:43.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:43.876+0000 7eff50c00640 1 --2- 172.21.15.60:0/12574757 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7eff2c06b990 0x7eff2c06de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:43.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:43.877+0000 7eff50c00640 1 --2- 172.21.15.60:0/12574757 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7eff2c06b990 0x7eff2c06de50 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7eff40036000 tx=0x7eff40008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:52:43.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:43.880+0000 7eff48c00640 1 -- 172.21.15.60:0/12574757 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7eff4c0736e0 con 0x7eff4c159450 2024-04-14T23:52:44.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:44.127+0000 7eff53600640 1 -- 172.21.15.60:0/12574757 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7eff4c076970 con 0x7eff4c159450 2024-04-14T23:52:44.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:44.128+0000 7eff48c00640 1 -- 172.21.15.60:0/12574757 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+6007 (secure 0 0 0) 0x7eff44022030 con 0x7eff4c159450 2024-04-14T23:52:44.133 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:52:44.133 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:52:40.560878+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4839},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4839":{"gid":4839,"name":"c","rank":0,"incarnation":15,"state":"up:rejoin","state_seq":20,"addr":"172.21.15.60:6835/1714133698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1714133698},{"type":"v1","addr":"172.21.15.60:6835","nonce":1714133698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-14T23:52:44.134 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 17 2024-04-14T23:52:44.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:44.131+0000 7eff53600640 1 -- 172.21.15.60:0/12574757 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7eff2c06b990 msgr2=0x7eff2c06de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:44.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:44.131+0000 7eff53600640 1 --2- 172.21.15.60:0/12574757 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7eff2c06b990 0x7eff2c06de50 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7eff40036000 tx=0x7eff40008030 comp rx=0 tx=0).stop 2024-04-14T23:52:44.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:44.131+0000 7eff53600640 1 -- 172.21.15.60:0/12574757 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7eff4c159450 msgr2=0x7eff4c1f4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:44.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:44.131+0000 7eff53600640 1 --2- 172.21.15.60:0/12574757 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7eff4c159450 0x7eff4c1f4bf0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7eff4400efb0 tx=0x7eff4400c490 comp rx=0 tx=0).stop 2024-04-14T23:52:44.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:44.131+0000 7eff53600640 1 -- 172.21.15.60:0/12574757 shutdown_connections 2024-04-14T23:52:44.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:44.131+0000 7eff53600640 1 --2- 172.21.15.60:0/12574757 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7eff2c06b990 0x7eff2c06de50 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:44.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:44.131+0000 7eff53600640 1 --2- 172.21.15.60:0/12574757 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7eff4c159450 0x7eff4c1f4bf0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:44.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:44.131+0000 7eff53600640 1 --2- 172.21.15.60:0/12574757 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7eff4c152cf0 0x7eff4c1f46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:44.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:44.131+0000 7eff53600640 1 --2- 172.21.15.60:0/12574757 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7eff4c0c63c0 0x7eff4c157320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:44.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:44.131+0000 7eff53600640 1 -- 172.21.15.60:0/12574757 >> 172.21.15.60:0/12574757 conn(0x7eff4c0cc170 msgr2=0x7eff4c158a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:52:44.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:44.131+0000 7eff53600640 1 -- 172.21.15.60:0/12574757 shutdown_connections 2024-04-14T23:52:44.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:44.131+0000 7eff53600640 1 -- 172.21.15.60:0/12574757 wait complete. 2024-04-14T23:52:44.157 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 17, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-14T23:48:42.700188+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 17, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4839": { "addr": "172.21.15.60:6835/1714133698", "addrs": { "addrvec": [ { "addr": "172.21.15.60:6834", "nonce": 1714133698, "type": "v2" }, { "addr": "172.21.15.60:6835", "nonce": 1714133698, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4839, "incarnation": 15, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:rejoin", "state_seq": 20 } }, "last_failure": 0, "last_failure_osd_epoch": 27, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-14T23:52:40.560878+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4839 } } } ], "standbys": [ { "addr": "172.21.15.60:6837/1527815634", "addrs": { "addrvec": [ { "addr": "172.21.15.60:6836", "nonce": 1527815634, "type": "v2" }, { "addr": "172.21.15.60:6837", "nonce": 1527815634, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4197, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6838/3958319486", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6835", "nonce": 3958319486, "type": "v2" }, { "addr": "172.21.15.186:6838", "nonce": 3958319486, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4220, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6837/3910951185", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6834", "nonce": 3910951185, "type": "v2" }, { "addr": "172.21.15.186:6837", "nonce": 3910951185, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4264, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6839/2155020404", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6836", "nonce": 2155020404, "type": "v2" }, { "addr": "172.21.15.186:6839", "nonce": 2155020404, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4267, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-14T23:52:44.955 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-14T23:52:44.963 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-14T23:52:46.158 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:52:46.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.386+0000 7ff614e00640 1 Processor -- start 2024-04-14T23:52:46.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.386+0000 7ff614e00640 1 -- start start 2024-04-14T23:52:46.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.388+0000 7ff614e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6100b1e00 0x7ff6100c6950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:46.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.388+0000 7ff614e00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff6100c3d90 0x7ff6100c41e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:46.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.388+0000 7ff614e00640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff610152fb0 con 0x7ff610073320 2024-04-14T23:52:46.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.388+0000 7ff614e00640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7ff6100c6540 con 0x7ff6100c3d90 2024-04-14T23:52:46.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.388+0000 7ff614e00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7ff6100b21e0 con 0x7ff6100b1e00 2024-04-14T23:52:46.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.389+0000 7ff60fe00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff6100c3d90 0x7ff6100c41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:46.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.389+0000 7ff60f400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6100b1e00 0x7ff6100c6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:46.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.389+0000 7ff60ea00640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7ff610073320 0x7ff6100c7fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.60:47310/0 (socket says 172.21.15.60:47310) 2024-04-14T23:52:46.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.389+0000 7ff60ea00640 1 -- 172.21.15.60:0/758833627 learned_addr learned my addr 172.21.15.60:0/758833627 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:52:46.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.389+0000 7ff60fe00640 1 -- 172.21.15.60:0/758833627 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6100b1e00 msgr2=0x7ff6100c6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:52:46.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.389+0000 7ff60fe00640 1 --2- 172.21.15.60:0/758833627 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6100b1e00 0x7ff6100c6950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:46.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.389+0000 7ff60fe00640 1 -- 172.21.15.60:0/758833627 >> v1:172.21.15.186:6789/0 conn(0x7ff610073320 legacy=0x7ff6100c7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:46.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.389+0000 7ff60e000640 1 -- 172.21.15.60:0/758833627 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3029762444 0 0) 0x7ff610152fb0 con 0x7ff610073320 2024-04-14T23:52:46.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.389+0000 7ff60fe00640 1 -- 172.21.15.60:0/758833627 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff610159310 con 0x7ff6100c3d90 2024-04-14T23:52:46.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.391+0000 7ff60fe00640 1 --2- 172.21.15.60:0/758833627 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6100c3d90 0x7ff6100c41e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff608009d90 tx=0x7ff60802bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d99c9d908be9ecf9 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:52:46.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.391+0000 7ff60e000640 1 -- 172.21.15.60:0/758833627 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff608039070 con 0x7ff6100c3d90 2024-04-14T23:52:46.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.391+0000 7ff60e000640 1 -- 172.21.15.60:0/758833627 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff60802cce0 con 0x7ff6100c3d90 2024-04-14T23:52:46.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.391+0000 7ff60e000640 1 -- 172.21.15.60:0/758833627 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff608034660 con 0x7ff6100c3d90 2024-04-14T23:52:46.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.392+0000 7ff614e00640 1 -- 172.21.15.60:0/758833627 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6100c3d90 msgr2=0x7ff6100c41e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:46.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.392+0000 7ff614e00640 1 --2- 172.21.15.60:0/758833627 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6100c3d90 0x7ff6100c41e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff608009d90 tx=0x7ff60802bf30 comp rx=0 tx=0).stop 2024-04-14T23:52:46.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.392+0000 7ff614e00640 1 -- 172.21.15.60:0/758833627 shutdown_connections 2024-04-14T23:52:46.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.392+0000 7ff614e00640 1 --2- 172.21.15.60:0/758833627 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6100c3d90 0x7ff6100c41e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:46.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.392+0000 7ff614e00640 1 --2- 172.21.15.60:0/758833627 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6100b1e00 0x7ff6100c6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:46.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.392+0000 7ff614e00640 1 -- 172.21.15.60:0/758833627 >> 172.21.15.60:0/758833627 conn(0x7ff6100c8b50 msgr2=0x7ff6100c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:52:46.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.392+0000 7ff614e00640 1 -- 172.21.15.60:0/758833627 shutdown_connections 2024-04-14T23:52:46.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.392+0000 7ff614e00640 1 -- 172.21.15.60:0/758833627 wait complete. 2024-04-14T23:52:46.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.394+0000 7ff614e00640 1 Processor -- start 2024-04-14T23:52:46.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.394+0000 7ff614e00640 1 -- start start 2024-04-14T23:52:46.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.395+0000 7ff614e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff610073320 0x7ff610157c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:46.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.395+0000 7ff614e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6100b1e00 0x7ff6101f1830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:46.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.395+0000 7ff614e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6100c3d90 0x7ff6101f1d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:46.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.395+0000 7ff614e00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff6100c38b0 con 0x7ff610073320 2024-04-14T23:52:46.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.395+0000 7ff614e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff610159890 con 0x7ff6100c3d90 2024-04-14T23:52:46.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.395+0000 7ff614e00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7ff6101595b0 con 0x7ff6100b1e00 2024-04-14T23:52:46.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.395+0000 7ff60ea00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6100b1e00 0x7ff6101f1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:46.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.395+0000 7ff60ea00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6100b1e00 0x7ff6101f1830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.60:56684/0 (socket says 172.21.15.60:56684) 2024-04-14T23:52:46.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.396+0000 7ff60ea00640 1 -- 172.21.15.60:0/2002578779 learned_addr learned my addr 172.21.15.60:0/2002578779 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:52:46.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.396+0000 7ff60f400640 1 --2- 172.21.15.60:0/2002578779 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff610073320 0x7ff610157c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:46.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.396+0000 7ff60fe00640 1 --2- 172.21.15.60:0/2002578779 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6100c3d90 0x7ff6101f1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:46.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.396+0000 7ff60ea00640 1 -- 172.21.15.60:0/2002578779 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6100c3d90 msgr2=0x7ff6101f1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:46.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.396+0000 7ff60ea00640 1 --2- 172.21.15.60:0/2002578779 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6100c3d90 0x7ff6101f1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:46.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.396+0000 7ff60ea00640 1 -- 172.21.15.60:0/2002578779 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff610073320 msgr2=0x7ff610157c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:46.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.396+0000 7ff60ea00640 1 --2- 172.21.15.60:0/2002578779 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff610073320 0x7ff610157c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:46.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.396+0000 7ff60ea00640 1 -- 172.21.15.60:0/2002578779 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff61014edf0 con 0x7ff6100b1e00 2024-04-14T23:52:46.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.396+0000 7ff60ea00640 1 --2- 172.21.15.60:0/2002578779 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6100b1e00 0x7ff6101f1830 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc00c460 tx=0x7ff5fc00c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:52:46.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.397+0000 7ff607400640 1 -- 172.21.15.60:0/2002578779 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5fc018070 con 0x7ff6100b1e00 2024-04-14T23:52:46.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.397+0000 7ff607400640 1 -- 172.21.15.60:0/2002578779 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5fc007500 con 0x7ff6100b1e00 2024-04-14T23:52:46.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.397+0000 7ff607400640 1 -- 172.21.15.60:0/2002578779 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5fc004a20 con 0x7ff6100b1e00 2024-04-14T23:52:46.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.397+0000 7ff614e00640 1 -- 172.21.15.60:0/2002578779 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff61014eb10 con 0x7ff6100b1e00 2024-04-14T23:52:46.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.397+0000 7ff614e00640 1 -- 172.21.15.60:0/2002578779 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff61014fd30 con 0x7ff6100b1e00 2024-04-14T23:52:46.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.398+0000 7ff614e00640 1 -- 172.21.15.60:0/2002578779 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff61004f540 con 0x7ff6100b1e00 2024-04-14T23:52:46.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.399+0000 7ff607400640 1 -- 172.21.15.60:0/2002578779 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7ff5fc004bc0 con 0x7ff6100b1e00 2024-04-14T23:52:46.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.400+0000 7ff607400640 1 --2- 172.21.15.60:0/2002578779 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff5f006b990 0x7ff5f006de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:46.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.400+0000 7ff607400640 1 -- 172.21.15.60:0/2002578779 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(27..27 src has 1..27) ==== 5997+0+0 (secure 0 0 0) 0x7ff5fc089d10 con 0x7ff6100b1e00 2024-04-14T23:52:46.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.400+0000 7ff60f400640 1 --2- 172.21.15.60:0/2002578779 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff5f006b990 0x7ff5f006de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:46.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.401+0000 7ff60f400640 1 --2- 172.21.15.60:0/2002578779 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff5f006b990 0x7ff5f006de50 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff600009b40 tx=0x7ff600009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:52:46.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.405+0000 7ff607400640 1 -- 172.21.15.60:0/2002578779 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff5fc08f020 con 0x7ff6100b1e00 2024-04-14T23:52:46.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.649+0000 7ff614e00640 1 -- 172.21.15.60:0/2002578779 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff6100c99e0 con 0x7ff6100b1e00 2024-04-14T23:52:46.658 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.650+0000 7ff607400640 1 -- 172.21.15.60:0/2002578779 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+6007 (secure 0 0 0) 0x7ff5fc058080 con 0x7ff6100b1e00 2024-04-14T23:52:46.658 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:52:46.658 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:52:40.560878+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4839},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4839":{"gid":4839,"name":"c","rank":0,"incarnation":15,"state":"up:rejoin","state_seq":20,"addr":"172.21.15.60:6835/1714133698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1714133698},{"type":"v1","addr":"172.21.15.60:6835","nonce":1714133698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-14T23:52:46.658 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 17 2024-04-14T23:52:46.659 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.654+0000 7ff614e00640 1 -- 172.21.15.60:0/2002578779 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff5f006b990 msgr2=0x7ff5f006de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:46.660 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.654+0000 7ff614e00640 1 --2- 172.21.15.60:0/2002578779 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff5f006b990 0x7ff5f006de50 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff600009b40 tx=0x7ff600009190 comp rx=0 tx=0).stop 2024-04-14T23:52:46.660 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.654+0000 7ff614e00640 1 -- 172.21.15.60:0/2002578779 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6100b1e00 msgr2=0x7ff6101f1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:46.660 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.654+0000 7ff614e00640 1 --2- 172.21.15.60:0/2002578779 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6100b1e00 0x7ff6101f1830 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc00c460 tx=0x7ff5fc00c920 comp rx=0 tx=0).stop 2024-04-14T23:52:46.660 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.654+0000 7ff614e00640 1 -- 172.21.15.60:0/2002578779 shutdown_connections 2024-04-14T23:52:46.660 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.654+0000 7ff614e00640 1 --2- 172.21.15.60:0/2002578779 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff5f006b990 0x7ff5f006de50 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:46.660 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.654+0000 7ff614e00640 1 --2- 172.21.15.60:0/2002578779 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6100c3d90 0x7ff6101f1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:46.660 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.654+0000 7ff614e00640 1 --2- 172.21.15.60:0/2002578779 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6100b1e00 0x7ff6101f1830 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:46.660 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.654+0000 7ff614e00640 1 --2- 172.21.15.60:0/2002578779 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff610073320 0x7ff610157c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:46.660 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.654+0000 7ff614e00640 1 -- 172.21.15.60:0/2002578779 >> 172.21.15.60:0/2002578779 conn(0x7ff6100c8b50 msgr2=0x7ff6100cab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:52:46.660 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.655+0000 7ff614e00640 1 -- 172.21.15.60:0/2002578779 shutdown_connections 2024-04-14T23:52:46.660 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:46.655+0000 7ff614e00640 1 -- 172.21.15.60:0/2002578779 wait complete. 2024-04-14T23:52:46.686 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-14T23:52:48.687 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:52:48.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:48.915+0000 7f05c3000640 1 Processor -- start 2024-04-14T23:52:48.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:48.915+0000 7f05c3000640 1 -- start start 2024-04-14T23:52:48.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:48.917+0000 7f05c3000640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f05bc0b1e00 0x7f05bc0c6950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:48.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:48.917+0000 7f05c3000640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05bc152fb0 con 0x7f05bc073320 2024-04-14T23:52:48.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:48.917+0000 7f05c3000640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05bc159310 con 0x7f05bc0c3d90 2024-04-14T23:52:48.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:48.917+0000 7f05c3000640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f05bc0c6540 con 0x7f05bc0b1e00 2024-04-14T23:52:48.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:48.917+0000 7f05c1200640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f05bc073320 0x7f05bc0c7fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:46620/0 (socket says 172.21.15.60:46620) 2024-04-14T23:52:48.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:48.917+0000 7f05c1200640 1 -- 172.21.15.60:0/1824623987 learned_addr learned my addr 172.21.15.60:0/1824623987 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:52:48.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:48.917+0000 7f05c1c00640 1 --2- 172.21.15.60:0/1824623987 >> v2:172.21.15.186:3300/0 conn(0x7f05bc0b1e00 0x7f05bc0c6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:48.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:48.918+0000 7f05bbe00640 1 -- 172.21.15.60:0/1824623987 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1543550452 0 0) 0x7f05bc159310 con 0x7f05bc0c3d90 2024-04-14T23:52:48.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:48.918+0000 7f05bbe00640 1 -- 172.21.15.60:0/1824623987 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f05a4003520 con 0x7f05bc0c3d90 2024-04-14T23:52:48.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:48.918+0000 7f05bbe00640 1 -- 172.21.15.60:0/1824623987 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2583551381 0 0) 0x7f05bc152fb0 con 0x7f05bc073320 2024-04-14T23:52:48.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:48.918+0000 7f05bbe00640 1 -- 172.21.15.60:0/1824623987 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f05bc159310 con 0x7f05bc073320 2024-04-14T23:52:48.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:48.918+0000 7f05c1c00640 1 -- 172.21.15.60:0/1824623987 >> v1:172.21.15.186:6789/0 conn(0x7f05bc0c3d90 legacy=0x7f05bc0c41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:48.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:48.918+0000 7f05c1c00640 1 -- 172.21.15.60:0/1824623987 >> v1:172.21.15.60:6789/0 conn(0x7f05bc073320 legacy=0x7f05bc0c7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:48.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:48.918+0000 7f05c1c00640 1 -- 172.21.15.60:0/1824623987 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05bc0c38b0 con 0x7f05bc0b1e00 2024-04-14T23:52:48.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:48.920+0000 7f05c1c00640 1 --2- 172.21.15.60:0/1824623987 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f05bc0b1e00 0x7f05bc0c6950 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f05bc1594e0 tx=0x7f05b002b9d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af2f18d8bd56b8f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:52:48.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:48.920+0000 7f05bbe00640 1 -- 172.21.15.60:0/1824623987 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05b0034070 con 0x7f05bc0b1e00 2024-04-14T23:52:48.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:48.920+0000 7f05bbe00640 1 -- 172.21.15.60:0/1824623987 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05b0030890 con 0x7f05bc0b1e00 2024-04-14T23:52:48.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:48.920+0000 7f05bbe00640 1 -- 172.21.15.60:0/1824623987 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05b0030c60 con 0x7f05bc0b1e00 2024-04-14T23:52:48.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:48.921+0000 7f05c3000640 1 -- 172.21.15.60:0/1824623987 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f05bc0b1e00 msgr2=0x7f05bc0c6950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:48.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:48.921+0000 7f05c3000640 1 --2- 172.21.15.60:0/1824623987 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f05bc0b1e00 0x7f05bc0c6950 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f05bc1594e0 tx=0x7f05b002b9d0 comp rx=0 tx=0).stop 2024-04-14T23:52:48.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:48.921+0000 7f05c3000640 1 -- 172.21.15.60:0/1824623987 shutdown_connections 2024-04-14T23:52:48.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:48.921+0000 7f05c3000640 1 --2- 172.21.15.60:0/1824623987 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f05bc0b1e00 0x7f05bc0c6950 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:48.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:48.921+0000 7f05c3000640 1 -- 172.21.15.60:0/1824623987 >> 172.21.15.60:0/1824623987 conn(0x7f05bc0c8b50 msgr2=0x7f05bc0c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:52:48.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:48.921+0000 7f05c3000640 1 -- 172.21.15.60:0/1824623987 shutdown_connections 2024-04-14T23:52:48.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:48.921+0000 7f05c3000640 1 -- 172.21.15.60:0/1824623987 wait complete. 2024-04-14T23:52:48.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:48.923+0000 7f05c3000640 1 Processor -- start 2024-04-14T23:52:48.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:48.923+0000 7f05c3000640 1 -- start start 2024-04-14T23:52:48.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:48.924+0000 7f05c3000640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f05bc073320 0x7f05bc1ee890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:48.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:48.924+0000 7f05c3000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f05bc0b1e00 0x7f05bc1efd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:48.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:48.924+0000 7f05c3000640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f05bc0c3d90 0x7f05bc14bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:48.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:48.924+0000 7f05c3000640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f05bc159890 con 0x7f05bc073320 2024-04-14T23:52:48.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:48.924+0000 7f05c3000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f05bc0c5f80 con 0x7f05bc0b1e00 2024-04-14T23:52:48.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:48.924+0000 7f05c3000640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f05bc0c6260 con 0x7f05bc0c3d90 2024-04-14T23:52:48.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:48.924+0000 7f05c1c00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f05bc073320 0x7f05bc1ee890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:48.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:48.924+0000 7f05c1c00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f05bc073320 0x7f05bc1ee890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41856/0 (socket says 172.21.15.60:41856) 2024-04-14T23:52:48.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:48.924+0000 7f05c1c00640 1 -- 172.21.15.60:0/161195770 learned_addr learned my addr 172.21.15.60:0/161195770 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:52:48.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:48.924+0000 7f05c1200640 1 --2- 172.21.15.60:0/161195770 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f05bc0b1e00 0x7f05bc1efd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:48.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:48.924+0000 7f05c2600640 1 --2- 172.21.15.60:0/161195770 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f05bc0c3d90 0x7f05bc14bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:48.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:48.925+0000 7f05c1c00640 1 -- 172.21.15.60:0/161195770 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f05bc0c3d90 msgr2=0x7f05bc14bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:48.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:48.925+0000 7f05c1c00640 1 --2- 172.21.15.60:0/161195770 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f05bc0c3d90 0x7f05bc14bba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:48.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:48.925+0000 7f05c1c00640 1 -- 172.21.15.60:0/161195770 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f05bc0b1e00 msgr2=0x7f05bc1efd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:48.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:48.925+0000 7f05c1c00640 1 --2- 172.21.15.60:0/161195770 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f05bc0b1e00 0x7f05bc1efd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:48.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:48.925+0000 7f05c1c00640 1 -- 172.21.15.60:0/161195770 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05bc14c0e0 con 0x7f05bc073320 2024-04-14T23:52:48.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:48.925+0000 7f05c1c00640 1 --2- 172.21.15.60:0/161195770 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f05bc073320 0x7f05bc1ee890 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f05b002bfe0 tx=0x7f05b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:52:48.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:48.926+0000 7f05b9600640 1 -- 172.21.15.60:0/161195770 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05b0048070 con 0x7f05bc073320 2024-04-14T23:52:48.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:48.926+0000 7f05b9600640 1 -- 172.21.15.60:0/161195770 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05b0030690 con 0x7f05bc073320 2024-04-14T23:52:48.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:48.926+0000 7f05b9600640 1 -- 172.21.15.60:0/161195770 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05b0034040 con 0x7f05bc073320 2024-04-14T23:52:48.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:48.926+0000 7f05c3000640 1 -- 172.21.15.60:0/161195770 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05bc1f2720 con 0x7f05bc073320 2024-04-14T23:52:48.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:48.926+0000 7f05c3000640 1 -- 172.21.15.60:0/161195770 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05bc1f2ce0 con 0x7f05bc073320 2024-04-14T23:52:48.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:48.927+0000 7f05c3000640 1 -- 172.21.15.60:0/161195770 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05bc04f540 con 0x7f05bc073320 2024-04-14T23:52:48.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:48.928+0000 7f05b9600640 1 -- 172.21.15.60:0/161195770 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f05b004b070 con 0x7f05bc073320 2024-04-14T23:52:48.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:48.929+0000 7f05b9600640 1 --2- 172.21.15.60:0/161195770 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f059c06b990 0x7f059c06de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:48.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:48.929+0000 7f05b9600640 1 -- 172.21.15.60:0/161195770 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5997+0+0 (secure 0 0 0) 0x7f05b00ac7a0 con 0x7f05bc073320 2024-04-14T23:52:48.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:48.929+0000 7f05c1200640 1 --2- 172.21.15.60:0/161195770 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f059c06b990 0x7f059c06de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:48.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:48.930+0000 7f05c1200640 1 --2- 172.21.15.60:0/161195770 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f059c06b990 0x7f059c06de50 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f05bc0c3850 tx=0x7f05a8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:52:48.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:48.934+0000 7f05b9600640 1 -- 172.21.15.60:0/161195770 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f05bc14c0e0 con 0x7f05bc073320 2024-04-14T23:52:49.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:49.171+0000 7f05c3000640 1 -- 172.21.15.60:0/161195770 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f05bc0c8480 con 0x7f05bc073320 2024-04-14T23:52:49.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:49.172+0000 7f05b9600640 1 -- 172.21.15.60:0/161195770 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+6013 (secure 0 0 0) 0x7f05b007aa90 con 0x7f05bc073320 2024-04-14T23:52:49.177 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:52:49.177 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:52:46.686071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4839},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4839":{"gid":4839,"name":"c","rank":0,"incarnation":15,"state":"up:clientreplay","state_seq":22,"addr":"172.21.15.60:6835/1714133698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1714133698},{"type":"v1","addr":"172.21.15.60:6835","nonce":1714133698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-14T23:52:49.177 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 18 2024-04-14T23:52:49.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:49.174+0000 7f05c3000640 1 -- 172.21.15.60:0/161195770 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f059c06b990 msgr2=0x7f059c06de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:49.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:49.174+0000 7f05c3000640 1 --2- 172.21.15.60:0/161195770 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f059c06b990 0x7f059c06de50 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f05bc0c3850 tx=0x7f05a8009040 comp rx=0 tx=0).stop 2024-04-14T23:52:49.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:49.174+0000 7f05c3000640 1 -- 172.21.15.60:0/161195770 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f05bc073320 msgr2=0x7f05bc1ee890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:49.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:49.174+0000 7f05c3000640 1 --2- 172.21.15.60:0/161195770 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f05bc073320 0x7f05bc1ee890 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f05b002bfe0 tx=0x7f05b00046c0 comp rx=0 tx=0).stop 2024-04-14T23:52:49.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:49.175+0000 7f05c3000640 1 -- 172.21.15.60:0/161195770 shutdown_connections 2024-04-14T23:52:49.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:49.175+0000 7f05c3000640 1 --2- 172.21.15.60:0/161195770 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f059c06b990 0x7f059c06de50 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:49.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:49.175+0000 7f05c3000640 1 --2- 172.21.15.60:0/161195770 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f05bc0c3d90 0x7f05bc14bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:49.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:49.175+0000 7f05c3000640 1 --2- 172.21.15.60:0/161195770 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f05bc0b1e00 0x7f05bc1efd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:49.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:49.175+0000 7f05c3000640 1 --2- 172.21.15.60:0/161195770 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f05bc073320 0x7f05bc1ee890 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:49.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:49.175+0000 7f05c3000640 1 -- 172.21.15.60:0/161195770 >> 172.21.15.60:0/161195770 conn(0x7f05bc0c8b50 msgr2=0x7f05bc1525e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:52:49.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:49.175+0000 7f05c3000640 1 -- 172.21.15.60:0/161195770 shutdown_connections 2024-04-14T23:52:49.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:49.175+0000 7f05c3000640 1 -- 172.21.15.60:0/161195770 wait complete. 2024-04-14T23:52:49.196 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-14T23:52:51.197 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:52:51.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.425+0000 7fae23800640 1 Processor -- start 2024-04-14T23:52:51.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.426+0000 7fae23800640 1 -- start start 2024-04-14T23:52:51.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.427+0000 7fae23800640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fae1c0cd160 0x7fae1c15ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:51.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.427+0000 7fae23800640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae1c0cd540 0x7fae1c0b3190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:51.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.427+0000 7fae23800640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fae1c0cb750 0x7fae1c154b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:51.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.427+0000 7fae23800640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fae1c15b370 con 0x7fae1c0cd160 2024-04-14T23:52:51.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.427+0000 7fae23800640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fae1c0b3f50 con 0x7fae1c0cb750 2024-04-14T23:52:51.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.427+0000 7fae23800640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fae1c0b39b0 con 0x7fae1c0cd540 2024-04-14T23:52:51.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.428+0000 7fae20e00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fae1c0cd160 0x7fae1c15ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:51.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.428+0000 7fae20e00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fae1c0cd160 0x7fae1c15ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41862/0 (socket says 172.21.15.60:41862) 2024-04-14T23:52:51.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.428+0000 7fae20e00640 1 -- 172.21.15.60:0/2289426195 learned_addr learned my addr 172.21.15.60:0/2289426195 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:52:51.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.428+0000 7fae1be00640 1 --2- 172.21.15.60:0/2289426195 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae1c0cd540 0x7fae1c0b3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:51.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.428+0000 7fae21800640 1 --2- 172.21.15.60:0/2289426195 >> v2:172.21.15.186:3300/0 conn(0x7fae1c0cb750 0x7fae1c154b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:51.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.428+0000 7fae20e00640 1 -- 172.21.15.60:0/2289426195 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae1c0cd540 msgr2=0x7fae1c0b3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:52:51.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.428+0000 7fae20e00640 1 --2- 172.21.15.60:0/2289426195 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae1c0cd540 0x7fae1c0b3190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:51.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.428+0000 7fae20e00640 1 -- 172.21.15.60:0/2289426195 >> v2:172.21.15.186:3300/0 conn(0x7fae1c0cb750 msgr2=0x7fae1c154b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:52:51.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.428+0000 7fae20e00640 1 --2- 172.21.15.60:0/2289426195 >> v2:172.21.15.186:3300/0 conn(0x7fae1c0cb750 0x7fae1c154b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:51.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.428+0000 7fae20e00640 1 -- 172.21.15.60:0/2289426195 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae1c0b3c70 con 0x7fae1c0cd160 2024-04-14T23:52:51.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.428+0000 7fae1be00640 1 --2- 172.21.15.60:0/2289426195 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae1c0cd540 0x7fae1c0b3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-14T23:52:51.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.429+0000 7fae20e00640 1 --2- 172.21.15.60:0/2289426195 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae1c0cd160 0x7fae1c15ae30 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fae1c0b6680 tx=0x7fae1002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e36b0db484f90f16 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:52:51.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.430+0000 7fae1b400640 1 -- 172.21.15.60:0/2289426195 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae10035020 con 0x7fae1c0cd160 2024-04-14T23:52:51.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.430+0000 7fae1b400640 1 -- 172.21.15.60:0/2289426195 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae10031640 con 0x7fae1c0cd160 2024-04-14T23:52:51.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.430+0000 7fae1b400640 1 -- 172.21.15.60:0/2289426195 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae1002d040 con 0x7fae1c0cd160 2024-04-14T23:52:51.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.430+0000 7fae23800640 1 -- 172.21.15.60:0/2289426195 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae1c0cd160 msgr2=0x7fae1c15ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:51.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.430+0000 7fae23800640 1 --2- 172.21.15.60:0/2289426195 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae1c0cd160 0x7fae1c15ae30 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fae1c0b6680 tx=0x7fae1002b8f0 comp rx=0 tx=0).stop 2024-04-14T23:52:51.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.431+0000 7fae23800640 1 -- 172.21.15.60:0/2289426195 shutdown_connections 2024-04-14T23:52:51.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.431+0000 7fae23800640 1 --2- 172.21.15.60:0/2289426195 >> v2:172.21.15.186:3300/0 conn(0x7fae1c0cb750 0x7fae1c154b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:51.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.431+0000 7fae23800640 1 --2- 172.21.15.60:0/2289426195 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae1c0cd540 0x7fae1c0b3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:51.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.431+0000 7fae23800640 1 --2- 172.21.15.60:0/2289426195 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae1c0cd160 0x7fae1c15ae30 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:51.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.431+0000 7fae23800640 1 -- 172.21.15.60:0/2289426195 >> 172.21.15.60:0/2289426195 conn(0x7fae1c066f10 msgr2=0x7fae1c0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:52:51.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.431+0000 7fae23800640 1 -- 172.21.15.60:0/2289426195 shutdown_connections 2024-04-14T23:52:51.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.431+0000 7fae23800640 1 -- 172.21.15.60:0/2289426195 wait complete. 2024-04-14T23:52:51.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.432+0000 7fae23800640 1 Processor -- start 2024-04-14T23:52:51.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.433+0000 7fae23800640 1 -- start start 2024-04-14T23:52:51.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.433+0000 7fae23800640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae1c0cb750 0x7fae1c1f16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:51.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.433+0000 7fae23800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae1c0cd160 0x7fae1c0ca280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:51.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.434+0000 7fae23800640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae1c0cd540 0x7fae1c0c9390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:51.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.434+0000 7fae23800640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fae1c15ea70 con 0x7fae1c0cd540 2024-04-14T23:52:51.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.434+0000 7fae23800640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fae1c15e790 con 0x7fae1c0cd160 2024-04-14T23:52:51.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.434+0000 7fae23800640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fae1c0b36d0 con 0x7fae1c0cb750 2024-04-14T23:52:51.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.434+0000 7fae21800640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae1c0cd540 0x7fae1c0c9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:51.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.434+0000 7fae21800640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae1c0cd540 0x7fae1c0c9390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41872/0 (socket says 172.21.15.60:41872) 2024-04-14T23:52:51.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.434+0000 7fae21800640 1 -- 172.21.15.60:0/961875125 learned_addr learned my addr 172.21.15.60:0/961875125 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:52:51.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.434+0000 7fae20e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae1c0cb750 0x7fae1c1f16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:51.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.434+0000 7fae1be00640 1 --2- 172.21.15.60:0/961875125 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae1c0cd160 0x7fae1c0ca280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:51.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.434+0000 7fae21800640 1 -- 172.21.15.60:0/961875125 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae1c0cb750 msgr2=0x7fae1c1f16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:51.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.434+0000 7fae21800640 1 --2- 172.21.15.60:0/961875125 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae1c0cb750 0x7fae1c1f16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:51.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.434+0000 7fae21800640 1 -- 172.21.15.60:0/961875125 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae1c0cd160 msgr2=0x7fae1c0ca280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:51.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.434+0000 7fae21800640 1 --2- 172.21.15.60:0/961875125 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae1c0cd160 0x7fae1c0ca280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:51.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.434+0000 7fae21800640 1 -- 172.21.15.60:0/961875125 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae1c0c8d50 con 0x7fae1c0cd540 2024-04-14T23:52:51.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.435+0000 7fae21800640 1 --2- 172.21.15.60:0/961875125 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae1c0cd540 0x7fae1c0c9390 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fae1400ef00 tx=0x7fae1400b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:52:51.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.435+0000 7fae18c00640 1 -- 172.21.15.60:0/961875125 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae14018020 con 0x7fae1c0cd540 2024-04-14T23:52:51.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.435+0000 7fae18c00640 1 -- 172.21.15.60:0/961875125 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae1400bd20 con 0x7fae1c0cd540 2024-04-14T23:52:51.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.435+0000 7fae18c00640 1 -- 172.21.15.60:0/961875125 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae14014640 con 0x7fae1c0cd540 2024-04-14T23:52:51.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.435+0000 7fae23800640 1 -- 172.21.15.60:0/961875125 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae1c0ce590 con 0x7fae1c0cd540 2024-04-14T23:52:51.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.435+0000 7fae23800640 1 -- 172.21.15.60:0/961875125 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae1c0ceb50 con 0x7fae1c0cd540 2024-04-14T23:52:51.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.436+0000 7fae23800640 1 -- 172.21.15.60:0/961875125 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae1c076970 con 0x7fae1c0cd540 2024-04-14T23:52:51.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.437+0000 7fae18c00640 1 -- 172.21.15.60:0/961875125 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7fae14005d10 con 0x7fae1c0cd540 2024-04-14T23:52:51.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.438+0000 7fae18c00640 1 --2- 172.21.15.60:0/961875125 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fadfc06b9c0 0x7fadfc06de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:51.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.438+0000 7fae18c00640 1 -- 172.21.15.60:0/961875125 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5997+0+0 (secure 0 0 0) 0x7fae1408b1f0 con 0x7fae1c0cd540 2024-04-14T23:52:51.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.439+0000 7fae20e00640 1 --2- 172.21.15.60:0/961875125 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fadfc06b9c0 0x7fadfc06de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:51.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.440+0000 7fae20e00640 1 --2- 172.21.15.60:0/961875125 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fadfc06b9c0 0x7fadfc06de80 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fae10036000 tx=0x7fae1003c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:52:51.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.443+0000 7fae18c00640 1 -- 172.21.15.60:0/961875125 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fae14022030 con 0x7fae1c0cd540 2024-04-14T23:52:51.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.682+0000 7fae23800640 1 -- 172.21.15.60:0/961875125 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fae1c0c3a00 con 0x7fae1c0cd540 2024-04-14T23:52:51.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.684+0000 7fae18c00640 1 -- 172.21.15.60:0/961875125 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+6013 (secure 0 0 0) 0x7fae140595f0 con 0x7fae1c0cd540 2024-04-14T23:52:51.689 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:52:51.689 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:52:46.686071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4839},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4839":{"gid":4839,"name":"c","rank":0,"incarnation":15,"state":"up:clientreplay","state_seq":22,"addr":"172.21.15.60:6835/1714133698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1714133698},{"type":"v1","addr":"172.21.15.60:6835","nonce":1714133698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-14T23:52:51.689 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 18 2024-04-14T23:52:51.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.686+0000 7fae23800640 1 -- 172.21.15.60:0/961875125 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fadfc06b9c0 msgr2=0x7fadfc06de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:51.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.686+0000 7fae23800640 1 --2- 172.21.15.60:0/961875125 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fadfc06b9c0 0x7fadfc06de80 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fae10036000 tx=0x7fae1003c040 comp rx=0 tx=0).stop 2024-04-14T23:52:51.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.686+0000 7fae23800640 1 -- 172.21.15.60:0/961875125 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae1c0cd540 msgr2=0x7fae1c0c9390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:51.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.686+0000 7fae23800640 1 --2- 172.21.15.60:0/961875125 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae1c0cd540 0x7fae1c0c9390 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fae1400ef00 tx=0x7fae1400b480 comp rx=0 tx=0).stop 2024-04-14T23:52:51.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.686+0000 7fae23800640 1 -- 172.21.15.60:0/961875125 shutdown_connections 2024-04-14T23:52:51.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.686+0000 7fae23800640 1 --2- 172.21.15.60:0/961875125 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fadfc06b9c0 0x7fadfc06de80 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:51.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.686+0000 7fae23800640 1 --2- 172.21.15.60:0/961875125 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae1c0cd540 0x7fae1c0c9390 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:51.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.686+0000 7fae23800640 1 --2- 172.21.15.60:0/961875125 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae1c0cd160 0x7fae1c0ca280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:51.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.686+0000 7fae23800640 1 --2- 172.21.15.60:0/961875125 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae1c0cb750 0x7fae1c1f16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:51.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.686+0000 7fae23800640 1 -- 172.21.15.60:0/961875125 >> 172.21.15.60:0/961875125 conn(0x7fae1c066f10 msgr2=0x7fae1c0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:52:51.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.687+0000 7fae23800640 1 -- 172.21.15.60:0/961875125 shutdown_connections 2024-04-14T23:52:51.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:51.687+0000 7fae23800640 1 -- 172.21.15.60:0/961875125 wait complete. 2024-04-14T23:52:51.712 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-14T23:52:53.712 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:52:53.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:53.933+0000 7fc607000640 1 Processor -- start 2024-04-14T23:52:53.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:53.933+0000 7fc607000640 1 -- start start 2024-04-14T23:52:53.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:53.935+0000 7fc607000640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc600151ae0 0x7fc6000c3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:53.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:53.935+0000 7fc607000640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fc6000c52b0 0x7fc60006e210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:53.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:53.935+0000 7fc607000640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6000c5710 con 0x7fc60015ae00 2024-04-14T23:52:53.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:53.935+0000 7fc607000640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fc6000ccf80 con 0x7fc6000c52b0 2024-04-14T23:52:53.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:53.935+0000 7fc607000640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fc600072dc0 con 0x7fc600151ae0 2024-04-14T23:52:53.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:53.935+0000 7fc5ff400640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fc6000c52b0 0x7fc60006e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:53.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:53.935+0000 7fc5ffe00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc600151ae0 0x7fc6000c3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:53.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:53.935+0000 7fc5ff400640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fc6000c52b0 0x7fc60006e210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:53570/0 (socket says 172.21.15.60:53570) 2024-04-14T23:52:53.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:53.935+0000 7fc5ff400640 1 -- 172.21.15.60:0/3218448063 learned_addr learned my addr 172.21.15.60:0/3218448063 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:52:53.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:53.936+0000 7fc5ff400640 1 -- 172.21.15.60:0/3218448063 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc600151ae0 msgr2=0x7fc6000c3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:52:53.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:53.936+0000 7fc5ff400640 1 --2- 172.21.15.60:0/3218448063 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc600151ae0 0x7fc6000c3910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:53.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:53.936+0000 7fc5ff400640 1 -- 172.21.15.60:0/3218448063 >> v1:172.21.15.60:6789/0 conn(0x7fc60015ae00 legacy=0x7fc60015b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:53.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:53.936+0000 7fc5ff400640 1 -- 172.21.15.60:0/3218448063 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6000730a0 con 0x7fc6000c52b0 2024-04-14T23:52:53.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:53.936+0000 7fc5ffe00640 1 --2- 172.21.15.60:0/3218448063 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc600151ae0 0x7fc6000c3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:52:53.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:53.937+0000 7fc5ff400640 1 --2- 172.21.15.60:0/3218448063 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6000c52b0 0x7fc60006e210 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc5e8009870 tx=0x7fc5e802ba10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=34bc6c7229f66858 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:52:53.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:53.937+0000 7fc5fea00640 1 -- 172.21.15.60:0/3218448063 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5e802f830 con 0x7fc6000c52b0 2024-04-14T23:52:53.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:53.937+0000 7fc5fea00640 1 -- 172.21.15.60:0/3218448063 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5e802f9d0 con 0x7fc6000c52b0 2024-04-14T23:52:53.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:53.937+0000 7fc5fea00640 1 -- 172.21.15.60:0/3218448063 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5e802fda0 con 0x7fc6000c52b0 2024-04-14T23:52:53.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:53.938+0000 7fc607000640 1 -- 172.21.15.60:0/3218448063 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6000c52b0 msgr2=0x7fc60006e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:53.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:53.938+0000 7fc607000640 1 --2- 172.21.15.60:0/3218448063 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6000c52b0 0x7fc60006e210 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc5e8009870 tx=0x7fc5e802ba10 comp rx=0 tx=0).stop 2024-04-14T23:52:53.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:53.938+0000 7fc607000640 1 -- 172.21.15.60:0/3218448063 shutdown_connections 2024-04-14T23:52:53.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:53.938+0000 7fc607000640 1 --2- 172.21.15.60:0/3218448063 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6000c52b0 0x7fc60006e210 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:53.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:53.938+0000 7fc607000640 1 --2- 172.21.15.60:0/3218448063 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc600151ae0 0x7fc6000c3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:53.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:53.938+0000 7fc607000640 1 -- 172.21.15.60:0/3218448063 >> 172.21.15.60:0/3218448063 conn(0x7fc6000cc170 msgr2=0x7fc6000cc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:52:53.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:53.938+0000 7fc607000640 1 -- 172.21.15.60:0/3218448063 shutdown_connections 2024-04-14T23:52:53.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:53.938+0000 7fc607000640 1 -- 172.21.15.60:0/3218448063 wait complete. 2024-04-14T23:52:53.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:53.939+0000 7fc607000640 1 Processor -- start 2024-04-14T23:52:53.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:53.940+0000 7fc607000640 1 -- start start 2024-04-14T23:52:53.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:53.940+0000 7fc607000640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6000c52b0 0x7fc6001641d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:53.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:53.941+0000 7fc607000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc600151ae0 0x7fc60013ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:53.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:53.941+0000 7fc607000640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc60015ae00 0x7fc600150780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:53.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:53.941+0000 7fc607000640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fc60015b960 con 0x7fc6000c52b0 2024-04-14T23:52:53.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:53.941+0000 7fc607000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc6000726c0 con 0x7fc600151ae0 2024-04-14T23:52:53.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:53.941+0000 7fc607000640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fc6000729a0 con 0x7fc60015ae00 2024-04-14T23:52:53.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:53.941+0000 7fc5ffe00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6000c52b0 0x7fc6001641d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:53.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:53.941+0000 7fc5ffe00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6000c52b0 0x7fc6001641d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53460/0 (socket says 172.21.15.60:53460) 2024-04-14T23:52:53.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:53.941+0000 7fc5ffe00640 1 -- 172.21.15.60:0/3363502755 learned_addr learned my addr 172.21.15.60:0/3363502755 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:52:53.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:53.941+0000 7fc605000640 1 --2- 172.21.15.60:0/3363502755 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc60015ae00 0x7fc600150780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:53.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:53.941+0000 7fc5ff400640 1 --2- 172.21.15.60:0/3363502755 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc600151ae0 0x7fc60013ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:53.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:53.941+0000 7fc605000640 1 -- 172.21.15.60:0/3363502755 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc600151ae0 msgr2=0x7fc60013ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:53.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:53.941+0000 7fc605000640 1 --2- 172.21.15.60:0/3363502755 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc600151ae0 0x7fc60013ff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:53.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:53.941+0000 7fc605000640 1 -- 172.21.15.60:0/3363502755 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6000c52b0 msgr2=0x7fc6001641d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:53.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:53.941+0000 7fc605000640 1 --2- 172.21.15.60:0/3363502755 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6000c52b0 0x7fc6001641d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:53.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:53.941+0000 7fc605000640 1 -- 172.21.15.60:0/3363502755 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc600164d10 con 0x7fc60015ae00 2024-04-14T23:52:53.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:53.942+0000 7fc5ffe00640 1 --2- 172.21.15.60:0/3363502755 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6000c52b0 0x7fc6001641d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:52:53.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:53.942+0000 7fc605000640 1 --2- 172.21.15.60:0/3363502755 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc60015ae00 0x7fc600150780 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc5f800c3c0 tx=0x7fc5f800c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:52:53.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:53.942+0000 7fc5f7e00640 1 -- 172.21.15.60:0/3363502755 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5f8017070 con 0x7fc60015ae00 2024-04-14T23:52:53.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:53.942+0000 7fc5f7e00640 1 -- 172.21.15.60:0/3363502755 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5f80052d0 con 0x7fc60015ae00 2024-04-14T23:52:53.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:53.942+0000 7fc5f7e00640 1 -- 172.21.15.60:0/3363502755 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5f8003830 con 0x7fc60015ae00 2024-04-14T23:52:53.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:53.942+0000 7fc607000640 1 -- 172.21.15.60:0/3363502755 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc600164fd0 con 0x7fc60015ae00 2024-04-14T23:52:53.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:53.942+0000 7fc607000640 1 -- 172.21.15.60:0/3363502755 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc60013e6e0 con 0x7fc60015ae00 2024-04-14T23:52:53.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:53.943+0000 7fc607000640 1 -- 172.21.15.60:0/3363502755 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc600076970 con 0x7fc60015ae00 2024-04-14T23:52:53.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:53.944+0000 7fc5f7e00640 1 -- 172.21.15.60:0/3363502755 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7fc5f801a070 con 0x7fc60015ae00 2024-04-14T23:52:53.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:53.945+0000 7fc5f7e00640 1 --2- 172.21.15.60:0/3363502755 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fc5dc06b990 0x7fc5dc06de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:52:53.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:53.945+0000 7fc5f7e00640 1 -- 172.21.15.60:0/3363502755 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(27..27 src has 1..27) ==== 5997+0+0 (secure 0 0 0) 0x7fc5f8089b30 con 0x7fc60015ae00 2024-04-14T23:52:53.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:53.945+0000 7fc5ffe00640 1 --2- 172.21.15.60:0/3363502755 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fc5dc06b990 0x7fc5dc06de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:52:53.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:53.946+0000 7fc5ffe00640 1 --2- 172.21.15.60:0/3363502755 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fc5dc06b990 0x7fc5dc06de50 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc5f0009bd0 tx=0x7fc5f000b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:52:53.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:53.950+0000 7fc5f7e00640 1 -- 172.21.15.60:0/3363502755 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc5f8057fe0 con 0x7fc60015ae00 2024-04-14T23:52:54.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:54.193+0000 7fc607000640 1 -- 172.21.15.60:0/3363502755 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc60013fa30 con 0x7fc60015ae00 2024-04-14T23:52:54.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:54.194+0000 7fc5f7e00640 1 -- 172.21.15.60:0/3363502755 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6014 (secure 0 0 0) 0x7fc5f805c080 con 0x7fc60015ae00 2024-04-14T23:52:54.201 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:52:54.201 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:52:53.627240+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4839},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4839":{"gid":4839,"name":"c","rank":0,"incarnation":15,"state":"up:active","state_seq":24,"addr":"172.21.15.60:6835/1714133698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1714133698},{"type":"v1","addr":"172.21.15.60:6835","nonce":1714133698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4839,"qdb_cluster":[4839]},"id":1}]} 2024-04-14T23:52:54.201 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 19 2024-04-14T23:52:54.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:54.198+0000 7fc607000640 1 -- 172.21.15.60:0/3363502755 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fc5dc06b990 msgr2=0x7fc5dc06de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:54.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:54.198+0000 7fc607000640 1 --2- 172.21.15.60:0/3363502755 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fc5dc06b990 0x7fc5dc06de50 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc5f0009bd0 tx=0x7fc5f000b040 comp rx=0 tx=0).stop 2024-04-14T23:52:54.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:54.198+0000 7fc607000640 1 -- 172.21.15.60:0/3363502755 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc60015ae00 msgr2=0x7fc600150780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:52:54.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:54.198+0000 7fc607000640 1 --2- 172.21.15.60:0/3363502755 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc60015ae00 0x7fc600150780 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc5f800c3c0 tx=0x7fc5f800c880 comp rx=0 tx=0).stop 2024-04-14T23:52:54.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:54.198+0000 7fc607000640 1 -- 172.21.15.60:0/3363502755 shutdown_connections 2024-04-14T23:52:54.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:54.198+0000 7fc607000640 1 --2- 172.21.15.60:0/3363502755 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fc5dc06b990 0x7fc5dc06de50 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:54.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:54.198+0000 7fc607000640 1 --2- 172.21.15.60:0/3363502755 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc60015ae00 0x7fc600150780 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:54.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:54.198+0000 7fc607000640 1 --2- 172.21.15.60:0/3363502755 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc600151ae0 0x7fc60013ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:54.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:54.198+0000 7fc607000640 1 --2- 172.21.15.60:0/3363502755 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6000c52b0 0x7fc6001641d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:52:54.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:54.198+0000 7fc607000640 1 -- 172.21.15.60:0/3363502755 >> 172.21.15.60:0/3363502755 conn(0x7fc6000cc170 msgr2=0x7fc6000717c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:52:54.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:54.198+0000 7fc607000640 1 -- 172.21.15.60:0/3363502755 shutdown_connections 2024-04-14T23:52:54.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:52:54.198+0000 7fc607000640 1 -- 172.21.15.60:0/3363502755 wait complete. 2024-04-14T23:52:54.225 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c has gained rank=0, replacing gid=4276 2024-04-14T23:52:54.225 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 8 secs before reviving mds.a 2024-04-14T23:52:58.673 INFO:tasks.workunit.client.0.smithi060.stdout: create 271/39 file after the snapshot 2024-04-14T23:52:58.939 INFO:tasks.workunit.client.0.smithi060.stdout: create 272/.snap/snap-subdir-test 2024-04-14T23:52:59.095 INFO:tasks.workunit.client.0.smithi060.stdout: create 272/30 file after the snapshot 2024-04-14T23:52:59.097 INFO:tasks.workunit.client.0.smithi060.stdout: create 272/31 file after the snapshot 2024-04-14T23:52:59.099 INFO:tasks.workunit.client.0.smithi060.stdout: create 272/32 file after the snapshot 2024-04-14T23:52:59.102 INFO:tasks.workunit.client.0.smithi060.stdout: create 272/33 file after the snapshot 2024-04-14T23:52:59.105 INFO:tasks.workunit.client.0.smithi060.stdout: create 272/34 file after the snapshot 2024-04-14T23:52:59.107 INFO:tasks.workunit.client.0.smithi060.stdout: create 272/35 file after the snapshot 2024-04-14T23:52:59.110 INFO:tasks.workunit.client.0.smithi060.stdout: create 272/36 file after the snapshot 2024-04-14T23:52:59.112 INFO:tasks.workunit.client.0.smithi060.stdout: create 272/37 file after the snapshot 2024-04-14T23:52:59.114 INFO:tasks.workunit.client.0.smithi060.stdout: create 272/38 file after the snapshot 2024-04-14T23:52:59.117 INFO:tasks.workunit.client.0.smithi060.stdout: create 272/39 file after the snapshot 2024-04-14T23:52:59.120 INFO:tasks.workunit.client.0.smithi060.stdout: create 273/.snap/snap-subdir-test 2024-04-14T23:52:59.237 INFO:tasks.workunit.client.0.smithi060.stdout: create 273/30 file after the snapshot 2024-04-14T23:52:59.239 INFO:tasks.workunit.client.0.smithi060.stdout: create 273/31 file after the snapshot 2024-04-14T23:52:59.241 INFO:tasks.workunit.client.0.smithi060.stdout: create 273/32 file after the snapshot 2024-04-14T23:52:59.243 INFO:tasks.workunit.client.0.smithi060.stdout: create 273/33 file after the snapshot 2024-04-14T23:52:59.244 INFO:tasks.workunit.client.0.smithi060.stdout: create 273/34 file after the snapshot 2024-04-14T23:52:59.246 INFO:tasks.workunit.client.0.smithi060.stdout: create 273/35 file after the snapshot 2024-04-14T23:52:59.250 INFO:tasks.workunit.client.0.smithi060.stdout: create 273/36 file after the snapshot 2024-04-14T23:52:59.251 INFO:tasks.workunit.client.0.smithi060.stdout: create 273/37 file after the snapshot 2024-04-14T23:52:59.255 INFO:tasks.workunit.client.0.smithi060.stdout: create 273/38 file after the snapshot 2024-04-14T23:52:59.257 INFO:tasks.workunit.client.0.smithi060.stdout: create 273/39 file after the snapshot 2024-04-14T23:52:59.260 INFO:tasks.workunit.client.0.smithi060.stdout: create 274/.snap/snap-subdir-test 2024-04-14T23:52:59.325 INFO:tasks.workunit.client.0.smithi060.stdout: create 274/30 file after the snapshot 2024-04-14T23:52:59.327 INFO:tasks.workunit.client.0.smithi060.stdout: create 274/31 file after the snapshot 2024-04-14T23:52:59.329 INFO:tasks.workunit.client.0.smithi060.stdout: create 274/32 file after the snapshot 2024-04-14T23:52:59.334 INFO:tasks.workunit.client.0.smithi060.stdout: create 274/33 file after the snapshot 2024-04-14T23:52:59.334 INFO:tasks.workunit.client.0.smithi060.stdout: create 274/34 file after the snapshot 2024-04-14T23:52:59.336 INFO:tasks.workunit.client.0.smithi060.stdout: create 274/35 file after the snapshot 2024-04-14T23:52:59.339 INFO:tasks.workunit.client.0.smithi060.stdout: create 274/36 file after the snapshot 2024-04-14T23:52:59.341 INFO:tasks.workunit.client.0.smithi060.stdout: create 274/37 file after the snapshot 2024-04-14T23:52:59.344 INFO:tasks.workunit.client.0.smithi060.stdout: create 274/38 file after the snapshot 2024-04-14T23:52:59.346 INFO:tasks.workunit.client.0.smithi060.stdout: create 274/39 file after the snapshot 2024-04-14T23:52:59.349 INFO:tasks.workunit.client.0.smithi060.stdout: create 275/.snap/snap-subdir-test 2024-04-14T23:52:59.404 INFO:tasks.workunit.client.0.smithi060.stdout: create 275/30 file after the snapshot 2024-04-14T23:52:59.407 INFO:tasks.workunit.client.0.smithi060.stdout: create 275/31 file after the snapshot 2024-04-14T23:52:59.410 INFO:tasks.workunit.client.0.smithi060.stdout: create 275/32 file after the snapshot 2024-04-14T23:52:59.413 INFO:tasks.workunit.client.0.smithi060.stdout: create 275/33 file after the snapshot 2024-04-14T23:52:59.415 INFO:tasks.workunit.client.0.smithi060.stdout: create 275/34 file after the snapshot 2024-04-14T23:52:59.418 INFO:tasks.workunit.client.0.smithi060.stdout: create 275/35 file after the snapshot 2024-04-14T23:52:59.421 INFO:tasks.workunit.client.0.smithi060.stdout: create 275/36 file after the snapshot 2024-04-14T23:52:59.424 INFO:tasks.workunit.client.0.smithi060.stdout: create 275/37 file after the snapshot 2024-04-14T23:52:59.426 INFO:tasks.workunit.client.0.smithi060.stdout: create 275/38 file after the snapshot 2024-04-14T23:52:59.429 INFO:tasks.workunit.client.0.smithi060.stdout: create 275/39 file after the snapshot 2024-04-14T23:52:59.432 INFO:tasks.workunit.client.0.smithi060.stdout: create 276/.snap/snap-subdir-test 2024-04-14T23:52:59.493 INFO:tasks.workunit.client.0.smithi060.stdout: create 276/30 file after the snapshot 2024-04-14T23:52:59.498 INFO:tasks.workunit.client.0.smithi060.stdout: create 276/31 file after the snapshot 2024-04-14T23:52:59.499 INFO:tasks.workunit.client.0.smithi060.stdout: create 276/32 file after the snapshot 2024-04-14T23:52:59.501 INFO:tasks.workunit.client.0.smithi060.stdout: create 276/33 file after the snapshot 2024-04-14T23:52:59.505 INFO:tasks.workunit.client.0.smithi060.stdout: create 276/34 file after the snapshot 2024-04-14T23:52:59.507 INFO:tasks.workunit.client.0.smithi060.stdout: create 276/35 file after the snapshot 2024-04-14T23:52:59.510 INFO:tasks.workunit.client.0.smithi060.stdout: create 276/36 file after the snapshot 2024-04-14T23:52:59.513 INFO:tasks.workunit.client.0.smithi060.stdout: create 276/37 file after the snapshot 2024-04-14T23:52:59.593 INFO:tasks.workunit.client.0.smithi060.stdout: create 276/38 file after the snapshot 2024-04-14T23:52:59.596 INFO:tasks.workunit.client.0.smithi060.stdout: create 276/39 file after the snapshot 2024-04-14T23:52:59.599 INFO:tasks.workunit.client.0.smithi060.stdout: create 277/.snap/snap-subdir-test 2024-04-14T23:52:59.662 INFO:tasks.workunit.client.0.smithi060.stdout: create 277/30 file after the snapshot 2024-04-14T23:52:59.667 INFO:tasks.workunit.client.0.smithi060.stdout: create 277/31 file after the snapshot 2024-04-14T23:52:59.668 INFO:tasks.workunit.client.0.smithi060.stdout: create 277/32 file after the snapshot 2024-04-14T23:52:59.669 INFO:tasks.workunit.client.0.smithi060.stdout: create 277/33 file after the snapshot 2024-04-14T23:52:59.670 INFO:tasks.workunit.client.0.smithi060.stdout: create 277/34 file after the snapshot 2024-04-14T23:52:59.671 INFO:tasks.workunit.client.0.smithi060.stdout: create 277/35 file after the snapshot 2024-04-14T23:52:59.673 INFO:tasks.workunit.client.0.smithi060.stdout: create 277/36 file after the snapshot 2024-04-14T23:52:59.675 INFO:tasks.workunit.client.0.smithi060.stdout: create 277/37 file after the snapshot 2024-04-14T23:52:59.677 INFO:tasks.workunit.client.0.smithi060.stdout: create 277/38 file after the snapshot 2024-04-14T23:52:59.679 INFO:tasks.workunit.client.0.smithi060.stdout: create 277/39 file after the snapshot 2024-04-14T23:52:59.682 INFO:tasks.workunit.client.0.smithi060.stdout: create 278/.snap/snap-subdir-test 2024-04-14T23:52:59.743 INFO:tasks.workunit.client.0.smithi060.stdout: create 278/30 file after the snapshot 2024-04-14T23:52:59.746 INFO:tasks.workunit.client.0.smithi060.stdout: create 278/31 file after the snapshot 2024-04-14T23:52:59.749 INFO:tasks.workunit.client.0.smithi060.stdout: create 278/32 file after the snapshot 2024-04-14T23:52:59.752 INFO:tasks.workunit.client.0.smithi060.stdout: create 278/33 file after the snapshot 2024-04-14T23:52:59.755 INFO:tasks.workunit.client.0.smithi060.stdout: create 278/34 file after the snapshot 2024-04-14T23:52:59.758 INFO:tasks.workunit.client.0.smithi060.stdout: create 278/35 file after the snapshot 2024-04-14T23:52:59.761 INFO:tasks.workunit.client.0.smithi060.stdout: create 278/36 file after the snapshot 2024-04-14T23:52:59.764 INFO:tasks.workunit.client.0.smithi060.stdout: create 278/37 file after the snapshot 2024-04-14T23:52:59.766 INFO:tasks.workunit.client.0.smithi060.stdout: create 278/38 file after the snapshot 2024-04-14T23:52:59.769 INFO:tasks.workunit.client.0.smithi060.stdout: create 278/39 file after the snapshot 2024-04-14T23:52:59.772 INFO:tasks.workunit.client.0.smithi060.stdout: create 279/.snap/snap-subdir-test 2024-04-14T23:52:59.838 INFO:tasks.workunit.client.0.smithi060.stdout: create 279/30 file after the snapshot 2024-04-14T23:52:59.842 INFO:tasks.workunit.client.0.smithi060.stdout: create 279/31 file after the snapshot 2024-04-14T23:52:59.843 INFO:tasks.workunit.client.0.smithi060.stdout: create 279/32 file after the snapshot 2024-04-14T23:52:59.844 INFO:tasks.workunit.client.0.smithi060.stdout: create 279/33 file after the snapshot 2024-04-14T23:52:59.846 INFO:tasks.workunit.client.0.smithi060.stdout: create 279/34 file after the snapshot 2024-04-14T23:52:59.847 INFO:tasks.workunit.client.0.smithi060.stdout: create 279/35 file after the snapshot 2024-04-14T23:52:59.850 INFO:tasks.workunit.client.0.smithi060.stdout: create 279/36 file after the snapshot 2024-04-14T23:52:59.853 INFO:tasks.workunit.client.0.smithi060.stdout: create 279/37 file after the snapshot 2024-04-14T23:52:59.855 INFO:tasks.workunit.client.0.smithi060.stdout: create 279/38 file after the snapshot 2024-04-14T23:52:59.858 INFO:tasks.workunit.client.0.smithi060.stdout: create 279/39 file after the snapshot 2024-04-14T23:52:59.861 INFO:tasks.workunit.client.0.smithi060.stdout: create 280/.snap/snap-subdir-test 2024-04-14T23:52:59.919 INFO:tasks.workunit.client.0.smithi060.stdout: create 280/30 file after the snapshot 2024-04-14T23:52:59.921 INFO:tasks.workunit.client.0.smithi060.stdout: create 280/31 file after the snapshot 2024-04-14T23:53:00.188 INFO:tasks.workunit.client.0.smithi060.stdout: create 280/32 file after the snapshot 2024-04-14T23:53:00.191 INFO:tasks.workunit.client.0.smithi060.stdout: create 280/33 file after the snapshot 2024-04-14T23:53:00.194 INFO:tasks.workunit.client.0.smithi060.stdout: create 280/34 file after the snapshot 2024-04-14T23:53:00.196 INFO:tasks.workunit.client.0.smithi060.stdout: create 280/35 file after the snapshot 2024-04-14T23:53:00.199 INFO:tasks.workunit.client.0.smithi060.stdout: create 280/36 file after the snapshot 2024-04-14T23:53:00.202 INFO:tasks.workunit.client.0.smithi060.stdout: create 280/37 file after the snapshot 2024-04-14T23:53:00.205 INFO:tasks.workunit.client.0.smithi060.stdout: create 280/38 file after the snapshot 2024-04-14T23:53:00.207 INFO:tasks.workunit.client.0.smithi060.stdout: create 280/39 file after the snapshot 2024-04-14T23:53:00.210 INFO:tasks.workunit.client.0.smithi060.stdout: create 281/.snap/snap-subdir-test 2024-04-14T23:53:00.272 INFO:tasks.workunit.client.0.smithi060.stdout: create 281/30 file after the snapshot 2024-04-14T23:53:00.275 INFO:tasks.workunit.client.0.smithi060.stdout: create 281/31 file after the snapshot 2024-04-14T23:53:00.278 INFO:tasks.workunit.client.0.smithi060.stdout: create 281/32 file after the snapshot 2024-04-14T23:53:00.281 INFO:tasks.workunit.client.0.smithi060.stdout: create 281/33 file after the snapshot 2024-04-14T23:53:00.284 INFO:tasks.workunit.client.0.smithi060.stdout: create 281/34 file after the snapshot 2024-04-14T23:53:00.288 INFO:tasks.workunit.client.0.smithi060.stdout: create 281/35 file after the snapshot 2024-04-14T23:53:00.292 INFO:tasks.workunit.client.0.smithi060.stdout: create 281/36 file after the snapshot 2024-04-14T23:53:00.296 INFO:tasks.workunit.client.0.smithi060.stdout: create 281/37 file after the snapshot 2024-04-14T23:53:00.300 INFO:tasks.workunit.client.0.smithi060.stdout: create 281/38 file after the snapshot 2024-04-14T23:53:00.304 INFO:tasks.workunit.client.0.smithi060.stdout: create 281/39 file after the snapshot 2024-04-14T23:53:00.308 INFO:tasks.workunit.client.0.smithi060.stdout: create 282/.snap/snap-subdir-test 2024-04-14T23:53:00.359 INFO:tasks.ceph.mds.c.smithi060.stderr:2024-04-14T23:53:00.354+0000 7f25b7800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-14T23:53:00.360 INFO:tasks.workunit.client.0.smithi060.stdout: create 282/30 file after the snapshot 2024-04-14T23:53:00.363 INFO:tasks.workunit.client.0.smithi060.stdout: create 282/31 file after the snapshot 2024-04-14T23:53:00.366 INFO:tasks.workunit.client.0.smithi060.stdout: create 282/32 file after the snapshot 2024-04-14T23:53:00.368 INFO:tasks.workunit.client.0.smithi060.stdout: create 282/33 file after the snapshot 2024-04-14T23:53:00.371 INFO:tasks.workunit.client.0.smithi060.stdout: create 282/34 file after the snapshot 2024-04-14T23:53:00.374 INFO:tasks.workunit.client.0.smithi060.stdout: create 282/35 file after the snapshot 2024-04-14T23:53:00.377 INFO:tasks.workunit.client.0.smithi060.stdout: create 282/36 file after the snapshot 2024-04-14T23:53:00.379 INFO:tasks.workunit.client.0.smithi060.stdout: create 282/37 file after the snapshot 2024-04-14T23:53:00.382 INFO:tasks.workunit.client.0.smithi060.stdout: create 282/38 file after the snapshot 2024-04-14T23:53:00.384 INFO:tasks.workunit.client.0.smithi060.stdout: create 282/39 file after the snapshot 2024-04-14T23:53:00.387 INFO:tasks.workunit.client.0.smithi060.stdout: create 283/.snap/snap-subdir-test 2024-04-14T23:53:00.639 INFO:tasks.workunit.client.0.smithi060.stdout: create 283/30 file after the snapshot 2024-04-14T23:53:00.641 INFO:tasks.workunit.client.0.smithi060.stdout: create 283/31 file after the snapshot 2024-04-14T23:53:00.644 INFO:tasks.workunit.client.0.smithi060.stdout: create 283/32 file after the snapshot 2024-04-14T23:53:00.646 INFO:tasks.workunit.client.0.smithi060.stdout: create 283/33 file after the snapshot 2024-04-14T23:53:00.648 INFO:tasks.workunit.client.0.smithi060.stdout: create 283/34 file after the snapshot 2024-04-14T23:53:00.649 INFO:tasks.workunit.client.0.smithi060.stdout: create 283/35 file after the snapshot 2024-04-14T23:53:00.651 INFO:tasks.workunit.client.0.smithi060.stdout: create 283/36 file after the snapshot 2024-04-14T23:53:00.653 INFO:tasks.workunit.client.0.smithi060.stdout: create 283/37 file after the snapshot 2024-04-14T23:53:00.656 INFO:tasks.workunit.client.0.smithi060.stdout: create 283/38 file after the snapshot 2024-04-14T23:53:00.658 INFO:tasks.workunit.client.0.smithi060.stdout: create 283/39 file after the snapshot 2024-04-14T23:53:00.662 INFO:tasks.workunit.client.0.smithi060.stdout: create 284/.snap/snap-subdir-test 2024-04-14T23:53:00.712 INFO:tasks.workunit.client.0.smithi060.stdout: create 284/30 file after the snapshot 2024-04-14T23:53:00.716 INFO:tasks.workunit.client.0.smithi060.stdout: create 284/31 file after the snapshot 2024-04-14T23:53:00.720 INFO:tasks.workunit.client.0.smithi060.stdout: create 284/32 file after the snapshot 2024-04-14T23:53:00.724 INFO:tasks.workunit.client.0.smithi060.stdout: create 284/33 file after the snapshot 2024-04-14T23:53:00.728 INFO:tasks.workunit.client.0.smithi060.stdout: create 284/34 file after the snapshot 2024-04-14T23:53:00.733 INFO:tasks.workunit.client.0.smithi060.stdout: create 284/35 file after the snapshot 2024-04-14T23:53:00.737 INFO:tasks.workunit.client.0.smithi060.stdout: create 284/36 file after the snapshot 2024-04-14T23:53:00.741 INFO:tasks.workunit.client.0.smithi060.stdout: create 284/37 file after the snapshot 2024-04-14T23:53:00.745 INFO:tasks.workunit.client.0.smithi060.stdout: create 284/38 file after the snapshot 2024-04-14T23:53:00.749 INFO:tasks.workunit.client.0.smithi060.stdout: create 284/39 file after the snapshot 2024-04-14T23:53:00.753 INFO:tasks.workunit.client.0.smithi060.stdout: create 285/.snap/snap-subdir-test 2024-04-14T23:53:00.883 INFO:tasks.workunit.client.0.smithi060.stdout: create 285/30 file after the snapshot 2024-04-14T23:53:00.886 INFO:tasks.workunit.client.0.smithi060.stdout: create 285/31 file after the snapshot 2024-04-14T23:53:00.889 INFO:tasks.workunit.client.0.smithi060.stdout: create 285/32 file after the snapshot 2024-04-14T23:53:00.891 INFO:tasks.workunit.client.0.smithi060.stdout: create 285/33 file after the snapshot 2024-04-14T23:53:00.895 INFO:tasks.workunit.client.0.smithi060.stdout: create 285/34 file after the snapshot 2024-04-14T23:53:00.900 INFO:tasks.workunit.client.0.smithi060.stdout: create 285/35 file after the snapshot 2024-04-14T23:53:00.904 INFO:tasks.workunit.client.0.smithi060.stdout: create 285/36 file after the snapshot 2024-04-14T23:53:00.908 INFO:tasks.workunit.client.0.smithi060.stdout: create 285/37 file after the snapshot 2024-04-14T23:53:00.912 INFO:tasks.workunit.client.0.smithi060.stdout: create 285/38 file after the snapshot 2024-04-14T23:53:00.917 INFO:tasks.workunit.client.0.smithi060.stdout: create 285/39 file after the snapshot 2024-04-14T23:53:00.921 INFO:tasks.workunit.client.0.smithi060.stdout: create 286/.snap/snap-subdir-test 2024-04-14T23:53:00.980 INFO:tasks.workunit.client.0.smithi060.stdout: create 286/30 file after the snapshot 2024-04-14T23:53:00.982 INFO:tasks.workunit.client.0.smithi060.stdout: create 286/31 file after the snapshot 2024-04-14T23:53:00.985 INFO:tasks.workunit.client.0.smithi060.stdout: create 286/32 file after the snapshot 2024-04-14T23:53:00.988 INFO:tasks.workunit.client.0.smithi060.stdout: create 286/33 file after the snapshot 2024-04-14T23:53:00.991 INFO:tasks.workunit.client.0.smithi060.stdout: create 286/34 file after the snapshot 2024-04-14T23:53:00.993 INFO:tasks.workunit.client.0.smithi060.stdout: create 286/35 file after the snapshot 2024-04-14T23:53:00.996 INFO:tasks.workunit.client.0.smithi060.stdout: create 286/36 file after the snapshot 2024-04-14T23:53:00.999 INFO:tasks.workunit.client.0.smithi060.stdout: create 286/37 file after the snapshot 2024-04-14T23:53:01.002 INFO:tasks.workunit.client.0.smithi060.stdout: create 286/38 file after the snapshot 2024-04-14T23:53:01.005 INFO:tasks.workunit.client.0.smithi060.stdout: create 286/39 file after the snapshot 2024-04-14T23:53:01.008 INFO:tasks.workunit.client.0.smithi060.stdout: create 287/.snap/snap-subdir-test 2024-04-14T23:53:01.065 INFO:tasks.workunit.client.0.smithi060.stdout: create 287/30 file after the snapshot 2024-04-14T23:53:01.067 INFO:tasks.workunit.client.0.smithi060.stdout: create 287/31 file after the snapshot 2024-04-14T23:53:01.070 INFO:tasks.workunit.client.0.smithi060.stdout: create 287/32 file after the snapshot 2024-04-14T23:53:01.073 INFO:tasks.workunit.client.0.smithi060.stdout: create 287/33 file after the snapshot 2024-04-14T23:53:01.075 INFO:tasks.workunit.client.0.smithi060.stdout: create 287/34 file after the snapshot 2024-04-14T23:53:01.078 INFO:tasks.workunit.client.0.smithi060.stdout: create 287/35 file after the snapshot 2024-04-14T23:53:01.081 INFO:tasks.workunit.client.0.smithi060.stdout: create 287/36 file after the snapshot 2024-04-14T23:53:01.083 INFO:tasks.workunit.client.0.smithi060.stdout: create 287/37 file after the snapshot 2024-04-14T23:53:01.086 INFO:tasks.workunit.client.0.smithi060.stdout: create 287/38 file after the snapshot 2024-04-14T23:53:01.089 INFO:tasks.workunit.client.0.smithi060.stdout: create 287/39 file after the snapshot 2024-04-14T23:53:01.092 INFO:tasks.workunit.client.0.smithi060.stdout: create 288/.snap/snap-subdir-test 2024-04-14T23:53:01.145 INFO:tasks.workunit.client.0.smithi060.stdout: create 288/30 file after the snapshot 2024-04-14T23:53:01.147 INFO:tasks.workunit.client.0.smithi060.stdout: create 288/31 file after the snapshot 2024-04-14T23:53:01.150 INFO:tasks.workunit.client.0.smithi060.stdout: create 288/32 file after the snapshot 2024-04-14T23:53:01.153 INFO:tasks.workunit.client.0.smithi060.stdout: create 288/33 file after the snapshot 2024-04-14T23:53:01.155 INFO:tasks.workunit.client.0.smithi060.stdout: create 288/34 file after the snapshot 2024-04-14T23:53:01.158 INFO:tasks.workunit.client.0.smithi060.stdout: create 288/35 file after the snapshot 2024-04-14T23:53:01.161 INFO:tasks.workunit.client.0.smithi060.stdout: create 288/36 file after the snapshot 2024-04-14T23:53:01.163 INFO:tasks.workunit.client.0.smithi060.stdout: create 288/37 file after the snapshot 2024-04-14T23:53:01.166 INFO:tasks.workunit.client.0.smithi060.stdout: create 288/38 file after the snapshot 2024-04-14T23:53:01.169 INFO:tasks.workunit.client.0.smithi060.stdout: create 288/39 file after the snapshot 2024-04-14T23:53:01.172 INFO:tasks.workunit.client.0.smithi060.stdout: create 289/.snap/snap-subdir-test 2024-04-14T23:53:01.215 INFO:tasks.workunit.client.0.smithi060.stdout: create 289/30 file after the snapshot 2024-04-14T23:53:01.218 INFO:tasks.workunit.client.0.smithi060.stdout: create 289/31 file after the snapshot 2024-04-14T23:53:01.499 INFO:tasks.workunit.client.0.smithi060.stdout: create 289/32 file after the snapshot 2024-04-14T23:53:01.501 INFO:tasks.workunit.client.0.smithi060.stdout: create 289/33 file after the snapshot 2024-04-14T23:53:01.504 INFO:tasks.workunit.client.0.smithi060.stdout: create 289/34 file after the snapshot 2024-04-14T23:53:01.507 INFO:tasks.workunit.client.0.smithi060.stdout: create 289/35 file after the snapshot 2024-04-14T23:53:01.510 INFO:tasks.workunit.client.0.smithi060.stdout: create 289/36 file after the snapshot 2024-04-14T23:53:01.512 INFO:tasks.workunit.client.0.smithi060.stdout: create 289/37 file after the snapshot 2024-04-14T23:53:01.515 INFO:tasks.workunit.client.0.smithi060.stdout: create 289/38 file after the snapshot 2024-04-14T23:53:01.518 INFO:tasks.workunit.client.0.smithi060.stdout: create 289/39 file after the snapshot 2024-04-14T23:53:01.521 INFO:tasks.workunit.client.0.smithi060.stdout: create 290/.snap/snap-subdir-test 2024-04-14T23:53:01.571 INFO:tasks.workunit.client.0.smithi060.stdout: create 290/30 file after the snapshot 2024-04-14T23:53:01.573 INFO:tasks.workunit.client.0.smithi060.stdout: create 290/31 file after the snapshot 2024-04-14T23:53:01.578 INFO:tasks.workunit.client.0.smithi060.stdout: create 290/32 file after the snapshot 2024-04-14T23:53:01.578 INFO:tasks.workunit.client.0.smithi060.stdout: create 290/33 file after the snapshot 2024-04-14T23:53:01.580 INFO:tasks.workunit.client.0.smithi060.stdout: create 290/34 file after the snapshot 2024-04-14T23:53:01.583 INFO:tasks.workunit.client.0.smithi060.stdout: create 290/35 file after the snapshot 2024-04-14T23:53:01.828 INFO:tasks.workunit.client.0.smithi060.stdout: create 290/36 file after the snapshot 2024-04-14T23:53:02.059 INFO:tasks.workunit.client.0.smithi060.stdout: create 290/37 file after the snapshot 2024-04-14T23:53:02.061 INFO:tasks.workunit.client.0.smithi060.stdout: create 290/38 file after the snapshot 2024-04-14T23:53:02.064 INFO:tasks.workunit.client.0.smithi060.stdout: create 290/39 file after the snapshot 2024-04-14T23:53:02.067 INFO:tasks.workunit.client.0.smithi060.stdout: create 291/.snap/snap-subdir-test 2024-04-14T23:53:02.113 INFO:tasks.workunit.client.0.smithi060.stdout: create 291/30 file after the snapshot 2024-04-14T23:53:02.115 INFO:tasks.workunit.client.0.smithi060.stdout: create 291/31 file after the snapshot 2024-04-14T23:53:02.118 INFO:tasks.workunit.client.0.smithi060.stdout: create 291/32 file after the snapshot 2024-04-14T23:53:02.121 INFO:tasks.workunit.client.0.smithi060.stdout: create 291/33 file after the snapshot 2024-04-14T23:53:02.124 INFO:tasks.workunit.client.0.smithi060.stdout: create 291/34 file after the snapshot 2024-04-14T23:53:02.127 INFO:tasks.workunit.client.0.smithi060.stdout: create 291/35 file after the snapshot 2024-04-14T23:53:02.129 INFO:tasks.workunit.client.0.smithi060.stdout: create 291/36 file after the snapshot 2024-04-14T23:53:02.132 INFO:tasks.workunit.client.0.smithi060.stdout: create 291/37 file after the snapshot 2024-04-14T23:53:02.135 INFO:tasks.workunit.client.0.smithi060.stdout: create 291/38 file after the snapshot 2024-04-14T23:53:02.138 INFO:tasks.workunit.client.0.smithi060.stdout: create 291/39 file after the snapshot 2024-04-14T23:53:02.141 INFO:tasks.workunit.client.0.smithi060.stdout: create 292/.snap/snap-subdir-test 2024-04-14T23:53:02.200 INFO:tasks.workunit.client.0.smithi060.stdout: create 292/30 file after the snapshot 2024-04-14T23:53:02.202 INFO:tasks.workunit.client.0.smithi060.stdout: create 292/31 file after the snapshot 2024-04-14T23:53:02.205 INFO:tasks.workunit.client.0.smithi060.stdout: create 292/32 file after the snapshot 2024-04-14T23:53:02.208 INFO:tasks.workunit.client.0.smithi060.stdout: create 292/33 file after the snapshot 2024-04-14T23:53:02.210 INFO:tasks.workunit.client.0.smithi060.stdout: create 292/34 file after the snapshot 2024-04-14T23:53:02.213 INFO:tasks.workunit.client.0.smithi060.stdout: create 292/35 file after the snapshot 2024-04-14T23:53:02.216 INFO:tasks.workunit.client.0.smithi060.stdout: create 292/36 file after the snapshot 2024-04-14T23:53:02.219 INFO:tasks.workunit.client.0.smithi060.stdout: create 292/37 file after the snapshot 2024-04-14T23:53:02.221 INFO:tasks.workunit.client.0.smithi060.stdout: create 292/38 file after the snapshot 2024-04-14T23:53:02.224 INFO:tasks.workunit.client.0.smithi060.stdout: create 292/39 file after the snapshot 2024-04-14T23:53:02.226 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2024-04-14T23:53:02.226 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-14T23:53:02.226 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-14T23:53:02.227 INFO:tasks.workunit.client.0.smithi060.stdout: create 293/.snap/snap-subdir-test 2024-04-14T23:53:02.234 INFO:tasks.ceph.mds.a:Started 2024-04-14T23:53:02.234 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:53:02.266 INFO:tasks.workunit.client.0.smithi060.stdout: create 293/30 file after the snapshot 2024-04-14T23:53:02.268 INFO:tasks.workunit.client.0.smithi060.stdout: create 293/31 file after the snapshot 2024-04-14T23:53:02.269 INFO:tasks.workunit.client.0.smithi060.stdout: create 293/32 file after the snapshot 2024-04-14T23:53:02.271 INFO:tasks.workunit.client.0.smithi060.stdout: create 293/33 file after the snapshot 2024-04-14T23:53:02.272 INFO:tasks.workunit.client.0.smithi060.stdout: create 293/34 file after the snapshot 2024-04-14T23:53:02.274 INFO:tasks.workunit.client.0.smithi060.stdout: create 293/35 file after the snapshot 2024-04-14T23:53:02.275 INFO:tasks.workunit.client.0.smithi060.stdout: create 293/36 file after the snapshot 2024-04-14T23:53:02.277 INFO:tasks.workunit.client.0.smithi060.stdout: create 293/37 file after the snapshot 2024-04-14T23:53:02.278 INFO:tasks.workunit.client.0.smithi060.stdout: create 293/38 file after the snapshot 2024-04-14T23:53:02.280 INFO:tasks.workunit.client.0.smithi060.stdout: create 293/39 file after the snapshot 2024-04-14T23:53:02.281 INFO:tasks.workunit.client.0.smithi060.stdout: create 294/.snap/snap-subdir-test 2024-04-14T23:53:02.312 INFO:tasks.workunit.client.0.smithi060.stdout: create 294/30 file after the snapshot 2024-04-14T23:53:02.313 INFO:tasks.workunit.client.0.smithi060.stdout: create 294/31 file after the snapshot 2024-04-14T23:53:02.315 INFO:tasks.workunit.client.0.smithi060.stdout: create 294/32 file after the snapshot 2024-04-14T23:53:02.317 INFO:tasks.workunit.client.0.smithi060.stdout: create 294/33 file after the snapshot 2024-04-14T23:53:02.318 INFO:tasks.workunit.client.0.smithi060.stdout: create 294/34 file after the snapshot 2024-04-14T23:53:02.319 INFO:tasks.workunit.client.0.smithi060.stdout: create 294/35 file after the snapshot 2024-04-14T23:53:02.320 INFO:tasks.workunit.client.0.smithi060.stdout: create 294/36 file after the snapshot 2024-04-14T23:53:02.321 INFO:tasks.workunit.client.0.smithi060.stdout: create 294/37 file after the snapshot 2024-04-14T23:53:02.322 INFO:tasks.workunit.client.0.smithi060.stdout: create 294/38 file after the snapshot 2024-04-14T23:53:02.323 INFO:tasks.workunit.client.0.smithi060.stdout: create 294/39 file after the snapshot 2024-04-14T23:53:02.324 INFO:tasks.workunit.client.0.smithi060.stdout: create 295/.snap/snap-subdir-test 2024-04-14T23:53:02.358 INFO:tasks.workunit.client.0.smithi060.stdout: create 295/30 file after the snapshot 2024-04-14T23:53:02.359 INFO:tasks.workunit.client.0.smithi060.stdout: create 295/31 file after the snapshot 2024-04-14T23:53:02.360 INFO:tasks.workunit.client.0.smithi060.stdout: create 295/32 file after the snapshot 2024-04-14T23:53:02.362 INFO:tasks.workunit.client.0.smithi060.stdout: create 295/33 file after the snapshot 2024-04-14T23:53:02.363 INFO:tasks.workunit.client.0.smithi060.stdout: create 295/34 file after the snapshot 2024-04-14T23:53:02.364 INFO:tasks.workunit.client.0.smithi060.stdout: create 295/35 file after the snapshot 2024-04-14T23:53:02.366 INFO:tasks.workunit.client.0.smithi060.stdout: create 295/36 file after the snapshot 2024-04-14T23:53:02.367 INFO:tasks.workunit.client.0.smithi060.stdout: create 295/37 file after the snapshot 2024-04-14T23:53:02.369 INFO:tasks.workunit.client.0.smithi060.stdout: create 295/38 file after the snapshot 2024-04-14T23:53:02.370 INFO:tasks.workunit.client.0.smithi060.stdout: create 295/39 file after the snapshot 2024-04-14T23:53:02.372 INFO:tasks.workunit.client.0.smithi060.stdout: create 296/.snap/snap-subdir-test 2024-04-14T23:53:02.403 INFO:tasks.ceph.mds.a.smithi060.stdout:starting mds.a at 2024-04-14T23:53:02.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.430+0000 7f68f9200640 1 Processor -- start 2024-04-14T23:53:02.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.430+0000 7f68f9200640 1 -- start start 2024-04-14T23:53:02.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.432+0000 7f68f9200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f68f415c9b0 0x7f68f40c7280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:53:02.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.432+0000 7f68f9200640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68f41530d0 con 0x7f68f40c63c0 2024-04-14T23:53:02.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.432+0000 7f68f9200640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68f40751f0 con 0x7f68f4152cf0 2024-04-14T23:53:02.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.432+0000 7f68f9200640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f68f40c8370 con 0x7f68f415c9b0 2024-04-14T23:53:02.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.432+0000 7f68e9400640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f68f40c63c0 0x7f68f40c6b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33334/0 (socket says 172.21.15.60:33334) 2024-04-14T23:53:02.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.432+0000 7f68e9400640 1 -- 172.21.15.60:0/2417233068 learned_addr learned my addr 172.21.15.60:0/2417233068 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:53:02.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.432+0000 7f68f2800640 1 --2- 172.21.15.60:0/2417233068 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f68f415c9b0 0x7f68f40c7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:53:02.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.433+0000 7f68f1400640 1 -- 172.21.15.60:0/2417233068 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3474061849 0 0) 0x7f68f41530d0 con 0x7f68f40c63c0 2024-04-14T23:53:02.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.433+0000 7f68f1400640 1 -- 172.21.15.60:0/2417233068 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f68d8003520 con 0x7f68f40c63c0 2024-04-14T23:53:02.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.433+0000 7f68f2800640 1 -- 172.21.15.60:0/2417233068 >> v1:172.21.15.186:6789/0 conn(0x7f68f4152cf0 legacy=0x7f68f415c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:53:02.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.433+0000 7f68f1400640 1 -- 172.21.15.60:0/2417233068 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3070593355 0 0) 0x7f68f40751f0 con 0x7f68f4152cf0 2024-04-14T23:53:02.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.433+0000 7f68f2800640 1 -- 172.21.15.60:0/2417233068 >> v1:172.21.15.60:6789/0 conn(0x7f68f40c63c0 legacy=0x7f68f40c6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:53:02.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.433+0000 7f68f2800640 1 -- 172.21.15.60:0/2417233068 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68f40c3810 con 0x7f68f415c9b0 2024-04-14T23:53:02.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.434+0000 7f68f2800640 1 --2- 172.21.15.60:0/2417233068 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f68f415c9b0 0x7f68f40c7280 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f68ec009dc0 tx=0x7f68ec02bfa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=521bb37568d85b49 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:53:02.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.434+0000 7f68f1400640 1 -- 172.21.15.60:0/2417233068 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68ec039070 con 0x7f68f415c9b0 2024-04-14T23:53:02.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.434+0000 7f68f1400640 1 -- 172.21.15.60:0/2417233068 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68ec02cd50 con 0x7f68f415c9b0 2024-04-14T23:53:02.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.434+0000 7f68f1400640 1 -- 172.21.15.60:0/2417233068 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68ec034660 con 0x7f68f415c9b0 2024-04-14T23:53:02.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.435+0000 7f68f9200640 1 -- 172.21.15.60:0/2417233068 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f68f415c9b0 msgr2=0x7f68f40c7280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:53:02.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.435+0000 7f68f9200640 1 --2- 172.21.15.60:0/2417233068 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f68f415c9b0 0x7f68f40c7280 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f68ec009dc0 tx=0x7f68ec02bfa0 comp rx=0 tx=0).stop 2024-04-14T23:53:02.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.435+0000 7f68f9200640 1 -- 172.21.15.60:0/2417233068 shutdown_connections 2024-04-14T23:53:02.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.435+0000 7f68f9200640 1 --2- 172.21.15.60:0/2417233068 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f68f415c9b0 0x7f68f40c7280 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:53:02.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.435+0000 7f68f9200640 1 -- 172.21.15.60:0/2417233068 >> 172.21.15.60:0/2417233068 conn(0x7f68f40cc170 msgr2=0x7f68f40cc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:53:02.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.435+0000 7f68f9200640 1 -- 172.21.15.60:0/2417233068 shutdown_connections 2024-04-14T23:53:02.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.435+0000 7f68f9200640 1 -- 172.21.15.60:0/2417233068 wait complete. 2024-04-14T23:53:02.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.437+0000 7f68f9200640 1 Processor -- start 2024-04-14T23:53:02.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.437+0000 7f68f9200640 1 -- start start 2024-04-14T23:53:02.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.438+0000 7f68f9200640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68f40c63c0 0x7f68f41f0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:53:02.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.438+0000 7f68f9200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f68f4152cf0 0x7f68f41f4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:53:02.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.438+0000 7f68f9200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f68f415c9b0 0x7f68f41434c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:53:02.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.438+0000 7f68f9200640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f68f40c88f0 con 0x7f68f40c63c0 2024-04-14T23:53:02.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.438+0000 7f68f9200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f68f40c8610 con 0x7f68f415c9b0 2024-04-14T23:53:02.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.438+0000 7f68f9200640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f68f40c77c0 con 0x7f68f4152cf0 2024-04-14T23:53:02.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.438+0000 7f68f1e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68f40c63c0 0x7f68f41f0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:53:02.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.438+0000 7f68f1e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68f40c63c0 0x7f68f41f0d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44784/0 (socket says 172.21.15.60:44784) 2024-04-14T23:53:02.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.438+0000 7f68f1e00640 1 -- 172.21.15.60:0/3983518001 learned_addr learned my addr 172.21.15.60:0/3983518001 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:53:02.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.438+0000 7f68e9400640 1 --2- 172.21.15.60:0/3983518001 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f68f4152cf0 0x7f68f41f4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:53:02.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.438+0000 7f68f2800640 1 --2- 172.21.15.60:0/3983518001 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f68f415c9b0 0x7f68f41434c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:53:02.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.439+0000 7f68f1e00640 1 -- 172.21.15.60:0/3983518001 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f68f4152cf0 msgr2=0x7f68f41f4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:53:02.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.439+0000 7f68f1e00640 1 --2- 172.21.15.60:0/3983518001 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f68f4152cf0 0x7f68f41f4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:53:02.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.439+0000 7f68f1e00640 1 -- 172.21.15.60:0/3983518001 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f68f415c9b0 msgr2=0x7f68f41434c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:53:02.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.439+0000 7f68f1e00640 1 --2- 172.21.15.60:0/3983518001 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f68f415c9b0 0x7f68f41434c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:53:02.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.439+0000 7f68f1e00640 1 -- 172.21.15.60:0/3983518001 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68f4151d80 con 0x7f68f40c63c0 2024-04-14T23:53:02.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.439+0000 7f68f1e00640 1 --2- 172.21.15.60:0/3983518001 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68f40c63c0 0x7f68f41f0d80 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f68e400c460 tx=0x7f68e400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:53:02.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.439+0000 7f68eaa00640 1 -- 172.21.15.60:0/3983518001 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68e4005480 con 0x7f68f40c63c0 2024-04-14T23:53:02.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.439+0000 7f68eaa00640 1 -- 172.21.15.60:0/3983518001 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68e40035f0 con 0x7f68f40c63c0 2024-04-14T23:53:02.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.439+0000 7f68f9200640 1 -- 172.21.15.60:0/3983518001 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68f4152060 con 0x7f68f40c63c0 2024-04-14T23:53:02.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.439+0000 7f68eaa00640 1 -- 172.21.15.60:0/3983518001 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68e4003a00 con 0x7f68f40c63c0 2024-04-14T23:53:02.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.439+0000 7f68f9200640 1 -- 172.21.15.60:0/3983518001 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68f41f4500 con 0x7f68f40c63c0 2024-04-14T23:53:02.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.441+0000 7f68f9200640 1 -- 172.21.15.60:0/3983518001 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68f41f42f0 con 0x7f68f40c63c0 2024-04-14T23:53:02.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.442+0000 7f68eaa00640 1 -- 172.21.15.60:0/3983518001 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f68e4003ba0 con 0x7f68f40c63c0 2024-04-14T23:53:02.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.443+0000 7f68eaa00640 1 --2- 172.21.15.60:0/3983518001 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f68d006bb30 0x7f68d006dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:53:02.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.443+0000 7f68eaa00640 1 -- 172.21.15.60:0/3983518001 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5997+0+0 (secure 0 0 0) 0x7f68e408a600 con 0x7f68f40c63c0 2024-04-14T23:53:02.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.443+0000 7f68e9400640 1 --2- 172.21.15.60:0/3983518001 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f68d006bb30 0x7f68d006dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:53:02.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.444+0000 7f68e9400640 1 --2- 172.21.15.60:0/3983518001 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f68d006bb30 0x7f68d006dff0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f68f4150c50 tx=0x7f68dc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:53:02.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.448+0000 7f68eaa00640 1 -- 172.21.15.60:0/3983518001 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f68e4058970 con 0x7f68f40c63c0 2024-04-14T23:53:02.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.694+0000 7f68f9200640 1 -- 172.21.15.60:0/3983518001 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f68f40c9a90 con 0x7f68f40c63c0 2024-04-14T23:53:02.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.696+0000 7f68eaa00640 1 -- 172.21.15.60:0/3983518001 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6839 (secure 0 0 0) 0x7f68e405ca10 con 0x7f68f40c63c0 2024-04-14T23:53:02.701 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:53:02.701 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4986,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/1971580590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1971580590},{"type":"v1","addr":"172.21.15.60:6833","nonce":1971580590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:52:53.627240+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4839},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4839":{"gid":4839,"name":"c","rank":0,"incarnation":15,"state":"up:active","state_seq":24,"addr":"172.21.15.60:6835/1714133698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1714133698},{"type":"v1","addr":"172.21.15.60:6835","nonce":1714133698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4839,"qdb_cluster":[4839]},"id":1}]} 2024-04-14T23:53:02.701 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 20 2024-04-14T23:53:02.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.698+0000 7f68f9200640 1 -- 172.21.15.60:0/3983518001 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f68d006bb30 msgr2=0x7f68d006dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:53:02.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.698+0000 7f68f9200640 1 --2- 172.21.15.60:0/3983518001 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f68d006bb30 0x7f68d006dff0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f68f4150c50 tx=0x7f68dc009040 comp rx=0 tx=0).stop 2024-04-14T23:53:02.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.698+0000 7f68f9200640 1 -- 172.21.15.60:0/3983518001 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68f40c63c0 msgr2=0x7f68f41f0d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:53:02.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.698+0000 7f68f9200640 1 --2- 172.21.15.60:0/3983518001 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68f40c63c0 0x7f68f41f0d80 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f68e400c460 tx=0x7f68e400c920 comp rx=0 tx=0).stop 2024-04-14T23:53:02.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.699+0000 7f68f9200640 1 -- 172.21.15.60:0/3983518001 shutdown_connections 2024-04-14T23:53:02.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.699+0000 7f68f9200640 1 --2- 172.21.15.60:0/3983518001 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f68d006bb30 0x7f68d006dff0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:53:02.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.699+0000 7f68f9200640 1 --2- 172.21.15.60:0/3983518001 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f68f415c9b0 0x7f68f41434c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:53:02.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.699+0000 7f68f9200640 1 --2- 172.21.15.60:0/3983518001 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f68f4152cf0 0x7f68f41f4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:53:02.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.699+0000 7f68f9200640 1 --2- 172.21.15.60:0/3983518001 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68f40c63c0 0x7f68f41f0d80 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:53:02.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.699+0000 7f68f9200640 1 -- 172.21.15.60:0/3983518001 >> 172.21.15.60:0/3983518001 conn(0x7f68f40cc170 msgr2=0x7f68f40c4440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:53:02.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.699+0000 7f68f9200640 1 -- 172.21.15.60:0/3983518001 shutdown_connections 2024-04-14T23:53:02.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:53:02.699+0000 7f68f9200640 1 -- 172.21.15.60:0/3983518001 wait complete. 2024-04-14T23:53:02.715 INFO:tasks.workunit.client.0.smithi060.stdout: create 296/30 file after the snapshot 2024-04-14T23:53:02.716 INFO:tasks.workunit.client.0.smithi060.stdout: create 296/31 file after the snapshot 2024-04-14T23:53:02.718 INFO:tasks.workunit.client.0.smithi060.stdout: create 296/32 file after the snapshot 2024-04-14T23:53:02.720 INFO:tasks.workunit.client.0.smithi060.stdout: create 296/33 file after the snapshot 2024-04-14T23:53:02.722 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2024-04-14T23:53:02.722 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 87 secs before thrashing 2024-04-14T23:53:02.722 INFO:tasks.workunit.client.0.smithi060.stdout: create 296/34 file after the snapshot 2024-04-14T23:53:02.723 INFO:tasks.workunit.client.0.smithi060.stdout: create 296/35 file after the snapshot 2024-04-14T23:53:02.725 INFO:tasks.workunit.client.0.smithi060.stdout: create 296/36 file after the snapshot 2024-04-14T23:53:02.727 INFO:tasks.workunit.client.0.smithi060.stdout: create 296/37 file after the snapshot 2024-04-14T23:53:02.729 INFO:tasks.workunit.client.0.smithi060.stdout: create 296/38 file after the snapshot 2024-04-14T23:53:02.730 INFO:tasks.workunit.client.0.smithi060.stdout: create 296/39 file after the snapshot 2024-04-14T23:53:02.732 INFO:tasks.workunit.client.0.smithi060.stdout: create 297/.snap/snap-subdir-test 2024-04-14T23:53:02.770 INFO:tasks.workunit.client.0.smithi060.stdout: create 297/30 file after the snapshot 2024-04-14T23:53:02.774 INFO:tasks.workunit.client.0.smithi060.stdout: create 297/31 file after the snapshot 2024-04-14T23:53:02.778 INFO:tasks.workunit.client.0.smithi060.stdout: create 297/32 file after the snapshot 2024-04-14T23:53:02.782 INFO:tasks.workunit.client.0.smithi060.stdout: create 297/33 file after the snapshot 2024-04-14T23:53:02.786 INFO:tasks.workunit.client.0.smithi060.stdout: create 297/34 file after the snapshot 2024-04-14T23:53:02.790 INFO:tasks.workunit.client.0.smithi060.stdout: create 297/35 file after the snapshot 2024-04-14T23:53:02.794 INFO:tasks.workunit.client.0.smithi060.stdout: create 297/36 file after the snapshot 2024-04-14T23:53:02.797 INFO:tasks.workunit.client.0.smithi060.stdout: create 297/37 file after the snapshot 2024-04-14T23:53:02.802 INFO:tasks.workunit.client.0.smithi060.stdout: create 297/38 file after the snapshot 2024-04-14T23:53:02.806 INFO:tasks.workunit.client.0.smithi060.stdout: create 297/39 file after the snapshot 2024-04-14T23:53:02.809 INFO:tasks.workunit.client.0.smithi060.stdout: create 298/.snap/snap-subdir-test 2024-04-14T23:53:02.854 INFO:tasks.workunit.client.0.smithi060.stdout: create 298/30 file after the snapshot 2024-04-14T23:53:02.858 INFO:tasks.workunit.client.0.smithi060.stdout: create 298/31 file after the snapshot 2024-04-14T23:53:02.862 INFO:tasks.workunit.client.0.smithi060.stdout: create 298/32 file after the snapshot 2024-04-14T23:53:02.866 INFO:tasks.workunit.client.0.smithi060.stdout: create 298/33 file after the snapshot 2024-04-14T23:53:02.869 INFO:tasks.workunit.client.0.smithi060.stdout: create 298/34 file after the snapshot 2024-04-14T23:53:02.872 INFO:tasks.workunit.client.0.smithi060.stdout: create 298/35 file after the snapshot 2024-04-14T23:53:02.874 INFO:tasks.workunit.client.0.smithi060.stdout: create 298/36 file after the snapshot 2024-04-14T23:53:02.877 INFO:tasks.workunit.client.0.smithi060.stdout: create 298/37 file after the snapshot 2024-04-14T23:53:02.880 INFO:tasks.workunit.client.0.smithi060.stdout: create 298/38 file after the snapshot 2024-04-14T23:53:02.883 INFO:tasks.workunit.client.0.smithi060.stdout: create 298/39 file after the snapshot 2024-04-14T23:53:02.885 INFO:tasks.workunit.client.0.smithi060.stdout: create 299/.snap/snap-subdir-test 2024-04-14T23:53:02.932 INFO:tasks.workunit.client.0.smithi060.stdout: create 299/30 file after the snapshot 2024-04-14T23:53:02.943 INFO:tasks.workunit.client.0.smithi060.stdout: create 299/31 file after the snapshot 2024-04-14T23:53:02.945 INFO:tasks.workunit.client.0.smithi060.stdout: create 299/32 file after the snapshot 2024-04-14T23:53:02.948 INFO:tasks.workunit.client.0.smithi060.stdout: create 299/33 file after the snapshot 2024-04-14T23:53:02.951 INFO:tasks.workunit.client.0.smithi060.stdout: create 299/34 file after the snapshot 2024-04-14T23:53:02.954 INFO:tasks.workunit.client.0.smithi060.stdout: create 299/35 file after the snapshot 2024-04-14T23:53:02.957 INFO:tasks.workunit.client.0.smithi060.stdout: create 299/36 file after the snapshot 2024-04-14T23:53:02.960 INFO:tasks.workunit.client.0.smithi060.stdout: create 299/37 file after the snapshot 2024-04-14T23:53:02.964 INFO:tasks.workunit.client.0.smithi060.stdout: create 299/38 file after the snapshot 2024-04-14T23:53:02.968 INFO:tasks.workunit.client.0.smithi060.stdout: create 299/39 file after the snapshot 2024-04-14T23:53:02.972 INFO:tasks.workunit.client.0.smithi060.stdout:Take second snapshot .snap/test2 2024-04-14T23:53:03.012 INFO:tasks.workunit.client.0.smithi060.stdout:Copy content of .snap/test1 to copyofsnap1 ... 2024-04-14T23:53:03.713 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1' -> 'copyofsnap1/test1' 2024-04-14T23:53:03.713 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/100' -> 'copyofsnap1/test1/100' 2024-04-14T23:53:03.713 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/100/10' -> 'copyofsnap1/test1/100/10' 2024-04-14T23:53:03.713 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/100/11' -> 'copyofsnap1/test1/100/11' 2024-04-14T23:53:03.713 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/100/12' -> 'copyofsnap1/test1/100/12' 2024-04-14T23:53:03.713 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/100/13' -> 'copyofsnap1/test1/100/13' 2024-04-14T23:53:03.713 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/100/14' -> 'copyofsnap1/test1/100/14' 2024-04-14T23:53:03.713 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/100/15' -> 'copyofsnap1/test1/100/15' 2024-04-14T23:53:03.713 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/100/16' -> 'copyofsnap1/test1/100/16' 2024-04-14T23:53:03.713 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/100/17' -> 'copyofsnap1/test1/100/17' 2024-04-14T23:53:03.713 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/100/18' -> 'copyofsnap1/test1/100/18' 2024-04-14T23:53:03.714 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/100/19' -> 'copyofsnap1/test1/100/19' 2024-04-14T23:53:03.714 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/100/20' -> 'copyofsnap1/test1/100/20' 2024-04-14T23:53:03.714 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/101' -> 'copyofsnap1/test1/101' 2024-04-14T23:53:03.714 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/101/10' -> 'copyofsnap1/test1/101/10' 2024-04-14T23:53:03.714 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/101/11' -> 'copyofsnap1/test1/101/11' 2024-04-14T23:53:03.714 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/101/12' -> 'copyofsnap1/test1/101/12' 2024-04-14T23:53:03.714 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/101/13' -> 'copyofsnap1/test1/101/13' 2024-04-14T23:53:03.714 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/101/14' -> 'copyofsnap1/test1/101/14' 2024-04-14T23:53:03.714 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/101/15' -> 'copyofsnap1/test1/101/15' 2024-04-14T23:53:03.714 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/101/16' -> 'copyofsnap1/test1/101/16' 2024-04-14T23:53:03.714 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/101/17' -> 'copyofsnap1/test1/101/17' 2024-04-14T23:53:03.714 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/101/18' -> 'copyofsnap1/test1/101/18' 2024-04-14T23:53:03.714 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/101/19' -> 'copyofsnap1/test1/101/19' 2024-04-14T23:53:03.714 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/101/20' -> 'copyofsnap1/test1/101/20' 2024-04-14T23:53:03.715 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/102' -> 'copyofsnap1/test1/102' 2024-04-14T23:53:03.715 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/102/10' -> 'copyofsnap1/test1/102/10' 2024-04-14T23:53:03.715 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/102/11' -> 'copyofsnap1/test1/102/11' 2024-04-14T23:53:03.715 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/102/12' -> 'copyofsnap1/test1/102/12' 2024-04-14T23:53:03.715 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/102/13' -> 'copyofsnap1/test1/102/13' 2024-04-14T23:53:03.715 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/102/14' -> 'copyofsnap1/test1/102/14' 2024-04-14T23:53:03.715 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/102/15' -> 'copyofsnap1/test1/102/15' 2024-04-14T23:53:03.715 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/102/16' -> 'copyofsnap1/test1/102/16' 2024-04-14T23:53:03.715 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/102/17' -> 'copyofsnap1/test1/102/17' 2024-04-14T23:53:03.715 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/102/18' -> 'copyofsnap1/test1/102/18' 2024-04-14T23:53:03.715 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/102/19' -> 'copyofsnap1/test1/102/19' 2024-04-14T23:53:03.715 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/102/20' -> 'copyofsnap1/test1/102/20' 2024-04-14T23:53:03.715 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/103' -> 'copyofsnap1/test1/103' 2024-04-14T23:53:03.715 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/103/10' -> 'copyofsnap1/test1/103/10' 2024-04-14T23:53:03.715 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/103/11' -> 'copyofsnap1/test1/103/11' 2024-04-14T23:53:03.716 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/103/12' -> 'copyofsnap1/test1/103/12' 2024-04-14T23:53:03.716 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/103/13' -> 'copyofsnap1/test1/103/13' 2024-04-14T23:53:03.716 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/103/14' -> 'copyofsnap1/test1/103/14' 2024-04-14T23:53:03.716 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/103/15' -> 'copyofsnap1/test1/103/15' 2024-04-14T23:53:03.716 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/103/16' -> 'copyofsnap1/test1/103/16' 2024-04-14T23:53:03.716 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/103/17' -> 'copyofsnap1/test1/103/17' 2024-04-14T23:53:03.716 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/103/18' -> 'copyofsnap1/test1/103/18' 2024-04-14T23:53:03.716 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/103/19' -> 'copyofsnap1/test1/103/19' 2024-04-14T23:53:03.716 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/103/20' -> 'copyofsnap1/test1/103/20' 2024-04-14T23:53:03.716 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/104' -> 'copyofsnap1/test1/104' 2024-04-14T23:53:03.716 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/104/10' -> 'copyofsnap1/test1/104/10' 2024-04-14T23:53:03.716 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/104/11' -> 'copyofsnap1/test1/104/11' 2024-04-14T23:53:03.716 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/104/12' -> 'copyofsnap1/test1/104/12' 2024-04-14T23:53:03.716 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/104/13' -> 'copyofsnap1/test1/104/13' 2024-04-14T23:53:03.716 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/104/14' -> 'copyofsnap1/test1/104/14' 2024-04-14T23:53:03.717 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/104/15' -> 'copyofsnap1/test1/104/15' 2024-04-14T23:53:03.717 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/104/16' -> 'copyofsnap1/test1/104/16' 2024-04-14T23:53:03.717 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/104/17' -> 'copyofsnap1/test1/104/17' 2024-04-14T23:53:03.717 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/104/18' -> 'copyofsnap1/test1/104/18' 2024-04-14T23:53:03.717 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/104/19' -> 'copyofsnap1/test1/104/19' 2024-04-14T23:53:03.717 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/104/20' -> 'copyofsnap1/test1/104/20' 2024-04-14T23:53:03.717 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/105' -> 'copyofsnap1/test1/105' 2024-04-14T23:53:03.717 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/105/10' -> 'copyofsnap1/test1/105/10' 2024-04-14T23:53:03.717 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/105/11' -> 'copyofsnap1/test1/105/11' 2024-04-14T23:53:03.717 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/105/12' -> 'copyofsnap1/test1/105/12' 2024-04-14T23:53:03.717 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/105/13' -> 'copyofsnap1/test1/105/13' 2024-04-14T23:53:03.717 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/105/14' -> 'copyofsnap1/test1/105/14' 2024-04-14T23:53:03.717 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/105/15' -> 'copyofsnap1/test1/105/15' 2024-04-14T23:53:03.717 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/105/16' -> 'copyofsnap1/test1/105/16' 2024-04-14T23:53:03.717 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/105/17' -> 'copyofsnap1/test1/105/17' 2024-04-14T23:53:03.718 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/105/18' -> 'copyofsnap1/test1/105/18' 2024-04-14T23:53:03.718 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/105/19' -> 'copyofsnap1/test1/105/19' 2024-04-14T23:53:03.718 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/105/20' -> 'copyofsnap1/test1/105/20' 2024-04-14T23:53:03.718 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/106' -> 'copyofsnap1/test1/106' 2024-04-14T23:53:03.718 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/106/10' -> 'copyofsnap1/test1/106/10' 2024-04-14T23:53:03.718 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/106/11' -> 'copyofsnap1/test1/106/11' 2024-04-14T23:53:03.718 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/106/12' -> 'copyofsnap1/test1/106/12' 2024-04-14T23:53:03.718 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/106/13' -> 'copyofsnap1/test1/106/13' 2024-04-14T23:53:03.718 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/106/14' -> 'copyofsnap1/test1/106/14' 2024-04-14T23:53:03.718 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/106/15' -> 'copyofsnap1/test1/106/15' 2024-04-14T23:53:03.718 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/106/16' -> 'copyofsnap1/test1/106/16' 2024-04-14T23:53:04.915 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/106/17' -> 'copyofsnap1/test1/106/17' 2024-04-14T23:53:04.915 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/106/18' -> 'copyofsnap1/test1/106/18' 2024-04-14T23:53:04.915 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/106/19' -> 'copyofsnap1/test1/106/19' 2024-04-14T23:53:04.915 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/106/20' -> 'copyofsnap1/test1/106/20' 2024-04-14T23:53:04.915 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/107' -> 'copyofsnap1/test1/107' 2024-04-14T23:53:04.915 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/107/10' -> 'copyofsnap1/test1/107/10' 2024-04-14T23:53:04.916 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/107/11' -> 'copyofsnap1/test1/107/11' 2024-04-14T23:53:04.916 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/107/12' -> 'copyofsnap1/test1/107/12' 2024-04-14T23:53:04.916 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/107/13' -> 'copyofsnap1/test1/107/13' 2024-04-14T23:53:04.916 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/107/14' -> 'copyofsnap1/test1/107/14' 2024-04-14T23:53:04.916 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/107/15' -> 'copyofsnap1/test1/107/15' 2024-04-14T23:53:04.916 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/107/16' -> 'copyofsnap1/test1/107/16' 2024-04-14T23:53:04.916 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/107/17' -> 'copyofsnap1/test1/107/17' 2024-04-14T23:53:04.916 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/107/18' -> 'copyofsnap1/test1/107/18' 2024-04-14T23:53:04.916 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/107/19' -> 'copyofsnap1/test1/107/19' 2024-04-14T23:53:04.916 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/107/20' -> 'copyofsnap1/test1/107/20' 2024-04-14T23:53:04.916 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/108' -> 'copyofsnap1/test1/108' 2024-04-14T23:53:04.916 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/108/10' -> 'copyofsnap1/test1/108/10' 2024-04-14T23:53:04.916 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/108/11' -> 'copyofsnap1/test1/108/11' 2024-04-14T23:53:04.916 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/108/12' -> 'copyofsnap1/test1/108/12' 2024-04-14T23:53:04.916 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/108/13' -> 'copyofsnap1/test1/108/13' 2024-04-14T23:53:04.917 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/108/14' -> 'copyofsnap1/test1/108/14' 2024-04-14T23:53:04.917 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/108/15' -> 'copyofsnap1/test1/108/15' 2024-04-14T23:53:04.917 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/108/16' -> 'copyofsnap1/test1/108/16' 2024-04-14T23:53:04.917 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/108/17' -> 'copyofsnap1/test1/108/17' 2024-04-14T23:53:04.917 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/108/18' -> 'copyofsnap1/test1/108/18' 2024-04-14T23:53:04.917 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/108/19' -> 'copyofsnap1/test1/108/19' 2024-04-14T23:53:04.917 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/108/20' -> 'copyofsnap1/test1/108/20' 2024-04-14T23:53:04.917 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/109' -> 'copyofsnap1/test1/109' 2024-04-14T23:53:04.917 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/109/10' -> 'copyofsnap1/test1/109/10' 2024-04-14T23:53:04.917 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/109/11' -> 'copyofsnap1/test1/109/11' 2024-04-14T23:53:04.917 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/109/12' -> 'copyofsnap1/test1/109/12' 2024-04-14T23:53:04.917 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/109/13' -> 'copyofsnap1/test1/109/13' 2024-04-14T23:53:04.917 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/109/14' -> 'copyofsnap1/test1/109/14' 2024-04-14T23:53:04.917 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/109/15' -> 'copyofsnap1/test1/109/15' 2024-04-14T23:53:04.917 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/109/16' -> 'copyofsnap1/test1/109/16' 2024-04-14T23:53:04.917 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/109/17' -> 'copyofsnap1/test1/109/17' 2024-04-14T23:53:04.918 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/109/18' -> 'copyofsnap1/test1/109/18' 2024-04-14T23:53:04.918 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/109/19' -> 'copyofsnap1/test1/109/19' 2024-04-14T23:53:04.918 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/109/20' -> 'copyofsnap1/test1/109/20' 2024-04-14T23:53:04.918 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/110' -> 'copyofsnap1/test1/110' 2024-04-14T23:53:04.918 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/110/10' -> 'copyofsnap1/test1/110/10' 2024-04-14T23:53:04.918 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/110/11' -> 'copyofsnap1/test1/110/11' 2024-04-14T23:53:04.918 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/110/12' -> 'copyofsnap1/test1/110/12' 2024-04-14T23:53:04.918 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/110/13' -> 'copyofsnap1/test1/110/13' 2024-04-14T23:53:04.918 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/110/14' -> 'copyofsnap1/test1/110/14' 2024-04-14T23:53:04.918 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/110/15' -> 'copyofsnap1/test1/110/15' 2024-04-14T23:53:04.918 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/110/16' -> 'copyofsnap1/test1/110/16' 2024-04-14T23:53:04.918 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/110/17' -> 'copyofsnap1/test1/110/17' 2024-04-14T23:53:04.918 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/110/18' -> 'copyofsnap1/test1/110/18' 2024-04-14T23:53:04.918 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/110/19' -> 'copyofsnap1/test1/110/19' 2024-04-14T23:53:04.919 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/110/20' -> 'copyofsnap1/test1/110/20' 2024-04-14T23:53:04.919 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/111' -> 'copyofsnap1/test1/111' 2024-04-14T23:53:04.919 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/111/10' -> 'copyofsnap1/test1/111/10' 2024-04-14T23:53:04.919 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/111/11' -> 'copyofsnap1/test1/111/11' 2024-04-14T23:53:04.919 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/111/12' -> 'copyofsnap1/test1/111/12' 2024-04-14T23:53:04.919 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/111/13' -> 'copyofsnap1/test1/111/13' 2024-04-14T23:53:04.919 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/111/14' -> 'copyofsnap1/test1/111/14' 2024-04-14T23:53:04.919 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/111/15' -> 'copyofsnap1/test1/111/15' 2024-04-14T23:53:04.919 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/111/16' -> 'copyofsnap1/test1/111/16' 2024-04-14T23:53:04.919 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/111/17' -> 'copyofsnap1/test1/111/17' 2024-04-14T23:53:04.919 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/111/18' -> 'copyofsnap1/test1/111/18' 2024-04-14T23:53:04.919 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/111/19' -> 'copyofsnap1/test1/111/19' 2024-04-14T23:53:04.919 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/111/20' -> 'copyofsnap1/test1/111/20' 2024-04-14T23:53:04.919 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/112' -> 'copyofsnap1/test1/112' 2024-04-14T23:53:04.919 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/112/10' -> 'copyofsnap1/test1/112/10' 2024-04-14T23:53:04.920 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/112/11' -> 'copyofsnap1/test1/112/11' 2024-04-14T23:53:04.920 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/112/12' -> 'copyofsnap1/test1/112/12' 2024-04-14T23:53:04.920 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/112/13' -> 'copyofsnap1/test1/112/13' 2024-04-14T23:53:04.920 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/112/14' -> 'copyofsnap1/test1/112/14' 2024-04-14T23:53:04.920 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/112/15' -> 'copyofsnap1/test1/112/15' 2024-04-14T23:53:04.920 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/112/16' -> 'copyofsnap1/test1/112/16' 2024-04-14T23:53:04.920 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/112/17' -> 'copyofsnap1/test1/112/17' 2024-04-14T23:53:04.920 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/112/18' -> 'copyofsnap1/test1/112/18' 2024-04-14T23:53:04.920 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/112/19' -> 'copyofsnap1/test1/112/19' 2024-04-14T23:53:04.920 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/112/20' -> 'copyofsnap1/test1/112/20' 2024-04-14T23:53:04.920 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/113' -> 'copyofsnap1/test1/113' 2024-04-14T23:53:04.920 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/113/10' -> 'copyofsnap1/test1/113/10' 2024-04-14T23:53:04.920 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/113/11' -> 'copyofsnap1/test1/113/11' 2024-04-14T23:53:04.920 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/113/12' -> 'copyofsnap1/test1/113/12' 2024-04-14T23:53:04.920 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/113/13' -> 'copyofsnap1/test1/113/13' 2024-04-14T23:53:05.711 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/113/14' -> 'copyofsnap1/test1/113/14' 2024-04-14T23:53:05.711 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/113/15' -> 'copyofsnap1/test1/113/15' 2024-04-14T23:53:05.711 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/113/16' -> 'copyofsnap1/test1/113/16' 2024-04-14T23:53:05.711 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/113/17' -> 'copyofsnap1/test1/113/17' 2024-04-14T23:53:05.711 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/113/18' -> 'copyofsnap1/test1/113/18' 2024-04-14T23:53:05.711 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/113/19' -> 'copyofsnap1/test1/113/19' 2024-04-14T23:53:05.711 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/113/20' -> 'copyofsnap1/test1/113/20' 2024-04-14T23:53:05.712 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/114' -> 'copyofsnap1/test1/114' 2024-04-14T23:53:05.712 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/114/10' -> 'copyofsnap1/test1/114/10' 2024-04-14T23:53:05.712 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/114/11' -> 'copyofsnap1/test1/114/11' 2024-04-14T23:53:05.712 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/114/12' -> 'copyofsnap1/test1/114/12' 2024-04-14T23:53:05.712 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/114/13' -> 'copyofsnap1/test1/114/13' 2024-04-14T23:53:05.712 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/114/14' -> 'copyofsnap1/test1/114/14' 2024-04-14T23:53:05.712 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/114/15' -> 'copyofsnap1/test1/114/15' 2024-04-14T23:53:05.712 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/114/16' -> 'copyofsnap1/test1/114/16' 2024-04-14T23:53:05.712 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/114/17' -> 'copyofsnap1/test1/114/17' 2024-04-14T23:53:05.712 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/114/18' -> 'copyofsnap1/test1/114/18' 2024-04-14T23:53:05.712 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/114/19' -> 'copyofsnap1/test1/114/19' 2024-04-14T23:53:05.713 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/114/20' -> 'copyofsnap1/test1/114/20' 2024-04-14T23:53:05.713 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/115' -> 'copyofsnap1/test1/115' 2024-04-14T23:53:05.713 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/115/10' -> 'copyofsnap1/test1/115/10' 2024-04-14T23:53:05.713 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/115/11' -> 'copyofsnap1/test1/115/11' 2024-04-14T23:53:05.713 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/115/12' -> 'copyofsnap1/test1/115/12' 2024-04-14T23:53:05.713 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/115/13' -> 'copyofsnap1/test1/115/13' 2024-04-14T23:53:05.713 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/115/14' -> 'copyofsnap1/test1/115/14' 2024-04-14T23:53:05.713 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/115/15' -> 'copyofsnap1/test1/115/15' 2024-04-14T23:53:05.713 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/115/16' -> 'copyofsnap1/test1/115/16' 2024-04-14T23:53:05.713 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/115/17' -> 'copyofsnap1/test1/115/17' 2024-04-14T23:53:05.713 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/115/18' -> 'copyofsnap1/test1/115/18' 2024-04-14T23:53:05.713 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/115/19' -> 'copyofsnap1/test1/115/19' 2024-04-14T23:53:05.713 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/115/20' -> 'copyofsnap1/test1/115/20' 2024-04-14T23:53:05.713 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/116' -> 'copyofsnap1/test1/116' 2024-04-14T23:53:05.714 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/116/10' -> 'copyofsnap1/test1/116/10' 2024-04-14T23:53:05.714 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/116/11' -> 'copyofsnap1/test1/116/11' 2024-04-14T23:53:05.714 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/116/12' -> 'copyofsnap1/test1/116/12' 2024-04-14T23:53:05.714 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/116/13' -> 'copyofsnap1/test1/116/13' 2024-04-14T23:53:05.714 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/116/14' -> 'copyofsnap1/test1/116/14' 2024-04-14T23:53:05.714 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/116/15' -> 'copyofsnap1/test1/116/15' 2024-04-14T23:53:05.714 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/116/16' -> 'copyofsnap1/test1/116/16' 2024-04-14T23:53:05.714 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/116/17' -> 'copyofsnap1/test1/116/17' 2024-04-14T23:53:05.714 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/116/18' -> 'copyofsnap1/test1/116/18' 2024-04-14T23:53:05.714 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/116/19' -> 'copyofsnap1/test1/116/19' 2024-04-14T23:53:05.714 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/116/20' -> 'copyofsnap1/test1/116/20' 2024-04-14T23:53:05.714 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/117' -> 'copyofsnap1/test1/117' 2024-04-14T23:53:05.714 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/117/10' -> 'copyofsnap1/test1/117/10' 2024-04-14T23:53:05.715 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/117/11' -> 'copyofsnap1/test1/117/11' 2024-04-14T23:53:05.715 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/117/12' -> 'copyofsnap1/test1/117/12' 2024-04-14T23:53:05.715 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/117/13' -> 'copyofsnap1/test1/117/13' 2024-04-14T23:53:05.715 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/117/14' -> 'copyofsnap1/test1/117/14' 2024-04-14T23:53:05.715 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/117/15' -> 'copyofsnap1/test1/117/15' 2024-04-14T23:53:05.715 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/117/16' -> 'copyofsnap1/test1/117/16' 2024-04-14T23:53:05.715 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/117/17' -> 'copyofsnap1/test1/117/17' 2024-04-14T23:53:05.715 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/117/18' -> 'copyofsnap1/test1/117/18' 2024-04-14T23:53:05.715 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/117/19' -> 'copyofsnap1/test1/117/19' 2024-04-14T23:53:05.715 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/117/20' -> 'copyofsnap1/test1/117/20' 2024-04-14T23:53:05.715 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/118' -> 'copyofsnap1/test1/118' 2024-04-14T23:53:05.715 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/118/10' -> 'copyofsnap1/test1/118/10' 2024-04-14T23:53:05.716 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/118/11' -> 'copyofsnap1/test1/118/11' 2024-04-14T23:53:05.716 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/118/12' -> 'copyofsnap1/test1/118/12' 2024-04-14T23:53:05.716 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/118/13' -> 'copyofsnap1/test1/118/13' 2024-04-14T23:53:05.716 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/118/14' -> 'copyofsnap1/test1/118/14' 2024-04-14T23:53:05.716 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/118/15' -> 'copyofsnap1/test1/118/15' 2024-04-14T23:53:05.716 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/118/16' -> 'copyofsnap1/test1/118/16' 2024-04-14T23:53:05.716 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/118/17' -> 'copyofsnap1/test1/118/17' 2024-04-14T23:53:05.716 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/118/18' -> 'copyofsnap1/test1/118/18' 2024-04-14T23:53:05.716 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/118/19' -> 'copyofsnap1/test1/118/19' 2024-04-14T23:53:05.716 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/118/20' -> 'copyofsnap1/test1/118/20' 2024-04-14T23:53:05.716 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/119' -> 'copyofsnap1/test1/119' 2024-04-14T23:53:05.716 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/119/10' -> 'copyofsnap1/test1/119/10' 2024-04-14T23:53:05.716 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/119/11' -> 'copyofsnap1/test1/119/11' 2024-04-14T23:53:05.716 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/119/12' -> 'copyofsnap1/test1/119/12' 2024-04-14T23:53:05.717 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/119/13' -> 'copyofsnap1/test1/119/13' 2024-04-14T23:53:05.717 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/119/14' -> 'copyofsnap1/test1/119/14' 2024-04-14T23:53:05.717 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/119/15' -> 'copyofsnap1/test1/119/15' 2024-04-14T23:53:05.717 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/119/16' -> 'copyofsnap1/test1/119/16' 2024-04-14T23:53:05.717 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/119/17' -> 'copyofsnap1/test1/119/17' 2024-04-14T23:53:05.717 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/119/18' -> 'copyofsnap1/test1/119/18' 2024-04-14T23:53:05.717 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/119/19' -> 'copyofsnap1/test1/119/19' 2024-04-14T23:53:05.717 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/119/20' -> 'copyofsnap1/test1/119/20' 2024-04-14T23:53:05.717 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/120' -> 'copyofsnap1/test1/120' 2024-04-14T23:53:05.717 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/120/10' -> 'copyofsnap1/test1/120/10' 2024-04-14T23:53:06.743 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/120/11' -> 'copyofsnap1/test1/120/11' 2024-04-14T23:53:06.744 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/120/12' -> 'copyofsnap1/test1/120/12' 2024-04-14T23:53:06.744 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/120/13' -> 'copyofsnap1/test1/120/13' 2024-04-14T23:53:06.744 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/120/14' -> 'copyofsnap1/test1/120/14' 2024-04-14T23:53:06.744 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/120/15' -> 'copyofsnap1/test1/120/15' 2024-04-14T23:53:06.744 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/120/16' -> 'copyofsnap1/test1/120/16' 2024-04-14T23:53:06.744 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/120/17' -> 'copyofsnap1/test1/120/17' 2024-04-14T23:53:06.744 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/120/18' -> 'copyofsnap1/test1/120/18' 2024-04-14T23:53:06.744 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/120/19' -> 'copyofsnap1/test1/120/19' 2024-04-14T23:53:06.744 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/120/20' -> 'copyofsnap1/test1/120/20' 2024-04-14T23:53:06.744 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/121' -> 'copyofsnap1/test1/121' 2024-04-14T23:53:06.744 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/121/10' -> 'copyofsnap1/test1/121/10' 2024-04-14T23:53:06.744 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/121/11' -> 'copyofsnap1/test1/121/11' 2024-04-14T23:53:06.745 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/121/12' -> 'copyofsnap1/test1/121/12' 2024-04-14T23:53:06.745 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/121/13' -> 'copyofsnap1/test1/121/13' 2024-04-14T23:53:06.745 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/121/14' -> 'copyofsnap1/test1/121/14' 2024-04-14T23:53:06.745 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/121/15' -> 'copyofsnap1/test1/121/15' 2024-04-14T23:53:06.745 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/121/16' -> 'copyofsnap1/test1/121/16' 2024-04-14T23:53:06.745 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/121/17' -> 'copyofsnap1/test1/121/17' 2024-04-14T23:53:06.745 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/121/18' -> 'copyofsnap1/test1/121/18' 2024-04-14T23:53:06.745 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/121/19' -> 'copyofsnap1/test1/121/19' 2024-04-14T23:53:06.745 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/121/20' -> 'copyofsnap1/test1/121/20' 2024-04-14T23:53:06.745 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/122' -> 'copyofsnap1/test1/122' 2024-04-14T23:53:06.745 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/122/10' -> 'copyofsnap1/test1/122/10' 2024-04-14T23:53:06.745 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/122/11' -> 'copyofsnap1/test1/122/11' 2024-04-14T23:53:06.745 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/122/12' -> 'copyofsnap1/test1/122/12' 2024-04-14T23:53:06.745 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/122/13' -> 'copyofsnap1/test1/122/13' 2024-04-14T23:53:06.745 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/122/14' -> 'copyofsnap1/test1/122/14' 2024-04-14T23:53:06.746 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/122/15' -> 'copyofsnap1/test1/122/15' 2024-04-14T23:53:06.746 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/122/16' -> 'copyofsnap1/test1/122/16' 2024-04-14T23:53:06.746 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/122/17' -> 'copyofsnap1/test1/122/17' 2024-04-14T23:53:06.746 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/122/18' -> 'copyofsnap1/test1/122/18' 2024-04-14T23:53:06.746 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/122/19' -> 'copyofsnap1/test1/122/19' 2024-04-14T23:53:06.746 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/122/20' -> 'copyofsnap1/test1/122/20' 2024-04-14T23:53:06.746 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/123' -> 'copyofsnap1/test1/123' 2024-04-14T23:53:06.746 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/123/10' -> 'copyofsnap1/test1/123/10' 2024-04-14T23:53:06.746 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/123/11' -> 'copyofsnap1/test1/123/11' 2024-04-14T23:53:06.746 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/123/12' -> 'copyofsnap1/test1/123/12' 2024-04-14T23:53:06.746 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/123/13' -> 'copyofsnap1/test1/123/13' 2024-04-14T23:53:06.746 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/123/14' -> 'copyofsnap1/test1/123/14' 2024-04-14T23:53:06.746 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/123/15' -> 'copyofsnap1/test1/123/15' 2024-04-14T23:53:06.746 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/123/16' -> 'copyofsnap1/test1/123/16' 2024-04-14T23:53:06.746 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/123/17' -> 'copyofsnap1/test1/123/17' 2024-04-14T23:53:06.747 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/123/18' -> 'copyofsnap1/test1/123/18' 2024-04-14T23:53:06.747 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/123/19' -> 'copyofsnap1/test1/123/19' 2024-04-14T23:53:06.747 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/123/20' -> 'copyofsnap1/test1/123/20' 2024-04-14T23:53:06.747 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/124' -> 'copyofsnap1/test1/124' 2024-04-14T23:53:06.747 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/124/10' -> 'copyofsnap1/test1/124/10' 2024-04-14T23:53:06.747 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/124/11' -> 'copyofsnap1/test1/124/11' 2024-04-14T23:53:06.747 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/124/12' -> 'copyofsnap1/test1/124/12' 2024-04-14T23:53:06.747 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/124/13' -> 'copyofsnap1/test1/124/13' 2024-04-14T23:53:06.747 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/124/14' -> 'copyofsnap1/test1/124/14' 2024-04-14T23:53:06.747 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/124/15' -> 'copyofsnap1/test1/124/15' 2024-04-14T23:53:06.747 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/124/16' -> 'copyofsnap1/test1/124/16' 2024-04-14T23:53:06.747 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/124/17' -> 'copyofsnap1/test1/124/17' 2024-04-14T23:53:06.747 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/124/18' -> 'copyofsnap1/test1/124/18' 2024-04-14T23:53:06.747 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/124/19' -> 'copyofsnap1/test1/124/19' 2024-04-14T23:53:06.747 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/124/20' -> 'copyofsnap1/test1/124/20' 2024-04-14T23:53:06.748 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/125' -> 'copyofsnap1/test1/125' 2024-04-14T23:53:06.748 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/125/10' -> 'copyofsnap1/test1/125/10' 2024-04-14T23:53:06.748 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/125/11' -> 'copyofsnap1/test1/125/11' 2024-04-14T23:53:06.748 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/125/12' -> 'copyofsnap1/test1/125/12' 2024-04-14T23:53:06.748 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/125/13' -> 'copyofsnap1/test1/125/13' 2024-04-14T23:53:06.748 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/125/14' -> 'copyofsnap1/test1/125/14' 2024-04-14T23:53:06.748 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/125/15' -> 'copyofsnap1/test1/125/15' 2024-04-14T23:53:06.748 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/125/16' -> 'copyofsnap1/test1/125/16' 2024-04-14T23:53:06.748 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/125/17' -> 'copyofsnap1/test1/125/17' 2024-04-14T23:53:06.748 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/125/18' -> 'copyofsnap1/test1/125/18' 2024-04-14T23:53:06.748 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/125/19' -> 'copyofsnap1/test1/125/19' 2024-04-14T23:53:06.749 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/125/20' -> 'copyofsnap1/test1/125/20' 2024-04-14T23:53:06.749 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/126' -> 'copyofsnap1/test1/126' 2024-04-14T23:53:06.749 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/126/10' -> 'copyofsnap1/test1/126/10' 2024-04-14T23:53:06.749 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/126/11' -> 'copyofsnap1/test1/126/11' 2024-04-14T23:53:06.749 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/126/12' -> 'copyofsnap1/test1/126/12' 2024-04-14T23:53:06.749 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/126/13' -> 'copyofsnap1/test1/126/13' 2024-04-14T23:53:06.749 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/126/14' -> 'copyofsnap1/test1/126/14' 2024-04-14T23:53:06.749 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/126/15' -> 'copyofsnap1/test1/126/15' 2024-04-14T23:53:06.749 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/126/16' -> 'copyofsnap1/test1/126/16' 2024-04-14T23:53:06.749 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/126/17' -> 'copyofsnap1/test1/126/17' 2024-04-14T23:53:06.749 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/126/18' -> 'copyofsnap1/test1/126/18' 2024-04-14T23:53:06.750 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/126/19' -> 'copyofsnap1/test1/126/19' 2024-04-14T23:53:07.096 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/126/20' -> 'copyofsnap1/test1/126/20' 2024-04-14T23:53:07.096 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/127' -> 'copyofsnap1/test1/127' 2024-04-14T23:53:07.096 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/127/10' -> 'copyofsnap1/test1/127/10' 2024-04-14T23:53:07.096 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/127/11' -> 'copyofsnap1/test1/127/11' 2024-04-14T23:53:07.096 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/127/12' -> 'copyofsnap1/test1/127/12' 2024-04-14T23:53:07.096 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/127/13' -> 'copyofsnap1/test1/127/13' 2024-04-14T23:53:07.096 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/127/14' -> 'copyofsnap1/test1/127/14' 2024-04-14T23:53:07.096 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/127/15' -> 'copyofsnap1/test1/127/15' 2024-04-14T23:53:07.096 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/127/16' -> 'copyofsnap1/test1/127/16' 2024-04-14T23:53:07.096 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/127/17' -> 'copyofsnap1/test1/127/17' 2024-04-14T23:53:07.096 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/127/18' -> 'copyofsnap1/test1/127/18' 2024-04-14T23:53:07.096 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/127/19' -> 'copyofsnap1/test1/127/19' 2024-04-14T23:53:07.096 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/127/20' -> 'copyofsnap1/test1/127/20' 2024-04-14T23:53:07.097 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/128' -> 'copyofsnap1/test1/128' 2024-04-14T23:53:07.097 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/128/10' -> 'copyofsnap1/test1/128/10' 2024-04-14T23:53:07.097 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/128/11' -> 'copyofsnap1/test1/128/11' 2024-04-14T23:53:07.097 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/128/12' -> 'copyofsnap1/test1/128/12' 2024-04-14T23:53:07.097 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/128/13' -> 'copyofsnap1/test1/128/13' 2024-04-14T23:53:07.097 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/128/14' -> 'copyofsnap1/test1/128/14' 2024-04-14T23:53:07.097 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/128/15' -> 'copyofsnap1/test1/128/15' 2024-04-14T23:53:07.097 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/128/16' -> 'copyofsnap1/test1/128/16' 2024-04-14T23:53:07.097 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/128/17' -> 'copyofsnap1/test1/128/17' 2024-04-14T23:53:07.097 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/128/18' -> 'copyofsnap1/test1/128/18' 2024-04-14T23:53:07.097 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/128/19' -> 'copyofsnap1/test1/128/19' 2024-04-14T23:53:07.097 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/128/20' -> 'copyofsnap1/test1/128/20' 2024-04-14T23:53:07.097 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/129' -> 'copyofsnap1/test1/129' 2024-04-14T23:53:07.097 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/129/10' -> 'copyofsnap1/test1/129/10' 2024-04-14T23:53:07.098 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/129/11' -> 'copyofsnap1/test1/129/11' 2024-04-14T23:53:07.098 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/129/12' -> 'copyofsnap1/test1/129/12' 2024-04-14T23:53:07.098 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/129/13' -> 'copyofsnap1/test1/129/13' 2024-04-14T23:53:07.098 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/129/14' -> 'copyofsnap1/test1/129/14' 2024-04-14T23:53:07.098 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/129/15' -> 'copyofsnap1/test1/129/15' 2024-04-14T23:53:07.098 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/129/16' -> 'copyofsnap1/test1/129/16' 2024-04-14T23:53:07.098 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/129/17' -> 'copyofsnap1/test1/129/17' 2024-04-14T23:53:07.098 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/129/18' -> 'copyofsnap1/test1/129/18' 2024-04-14T23:53:07.098 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/129/19' -> 'copyofsnap1/test1/129/19' 2024-04-14T23:53:07.098 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/129/20' -> 'copyofsnap1/test1/129/20' 2024-04-14T23:53:07.098 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/130' -> 'copyofsnap1/test1/130' 2024-04-14T23:53:07.098 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/130/10' -> 'copyofsnap1/test1/130/10' 2024-04-14T23:53:07.098 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/130/11' -> 'copyofsnap1/test1/130/11' 2024-04-14T23:53:07.098 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/130/12' -> 'copyofsnap1/test1/130/12' 2024-04-14T23:53:07.099 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/130/13' -> 'copyofsnap1/test1/130/13' 2024-04-14T23:53:07.099 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/130/14' -> 'copyofsnap1/test1/130/14' 2024-04-14T23:53:07.099 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/130/15' -> 'copyofsnap1/test1/130/15' 2024-04-14T23:53:07.099 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/130/16' -> 'copyofsnap1/test1/130/16' 2024-04-14T23:53:07.099 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/130/17' -> 'copyofsnap1/test1/130/17' 2024-04-14T23:53:07.099 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/130/18' -> 'copyofsnap1/test1/130/18' 2024-04-14T23:53:07.099 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/130/19' -> 'copyofsnap1/test1/130/19' 2024-04-14T23:53:07.099 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/130/20' -> 'copyofsnap1/test1/130/20' 2024-04-14T23:53:07.099 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/131' -> 'copyofsnap1/test1/131' 2024-04-14T23:53:07.099 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/131/10' -> 'copyofsnap1/test1/131/10' 2024-04-14T23:53:07.099 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/131/11' -> 'copyofsnap1/test1/131/11' 2024-04-14T23:53:07.099 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/131/12' -> 'copyofsnap1/test1/131/12' 2024-04-14T23:53:07.099 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/131/13' -> 'copyofsnap1/test1/131/13' 2024-04-14T23:53:07.099 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/131/14' -> 'copyofsnap1/test1/131/14' 2024-04-14T23:53:07.100 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/131/15' -> 'copyofsnap1/test1/131/15' 2024-04-14T23:53:07.100 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/131/16' -> 'copyofsnap1/test1/131/16' 2024-04-14T23:53:07.100 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/131/17' -> 'copyofsnap1/test1/131/17' 2024-04-14T23:53:07.100 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/131/18' -> 'copyofsnap1/test1/131/18' 2024-04-14T23:53:07.101 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/131/19' -> 'copyofsnap1/test1/131/19' 2024-04-14T23:53:07.101 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/131/20' -> 'copyofsnap1/test1/131/20' 2024-04-14T23:53:07.101 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/132' -> 'copyofsnap1/test1/132' 2024-04-14T23:53:07.101 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/132/10' -> 'copyofsnap1/test1/132/10' 2024-04-14T23:53:07.101 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/132/11' -> 'copyofsnap1/test1/132/11' 2024-04-14T23:53:07.102 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/132/12' -> 'copyofsnap1/test1/132/12' 2024-04-14T23:53:07.102 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/132/13' -> 'copyofsnap1/test1/132/13' 2024-04-14T23:53:07.102 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/132/14' -> 'copyofsnap1/test1/132/14' 2024-04-14T23:53:07.102 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/132/15' -> 'copyofsnap1/test1/132/15' 2024-04-14T23:53:07.102 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/132/16' -> 'copyofsnap1/test1/132/16' 2024-04-14T23:53:07.102 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/132/17' -> 'copyofsnap1/test1/132/17' 2024-04-14T23:53:07.102 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/132/18' -> 'copyofsnap1/test1/132/18' 2024-04-14T23:53:07.102 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/132/19' -> 'copyofsnap1/test1/132/19' 2024-04-14T23:53:07.102 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/132/20' -> 'copyofsnap1/test1/132/20' 2024-04-14T23:53:07.102 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/133' -> 'copyofsnap1/test1/133' 2024-04-14T23:53:07.102 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/133/10' -> 'copyofsnap1/test1/133/10' 2024-04-14T23:53:07.102 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/133/11' -> 'copyofsnap1/test1/133/11' 2024-04-14T23:53:07.102 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/133/12' -> 'copyofsnap1/test1/133/12' 2024-04-14T23:53:07.102 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/133/13' -> 'copyofsnap1/test1/133/13' 2024-04-14T23:53:07.102 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/133/14' -> 'copyofsnap1/test1/133/14' 2024-04-14T23:53:07.103 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/133/15' -> 'copyofsnap1/test1/133/15' 2024-04-14T23:53:07.103 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/133/16' -> 'copyofsnap1/test1/133/16' 2024-04-14T23:53:07.440 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/133/17' -> 'copyofsnap1/test1/133/17' 2024-04-14T23:53:07.440 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/133/18' -> 'copyofsnap1/test1/133/18' 2024-04-14T23:53:07.440 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/133/19' -> 'copyofsnap1/test1/133/19' 2024-04-14T23:53:07.440 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/133/20' -> 'copyofsnap1/test1/133/20' 2024-04-14T23:53:07.440 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/134' -> 'copyofsnap1/test1/134' 2024-04-14T23:53:07.440 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/134/10' -> 'copyofsnap1/test1/134/10' 2024-04-14T23:53:07.440 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/134/11' -> 'copyofsnap1/test1/134/11' 2024-04-14T23:53:07.441 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/134/12' -> 'copyofsnap1/test1/134/12' 2024-04-14T23:53:07.441 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/134/13' -> 'copyofsnap1/test1/134/13' 2024-04-14T23:53:07.441 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/134/14' -> 'copyofsnap1/test1/134/14' 2024-04-14T23:53:07.441 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/134/15' -> 'copyofsnap1/test1/134/15' 2024-04-14T23:53:07.441 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/134/16' -> 'copyofsnap1/test1/134/16' 2024-04-14T23:53:07.441 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/134/17' -> 'copyofsnap1/test1/134/17' 2024-04-14T23:53:07.441 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/134/18' -> 'copyofsnap1/test1/134/18' 2024-04-14T23:53:07.441 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/134/19' -> 'copyofsnap1/test1/134/19' 2024-04-14T23:53:07.441 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/134/20' -> 'copyofsnap1/test1/134/20' 2024-04-14T23:53:07.441 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/135' -> 'copyofsnap1/test1/135' 2024-04-14T23:53:07.441 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/135/10' -> 'copyofsnap1/test1/135/10' 2024-04-14T23:53:07.441 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/135/11' -> 'copyofsnap1/test1/135/11' 2024-04-14T23:53:07.441 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/135/12' -> 'copyofsnap1/test1/135/12' 2024-04-14T23:53:07.441 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/135/13' -> 'copyofsnap1/test1/135/13' 2024-04-14T23:53:07.442 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/135/14' -> 'copyofsnap1/test1/135/14' 2024-04-14T23:53:07.442 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/135/15' -> 'copyofsnap1/test1/135/15' 2024-04-14T23:53:07.442 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/135/16' -> 'copyofsnap1/test1/135/16' 2024-04-14T23:53:07.442 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/135/17' -> 'copyofsnap1/test1/135/17' 2024-04-14T23:53:07.442 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/135/18' -> 'copyofsnap1/test1/135/18' 2024-04-14T23:53:07.442 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/135/19' -> 'copyofsnap1/test1/135/19' 2024-04-14T23:53:07.442 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/135/20' -> 'copyofsnap1/test1/135/20' 2024-04-14T23:53:07.442 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/136' -> 'copyofsnap1/test1/136' 2024-04-14T23:53:07.442 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/136/10' -> 'copyofsnap1/test1/136/10' 2024-04-14T23:53:07.442 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/136/11' -> 'copyofsnap1/test1/136/11' 2024-04-14T23:53:07.442 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/136/12' -> 'copyofsnap1/test1/136/12' 2024-04-14T23:53:07.442 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/136/13' -> 'copyofsnap1/test1/136/13' 2024-04-14T23:53:07.442 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/136/14' -> 'copyofsnap1/test1/136/14' 2024-04-14T23:53:07.442 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/136/15' -> 'copyofsnap1/test1/136/15' 2024-04-14T23:53:07.442 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/136/16' -> 'copyofsnap1/test1/136/16' 2024-04-14T23:53:07.443 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/136/17' -> 'copyofsnap1/test1/136/17' 2024-04-14T23:53:07.443 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/136/18' -> 'copyofsnap1/test1/136/18' 2024-04-14T23:53:07.443 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/136/19' -> 'copyofsnap1/test1/136/19' 2024-04-14T23:53:07.443 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/136/20' -> 'copyofsnap1/test1/136/20' 2024-04-14T23:53:07.443 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/137' -> 'copyofsnap1/test1/137' 2024-04-14T23:53:07.443 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/137/10' -> 'copyofsnap1/test1/137/10' 2024-04-14T23:53:07.443 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/137/11' -> 'copyofsnap1/test1/137/11' 2024-04-14T23:53:07.443 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/137/12' -> 'copyofsnap1/test1/137/12' 2024-04-14T23:53:07.443 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/137/13' -> 'copyofsnap1/test1/137/13' 2024-04-14T23:53:07.443 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/137/14' -> 'copyofsnap1/test1/137/14' 2024-04-14T23:53:07.443 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/137/15' -> 'copyofsnap1/test1/137/15' 2024-04-14T23:53:07.443 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/137/16' -> 'copyofsnap1/test1/137/16' 2024-04-14T23:53:07.443 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/137/17' -> 'copyofsnap1/test1/137/17' 2024-04-14T23:53:07.443 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/137/18' -> 'copyofsnap1/test1/137/18' 2024-04-14T23:53:07.444 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/137/19' -> 'copyofsnap1/test1/137/19' 2024-04-14T23:53:07.444 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/137/20' -> 'copyofsnap1/test1/137/20' 2024-04-14T23:53:07.444 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/138' -> 'copyofsnap1/test1/138' 2024-04-14T23:53:07.444 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/138/10' -> 'copyofsnap1/test1/138/10' 2024-04-14T23:53:07.444 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/138/11' -> 'copyofsnap1/test1/138/11' 2024-04-14T23:53:07.444 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/138/12' -> 'copyofsnap1/test1/138/12' 2024-04-14T23:53:07.444 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/138/13' -> 'copyofsnap1/test1/138/13' 2024-04-14T23:53:07.444 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/138/14' -> 'copyofsnap1/test1/138/14' 2024-04-14T23:53:07.444 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/138/15' -> 'copyofsnap1/test1/138/15' 2024-04-14T23:53:07.444 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/138/16' -> 'copyofsnap1/test1/138/16' 2024-04-14T23:53:07.444 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/138/17' -> 'copyofsnap1/test1/138/17' 2024-04-14T23:53:07.444 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/138/18' -> 'copyofsnap1/test1/138/18' 2024-04-14T23:53:07.444 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/138/19' -> 'copyofsnap1/test1/138/19' 2024-04-14T23:53:07.444 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/138/20' -> 'copyofsnap1/test1/138/20' 2024-04-14T23:53:07.444 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/139' -> 'copyofsnap1/test1/139' 2024-04-14T23:53:07.444 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/139/10' -> 'copyofsnap1/test1/139/10' 2024-04-14T23:53:07.445 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/139/11' -> 'copyofsnap1/test1/139/11' 2024-04-14T23:53:07.445 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/139/12' -> 'copyofsnap1/test1/139/12' 2024-04-14T23:53:07.445 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/139/13' -> 'copyofsnap1/test1/139/13' 2024-04-14T23:53:07.445 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/139/14' -> 'copyofsnap1/test1/139/14' 2024-04-14T23:53:07.445 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/139/15' -> 'copyofsnap1/test1/139/15' 2024-04-14T23:53:07.445 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/139/16' -> 'copyofsnap1/test1/139/16' 2024-04-14T23:53:07.445 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/139/17' -> 'copyofsnap1/test1/139/17' 2024-04-14T23:53:07.445 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/139/18' -> 'copyofsnap1/test1/139/18' 2024-04-14T23:53:07.445 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/139/19' -> 'copyofsnap1/test1/139/19' 2024-04-14T23:53:07.445 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/139/20' -> 'copyofsnap1/test1/139/20' 2024-04-14T23:53:07.445 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/140' -> 'copyofsnap1/test1/140' 2024-04-14T23:53:07.445 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/140/10' -> 'copyofsnap1/test1/140/10' 2024-04-14T23:53:07.445 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/140/11' -> 'copyofsnap1/test1/140/11' 2024-04-14T23:53:07.445 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/140/12' -> 'copyofsnap1/test1/140/12' 2024-04-14T23:53:07.445 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/140/13' -> 'copyofsnap1/test1/140/13' 2024-04-14T23:53:07.798 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/140/14' -> 'copyofsnap1/test1/140/14' 2024-04-14T23:53:07.799 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/140/15' -> 'copyofsnap1/test1/140/15' 2024-04-14T23:53:07.799 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/140/16' -> 'copyofsnap1/test1/140/16' 2024-04-14T23:53:07.799 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/140/17' -> 'copyofsnap1/test1/140/17' 2024-04-14T23:53:07.799 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/140/18' -> 'copyofsnap1/test1/140/18' 2024-04-14T23:53:07.799 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/140/19' -> 'copyofsnap1/test1/140/19' 2024-04-14T23:53:07.799 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/140/20' -> 'copyofsnap1/test1/140/20' 2024-04-14T23:53:07.799 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/141' -> 'copyofsnap1/test1/141' 2024-04-14T23:53:07.799 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/141/10' -> 'copyofsnap1/test1/141/10' 2024-04-14T23:53:07.799 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/141/11' -> 'copyofsnap1/test1/141/11' 2024-04-14T23:53:07.799 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/141/12' -> 'copyofsnap1/test1/141/12' 2024-04-14T23:53:07.799 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/141/13' -> 'copyofsnap1/test1/141/13' 2024-04-14T23:53:07.799 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/141/14' -> 'copyofsnap1/test1/141/14' 2024-04-14T23:53:07.800 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/141/15' -> 'copyofsnap1/test1/141/15' 2024-04-14T23:53:07.800 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/141/16' -> 'copyofsnap1/test1/141/16' 2024-04-14T23:53:07.800 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/141/17' -> 'copyofsnap1/test1/141/17' 2024-04-14T23:53:07.800 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/141/18' -> 'copyofsnap1/test1/141/18' 2024-04-14T23:53:07.800 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/141/19' -> 'copyofsnap1/test1/141/19' 2024-04-14T23:53:07.800 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/141/20' -> 'copyofsnap1/test1/141/20' 2024-04-14T23:53:07.800 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/142' -> 'copyofsnap1/test1/142' 2024-04-14T23:53:07.800 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/142/10' -> 'copyofsnap1/test1/142/10' 2024-04-14T23:53:07.800 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/142/11' -> 'copyofsnap1/test1/142/11' 2024-04-14T23:53:07.800 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/142/12' -> 'copyofsnap1/test1/142/12' 2024-04-14T23:53:07.800 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/142/13' -> 'copyofsnap1/test1/142/13' 2024-04-14T23:53:07.800 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/142/14' -> 'copyofsnap1/test1/142/14' 2024-04-14T23:53:07.800 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/142/15' -> 'copyofsnap1/test1/142/15' 2024-04-14T23:53:07.800 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/142/16' -> 'copyofsnap1/test1/142/16' 2024-04-14T23:53:07.801 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/142/17' -> 'copyofsnap1/test1/142/17' 2024-04-14T23:53:07.801 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/142/18' -> 'copyofsnap1/test1/142/18' 2024-04-14T23:53:07.801 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/142/19' -> 'copyofsnap1/test1/142/19' 2024-04-14T23:53:07.801 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/142/20' -> 'copyofsnap1/test1/142/20' 2024-04-14T23:53:07.801 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/143' -> 'copyofsnap1/test1/143' 2024-04-14T23:53:07.801 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/143/10' -> 'copyofsnap1/test1/143/10' 2024-04-14T23:53:07.801 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/143/11' -> 'copyofsnap1/test1/143/11' 2024-04-14T23:53:07.801 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/143/12' -> 'copyofsnap1/test1/143/12' 2024-04-14T23:53:07.801 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/143/13' -> 'copyofsnap1/test1/143/13' 2024-04-14T23:53:07.801 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/143/14' -> 'copyofsnap1/test1/143/14' 2024-04-14T23:53:07.801 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/143/15' -> 'copyofsnap1/test1/143/15' 2024-04-14T23:53:07.801 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/143/16' -> 'copyofsnap1/test1/143/16' 2024-04-14T23:53:07.801 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/143/17' -> 'copyofsnap1/test1/143/17' 2024-04-14T23:53:07.801 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/143/18' -> 'copyofsnap1/test1/143/18' 2024-04-14T23:53:07.802 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/143/19' -> 'copyofsnap1/test1/143/19' 2024-04-14T23:53:07.802 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/143/20' -> 'copyofsnap1/test1/143/20' 2024-04-14T23:53:07.802 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/144' -> 'copyofsnap1/test1/144' 2024-04-14T23:53:07.802 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/144/10' -> 'copyofsnap1/test1/144/10' 2024-04-14T23:53:07.802 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/144/11' -> 'copyofsnap1/test1/144/11' 2024-04-14T23:53:07.802 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/144/12' -> 'copyofsnap1/test1/144/12' 2024-04-14T23:53:07.802 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/144/13' -> 'copyofsnap1/test1/144/13' 2024-04-14T23:53:07.802 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/144/14' -> 'copyofsnap1/test1/144/14' 2024-04-14T23:53:07.802 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/144/15' -> 'copyofsnap1/test1/144/15' 2024-04-14T23:53:07.802 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/144/16' -> 'copyofsnap1/test1/144/16' 2024-04-14T23:53:07.802 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/144/17' -> 'copyofsnap1/test1/144/17' 2024-04-14T23:53:07.802 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/144/18' -> 'copyofsnap1/test1/144/18' 2024-04-14T23:53:07.802 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/144/19' -> 'copyofsnap1/test1/144/19' 2024-04-14T23:53:07.802 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/144/20' -> 'copyofsnap1/test1/144/20' 2024-04-14T23:53:07.802 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/145' -> 'copyofsnap1/test1/145' 2024-04-14T23:53:07.803 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/145/10' -> 'copyofsnap1/test1/145/10' 2024-04-14T23:53:07.803 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/145/11' -> 'copyofsnap1/test1/145/11' 2024-04-14T23:53:07.803 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/145/12' -> 'copyofsnap1/test1/145/12' 2024-04-14T23:53:07.803 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/145/13' -> 'copyofsnap1/test1/145/13' 2024-04-14T23:53:07.803 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/145/14' -> 'copyofsnap1/test1/145/14' 2024-04-14T23:53:07.803 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/145/15' -> 'copyofsnap1/test1/145/15' 2024-04-14T23:53:07.803 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/145/16' -> 'copyofsnap1/test1/145/16' 2024-04-14T23:53:07.803 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/145/17' -> 'copyofsnap1/test1/145/17' 2024-04-14T23:53:07.803 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/145/18' -> 'copyofsnap1/test1/145/18' 2024-04-14T23:53:07.803 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/145/19' -> 'copyofsnap1/test1/145/19' 2024-04-14T23:53:07.803 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/145/20' -> 'copyofsnap1/test1/145/20' 2024-04-14T23:53:07.803 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/146' -> 'copyofsnap1/test1/146' 2024-04-14T23:53:07.803 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/146/10' -> 'copyofsnap1/test1/146/10' 2024-04-14T23:53:07.803 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/146/11' -> 'copyofsnap1/test1/146/11' 2024-04-14T23:53:07.804 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/146/12' -> 'copyofsnap1/test1/146/12' 2024-04-14T23:53:07.804 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/146/13' -> 'copyofsnap1/test1/146/13' 2024-04-14T23:53:07.804 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/146/14' -> 'copyofsnap1/test1/146/14' 2024-04-14T23:53:07.804 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/146/15' -> 'copyofsnap1/test1/146/15' 2024-04-14T23:53:07.804 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/146/16' -> 'copyofsnap1/test1/146/16' 2024-04-14T23:53:07.804 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/146/17' -> 'copyofsnap1/test1/146/17' 2024-04-14T23:53:07.804 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/146/18' -> 'copyofsnap1/test1/146/18' 2024-04-14T23:53:07.804 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/146/19' -> 'copyofsnap1/test1/146/19' 2024-04-14T23:53:07.804 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/146/20' -> 'copyofsnap1/test1/146/20' 2024-04-14T23:53:07.804 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/147' -> 'copyofsnap1/test1/147' 2024-04-14T23:53:07.804 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/147/10' -> 'copyofsnap1/test1/147/10' 2024-04-14T23:53:07.804 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/147/11' -> 'copyofsnap1/test1/147/11' 2024-04-14T23:53:10.818 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/147/12' -> 'copyofsnap1/test1/147/12' 2024-04-14T23:53:10.818 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/147/13' -> 'copyofsnap1/test1/147/13' 2024-04-14T23:53:10.818 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/147/14' -> 'copyofsnap1/test1/147/14' 2024-04-14T23:53:10.818 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/147/15' -> 'copyofsnap1/test1/147/15' 2024-04-14T23:53:10.818 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/147/16' -> 'copyofsnap1/test1/147/16' 2024-04-14T23:53:10.818 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/147/17' -> 'copyofsnap1/test1/147/17' 2024-04-14T23:53:10.818 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/147/18' -> 'copyofsnap1/test1/147/18' 2024-04-14T23:53:10.818 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/147/19' -> 'copyofsnap1/test1/147/19' 2024-04-14T23:53:10.819 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/147/20' -> 'copyofsnap1/test1/147/20' 2024-04-14T23:53:10.819 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/148' -> 'copyofsnap1/test1/148' 2024-04-14T23:53:10.819 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/148/11' -> 'copyofsnap1/test1/148/11' 2024-04-14T23:53:10.819 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/148/10' -> 'copyofsnap1/test1/148/10' 2024-04-14T23:53:10.819 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/148/12' -> 'copyofsnap1/test1/148/12' 2024-04-14T23:53:10.819 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/148/13' -> 'copyofsnap1/test1/148/13' 2024-04-14T23:53:10.819 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/148/14' -> 'copyofsnap1/test1/148/14' 2024-04-14T23:53:10.819 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/148/15' -> 'copyofsnap1/test1/148/15' 2024-04-14T23:53:10.819 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/148/16' -> 'copyofsnap1/test1/148/16' 2024-04-14T23:53:10.819 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/148/17' -> 'copyofsnap1/test1/148/17' 2024-04-14T23:53:10.819 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/148/18' -> 'copyofsnap1/test1/148/18' 2024-04-14T23:53:10.819 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/148/19' -> 'copyofsnap1/test1/148/19' 2024-04-14T23:53:10.819 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/148/20' -> 'copyofsnap1/test1/148/20' 2024-04-14T23:53:10.819 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/149' -> 'copyofsnap1/test1/149' 2024-04-14T23:53:10.819 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/149/10' -> 'copyofsnap1/test1/149/10' 2024-04-14T23:53:10.820 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/149/11' -> 'copyofsnap1/test1/149/11' 2024-04-14T23:53:10.820 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/149/12' -> 'copyofsnap1/test1/149/12' 2024-04-14T23:53:10.820 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/149/13' -> 'copyofsnap1/test1/149/13' 2024-04-14T23:53:10.820 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/149/14' -> 'copyofsnap1/test1/149/14' 2024-04-14T23:53:10.820 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/149/15' -> 'copyofsnap1/test1/149/15' 2024-04-14T23:53:10.820 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/149/16' -> 'copyofsnap1/test1/149/16' 2024-04-14T23:53:10.820 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/149/17' -> 'copyofsnap1/test1/149/17' 2024-04-14T23:53:10.820 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/149/18' -> 'copyofsnap1/test1/149/18' 2024-04-14T23:53:10.820 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/149/19' -> 'copyofsnap1/test1/149/19' 2024-04-14T23:53:10.820 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/149/20' -> 'copyofsnap1/test1/149/20' 2024-04-14T23:53:10.820 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/150' -> 'copyofsnap1/test1/150' 2024-04-14T23:53:10.820 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/150/10' -> 'copyofsnap1/test1/150/10' 2024-04-14T23:53:10.820 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/150/11' -> 'copyofsnap1/test1/150/11' 2024-04-14T23:53:10.820 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/150/12' -> 'copyofsnap1/test1/150/12' 2024-04-14T23:53:10.820 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/150/13' -> 'copyofsnap1/test1/150/13' 2024-04-14T23:53:10.820 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/150/14' -> 'copyofsnap1/test1/150/14' 2024-04-14T23:53:10.821 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/150/15' -> 'copyofsnap1/test1/150/15' 2024-04-14T23:53:10.821 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/150/16' -> 'copyofsnap1/test1/150/16' 2024-04-14T23:53:10.821 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/150/17' -> 'copyofsnap1/test1/150/17' 2024-04-14T23:53:10.821 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/150/18' -> 'copyofsnap1/test1/150/18' 2024-04-14T23:53:10.821 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/150/19' -> 'copyofsnap1/test1/150/19' 2024-04-14T23:53:10.821 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/150/20' -> 'copyofsnap1/test1/150/20' 2024-04-14T23:53:10.821 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/151' -> 'copyofsnap1/test1/151' 2024-04-14T23:53:10.821 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/151/10' -> 'copyofsnap1/test1/151/10' 2024-04-14T23:53:10.821 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/151/11' -> 'copyofsnap1/test1/151/11' 2024-04-14T23:53:10.821 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/151/12' -> 'copyofsnap1/test1/151/12' 2024-04-14T23:53:10.821 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/151/13' -> 'copyofsnap1/test1/151/13' 2024-04-14T23:53:10.821 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/151/14' -> 'copyofsnap1/test1/151/14' 2024-04-14T23:53:10.821 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/151/15' -> 'copyofsnap1/test1/151/15' 2024-04-14T23:53:10.821 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/151/16' -> 'copyofsnap1/test1/151/16' 2024-04-14T23:53:10.821 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/151/17' -> 'copyofsnap1/test1/151/17' 2024-04-14T23:53:10.822 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/151/18' -> 'copyofsnap1/test1/151/18' 2024-04-14T23:53:10.822 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/151/19' -> 'copyofsnap1/test1/151/19' 2024-04-14T23:53:10.822 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/151/20' -> 'copyofsnap1/test1/151/20' 2024-04-14T23:53:10.822 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/152' -> 'copyofsnap1/test1/152' 2024-04-14T23:53:10.822 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/152/10' -> 'copyofsnap1/test1/152/10' 2024-04-14T23:53:10.822 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/152/11' -> 'copyofsnap1/test1/152/11' 2024-04-14T23:53:10.822 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/152/12' -> 'copyofsnap1/test1/152/12' 2024-04-14T23:53:10.822 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/152/13' -> 'copyofsnap1/test1/152/13' 2024-04-14T23:53:10.822 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/152/14' -> 'copyofsnap1/test1/152/14' 2024-04-14T23:53:10.822 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/152/15' -> 'copyofsnap1/test1/152/15' 2024-04-14T23:53:10.822 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/152/16' -> 'copyofsnap1/test1/152/16' 2024-04-14T23:53:10.822 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/152/17' -> 'copyofsnap1/test1/152/17' 2024-04-14T23:53:10.822 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/152/18' -> 'copyofsnap1/test1/152/18' 2024-04-14T23:53:10.822 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/152/19' -> 'copyofsnap1/test1/152/19' 2024-04-14T23:53:10.823 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/152/20' -> 'copyofsnap1/test1/152/20' 2024-04-14T23:53:10.823 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/153' -> 'copyofsnap1/test1/153' 2024-04-14T23:53:10.823 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/153/10' -> 'copyofsnap1/test1/153/10' 2024-04-14T23:53:10.823 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/153/11' -> 'copyofsnap1/test1/153/11' 2024-04-14T23:53:10.823 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/153/12' -> 'copyofsnap1/test1/153/12' 2024-04-14T23:53:10.823 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/153/13' -> 'copyofsnap1/test1/153/13' 2024-04-14T23:53:10.823 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/153/14' -> 'copyofsnap1/test1/153/14' 2024-04-14T23:53:10.823 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/153/15' -> 'copyofsnap1/test1/153/15' 2024-04-14T23:53:10.823 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/153/16' -> 'copyofsnap1/test1/153/16' 2024-04-14T23:53:10.823 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/153/17' -> 'copyofsnap1/test1/153/17' 2024-04-14T23:53:10.823 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/153/18' -> 'copyofsnap1/test1/153/18' 2024-04-14T23:53:10.823 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/153/19' -> 'copyofsnap1/test1/153/19' 2024-04-14T23:53:10.823 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/153/20' -> 'copyofsnap1/test1/153/20' 2024-04-14T23:53:11.189 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/154' -> 'copyofsnap1/test1/154' 2024-04-14T23:53:11.189 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/154/10' -> 'copyofsnap1/test1/154/10' 2024-04-14T23:53:11.189 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/154/11' -> 'copyofsnap1/test1/154/11' 2024-04-14T23:53:11.190 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/154/12' -> 'copyofsnap1/test1/154/12' 2024-04-14T23:53:11.190 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/154/13' -> 'copyofsnap1/test1/154/13' 2024-04-14T23:53:11.190 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/154/14' -> 'copyofsnap1/test1/154/14' 2024-04-14T23:53:11.190 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/154/15' -> 'copyofsnap1/test1/154/15' 2024-04-14T23:53:11.190 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/154/16' -> 'copyofsnap1/test1/154/16' 2024-04-14T23:53:11.190 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/154/17' -> 'copyofsnap1/test1/154/17' 2024-04-14T23:53:11.190 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/154/18' -> 'copyofsnap1/test1/154/18' 2024-04-14T23:53:11.190 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/154/19' -> 'copyofsnap1/test1/154/19' 2024-04-14T23:53:11.190 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/154/20' -> 'copyofsnap1/test1/154/20' 2024-04-14T23:53:11.190 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/155' -> 'copyofsnap1/test1/155' 2024-04-14T23:53:11.190 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/155/10' -> 'copyofsnap1/test1/155/10' 2024-04-14T23:53:11.190 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/155/11' -> 'copyofsnap1/test1/155/11' 2024-04-14T23:53:11.190 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/155/12' -> 'copyofsnap1/test1/155/12' 2024-04-14T23:53:11.190 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/155/13' -> 'copyofsnap1/test1/155/13' 2024-04-14T23:53:11.191 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/155/14' -> 'copyofsnap1/test1/155/14' 2024-04-14T23:53:11.191 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/155/15' -> 'copyofsnap1/test1/155/15' 2024-04-14T23:53:11.191 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/155/16' -> 'copyofsnap1/test1/155/16' 2024-04-14T23:53:11.191 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/155/17' -> 'copyofsnap1/test1/155/17' 2024-04-14T23:53:11.191 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/155/18' -> 'copyofsnap1/test1/155/18' 2024-04-14T23:53:11.191 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/155/19' -> 'copyofsnap1/test1/155/19' 2024-04-14T23:53:11.191 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/155/20' -> 'copyofsnap1/test1/155/20' 2024-04-14T23:53:11.191 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/156' -> 'copyofsnap1/test1/156' 2024-04-14T23:53:11.191 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/156/10' -> 'copyofsnap1/test1/156/10' 2024-04-14T23:53:11.191 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/156/11' -> 'copyofsnap1/test1/156/11' 2024-04-14T23:53:11.191 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/156/12' -> 'copyofsnap1/test1/156/12' 2024-04-14T23:53:11.191 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/156/13' -> 'copyofsnap1/test1/156/13' 2024-04-14T23:53:11.191 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/156/14' -> 'copyofsnap1/test1/156/14' 2024-04-14T23:53:11.191 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/156/15' -> 'copyofsnap1/test1/156/15' 2024-04-14T23:53:11.192 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/156/16' -> 'copyofsnap1/test1/156/16' 2024-04-14T23:53:11.192 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/156/17' -> 'copyofsnap1/test1/156/17' 2024-04-14T23:53:11.192 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/156/18' -> 'copyofsnap1/test1/156/18' 2024-04-14T23:53:11.192 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/156/19' -> 'copyofsnap1/test1/156/19' 2024-04-14T23:53:11.192 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/156/20' -> 'copyofsnap1/test1/156/20' 2024-04-14T23:53:11.192 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/157' -> 'copyofsnap1/test1/157' 2024-04-14T23:53:11.192 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/157/10' -> 'copyofsnap1/test1/157/10' 2024-04-14T23:53:11.192 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/157/11' -> 'copyofsnap1/test1/157/11' 2024-04-14T23:53:11.192 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/157/12' -> 'copyofsnap1/test1/157/12' 2024-04-14T23:53:11.192 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/157/13' -> 'copyofsnap1/test1/157/13' 2024-04-14T23:53:11.192 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/157/14' -> 'copyofsnap1/test1/157/14' 2024-04-14T23:53:11.192 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/157/15' -> 'copyofsnap1/test1/157/15' 2024-04-14T23:53:11.192 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/157/16' -> 'copyofsnap1/test1/157/16' 2024-04-14T23:53:11.192 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/157/17' -> 'copyofsnap1/test1/157/17' 2024-04-14T23:53:11.192 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/157/18' -> 'copyofsnap1/test1/157/18' 2024-04-14T23:53:11.193 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/157/19' -> 'copyofsnap1/test1/157/19' 2024-04-14T23:53:11.193 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/157/20' -> 'copyofsnap1/test1/157/20' 2024-04-14T23:53:11.193 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/158' -> 'copyofsnap1/test1/158' 2024-04-14T23:53:11.193 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/158/10' -> 'copyofsnap1/test1/158/10' 2024-04-14T23:53:11.193 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/158/11' -> 'copyofsnap1/test1/158/11' 2024-04-14T23:53:11.193 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/158/12' -> 'copyofsnap1/test1/158/12' 2024-04-14T23:53:11.193 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/158/13' -> 'copyofsnap1/test1/158/13' 2024-04-14T23:53:11.193 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/158/14' -> 'copyofsnap1/test1/158/14' 2024-04-14T23:53:11.193 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/158/15' -> 'copyofsnap1/test1/158/15' 2024-04-14T23:53:11.193 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/158/16' -> 'copyofsnap1/test1/158/16' 2024-04-14T23:53:11.193 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/158/17' -> 'copyofsnap1/test1/158/17' 2024-04-14T23:53:11.193 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/158/18' -> 'copyofsnap1/test1/158/18' 2024-04-14T23:53:11.193 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/158/19' -> 'copyofsnap1/test1/158/19' 2024-04-14T23:53:11.193 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/158/20' -> 'copyofsnap1/test1/158/20' 2024-04-14T23:53:11.194 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/159' -> 'copyofsnap1/test1/159' 2024-04-14T23:53:11.194 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/159/10' -> 'copyofsnap1/test1/159/10' 2024-04-14T23:53:11.194 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/159/11' -> 'copyofsnap1/test1/159/11' 2024-04-14T23:53:11.194 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/159/12' -> 'copyofsnap1/test1/159/12' 2024-04-14T23:53:11.194 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/159/13' -> 'copyofsnap1/test1/159/13' 2024-04-14T23:53:11.194 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/159/14' -> 'copyofsnap1/test1/159/14' 2024-04-14T23:53:11.194 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/159/15' -> 'copyofsnap1/test1/159/15' 2024-04-14T23:53:11.194 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/159/16' -> 'copyofsnap1/test1/159/16' 2024-04-14T23:53:11.194 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/159/17' -> 'copyofsnap1/test1/159/17' 2024-04-14T23:53:11.194 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/159/18' -> 'copyofsnap1/test1/159/18' 2024-04-14T23:53:11.194 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/159/19' -> 'copyofsnap1/test1/159/19' 2024-04-14T23:53:11.194 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/159/20' -> 'copyofsnap1/test1/159/20' 2024-04-14T23:53:11.194 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/160' -> 'copyofsnap1/test1/160' 2024-04-14T23:53:11.195 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/160/10' -> 'copyofsnap1/test1/160/10' 2024-04-14T23:53:11.195 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/160/11' -> 'copyofsnap1/test1/160/11' 2024-04-14T23:53:11.195 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/160/12' -> 'copyofsnap1/test1/160/12' 2024-04-14T23:53:11.195 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/160/13' -> 'copyofsnap1/test1/160/13' 2024-04-14T23:53:11.195 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/160/14' -> 'copyofsnap1/test1/160/14' 2024-04-14T23:53:11.195 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/160/15' -> 'copyofsnap1/test1/160/15' 2024-04-14T23:53:11.195 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/160/16' -> 'copyofsnap1/test1/160/16' 2024-04-14T23:53:11.195 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/160/17' -> 'copyofsnap1/test1/160/17' 2024-04-14T23:53:12.623 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/160/18' -> 'copyofsnap1/test1/160/18' 2024-04-14T23:53:12.623 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/160/19' -> 'copyofsnap1/test1/160/19' 2024-04-14T23:53:12.623 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/160/20' -> 'copyofsnap1/test1/160/20' 2024-04-14T23:53:12.623 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/161' -> 'copyofsnap1/test1/161' 2024-04-14T23:53:12.623 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/161/10' -> 'copyofsnap1/test1/161/10' 2024-04-14T23:53:12.623 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/161/11' -> 'copyofsnap1/test1/161/11' 2024-04-14T23:53:12.623 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/161/12' -> 'copyofsnap1/test1/161/12' 2024-04-14T23:53:12.623 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/161/13' -> 'copyofsnap1/test1/161/13' 2024-04-14T23:53:12.623 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/161/14' -> 'copyofsnap1/test1/161/14' 2024-04-14T23:53:12.624 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/161/15' -> 'copyofsnap1/test1/161/15' 2024-04-14T23:53:12.624 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/161/16' -> 'copyofsnap1/test1/161/16' 2024-04-14T23:53:12.624 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/161/17' -> 'copyofsnap1/test1/161/17' 2024-04-14T23:53:12.624 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/161/18' -> 'copyofsnap1/test1/161/18' 2024-04-14T23:53:12.624 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/161/19' -> 'copyofsnap1/test1/161/19' 2024-04-14T23:53:12.624 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/161/20' -> 'copyofsnap1/test1/161/20' 2024-04-14T23:53:12.624 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/162' -> 'copyofsnap1/test1/162' 2024-04-14T23:53:12.624 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/162/10' -> 'copyofsnap1/test1/162/10' 2024-04-14T23:53:12.624 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/162/11' -> 'copyofsnap1/test1/162/11' 2024-04-14T23:53:12.624 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/162/12' -> 'copyofsnap1/test1/162/12' 2024-04-14T23:53:12.624 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/162/13' -> 'copyofsnap1/test1/162/13' 2024-04-14T23:53:12.624 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/162/14' -> 'copyofsnap1/test1/162/14' 2024-04-14T23:53:12.624 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/162/15' -> 'copyofsnap1/test1/162/15' 2024-04-14T23:53:12.624 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/162/16' -> 'copyofsnap1/test1/162/16' 2024-04-14T23:53:12.624 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/162/17' -> 'copyofsnap1/test1/162/17' 2024-04-14T23:53:12.625 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/162/18' -> 'copyofsnap1/test1/162/18' 2024-04-14T23:53:12.625 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/162/19' -> 'copyofsnap1/test1/162/19' 2024-04-14T23:53:12.625 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/162/20' -> 'copyofsnap1/test1/162/20' 2024-04-14T23:53:12.625 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/163' -> 'copyofsnap1/test1/163' 2024-04-14T23:53:12.625 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/163/10' -> 'copyofsnap1/test1/163/10' 2024-04-14T23:53:12.625 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/163/11' -> 'copyofsnap1/test1/163/11' 2024-04-14T23:53:12.625 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/163/12' -> 'copyofsnap1/test1/163/12' 2024-04-14T23:53:12.625 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/163/13' -> 'copyofsnap1/test1/163/13' 2024-04-14T23:53:12.625 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/163/14' -> 'copyofsnap1/test1/163/14' 2024-04-14T23:53:12.625 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/163/15' -> 'copyofsnap1/test1/163/15' 2024-04-14T23:53:12.625 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/163/16' -> 'copyofsnap1/test1/163/16' 2024-04-14T23:53:12.625 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/163/17' -> 'copyofsnap1/test1/163/17' 2024-04-14T23:53:12.625 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/163/18' -> 'copyofsnap1/test1/163/18' 2024-04-14T23:53:12.625 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/163/19' -> 'copyofsnap1/test1/163/19' 2024-04-14T23:53:12.626 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/163/20' -> 'copyofsnap1/test1/163/20' 2024-04-14T23:53:12.626 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/164' -> 'copyofsnap1/test1/164' 2024-04-14T23:53:12.626 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/164/10' -> 'copyofsnap1/test1/164/10' 2024-04-14T23:53:12.626 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/164/11' -> 'copyofsnap1/test1/164/11' 2024-04-14T23:53:12.626 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/164/12' -> 'copyofsnap1/test1/164/12' 2024-04-14T23:53:12.626 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/164/13' -> 'copyofsnap1/test1/164/13' 2024-04-14T23:53:12.626 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/164/14' -> 'copyofsnap1/test1/164/14' 2024-04-14T23:53:12.626 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/164/15' -> 'copyofsnap1/test1/164/15' 2024-04-14T23:53:12.626 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/164/16' -> 'copyofsnap1/test1/164/16' 2024-04-14T23:53:12.626 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/164/17' -> 'copyofsnap1/test1/164/17' 2024-04-14T23:53:12.626 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/164/18' -> 'copyofsnap1/test1/164/18' 2024-04-14T23:53:12.626 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/164/19' -> 'copyofsnap1/test1/164/19' 2024-04-14T23:53:12.626 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/164/20' -> 'copyofsnap1/test1/164/20' 2024-04-14T23:53:12.626 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/165' -> 'copyofsnap1/test1/165' 2024-04-14T23:53:12.627 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/165/10' -> 'copyofsnap1/test1/165/10' 2024-04-14T23:53:12.627 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/165/11' -> 'copyofsnap1/test1/165/11' 2024-04-14T23:53:12.627 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/165/12' -> 'copyofsnap1/test1/165/12' 2024-04-14T23:53:12.627 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/165/13' -> 'copyofsnap1/test1/165/13' 2024-04-14T23:53:12.627 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/165/14' -> 'copyofsnap1/test1/165/14' 2024-04-14T23:53:12.627 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/165/15' -> 'copyofsnap1/test1/165/15' 2024-04-14T23:53:12.627 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/165/16' -> 'copyofsnap1/test1/165/16' 2024-04-14T23:53:12.627 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/165/17' -> 'copyofsnap1/test1/165/17' 2024-04-14T23:53:12.627 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/165/18' -> 'copyofsnap1/test1/165/18' 2024-04-14T23:53:12.627 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/165/19' -> 'copyofsnap1/test1/165/19' 2024-04-14T23:53:12.627 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/165/20' -> 'copyofsnap1/test1/165/20' 2024-04-14T23:53:12.627 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/166' -> 'copyofsnap1/test1/166' 2024-04-14T23:53:12.627 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/166/10' -> 'copyofsnap1/test1/166/10' 2024-04-14T23:53:12.627 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/166/11' -> 'copyofsnap1/test1/166/11' 2024-04-14T23:53:12.627 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/166/12' -> 'copyofsnap1/test1/166/12' 2024-04-14T23:53:12.627 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/166/13' -> 'copyofsnap1/test1/166/13' 2024-04-14T23:53:12.628 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/166/14' -> 'copyofsnap1/test1/166/14' 2024-04-14T23:53:12.628 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/166/15' -> 'copyofsnap1/test1/166/15' 2024-04-14T23:53:12.628 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/166/16' -> 'copyofsnap1/test1/166/16' 2024-04-14T23:53:12.628 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/166/17' -> 'copyofsnap1/test1/166/17' 2024-04-14T23:53:12.628 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/166/18' -> 'copyofsnap1/test1/166/18' 2024-04-14T23:53:12.628 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/166/19' -> 'copyofsnap1/test1/166/19' 2024-04-14T23:53:12.628 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/166/20' -> 'copyofsnap1/test1/166/20' 2024-04-14T23:53:12.628 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/167' -> 'copyofsnap1/test1/167' 2024-04-14T23:53:12.628 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/167/10' -> 'copyofsnap1/test1/167/10' 2024-04-14T23:53:12.628 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/167/11' -> 'copyofsnap1/test1/167/11' 2024-04-14T23:53:12.628 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/167/12' -> 'copyofsnap1/test1/167/12' 2024-04-14T23:53:12.628 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/167/13' -> 'copyofsnap1/test1/167/13' 2024-04-14T23:53:12.628 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/167/14' -> 'copyofsnap1/test1/167/14' 2024-04-14T23:53:12.971 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/167/15' -> 'copyofsnap1/test1/167/15' 2024-04-14T23:53:12.971 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/167/16' -> 'copyofsnap1/test1/167/16' 2024-04-14T23:53:12.971 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/167/17' -> 'copyofsnap1/test1/167/17' 2024-04-14T23:53:12.971 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/167/18' -> 'copyofsnap1/test1/167/18' 2024-04-14T23:53:12.971 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/167/19' -> 'copyofsnap1/test1/167/19' 2024-04-14T23:53:12.971 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/167/20' -> 'copyofsnap1/test1/167/20' 2024-04-14T23:53:12.972 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/168' -> 'copyofsnap1/test1/168' 2024-04-14T23:53:12.972 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/168/10' -> 'copyofsnap1/test1/168/10' 2024-04-14T23:53:12.972 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/168/11' -> 'copyofsnap1/test1/168/11' 2024-04-14T23:53:12.972 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/168/12' -> 'copyofsnap1/test1/168/12' 2024-04-14T23:53:12.972 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/168/13' -> 'copyofsnap1/test1/168/13' 2024-04-14T23:53:12.972 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/168/14' -> 'copyofsnap1/test1/168/14' 2024-04-14T23:53:12.972 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/168/15' -> 'copyofsnap1/test1/168/15' 2024-04-14T23:53:12.972 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/168/16' -> 'copyofsnap1/test1/168/16' 2024-04-14T23:53:12.972 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/168/17' -> 'copyofsnap1/test1/168/17' 2024-04-14T23:53:12.972 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/168/18' -> 'copyofsnap1/test1/168/18' 2024-04-14T23:53:12.972 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/168/19' -> 'copyofsnap1/test1/168/19' 2024-04-14T23:53:12.972 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/168/20' -> 'copyofsnap1/test1/168/20' 2024-04-14T23:53:12.972 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/169' -> 'copyofsnap1/test1/169' 2024-04-14T23:53:12.972 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/169/10' -> 'copyofsnap1/test1/169/10' 2024-04-14T23:53:12.972 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/169/11' -> 'copyofsnap1/test1/169/11' 2024-04-14T23:53:12.973 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/169/12' -> 'copyofsnap1/test1/169/12' 2024-04-14T23:53:12.973 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/169/13' -> 'copyofsnap1/test1/169/13' 2024-04-14T23:53:12.973 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/169/14' -> 'copyofsnap1/test1/169/14' 2024-04-14T23:53:12.973 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/169/15' -> 'copyofsnap1/test1/169/15' 2024-04-14T23:53:12.973 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/169/16' -> 'copyofsnap1/test1/169/16' 2024-04-14T23:53:12.973 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/169/17' -> 'copyofsnap1/test1/169/17' 2024-04-14T23:53:12.973 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/169/18' -> 'copyofsnap1/test1/169/18' 2024-04-14T23:53:12.973 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/169/19' -> 'copyofsnap1/test1/169/19' 2024-04-14T23:53:12.973 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/169/20' -> 'copyofsnap1/test1/169/20' 2024-04-14T23:53:12.973 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/170' -> 'copyofsnap1/test1/170' 2024-04-14T23:53:12.973 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/170/10' -> 'copyofsnap1/test1/170/10' 2024-04-14T23:53:12.973 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/170/11' -> 'copyofsnap1/test1/170/11' 2024-04-14T23:53:12.973 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/170/12' -> 'copyofsnap1/test1/170/12' 2024-04-14T23:53:12.973 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/170/13' -> 'copyofsnap1/test1/170/13' 2024-04-14T23:53:12.973 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/170/14' -> 'copyofsnap1/test1/170/14' 2024-04-14T23:53:12.974 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/170/15' -> 'copyofsnap1/test1/170/15' 2024-04-14T23:53:12.974 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/170/16' -> 'copyofsnap1/test1/170/16' 2024-04-14T23:53:12.974 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/170/17' -> 'copyofsnap1/test1/170/17' 2024-04-14T23:53:12.974 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/170/18' -> 'copyofsnap1/test1/170/18' 2024-04-14T23:53:12.974 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/170/19' -> 'copyofsnap1/test1/170/19' 2024-04-14T23:53:12.974 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/170/20' -> 'copyofsnap1/test1/170/20' 2024-04-14T23:53:12.974 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/171' -> 'copyofsnap1/test1/171' 2024-04-14T23:53:12.974 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/171/10' -> 'copyofsnap1/test1/171/10' 2024-04-14T23:53:12.974 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/171/11' -> 'copyofsnap1/test1/171/11' 2024-04-14T23:53:12.974 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/171/12' -> 'copyofsnap1/test1/171/12' 2024-04-14T23:53:12.974 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/171/13' -> 'copyofsnap1/test1/171/13' 2024-04-14T23:53:12.974 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/171/14' -> 'copyofsnap1/test1/171/14' 2024-04-14T23:53:12.974 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/171/15' -> 'copyofsnap1/test1/171/15' 2024-04-14T23:53:12.974 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/171/16' -> 'copyofsnap1/test1/171/16' 2024-04-14T23:53:12.975 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/171/17' -> 'copyofsnap1/test1/171/17' 2024-04-14T23:53:12.975 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/171/18' -> 'copyofsnap1/test1/171/18' 2024-04-14T23:53:12.975 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/171/19' -> 'copyofsnap1/test1/171/19' 2024-04-14T23:53:12.975 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/171/20' -> 'copyofsnap1/test1/171/20' 2024-04-14T23:53:12.975 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/172' -> 'copyofsnap1/test1/172' 2024-04-14T23:53:12.975 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/172/10' -> 'copyofsnap1/test1/172/10' 2024-04-14T23:53:12.975 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/172/11' -> 'copyofsnap1/test1/172/11' 2024-04-14T23:53:12.975 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/172/12' -> 'copyofsnap1/test1/172/12' 2024-04-14T23:53:12.975 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/172/13' -> 'copyofsnap1/test1/172/13' 2024-04-14T23:53:12.975 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/172/14' -> 'copyofsnap1/test1/172/14' 2024-04-14T23:53:12.975 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/172/15' -> 'copyofsnap1/test1/172/15' 2024-04-14T23:53:12.975 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/172/16' -> 'copyofsnap1/test1/172/16' 2024-04-14T23:53:12.975 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/172/17' -> 'copyofsnap1/test1/172/17' 2024-04-14T23:53:12.975 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/172/18' -> 'copyofsnap1/test1/172/18' 2024-04-14T23:53:12.976 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/172/19' -> 'copyofsnap1/test1/172/19' 2024-04-14T23:53:12.976 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/172/20' -> 'copyofsnap1/test1/172/20' 2024-04-14T23:53:12.976 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/173' -> 'copyofsnap1/test1/173' 2024-04-14T23:53:12.976 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/173/10' -> 'copyofsnap1/test1/173/10' 2024-04-14T23:53:12.976 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/173/11' -> 'copyofsnap1/test1/173/11' 2024-04-14T23:53:12.976 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/173/12' -> 'copyofsnap1/test1/173/12' 2024-04-14T23:53:12.976 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/173/13' -> 'copyofsnap1/test1/173/13' 2024-04-14T23:53:12.976 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/173/14' -> 'copyofsnap1/test1/173/14' 2024-04-14T23:53:12.976 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/173/15' -> 'copyofsnap1/test1/173/15' 2024-04-14T23:53:12.976 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/173/16' -> 'copyofsnap1/test1/173/16' 2024-04-14T23:53:12.976 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/173/17' -> 'copyofsnap1/test1/173/17' 2024-04-14T23:53:12.976 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/173/18' -> 'copyofsnap1/test1/173/18' 2024-04-14T23:53:12.976 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/173/19' -> 'copyofsnap1/test1/173/19' 2024-04-14T23:53:12.976 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/173/20' -> 'copyofsnap1/test1/173/20' 2024-04-14T23:53:12.976 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/174' -> 'copyofsnap1/test1/174' 2024-04-14T23:53:12.977 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/174/10' -> 'copyofsnap1/test1/174/10' 2024-04-14T23:53:12.977 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/174/11' -> 'copyofsnap1/test1/174/11' 2024-04-14T23:53:14.009 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/174/12' -> 'copyofsnap1/test1/174/12' 2024-04-14T23:53:14.009 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/174/13' -> 'copyofsnap1/test1/174/13' 2024-04-14T23:53:14.010 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/174/14' -> 'copyofsnap1/test1/174/14' 2024-04-14T23:53:14.010 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/174/15' -> 'copyofsnap1/test1/174/15' 2024-04-14T23:53:14.010 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/174/16' -> 'copyofsnap1/test1/174/16' 2024-04-14T23:53:14.010 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/174/17' -> 'copyofsnap1/test1/174/17' 2024-04-14T23:53:14.010 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/174/18' -> 'copyofsnap1/test1/174/18' 2024-04-14T23:53:14.010 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/174/19' -> 'copyofsnap1/test1/174/19' 2024-04-14T23:53:14.010 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/174/20' -> 'copyofsnap1/test1/174/20' 2024-04-14T23:53:14.010 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/175' -> 'copyofsnap1/test1/175' 2024-04-14T23:53:14.010 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/175/10' -> 'copyofsnap1/test1/175/10' 2024-04-14T23:53:14.010 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/175/11' -> 'copyofsnap1/test1/175/11' 2024-04-14T23:53:14.010 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/175/12' -> 'copyofsnap1/test1/175/12' 2024-04-14T23:53:14.010 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/175/13' -> 'copyofsnap1/test1/175/13' 2024-04-14T23:53:14.010 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/175/14' -> 'copyofsnap1/test1/175/14' 2024-04-14T23:53:14.011 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/175/15' -> 'copyofsnap1/test1/175/15' 2024-04-14T23:53:14.011 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/175/16' -> 'copyofsnap1/test1/175/16' 2024-04-14T23:53:14.011 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/175/17' -> 'copyofsnap1/test1/175/17' 2024-04-14T23:53:14.011 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/175/18' -> 'copyofsnap1/test1/175/18' 2024-04-14T23:53:14.011 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/175/19' -> 'copyofsnap1/test1/175/19' 2024-04-14T23:53:14.011 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/175/20' -> 'copyofsnap1/test1/175/20' 2024-04-14T23:53:14.011 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/176' -> 'copyofsnap1/test1/176' 2024-04-14T23:53:14.011 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/176/10' -> 'copyofsnap1/test1/176/10' 2024-04-14T23:53:14.011 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/176/11' -> 'copyofsnap1/test1/176/11' 2024-04-14T23:53:14.011 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/176/12' -> 'copyofsnap1/test1/176/12' 2024-04-14T23:53:14.011 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/176/13' -> 'copyofsnap1/test1/176/13' 2024-04-14T23:53:14.011 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/176/14' -> 'copyofsnap1/test1/176/14' 2024-04-14T23:53:14.011 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/176/15' -> 'copyofsnap1/test1/176/15' 2024-04-14T23:53:14.011 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/176/16' -> 'copyofsnap1/test1/176/16' 2024-04-14T23:53:14.011 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/176/17' -> 'copyofsnap1/test1/176/17' 2024-04-14T23:53:14.012 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/176/18' -> 'copyofsnap1/test1/176/18' 2024-04-14T23:53:14.012 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/176/19' -> 'copyofsnap1/test1/176/19' 2024-04-14T23:53:14.012 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/176/20' -> 'copyofsnap1/test1/176/20' 2024-04-14T23:53:14.012 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/177' -> 'copyofsnap1/test1/177' 2024-04-14T23:53:14.012 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/177/10' -> 'copyofsnap1/test1/177/10' 2024-04-14T23:53:14.012 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/177/11' -> 'copyofsnap1/test1/177/11' 2024-04-14T23:53:14.012 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/177/12' -> 'copyofsnap1/test1/177/12' 2024-04-14T23:53:14.012 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/177/13' -> 'copyofsnap1/test1/177/13' 2024-04-14T23:53:14.012 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/177/14' -> 'copyofsnap1/test1/177/14' 2024-04-14T23:53:14.012 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/177/15' -> 'copyofsnap1/test1/177/15' 2024-04-14T23:53:14.012 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/177/16' -> 'copyofsnap1/test1/177/16' 2024-04-14T23:53:14.012 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/177/17' -> 'copyofsnap1/test1/177/17' 2024-04-14T23:53:14.012 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/177/18' -> 'copyofsnap1/test1/177/18' 2024-04-14T23:53:14.012 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/177/19' -> 'copyofsnap1/test1/177/19' 2024-04-14T23:53:14.012 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/177/20' -> 'copyofsnap1/test1/177/20' 2024-04-14T23:53:14.013 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/178' -> 'copyofsnap1/test1/178' 2024-04-14T23:53:14.013 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/178/10' -> 'copyofsnap1/test1/178/10' 2024-04-14T23:53:14.013 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/178/11' -> 'copyofsnap1/test1/178/11' 2024-04-14T23:53:14.013 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/178/12' -> 'copyofsnap1/test1/178/12' 2024-04-14T23:53:14.013 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/178/13' -> 'copyofsnap1/test1/178/13' 2024-04-14T23:53:14.013 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/178/14' -> 'copyofsnap1/test1/178/14' 2024-04-14T23:53:14.013 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/178/15' -> 'copyofsnap1/test1/178/15' 2024-04-14T23:53:14.013 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/178/16' -> 'copyofsnap1/test1/178/16' 2024-04-14T23:53:14.013 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/178/17' -> 'copyofsnap1/test1/178/17' 2024-04-14T23:53:14.013 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/178/18' -> 'copyofsnap1/test1/178/18' 2024-04-14T23:53:14.013 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/178/19' -> 'copyofsnap1/test1/178/19' 2024-04-14T23:53:14.013 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/178/20' -> 'copyofsnap1/test1/178/20' 2024-04-14T23:53:14.013 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/179' -> 'copyofsnap1/test1/179' 2024-04-14T23:53:14.013 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/179/10' -> 'copyofsnap1/test1/179/10' 2024-04-14T23:53:14.014 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/179/11' -> 'copyofsnap1/test1/179/11' 2024-04-14T23:53:14.014 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/179/12' -> 'copyofsnap1/test1/179/12' 2024-04-14T23:53:14.014 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/179/13' -> 'copyofsnap1/test1/179/13' 2024-04-14T23:53:14.014 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/179/14' -> 'copyofsnap1/test1/179/14' 2024-04-14T23:53:14.014 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/179/15' -> 'copyofsnap1/test1/179/15' 2024-04-14T23:53:14.014 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/179/16' -> 'copyofsnap1/test1/179/16' 2024-04-14T23:53:14.014 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/179/17' -> 'copyofsnap1/test1/179/17' 2024-04-14T23:53:14.014 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/179/18' -> 'copyofsnap1/test1/179/18' 2024-04-14T23:53:14.014 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/179/19' -> 'copyofsnap1/test1/179/19' 2024-04-14T23:53:14.014 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/179/20' -> 'copyofsnap1/test1/179/20' 2024-04-14T23:53:14.014 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/180' -> 'copyofsnap1/test1/180' 2024-04-14T23:53:14.014 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/180/10' -> 'copyofsnap1/test1/180/10' 2024-04-14T23:53:14.014 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/180/11' -> 'copyofsnap1/test1/180/11' 2024-04-14T23:53:14.014 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/180/12' -> 'copyofsnap1/test1/180/12' 2024-04-14T23:53:14.015 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/180/13' -> 'copyofsnap1/test1/180/13' 2024-04-14T23:53:14.015 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/180/14' -> 'copyofsnap1/test1/180/14' 2024-04-14T23:53:14.015 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/180/15' -> 'copyofsnap1/test1/180/15' 2024-04-14T23:53:14.015 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/180/16' -> 'copyofsnap1/test1/180/16' 2024-04-14T23:53:14.015 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/180/17' -> 'copyofsnap1/test1/180/17' 2024-04-14T23:53:14.015 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/180/18' -> 'copyofsnap1/test1/180/18' 2024-04-14T23:53:14.015 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/180/19' -> 'copyofsnap1/test1/180/19' 2024-04-14T23:53:14.015 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/180/20' -> 'copyofsnap1/test1/180/20' 2024-04-14T23:53:14.907 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/181' -> 'copyofsnap1/test1/181' 2024-04-14T23:53:14.907 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/181/10' -> 'copyofsnap1/test1/181/10' 2024-04-14T23:53:14.907 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/181/11' -> 'copyofsnap1/test1/181/11' 2024-04-14T23:53:14.907 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/181/12' -> 'copyofsnap1/test1/181/12' 2024-04-14T23:53:14.907 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/181/13' -> 'copyofsnap1/test1/181/13' 2024-04-14T23:53:14.907 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/181/14' -> 'copyofsnap1/test1/181/14' 2024-04-14T23:53:14.907 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/181/15' -> 'copyofsnap1/test1/181/15' 2024-04-14T23:53:14.907 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/181/16' -> 'copyofsnap1/test1/181/16' 2024-04-14T23:53:14.907 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/181/17' -> 'copyofsnap1/test1/181/17' 2024-04-14T23:53:14.907 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/181/18' -> 'copyofsnap1/test1/181/18' 2024-04-14T23:53:14.908 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/181/19' -> 'copyofsnap1/test1/181/19' 2024-04-14T23:53:14.908 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/181/20' -> 'copyofsnap1/test1/181/20' 2024-04-14T23:53:14.908 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/182' -> 'copyofsnap1/test1/182' 2024-04-14T23:53:14.908 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/182/10' -> 'copyofsnap1/test1/182/10' 2024-04-14T23:53:14.908 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/182/11' -> 'copyofsnap1/test1/182/11' 2024-04-14T23:53:14.908 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/182/12' -> 'copyofsnap1/test1/182/12' 2024-04-14T23:53:14.908 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/182/13' -> 'copyofsnap1/test1/182/13' 2024-04-14T23:53:14.908 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/182/14' -> 'copyofsnap1/test1/182/14' 2024-04-14T23:53:14.908 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/182/15' -> 'copyofsnap1/test1/182/15' 2024-04-14T23:53:14.908 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/182/16' -> 'copyofsnap1/test1/182/16' 2024-04-14T23:53:14.908 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/182/17' -> 'copyofsnap1/test1/182/17' 2024-04-14T23:53:14.908 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/182/18' -> 'copyofsnap1/test1/182/18' 2024-04-14T23:53:14.908 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/182/19' -> 'copyofsnap1/test1/182/19' 2024-04-14T23:53:14.908 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/182/20' -> 'copyofsnap1/test1/182/20' 2024-04-14T23:53:14.909 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/183' -> 'copyofsnap1/test1/183' 2024-04-14T23:53:14.909 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/183/10' -> 'copyofsnap1/test1/183/10' 2024-04-14T23:53:14.909 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/183/11' -> 'copyofsnap1/test1/183/11' 2024-04-14T23:53:14.909 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/183/12' -> 'copyofsnap1/test1/183/12' 2024-04-14T23:53:14.909 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/183/13' -> 'copyofsnap1/test1/183/13' 2024-04-14T23:53:14.909 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/183/14' -> 'copyofsnap1/test1/183/14' 2024-04-14T23:53:14.909 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/183/15' -> 'copyofsnap1/test1/183/15' 2024-04-14T23:53:14.909 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/183/16' -> 'copyofsnap1/test1/183/16' 2024-04-14T23:53:14.909 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/183/17' -> 'copyofsnap1/test1/183/17' 2024-04-14T23:53:14.909 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/183/18' -> 'copyofsnap1/test1/183/18' 2024-04-14T23:53:14.909 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/183/19' -> 'copyofsnap1/test1/183/19' 2024-04-14T23:53:14.909 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/183/20' -> 'copyofsnap1/test1/183/20' 2024-04-14T23:53:14.909 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/184' -> 'copyofsnap1/test1/184' 2024-04-14T23:53:14.909 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/184/10' -> 'copyofsnap1/test1/184/10' 2024-04-14T23:53:14.909 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/184/11' -> 'copyofsnap1/test1/184/11' 2024-04-14T23:53:14.910 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/184/12' -> 'copyofsnap1/test1/184/12' 2024-04-14T23:53:14.910 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/184/13' -> 'copyofsnap1/test1/184/13' 2024-04-14T23:53:14.910 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/184/14' -> 'copyofsnap1/test1/184/14' 2024-04-14T23:53:14.910 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/184/15' -> 'copyofsnap1/test1/184/15' 2024-04-14T23:53:14.910 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/184/16' -> 'copyofsnap1/test1/184/16' 2024-04-14T23:53:14.910 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/184/17' -> 'copyofsnap1/test1/184/17' 2024-04-14T23:53:14.910 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/184/18' -> 'copyofsnap1/test1/184/18' 2024-04-14T23:53:14.910 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/184/19' -> 'copyofsnap1/test1/184/19' 2024-04-14T23:53:14.910 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/184/20' -> 'copyofsnap1/test1/184/20' 2024-04-14T23:53:14.910 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/185' -> 'copyofsnap1/test1/185' 2024-04-14T23:53:14.910 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/185/10' -> 'copyofsnap1/test1/185/10' 2024-04-14T23:53:14.910 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/185/11' -> 'copyofsnap1/test1/185/11' 2024-04-14T23:53:14.910 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/185/12' -> 'copyofsnap1/test1/185/12' 2024-04-14T23:53:14.910 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/185/13' -> 'copyofsnap1/test1/185/13' 2024-04-14T23:53:14.911 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/185/14' -> 'copyofsnap1/test1/185/14' 2024-04-14T23:53:14.911 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/185/15' -> 'copyofsnap1/test1/185/15' 2024-04-14T23:53:14.911 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/185/16' -> 'copyofsnap1/test1/185/16' 2024-04-14T23:53:14.911 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/185/17' -> 'copyofsnap1/test1/185/17' 2024-04-14T23:53:14.911 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/185/18' -> 'copyofsnap1/test1/185/18' 2024-04-14T23:53:14.911 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/185/19' -> 'copyofsnap1/test1/185/19' 2024-04-14T23:53:14.911 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/185/20' -> 'copyofsnap1/test1/185/20' 2024-04-14T23:53:14.911 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/186' -> 'copyofsnap1/test1/186' 2024-04-14T23:53:14.911 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/186/10' -> 'copyofsnap1/test1/186/10' 2024-04-14T23:53:14.911 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/186/11' -> 'copyofsnap1/test1/186/11' 2024-04-14T23:53:14.911 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/186/12' -> 'copyofsnap1/test1/186/12' 2024-04-14T23:53:14.911 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/186/13' -> 'copyofsnap1/test1/186/13' 2024-04-14T23:53:14.911 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/186/14' -> 'copyofsnap1/test1/186/14' 2024-04-14T23:53:14.911 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/186/15' -> 'copyofsnap1/test1/186/15' 2024-04-14T23:53:14.911 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/186/16' -> 'copyofsnap1/test1/186/16' 2024-04-14T23:53:14.911 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/186/17' -> 'copyofsnap1/test1/186/17' 2024-04-14T23:53:14.912 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/186/18' -> 'copyofsnap1/test1/186/18' 2024-04-14T23:53:14.912 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/186/19' -> 'copyofsnap1/test1/186/19' 2024-04-14T23:53:14.912 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/186/20' -> 'copyofsnap1/test1/186/20' 2024-04-14T23:53:14.912 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/187' -> 'copyofsnap1/test1/187' 2024-04-14T23:53:14.912 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/187/10' -> 'copyofsnap1/test1/187/10' 2024-04-14T23:53:14.912 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/187/11' -> 'copyofsnap1/test1/187/11' 2024-04-14T23:53:14.912 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/187/12' -> 'copyofsnap1/test1/187/12' 2024-04-14T23:53:14.912 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/187/13' -> 'copyofsnap1/test1/187/13' 2024-04-14T23:53:14.912 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/187/14' -> 'copyofsnap1/test1/187/14' 2024-04-14T23:53:14.912 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/187/15' -> 'copyofsnap1/test1/187/15' 2024-04-14T23:53:14.912 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/187/16' -> 'copyofsnap1/test1/187/16' 2024-04-14T23:53:14.912 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/187/17' -> 'copyofsnap1/test1/187/17' 2024-04-14T23:53:15.088 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-14T23:53:15.099 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-14T23:53:15.483 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/187/18' -> 'copyofsnap1/test1/187/18' 2024-04-14T23:53:15.483 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/187/19' -> 'copyofsnap1/test1/187/19' 2024-04-14T23:53:15.483 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/187/20' -> 'copyofsnap1/test1/187/20' 2024-04-14T23:53:15.483 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/188' -> 'copyofsnap1/test1/188' 2024-04-14T23:53:15.483 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/188/10' -> 'copyofsnap1/test1/188/10' 2024-04-14T23:53:15.483 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/188/11' -> 'copyofsnap1/test1/188/11' 2024-04-14T23:53:15.483 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/188/12' -> 'copyofsnap1/test1/188/12' 2024-04-14T23:53:15.483 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/188/13' -> 'copyofsnap1/test1/188/13' 2024-04-14T23:53:15.483 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/188/14' -> 'copyofsnap1/test1/188/14' 2024-04-14T23:53:15.483 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/188/15' -> 'copyofsnap1/test1/188/15' 2024-04-14T23:53:15.483 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/188/16' -> 'copyofsnap1/test1/188/16' 2024-04-14T23:53:15.484 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/188/17' -> 'copyofsnap1/test1/188/17' 2024-04-14T23:53:15.484 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/188/18' -> 'copyofsnap1/test1/188/18' 2024-04-14T23:53:15.484 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/188/19' -> 'copyofsnap1/test1/188/19' 2024-04-14T23:53:15.484 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/188/20' -> 'copyofsnap1/test1/188/20' 2024-04-14T23:53:15.484 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/189' -> 'copyofsnap1/test1/189' 2024-04-14T23:53:15.484 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/189/10' -> 'copyofsnap1/test1/189/10' 2024-04-14T23:53:15.484 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/189/11' -> 'copyofsnap1/test1/189/11' 2024-04-14T23:53:15.484 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/189/12' -> 'copyofsnap1/test1/189/12' 2024-04-14T23:53:15.484 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/189/13' -> 'copyofsnap1/test1/189/13' 2024-04-14T23:53:15.485 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/189/14' -> 'copyofsnap1/test1/189/14' 2024-04-14T23:53:15.485 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/189/15' -> 'copyofsnap1/test1/189/15' 2024-04-14T23:53:15.485 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/189/16' -> 'copyofsnap1/test1/189/16' 2024-04-14T23:53:15.485 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/189/17' -> 'copyofsnap1/test1/189/17' 2024-04-14T23:53:15.485 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/189/18' -> 'copyofsnap1/test1/189/18' 2024-04-14T23:53:15.485 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/189/19' -> 'copyofsnap1/test1/189/19' 2024-04-14T23:53:15.485 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/189/20' -> 'copyofsnap1/test1/189/20' 2024-04-14T23:53:15.485 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/190' -> 'copyofsnap1/test1/190' 2024-04-14T23:53:15.485 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/190/10' -> 'copyofsnap1/test1/190/10' 2024-04-14T23:53:15.485 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/190/11' -> 'copyofsnap1/test1/190/11' 2024-04-14T23:53:15.485 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/190/12' -> 'copyofsnap1/test1/190/12' 2024-04-14T23:53:15.485 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/190/13' -> 'copyofsnap1/test1/190/13' 2024-04-14T23:53:15.486 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/190/14' -> 'copyofsnap1/test1/190/14' 2024-04-14T23:53:15.486 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/190/15' -> 'copyofsnap1/test1/190/15' 2024-04-14T23:53:15.486 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/190/16' -> 'copyofsnap1/test1/190/16' 2024-04-14T23:53:15.486 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/190/17' -> 'copyofsnap1/test1/190/17' 2024-04-14T23:53:15.486 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/190/18' -> 'copyofsnap1/test1/190/18' 2024-04-14T23:53:15.486 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/190/19' -> 'copyofsnap1/test1/190/19' 2024-04-14T23:53:15.486 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/190/20' -> 'copyofsnap1/test1/190/20' 2024-04-14T23:53:15.486 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/191' -> 'copyofsnap1/test1/191' 2024-04-14T23:53:15.486 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/191/10' -> 'copyofsnap1/test1/191/10' 2024-04-14T23:53:15.487 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/191/11' -> 'copyofsnap1/test1/191/11' 2024-04-14T23:53:15.487 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/191/12' -> 'copyofsnap1/test1/191/12' 2024-04-14T23:53:15.487 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/191/13' -> 'copyofsnap1/test1/191/13' 2024-04-14T23:53:15.487 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/191/14' -> 'copyofsnap1/test1/191/14' 2024-04-14T23:53:15.487 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/191/15' -> 'copyofsnap1/test1/191/15' 2024-04-14T23:53:15.487 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/191/16' -> 'copyofsnap1/test1/191/16' 2024-04-14T23:53:15.487 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/191/17' -> 'copyofsnap1/test1/191/17' 2024-04-14T23:53:15.487 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/191/18' -> 'copyofsnap1/test1/191/18' 2024-04-14T23:53:15.487 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/191/19' -> 'copyofsnap1/test1/191/19' 2024-04-14T23:53:15.487 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/191/20' -> 'copyofsnap1/test1/191/20' 2024-04-14T23:53:15.487 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/192' -> 'copyofsnap1/test1/192' 2024-04-14T23:53:15.488 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/192/10' -> 'copyofsnap1/test1/192/10' 2024-04-14T23:53:15.488 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/192/11' -> 'copyofsnap1/test1/192/11' 2024-04-14T23:53:15.488 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/192/12' -> 'copyofsnap1/test1/192/12' 2024-04-14T23:53:15.488 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/192/13' -> 'copyofsnap1/test1/192/13' 2024-04-14T23:53:15.488 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/192/14' -> 'copyofsnap1/test1/192/14' 2024-04-14T23:53:15.488 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/192/15' -> 'copyofsnap1/test1/192/15' 2024-04-14T23:53:15.488 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/192/16' -> 'copyofsnap1/test1/192/16' 2024-04-14T23:53:15.488 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/192/17' -> 'copyofsnap1/test1/192/17' 2024-04-14T23:53:15.488 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/192/18' -> 'copyofsnap1/test1/192/18' 2024-04-14T23:53:15.488 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/192/19' -> 'copyofsnap1/test1/192/19' 2024-04-14T23:53:15.488 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/192/20' -> 'copyofsnap1/test1/192/20' 2024-04-14T23:53:15.488 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/193' -> 'copyofsnap1/test1/193' 2024-04-14T23:53:15.488 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/193/10' -> 'copyofsnap1/test1/193/10' 2024-04-14T23:53:15.489 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/193/11' -> 'copyofsnap1/test1/193/11' 2024-04-14T23:53:15.489 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/193/12' -> 'copyofsnap1/test1/193/12' 2024-04-14T23:53:15.489 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/193/13' -> 'copyofsnap1/test1/193/13' 2024-04-14T23:53:15.489 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/193/14' -> 'copyofsnap1/test1/193/14' 2024-04-14T23:53:15.489 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/193/15' -> 'copyofsnap1/test1/193/15' 2024-04-14T23:53:15.489 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/193/16' -> 'copyofsnap1/test1/193/16' 2024-04-14T23:53:15.489 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/193/17' -> 'copyofsnap1/test1/193/17' 2024-04-14T23:53:15.489 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/193/18' -> 'copyofsnap1/test1/193/18' 2024-04-14T23:53:15.489 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/193/19' -> 'copyofsnap1/test1/193/19' 2024-04-14T23:53:15.489 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/193/20' -> 'copyofsnap1/test1/193/20' 2024-04-14T23:53:15.489 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/194' -> 'copyofsnap1/test1/194' 2024-04-14T23:53:15.489 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/194/10' -> 'copyofsnap1/test1/194/10' 2024-04-14T23:53:15.489 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/194/11' -> 'copyofsnap1/test1/194/11' 2024-04-14T23:53:15.489 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/194/12' -> 'copyofsnap1/test1/194/12' 2024-04-14T23:53:15.489 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/194/13' -> 'copyofsnap1/test1/194/13' 2024-04-14T23:53:15.489 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/194/14' -> 'copyofsnap1/test1/194/14' 2024-04-14T23:53:15.770 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/194/15' -> 'copyofsnap1/test1/194/15' 2024-04-14T23:53:15.770 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/194/16' -> 'copyofsnap1/test1/194/16' 2024-04-14T23:53:15.771 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/194/17' -> 'copyofsnap1/test1/194/17' 2024-04-14T23:53:15.771 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/194/18' -> 'copyofsnap1/test1/194/18' 2024-04-14T23:53:15.771 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/194/19' -> 'copyofsnap1/test1/194/19' 2024-04-14T23:53:15.771 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/194/20' -> 'copyofsnap1/test1/194/20' 2024-04-14T23:53:15.771 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/195' -> 'copyofsnap1/test1/195' 2024-04-14T23:53:15.771 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/195/10' -> 'copyofsnap1/test1/195/10' 2024-04-14T23:53:15.771 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/195/11' -> 'copyofsnap1/test1/195/11' 2024-04-14T23:53:15.771 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/195/12' -> 'copyofsnap1/test1/195/12' 2024-04-14T23:53:15.771 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/195/13' -> 'copyofsnap1/test1/195/13' 2024-04-14T23:53:15.771 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/195/14' -> 'copyofsnap1/test1/195/14' 2024-04-14T23:53:15.771 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/195/15' -> 'copyofsnap1/test1/195/15' 2024-04-14T23:53:15.772 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/195/16' -> 'copyofsnap1/test1/195/16' 2024-04-14T23:53:15.772 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/195/17' -> 'copyofsnap1/test1/195/17' 2024-04-14T23:53:15.772 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/195/18' -> 'copyofsnap1/test1/195/18' 2024-04-14T23:53:15.772 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/195/19' -> 'copyofsnap1/test1/195/19' 2024-04-14T23:53:15.772 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/195/20' -> 'copyofsnap1/test1/195/20' 2024-04-14T23:53:15.772 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/196' -> 'copyofsnap1/test1/196' 2024-04-14T23:53:15.772 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/196/10' -> 'copyofsnap1/test1/196/10' 2024-04-14T23:53:15.772 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/196/11' -> 'copyofsnap1/test1/196/11' 2024-04-14T23:53:15.772 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/196/12' -> 'copyofsnap1/test1/196/12' 2024-04-14T23:53:15.772 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/196/13' -> 'copyofsnap1/test1/196/13' 2024-04-14T23:53:15.772 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/196/14' -> 'copyofsnap1/test1/196/14' 2024-04-14T23:53:15.773 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/196/15' -> 'copyofsnap1/test1/196/15' 2024-04-14T23:53:15.773 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/196/16' -> 'copyofsnap1/test1/196/16' 2024-04-14T23:53:15.773 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/196/17' -> 'copyofsnap1/test1/196/17' 2024-04-14T23:53:15.773 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/196/18' -> 'copyofsnap1/test1/196/18' 2024-04-14T23:53:15.773 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/196/19' -> 'copyofsnap1/test1/196/19' 2024-04-14T23:53:15.773 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/196/20' -> 'copyofsnap1/test1/196/20' 2024-04-14T23:53:15.773 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/197' -> 'copyofsnap1/test1/197' 2024-04-14T23:53:15.773 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/197/10' -> 'copyofsnap1/test1/197/10' 2024-04-14T23:53:15.773 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/197/11' -> 'copyofsnap1/test1/197/11' 2024-04-14T23:53:15.773 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/197/12' -> 'copyofsnap1/test1/197/12' 2024-04-14T23:53:15.773 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/197/13' -> 'copyofsnap1/test1/197/13' 2024-04-14T23:53:15.773 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/197/14' -> 'copyofsnap1/test1/197/14' 2024-04-14T23:53:15.774 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/197/15' -> 'copyofsnap1/test1/197/15' 2024-04-14T23:53:15.774 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/197/16' -> 'copyofsnap1/test1/197/16' 2024-04-14T23:53:15.774 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/197/17' -> 'copyofsnap1/test1/197/17' 2024-04-14T23:53:15.774 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/197/18' -> 'copyofsnap1/test1/197/18' 2024-04-14T23:53:15.774 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/197/19' -> 'copyofsnap1/test1/197/19' 2024-04-14T23:53:15.774 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/197/20' -> 'copyofsnap1/test1/197/20' 2024-04-14T23:53:15.774 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/198' -> 'copyofsnap1/test1/198' 2024-04-14T23:53:15.774 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/198/10' -> 'copyofsnap1/test1/198/10' 2024-04-14T23:53:15.774 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/198/11' -> 'copyofsnap1/test1/198/11' 2024-04-14T23:53:15.774 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/198/12' -> 'copyofsnap1/test1/198/12' 2024-04-14T23:53:15.774 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/198/13' -> 'copyofsnap1/test1/198/13' 2024-04-14T23:53:15.774 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/198/14' -> 'copyofsnap1/test1/198/14' 2024-04-14T23:53:15.774 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/198/15' -> 'copyofsnap1/test1/198/15' 2024-04-14T23:53:15.775 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/198/16' -> 'copyofsnap1/test1/198/16' 2024-04-14T23:53:15.775 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/198/17' -> 'copyofsnap1/test1/198/17' 2024-04-14T23:53:15.775 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/198/18' -> 'copyofsnap1/test1/198/18' 2024-04-14T23:53:15.775 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/198/19' -> 'copyofsnap1/test1/198/19' 2024-04-14T23:53:15.775 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/198/20' -> 'copyofsnap1/test1/198/20' 2024-04-14T23:53:15.775 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/199' -> 'copyofsnap1/test1/199' 2024-04-14T23:53:15.775 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/199/10' -> 'copyofsnap1/test1/199/10' 2024-04-14T23:53:15.775 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/199/11' -> 'copyofsnap1/test1/199/11' 2024-04-14T23:53:15.776 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/199/12' -> 'copyofsnap1/test1/199/12' 2024-04-14T23:53:15.776 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/199/13' -> 'copyofsnap1/test1/199/13' 2024-04-14T23:53:15.776 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/199/14' -> 'copyofsnap1/test1/199/14' 2024-04-14T23:53:15.776 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/199/15' -> 'copyofsnap1/test1/199/15' 2024-04-14T23:53:15.776 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/199/16' -> 'copyofsnap1/test1/199/16' 2024-04-14T23:53:15.776 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/199/17' -> 'copyofsnap1/test1/199/17' 2024-04-14T23:53:15.776 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/199/18' -> 'copyofsnap1/test1/199/18' 2024-04-14T23:53:15.776 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/199/19' -> 'copyofsnap1/test1/199/19' 2024-04-14T23:53:15.776 INFO:tasks.workunit.client.0.smithi060.stdout:'.snap/test1/199/20' -> 'copyofsnap1/test1/199/20' 2024-04-14T23:53:15.776 INFO:tasks.workunit.client.0.smithi060.stdout:Take third snapshot .snap/test3 2024-04-14T23:53:16.021 INFO:tasks.workunit.client.0.smithi060.stdout:Delete the snapshots... 2024-04-14T23:53:18.602 INFO:tasks.workunit.client.0.smithi060.stdout:.//.snap/test1 2024-04-14T23:53:18.603 INFO:tasks.workunit.client.0.smithi060.stdout:.//.snap/test2 2024-04-14T23:53:18.603 INFO:tasks.workunit.client.0.smithi060.stdout:.//.snap/test3 2024-04-14T23:53:18.616 INFO:tasks.workunit.client.0.smithi060.stdout:./274/.snap/snap-subdir-test 2024-04-14T23:53:18.633 INFO:tasks.workunit.client.0.smithi060.stdout:./166/.snap/snap-subdir-test 2024-04-14T23:53:18.649 INFO:tasks.workunit.client.0.smithi060.stdout:./256/.snap/snap-subdir-test 2024-04-14T23:53:18.665 INFO:tasks.workunit.client.0.smithi060.stdout:./247/.snap/snap-subdir-test 2024-04-14T23:53:18.954 INFO:tasks.workunit.client.0.smithi060.stdout:./248/.snap/snap-subdir-test 2024-04-14T23:53:18.965 INFO:tasks.workunit.client.0.smithi060.stdout:./147/.snap/snap-subdir-test 2024-04-14T23:53:18.979 INFO:tasks.workunit.client.0.smithi060.stdout:./151/.snap/snap-subdir-test 2024-04-14T23:53:18.994 INFO:tasks.workunit.client.0.smithi060.stdout:./170/.snap/snap-subdir-test 2024-04-14T23:53:19.842 INFO:tasks.workunit.client.0.smithi060.stdout:./167/.snap/snap-subdir-test 2024-04-14T23:53:19.855 INFO:tasks.workunit.client.0.smithi060.stdout:./140/.snap/snap-subdir-test 2024-04-14T23:53:19.870 INFO:tasks.workunit.client.0.smithi060.stdout:./142/.snap/snap-subdir-test 2024-04-14T23:53:19.886 INFO:tasks.workunit.client.0.smithi060.stdout:./150/.snap/snap-subdir-test 2024-04-14T23:53:19.897 INFO:tasks.workunit.client.0.smithi060.stdout:./273/.snap/snap-subdir-test 2024-04-14T23:53:19.908 INFO:tasks.workunit.client.0.smithi060.stdout:./269/.snap/snap-subdir-test 2024-04-14T23:53:19.918 INFO:tasks.workunit.client.0.smithi060.stdout:./105/.snap/snap-subdir-test 2024-04-14T23:53:19.935 INFO:tasks.workunit.client.0.smithi060.stdout:./210/.snap/snap-subdir-test 2024-04-14T23:53:20.513 INFO:tasks.workunit.client.0.smithi060.stdout:./265/.snap/snap-subdir-test 2024-04-14T23:53:20.532 INFO:tasks.workunit.client.0.smithi060.stdout:./218/.snap/snap-subdir-test 2024-04-14T23:53:20.550 INFO:tasks.workunit.client.0.smithi060.stdout:./290/.snap/snap-subdir-test 2024-04-14T23:53:20.569 INFO:tasks.workunit.client.0.smithi060.stdout:./149/.snap/snap-subdir-test 2024-04-14T23:53:20.585 INFO:tasks.workunit.client.0.smithi060.stdout:./109/.snap/snap-subdir-test 2024-04-14T23:53:20.603 INFO:tasks.workunit.client.0.smithi060.stdout:./292/.snap/snap-subdir-test 2024-04-14T23:53:20.620 INFO:tasks.workunit.client.0.smithi060.stdout:./242/.snap/snap-subdir-test 2024-04-14T23:53:20.640 INFO:tasks.workunit.client.0.smithi060.stdout:./158/.snap/snap-subdir-test 2024-04-14T23:53:20.660 INFO:tasks.workunit.client.0.smithi060.stdout:./243/.snap/snap-subdir-test 2024-04-14T23:53:20.679 INFO:tasks.workunit.client.0.smithi060.stdout:./106/.snap/snap-subdir-test 2024-04-14T23:53:20.754 INFO:tasks.workunit.client.0.smithi060.stdout:./250/.snap/snap-subdir-test 2024-04-14T23:53:20.773 INFO:tasks.workunit.client.0.smithi060.stdout:./221/.snap/snap-subdir-test 2024-04-14T23:53:21.059 INFO:tasks.workunit.client.0.smithi060.stdout:./244/.snap/snap-subdir-test 2024-04-14T23:53:21.080 INFO:tasks.workunit.client.0.smithi060.stdout:./258/.snap/snap-subdir-test 2024-04-14T23:53:21.101 INFO:tasks.workunit.client.0.smithi060.stdout:./268/.snap/snap-subdir-test 2024-04-14T23:53:21.121 INFO:tasks.workunit.client.0.smithi060.stdout:./189/.snap/snap-subdir-test 2024-04-14T23:53:21.138 INFO:tasks.workunit.client.0.smithi060.stdout:./285/.snap/snap-subdir-test 2024-04-14T23:53:21.157 INFO:tasks.workunit.client.0.smithi060.stdout:./208/.snap/snap-subdir-test 2024-04-14T23:53:21.174 INFO:tasks.workunit.client.0.smithi060.stdout:./288/.snap/snap-subdir-test 2024-04-14T23:53:21.191 INFO:tasks.workunit.client.0.smithi060.stdout:./209/.snap/snap-subdir-test 2024-04-14T23:53:21.211 INFO:tasks.workunit.client.0.smithi060.stdout:./253/.snap/snap-subdir-test 2024-04-14T23:53:21.230 INFO:tasks.workunit.client.0.smithi060.stdout:./249/.snap/snap-subdir-test 2024-04-14T23:53:21.249 INFO:tasks.workunit.client.0.smithi060.stdout:./133/.snap/snap-subdir-test 2024-04-14T23:53:22.038 INFO:tasks.workunit.client.0.smithi060.stdout:./291/.snap/snap-subdir-test 2024-04-14T23:53:22.057 INFO:tasks.workunit.client.0.smithi060.stdout:./226/.snap/snap-subdir-test 2024-04-14T23:53:22.074 INFO:tasks.workunit.client.0.smithi060.stdout:./173/.snap/snap-subdir-test 2024-04-14T23:53:22.105 INFO:tasks.workunit.client.0.smithi060.stdout:./174/.snap/snap-subdir-test 2024-04-14T23:53:22.109 INFO:tasks.workunit.client.0.smithi060.stdout:./299/.snap/snap-subdir-test 2024-04-14T23:53:22.121 INFO:tasks.workunit.client.0.smithi060.stdout:./262/.snap/snap-subdir-test 2024-04-14T23:53:22.138 INFO:tasks.workunit.client.0.smithi060.stdout:./231/.snap/snap-subdir-test 2024-04-14T23:53:22.209 INFO:tasks.workunit.client.0.smithi060.stdout:./214/.snap/snap-subdir-test 2024-04-14T23:53:22.485 INFO:tasks.workunit.client.0.smithi060.stdout:./281/.snap/snap-subdir-test 2024-04-14T23:53:22.499 INFO:tasks.workunit.client.0.smithi060.stdout:./289/.snap/snap-subdir-test 2024-04-14T23:53:22.511 INFO:tasks.workunit.client.0.smithi060.stdout:./283/.snap/snap-subdir-test 2024-04-14T23:53:22.526 INFO:tasks.workunit.client.0.smithi060.stdout:./232/.snap/snap-subdir-test 2024-04-14T23:53:22.554 INFO:tasks.workunit.client.0.smithi060.stdout:./298/.snap/snap-subdir-test 2024-04-14T23:53:22.559 INFO:tasks.workunit.client.0.smithi060.stdout:./233/.snap/snap-subdir-test 2024-04-14T23:53:22.821 INFO:tasks.workunit.client.0.smithi060.stdout:./155/.snap/snap-subdir-test 2024-04-14T23:53:23.096 INFO:tasks.workunit.client.0.smithi060.stdout:./114/.snap/snap-subdir-test 2024-04-14T23:53:23.117 INFO:tasks.workunit.client.0.smithi060.stdout:./159/.snap/snap-subdir-test 2024-04-14T23:53:23.135 INFO:tasks.workunit.client.0.smithi060.stdout:./187/.snap/snap-subdir-test 2024-04-14T23:53:23.154 INFO:tasks.workunit.client.0.smithi060.stdout:./185/.snap/snap-subdir-test 2024-04-14T23:53:23.167 INFO:tasks.workunit.client.0.smithi060.stdout:./293/.snap/snap-subdir-test 2024-04-14T23:53:23.180 INFO:tasks.workunit.client.0.smithi060.stdout:./228/.snap/snap-subdir-test 2024-04-14T23:53:23.194 INFO:tasks.workunit.client.0.smithi060.stdout:./120/.snap/snap-subdir-test 2024-04-14T23:53:23.213 INFO:tasks.workunit.client.0.smithi060.stdout:./110/.snap/snap-subdir-test 2024-04-14T23:53:23.233 INFO:tasks.workunit.client.0.smithi060.stdout:./267/.snap/snap-subdir-test 2024-04-14T23:53:23.250 INFO:tasks.workunit.client.0.smithi060.stdout:./202/.snap/snap-subdir-test 2024-04-14T23:53:23.263 INFO:tasks.workunit.client.0.smithi060.stdout:./137/.snap/snap-subdir-test 2024-04-14T23:53:23.279 INFO:tasks.workunit.client.0.smithi060.stdout:./246/.snap/snap-subdir-test 2024-04-14T23:53:23.294 INFO:tasks.workunit.client.0.smithi060.stdout:./219/.snap/snap-subdir-test 2024-04-14T23:53:23.307 INFO:tasks.workunit.client.0.smithi060.stdout:./198/.snap/snap-subdir-test 2024-04-14T23:53:23.321 INFO:tasks.workunit.client.0.smithi060.stdout:./136/.snap/snap-subdir-test 2024-04-14T23:53:23.335 INFO:tasks.workunit.client.0.smithi060.stdout:./270/.snap/snap-subdir-test 2024-04-14T23:53:23.347 INFO:tasks.workunit.client.0.smithi060.stdout:./125/.snap/snap-subdir-test 2024-04-14T23:53:23.687 INFO:tasks.workunit.client.0.smithi060.stdout:./194/.snap/snap-subdir-test 2024-04-14T23:53:23.701 INFO:tasks.workunit.client.0.smithi060.stdout:./119/.snap/snap-subdir-test 2024-04-14T23:53:23.714 INFO:tasks.workunit.client.0.smithi060.stdout:./297/.snap/snap-subdir-test 2024-04-14T23:53:23.728 INFO:tasks.workunit.client.0.smithi060.stdout:./230/.snap/snap-subdir-test 2024-04-14T23:53:23.743 INFO:tasks.workunit.client.0.smithi060.stdout:./260/.snap/snap-subdir-test 2024-04-14T23:53:23.758 INFO:tasks.workunit.client.0.smithi060.stdout:./157/.snap/snap-subdir-test 2024-04-14T23:53:23.774 INFO:tasks.workunit.client.0.smithi060.stdout:./172/.snap/snap-subdir-test 2024-04-14T23:53:23.786 INFO:tasks.workunit.client.0.smithi060.stdout:./277/.snap/snap-subdir-test 2024-04-14T23:53:23.799 INFO:tasks.workunit.client.0.smithi060.stdout:./186/.snap/snap-subdir-test 2024-04-14T23:53:23.812 INFO:tasks.workunit.client.0.smithi060.stdout:./237/.snap/snap-subdir-test 2024-04-14T23:53:23.825 INFO:tasks.workunit.client.0.smithi060.stdout:./190/.snap/snap-subdir-test 2024-04-14T23:53:23.837 INFO:tasks.workunit.client.0.smithi060.stdout:./213/.snap/snap-subdir-test 2024-04-14T23:53:23.853 INFO:tasks.workunit.client.0.smithi060.stdout:./104/.snap/snap-subdir-test 2024-04-14T23:53:23.869 INFO:tasks.workunit.client.0.smithi060.stdout:./164/.snap/snap-subdir-test 2024-04-14T23:53:23.884 INFO:tasks.workunit.client.0.smithi060.stdout:./112/.snap/snap-subdir-test 2024-04-14T23:53:25.288 INFO:tasks.workunit.client.0.smithi060.stdout:./295/.snap/snap-subdir-test 2024-04-14T23:53:25.302 INFO:tasks.workunit.client.0.smithi060.stdout:./272/.snap/snap-subdir-test 2024-04-14T23:53:25.318 INFO:tasks.workunit.client.0.smithi060.stdout:./100/.snap/snap-subdir-test 2024-04-14T23:53:25.333 INFO:tasks.workunit.client.0.smithi060.stdout:./134/.snap/snap-subdir-test 2024-04-14T23:53:25.346 INFO:tasks.workunit.client.0.smithi060.stdout:./200/.snap/snap-subdir-test 2024-04-14T23:53:25.359 INFO:tasks.workunit.client.0.smithi060.stdout:./115/.snap/snap-subdir-test 2024-04-14T23:53:25.371 INFO:tasks.workunit.client.0.smithi060.stdout:./284/.snap/snap-subdir-test 2024-04-14T23:53:25.384 INFO:tasks.workunit.client.0.smithi060.stdout:./280/.snap/snap-subdir-test 2024-04-14T23:53:25.399 INFO:tasks.workunit.client.0.smithi060.stdout:./116/.snap/snap-subdir-test 2024-04-14T23:53:25.415 INFO:tasks.workunit.client.0.smithi060.stdout:./245/.snap/snap-subdir-test 2024-04-14T23:53:25.430 INFO:tasks.workunit.client.0.smithi060.stdout:./195/.snap/snap-subdir-test 2024-04-14T23:53:25.444 INFO:tasks.workunit.client.0.smithi060.stdout:./138/.snap/snap-subdir-test 2024-04-14T23:53:25.459 INFO:tasks.workunit.client.0.smithi060.stdout:./135/.snap/snap-subdir-test 2024-04-14T23:53:25.476 INFO:tasks.workunit.client.0.smithi060.stdout:./117/.snap/snap-subdir-test 2024-04-14T23:53:25.491 INFO:tasks.workunit.client.0.smithi060.stdout:./241/.snap/snap-subdir-test 2024-04-14T23:53:25.506 INFO:tasks.workunit.client.0.smithi060.stdout:./143/.snap/snap-subdir-test 2024-04-14T23:53:25.522 INFO:tasks.workunit.client.0.smithi060.stdout:./180/.snap/snap-subdir-test 2024-04-14T23:53:25.841 INFO:tasks.workunit.client.0.smithi060.stdout:./126/.snap/snap-subdir-test 2024-04-14T23:53:25.857 INFO:tasks.workunit.client.0.smithi060.stdout:./161/.snap/snap-subdir-test 2024-04-14T23:53:27.650 INFO:tasks.workunit.client.0.smithi060.stdout:./204/.snap/snap-subdir-test 2024-04-14T23:53:27.667 INFO:tasks.workunit.client.0.smithi060.stdout:./176/.snap/snap-subdir-test 2024-04-14T23:53:27.685 INFO:tasks.workunit.client.0.smithi060.stdout:./255/.snap/snap-subdir-test 2024-04-14T23:53:27.899 INFO:tasks.workunit.client.0.smithi060.stdout:./215/.snap/snap-subdir-test 2024-04-14T23:53:27.910 INFO:tasks.workunit.client.0.smithi060.stdout:./111/.snap/snap-subdir-test 2024-04-14T23:53:27.923 INFO:tasks.workunit.client.0.smithi060.stdout:./238/.snap/snap-subdir-test 2024-04-14T23:53:27.933 INFO:tasks.workunit.client.0.smithi060.stdout:./191/.snap/snap-subdir-test 2024-04-14T23:53:27.945 INFO:tasks.workunit.client.0.smithi060.stdout:./184/.snap/snap-subdir-test 2024-04-14T23:53:27.961 INFO:tasks.workunit.client.0.smithi060.stdout:./192/.snap/snap-subdir-test 2024-04-14T23:53:27.979 INFO:tasks.workunit.client.0.smithi060.stdout:./103/.snap/snap-subdir-test 2024-04-14T23:53:27.999 INFO:tasks.workunit.client.0.smithi060.stdout:./128/.snap/snap-subdir-test 2024-04-14T23:53:28.018 INFO:tasks.workunit.client.0.smithi060.stdout:./229/.snap/snap-subdir-test 2024-04-14T23:53:28.040 INFO:tasks.workunit.client.0.smithi060.stdout:./144/.snap/snap-subdir-test 2024-04-14T23:53:28.060 INFO:tasks.workunit.client.0.smithi060.stdout:./146/.snap/snap-subdir-test 2024-04-14T23:53:28.079 INFO:tasks.workunit.client.0.smithi060.stdout:./227/.snap/snap-subdir-test 2024-04-14T23:53:28.096 INFO:tasks.workunit.client.0.smithi060.stdout:./278/.snap/snap-subdir-test 2024-04-14T23:53:28.116 INFO:tasks.workunit.client.0.smithi060.stdout:./141/.snap/snap-subdir-test 2024-04-14T23:53:28.133 INFO:tasks.workunit.client.0.smithi060.stdout:./234/.snap/snap-subdir-test 2024-04-14T23:53:28.152 INFO:tasks.workunit.client.0.smithi060.stdout:./225/.snap/snap-subdir-test 2024-04-14T23:53:28.175 INFO:tasks.workunit.client.0.smithi060.stdout:./182/.snap/snap-subdir-test 2024-04-14T23:53:28.189 INFO:tasks.workunit.client.0.smithi060.stdout:./107/.snap/snap-subdir-test 2024-04-14T23:53:28.207 INFO:tasks.workunit.client.0.smithi060.stdout:./178/.snap/snap-subdir-test 2024-04-14T23:53:28.228 INFO:tasks.workunit.client.0.smithi060.stdout:./266/.snap/snap-subdir-test 2024-04-14T23:53:28.247 INFO:tasks.workunit.client.0.smithi060.stdout:./240/.snap/snap-subdir-test 2024-04-14T23:53:28.262 INFO:tasks.workunit.client.0.smithi060.stdout:./156/.snap/snap-subdir-test 2024-04-14T23:53:28.276 INFO:tasks.workunit.client.0.smithi060.stdout:./131/.snap/snap-subdir-test 2024-04-14T23:53:28.292 INFO:tasks.workunit.client.0.smithi060.stdout:./123/.snap/snap-subdir-test 2024-04-14T23:53:28.307 INFO:tasks.workunit.client.0.smithi060.stdout:./160/.snap/snap-subdir-test 2024-04-14T23:53:28.321 INFO:tasks.workunit.client.0.smithi060.stdout:./165/.snap/snap-subdir-test 2024-04-14T23:53:28.335 INFO:tasks.workunit.client.0.smithi060.stdout:./101/.snap/snap-subdir-test 2024-04-14T23:53:28.351 INFO:tasks.workunit.client.0.smithi060.stdout:./148/.snap/snap-subdir-test 2024-04-14T23:53:28.365 INFO:tasks.workunit.client.0.smithi060.stdout:./168/.snap/snap-subdir-test 2024-04-14T23:53:28.379 INFO:tasks.workunit.client.0.smithi060.stdout:./179/.snap/snap-subdir-test 2024-04-14T23:53:28.392 INFO:tasks.workunit.client.0.smithi060.stdout:./257/.snap/snap-subdir-test 2024-04-14T23:53:28.406 INFO:tasks.workunit.client.0.smithi060.stdout:./252/.snap/snap-subdir-test 2024-04-14T23:53:28.422 INFO:tasks.workunit.client.0.smithi060.stdout:./132/.snap/snap-subdir-test 2024-04-14T23:53:28.438 INFO:tasks.workunit.client.0.smithi060.stdout:./236/.snap/snap-subdir-test 2024-04-14T23:53:28.451 INFO:tasks.workunit.client.0.smithi060.stdout:./211/.snap/snap-subdir-test 2024-04-14T23:53:28.466 INFO:tasks.workunit.client.0.smithi060.stdout:./199/.snap/snap-subdir-test 2024-04-14T23:53:28.480 INFO:tasks.workunit.client.0.smithi060.stdout:./205/.snap/snap-subdir-test 2024-04-14T23:53:28.494 INFO:tasks.workunit.client.0.smithi060.stdout:./163/.snap/snap-subdir-test 2024-04-14T23:53:28.506 INFO:tasks.workunit.client.0.smithi060.stdout:./261/.snap/snap-subdir-test 2024-04-14T23:53:28.522 INFO:tasks.workunit.client.0.smithi060.stdout:./201/.snap/snap-subdir-test 2024-04-14T23:53:28.535 INFO:tasks.workunit.client.0.smithi060.stdout:./282/.snap/snap-subdir-test 2024-04-14T23:53:28.546 INFO:tasks.workunit.client.0.smithi060.stdout:./296/.snap/snap-subdir-test 2024-04-14T23:53:28.560 INFO:tasks.workunit.client.0.smithi060.stdout:./217/.snap/snap-subdir-test 2024-04-14T23:53:28.576 INFO:tasks.workunit.client.0.smithi060.stdout:./153/.snap/snap-subdir-test 2024-04-14T23:53:28.589 INFO:tasks.workunit.client.0.smithi060.stdout:./206/.snap/snap-subdir-test 2024-04-14T23:53:28.601 INFO:tasks.workunit.client.0.smithi060.stdout:./122/.snap/snap-subdir-test 2024-04-14T23:53:28.616 INFO:tasks.workunit.client.0.smithi060.stdout:./162/.snap/snap-subdir-test 2024-04-14T23:53:28.628 INFO:tasks.workunit.client.0.smithi060.stdout:./197/.snap/snap-subdir-test 2024-04-14T23:53:28.642 INFO:tasks.workunit.client.0.smithi060.stdout:./216/.snap/snap-subdir-test 2024-04-14T23:53:30.478 INFO:tasks.workunit.client.0.smithi060.stdout:./254/.snap/snap-subdir-test 2024-04-14T23:53:30.491 INFO:tasks.workunit.client.0.smithi060.stdout:./113/.snap/snap-subdir-test 2024-04-14T23:53:30.776 INFO:tasks.workunit.client.0.smithi060.stdout:./152/.snap/snap-subdir-test 2024-04-14T23:53:30.794 INFO:tasks.workunit.client.0.smithi060.stdout:./121/.snap/snap-subdir-test 2024-04-14T23:53:30.810 INFO:tasks.workunit.client.0.smithi060.stdout:./118/.snap/snap-subdir-test 2024-04-14T23:53:30.827 INFO:tasks.workunit.client.0.smithi060.stdout:./196/.snap/snap-subdir-test 2024-04-14T23:53:30.843 INFO:tasks.workunit.client.0.smithi060.stdout:./124/.snap/snap-subdir-test 2024-04-14T23:53:30.859 INFO:tasks.workunit.client.0.smithi060.stdout:./239/.snap/snap-subdir-test 2024-04-14T23:53:30.874 INFO:tasks.workunit.client.0.smithi060.stdout:./188/.snap/snap-subdir-test 2024-04-14T23:53:30.889 INFO:tasks.workunit.client.0.smithi060.stdout:./287/.snap/snap-subdir-test 2024-04-14T23:53:30.902 INFO:tasks.workunit.client.0.smithi060.stdout:./275/.snap/snap-subdir-test 2024-04-14T23:53:30.917 INFO:tasks.workunit.client.0.smithi060.stdout:./259/.snap/snap-subdir-test 2024-04-14T23:53:30.933 INFO:tasks.workunit.client.0.smithi060.stdout:./224/.snap/snap-subdir-test 2024-04-14T23:53:30.950 INFO:tasks.workunit.client.0.smithi060.stdout:./264/.snap/snap-subdir-test 2024-04-14T23:53:31.073 INFO:tasks.workunit.client.0.smithi060.stdout:./145/.snap/snap-subdir-test 2024-04-14T23:53:31.087 INFO:tasks.workunit.client.0.smithi060.stdout:./279/.snap/snap-subdir-test 2024-04-14T23:53:31.102 INFO:tasks.workunit.client.0.smithi060.stdout:./263/.snap/snap-subdir-test 2024-04-14T23:53:31.113 INFO:tasks.workunit.client.0.smithi060.stdout:./130/.snap/snap-subdir-test 2024-04-14T23:53:31.124 INFO:tasks.workunit.client.0.smithi060.stdout:./193/.snap/snap-subdir-test 2024-04-14T23:53:31.137 INFO:tasks.workunit.client.0.smithi060.stdout:./223/.snap/snap-subdir-test 2024-04-14T23:53:31.151 INFO:tasks.workunit.client.0.smithi060.stdout:./207/.snap/snap-subdir-test 2024-04-14T23:53:31.160 INFO:tasks.workunit.client.0.smithi060.stdout:./276/.snap/snap-subdir-test 2024-04-14T23:53:31.178 INFO:tasks.workunit.client.0.smithi060.stdout:./271/.snap/snap-subdir-test 2024-04-14T23:53:31.197 INFO:tasks.workunit.client.0.smithi060.stdout:./251/.snap/snap-subdir-test 2024-04-14T23:53:31.216 INFO:tasks.workunit.client.0.smithi060.stdout:./175/.snap/snap-subdir-test 2024-04-14T23:53:31.236 INFO:tasks.workunit.client.0.smithi060.stdout:./222/.snap/snap-subdir-test 2024-04-14T23:53:31.251 INFO:tasks.workunit.client.0.smithi060.stdout:./220/.snap/snap-subdir-test 2024-04-14T23:53:31.263 INFO:tasks.workunit.client.0.smithi060.stdout:./203/.snap/snap-subdir-test 2024-04-14T23:53:31.278 INFO:tasks.workunit.client.0.smithi060.stdout:./181/.snap/snap-subdir-test 2024-04-14T23:53:31.293 INFO:tasks.workunit.client.0.smithi060.stdout:./286/.snap/snap-subdir-test 2024-04-14T23:53:31.313 INFO:tasks.workunit.client.0.smithi060.stdout:./127/.snap/snap-subdir-test 2024-04-14T23:53:31.332 INFO:tasks.workunit.client.0.smithi060.stdout:./108/.snap/snap-subdir-test 2024-04-14T23:53:31.351 INFO:tasks.workunit.client.0.smithi060.stdout:./183/.snap/snap-subdir-test 2024-04-14T23:53:31.369 INFO:tasks.workunit.client.0.smithi060.stdout:./294/.snap/snap-subdir-test 2024-04-14T23:53:31.388 INFO:tasks.workunit.client.0.smithi060.stdout:./235/.snap/snap-subdir-test 2024-04-14T23:53:31.408 INFO:tasks.workunit.client.0.smithi060.stdout:./177/.snap/snap-subdir-test 2024-04-14T23:53:31.428 INFO:tasks.workunit.client.0.smithi060.stdout:./169/.snap/snap-subdir-test 2024-04-14T23:53:31.448 INFO:tasks.workunit.client.0.smithi060.stdout:./212/.snap/snap-subdir-test 2024-04-14T23:53:31.467 INFO:tasks.workunit.client.0.smithi060.stdout:./154/.snap/snap-subdir-test 2024-04-14T23:53:31.487 INFO:tasks.workunit.client.0.smithi060.stdout:./171/.snap/snap-subdir-test 2024-04-14T23:53:31.507 INFO:tasks.workunit.client.0.smithi060.stdout:./139/.snap/snap-subdir-test 2024-04-14T23:53:31.527 INFO:tasks.workunit.client.0.smithi060.stdout:./102/.snap/snap-subdir-test 2024-04-14T23:53:31.547 INFO:tasks.workunit.client.0.smithi060.stdout:./129/.snap/snap-subdir-test 2024-04-14T23:53:37.732 INFO:tasks.workunit.client.0.smithi060.stdout:Delete all the files and directories ... 2024-04-14T23:53:39.394 INFO:tasks.workunit.client.0.smithi060.stdout:removed './100/19' 2024-04-14T23:53:39.394 INFO:tasks.workunit.client.0.smithi060.stdout:removed './100/15' 2024-04-14T23:53:39.394 INFO:tasks.workunit.client.0.smithi060.stdout:removed './100/31' 2024-04-14T23:53:39.394 INFO:tasks.workunit.client.0.smithi060.stdout:removed './100/36' 2024-04-14T23:53:39.394 INFO:tasks.workunit.client.0.smithi060.stdout:removed './100/20' 2024-04-14T23:53:39.394 INFO:tasks.workunit.client.0.smithi060.stdout:removed './100/12' 2024-04-14T23:53:39.394 INFO:tasks.workunit.client.0.smithi060.stdout:removed './100/39' 2024-04-14T23:53:39.394 INFO:tasks.workunit.client.0.smithi060.stdout:removed './100/10' 2024-04-14T23:53:39.394 INFO:tasks.workunit.client.0.smithi060.stdout:removed './100/34' 2024-04-14T23:53:39.394 INFO:tasks.workunit.client.0.smithi060.stdout:removed './100/38' 2024-04-14T23:53:39.394 INFO:tasks.workunit.client.0.smithi060.stdout:removed './100/11' 2024-04-14T23:53:39.395 INFO:tasks.workunit.client.0.smithi060.stdout:removed './100/16' 2024-04-14T23:53:39.395 INFO:tasks.workunit.client.0.smithi060.stdout:removed './100/32' 2024-04-14T23:53:39.395 INFO:tasks.workunit.client.0.smithi060.stdout:removed './100/30' 2024-04-14T23:53:39.395 INFO:tasks.workunit.client.0.smithi060.stdout:removed './100/14' 2024-04-14T23:53:39.395 INFO:tasks.workunit.client.0.smithi060.stdout:removed './100/18' 2024-04-14T23:53:39.395 INFO:tasks.workunit.client.0.smithi060.stdout:removed './100/13' 2024-04-14T23:53:39.395 INFO:tasks.workunit.client.0.smithi060.stdout:removed './100/33' 2024-04-14T23:53:39.395 INFO:tasks.workunit.client.0.smithi060.stdout:removed './100/17' 2024-04-14T23:53:39.395 INFO:tasks.workunit.client.0.smithi060.stdout:removed './100/35' 2024-04-14T23:53:39.395 INFO:tasks.workunit.client.0.smithi060.stdout:removed './100/37' 2024-04-14T23:53:39.395 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './100' 2024-04-14T23:53:39.395 INFO:tasks.workunit.client.0.smithi060.stdout:removed './101/19' 2024-04-14T23:53:39.395 INFO:tasks.workunit.client.0.smithi060.stdout:removed './101/15' 2024-04-14T23:53:39.395 INFO:tasks.workunit.client.0.smithi060.stdout:removed './101/31' 2024-04-14T23:53:39.395 INFO:tasks.workunit.client.0.smithi060.stdout:removed './101/36' 2024-04-14T23:53:39.396 INFO:tasks.workunit.client.0.smithi060.stdout:removed './101/20' 2024-04-14T23:53:39.396 INFO:tasks.workunit.client.0.smithi060.stdout:removed './101/12' 2024-04-14T23:53:39.396 INFO:tasks.workunit.client.0.smithi060.stdout:removed './101/39' 2024-04-14T23:53:39.396 INFO:tasks.workunit.client.0.smithi060.stdout:removed './101/10' 2024-04-14T23:53:39.396 INFO:tasks.workunit.client.0.smithi060.stdout:removed './101/34' 2024-04-14T23:53:39.396 INFO:tasks.workunit.client.0.smithi060.stdout:removed './101/38' 2024-04-14T23:53:39.396 INFO:tasks.workunit.client.0.smithi060.stdout:removed './101/11' 2024-04-14T23:53:39.396 INFO:tasks.workunit.client.0.smithi060.stdout:removed './101/16' 2024-04-14T23:53:39.396 INFO:tasks.workunit.client.0.smithi060.stdout:removed './101/32' 2024-04-14T23:53:39.396 INFO:tasks.workunit.client.0.smithi060.stdout:removed './101/30' 2024-04-14T23:53:39.396 INFO:tasks.workunit.client.0.smithi060.stdout:removed './101/14' 2024-04-14T23:53:39.396 INFO:tasks.workunit.client.0.smithi060.stdout:removed './101/18' 2024-04-14T23:53:39.396 INFO:tasks.workunit.client.0.smithi060.stdout:removed './101/13' 2024-04-14T23:53:39.397 INFO:tasks.workunit.client.0.smithi060.stdout:removed './101/33' 2024-04-14T23:53:39.397 INFO:tasks.workunit.client.0.smithi060.stdout:removed './101/17' 2024-04-14T23:53:39.397 INFO:tasks.workunit.client.0.smithi060.stdout:removed './101/35' 2024-04-14T23:53:39.397 INFO:tasks.workunit.client.0.smithi060.stdout:removed './101/37' 2024-04-14T23:53:39.397 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './101' 2024-04-14T23:53:39.397 INFO:tasks.workunit.client.0.smithi060.stdout:removed './102/19' 2024-04-14T23:53:39.397 INFO:tasks.workunit.client.0.smithi060.stdout:removed './102/15' 2024-04-14T23:53:39.397 INFO:tasks.workunit.client.0.smithi060.stdout:removed './102/31' 2024-04-14T23:53:39.397 INFO:tasks.workunit.client.0.smithi060.stdout:removed './102/36' 2024-04-14T23:53:39.397 INFO:tasks.workunit.client.0.smithi060.stdout:removed './102/20' 2024-04-14T23:53:39.397 INFO:tasks.workunit.client.0.smithi060.stdout:removed './102/12' 2024-04-14T23:53:39.397 INFO:tasks.workunit.client.0.smithi060.stdout:removed './102/39' 2024-04-14T23:53:39.397 INFO:tasks.workunit.client.0.smithi060.stdout:removed './102/10' 2024-04-14T23:53:39.397 INFO:tasks.workunit.client.0.smithi060.stdout:removed './102/34' 2024-04-14T23:53:39.397 INFO:tasks.workunit.client.0.smithi060.stdout:removed './102/38' 2024-04-14T23:53:39.397 INFO:tasks.workunit.client.0.smithi060.stdout:removed './102/11' 2024-04-14T23:53:39.398 INFO:tasks.workunit.client.0.smithi060.stdout:removed './102/16' 2024-04-14T23:53:39.398 INFO:tasks.workunit.client.0.smithi060.stdout:removed './102/32' 2024-04-14T23:53:39.398 INFO:tasks.workunit.client.0.smithi060.stdout:removed './102/30' 2024-04-14T23:53:39.398 INFO:tasks.workunit.client.0.smithi060.stdout:removed './102/14' 2024-04-14T23:53:39.398 INFO:tasks.workunit.client.0.smithi060.stdout:removed './102/18' 2024-04-14T23:53:39.398 INFO:tasks.workunit.client.0.smithi060.stdout:removed './102/13' 2024-04-14T23:53:39.398 INFO:tasks.workunit.client.0.smithi060.stdout:removed './102/33' 2024-04-14T23:53:39.398 INFO:tasks.workunit.client.0.smithi060.stdout:removed './102/17' 2024-04-14T23:53:39.398 INFO:tasks.workunit.client.0.smithi060.stdout:removed './102/35' 2024-04-14T23:53:39.398 INFO:tasks.workunit.client.0.smithi060.stdout:removed './102/37' 2024-04-14T23:53:39.398 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './102' 2024-04-14T23:53:39.398 INFO:tasks.workunit.client.0.smithi060.stdout:removed './103/19' 2024-04-14T23:53:39.398 INFO:tasks.workunit.client.0.smithi060.stdout:removed './103/15' 2024-04-14T23:53:39.398 INFO:tasks.workunit.client.0.smithi060.stdout:removed './103/31' 2024-04-14T23:53:39.399 INFO:tasks.workunit.client.0.smithi060.stdout:removed './103/36' 2024-04-14T23:53:39.399 INFO:tasks.workunit.client.0.smithi060.stdout:removed './103/20' 2024-04-14T23:53:39.399 INFO:tasks.workunit.client.0.smithi060.stdout:removed './103/12' 2024-04-14T23:53:39.399 INFO:tasks.workunit.client.0.smithi060.stdout:removed './103/39' 2024-04-14T23:53:39.399 INFO:tasks.workunit.client.0.smithi060.stdout:removed './103/10' 2024-04-14T23:53:39.399 INFO:tasks.workunit.client.0.smithi060.stdout:removed './103/34' 2024-04-14T23:53:39.399 INFO:tasks.workunit.client.0.smithi060.stdout:removed './103/38' 2024-04-14T23:53:39.399 INFO:tasks.workunit.client.0.smithi060.stdout:removed './103/11' 2024-04-14T23:53:39.399 INFO:tasks.workunit.client.0.smithi060.stdout:removed './103/16' 2024-04-14T23:53:39.399 INFO:tasks.workunit.client.0.smithi060.stdout:removed './103/32' 2024-04-14T23:53:39.399 INFO:tasks.workunit.client.0.smithi060.stdout:removed './103/30' 2024-04-14T23:53:39.399 INFO:tasks.workunit.client.0.smithi060.stdout:removed './103/14' 2024-04-14T23:53:39.399 INFO:tasks.workunit.client.0.smithi060.stdout:removed './103/18' 2024-04-14T23:53:39.399 INFO:tasks.workunit.client.0.smithi060.stdout:removed './103/13' 2024-04-14T23:53:39.399 INFO:tasks.workunit.client.0.smithi060.stdout:removed './103/33' 2024-04-14T23:53:39.400 INFO:tasks.workunit.client.0.smithi060.stdout:removed './103/17' 2024-04-14T23:53:39.400 INFO:tasks.workunit.client.0.smithi060.stdout:removed './103/35' 2024-04-14T23:53:39.400 INFO:tasks.workunit.client.0.smithi060.stdout:removed './103/37' 2024-04-14T23:53:39.400 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './103' 2024-04-14T23:53:39.400 INFO:tasks.workunit.client.0.smithi060.stdout:removed './104/19' 2024-04-14T23:53:39.400 INFO:tasks.workunit.client.0.smithi060.stdout:removed './104/15' 2024-04-14T23:53:39.400 INFO:tasks.workunit.client.0.smithi060.stdout:removed './104/31' 2024-04-14T23:53:39.400 INFO:tasks.workunit.client.0.smithi060.stdout:removed './104/36' 2024-04-14T23:53:39.400 INFO:tasks.workunit.client.0.smithi060.stdout:removed './104/20' 2024-04-14T23:53:39.400 INFO:tasks.workunit.client.0.smithi060.stdout:removed './104/12' 2024-04-14T23:53:39.400 INFO:tasks.workunit.client.0.smithi060.stdout:removed './104/39' 2024-04-14T23:53:39.400 INFO:tasks.workunit.client.0.smithi060.stdout:removed './104/10' 2024-04-14T23:53:39.400 INFO:tasks.workunit.client.0.smithi060.stdout:removed './104/34' 2024-04-14T23:53:39.400 INFO:tasks.workunit.client.0.smithi060.stdout:removed './104/38' 2024-04-14T23:53:39.400 INFO:tasks.workunit.client.0.smithi060.stdout:removed './104/11' 2024-04-14T23:53:39.401 INFO:tasks.workunit.client.0.smithi060.stdout:removed './104/16' 2024-04-14T23:53:39.401 INFO:tasks.workunit.client.0.smithi060.stdout:removed './104/32' 2024-04-14T23:53:39.401 INFO:tasks.workunit.client.0.smithi060.stdout:removed './104/30' 2024-04-14T23:53:39.401 INFO:tasks.workunit.client.0.smithi060.stdout:removed './104/14' 2024-04-14T23:53:39.401 INFO:tasks.workunit.client.0.smithi060.stdout:removed './104/18' 2024-04-14T23:53:39.401 INFO:tasks.workunit.client.0.smithi060.stdout:removed './104/13' 2024-04-14T23:53:39.401 INFO:tasks.workunit.client.0.smithi060.stdout:removed './104/33' 2024-04-14T23:53:39.401 INFO:tasks.workunit.client.0.smithi060.stdout:removed './104/17' 2024-04-14T23:53:39.401 INFO:tasks.workunit.client.0.smithi060.stdout:removed './104/35' 2024-04-14T23:53:39.401 INFO:tasks.workunit.client.0.smithi060.stdout:removed './104/37' 2024-04-14T23:53:39.401 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './104' 2024-04-14T23:53:39.401 INFO:tasks.workunit.client.0.smithi060.stdout:removed './105/19' 2024-04-14T23:53:39.401 INFO:tasks.workunit.client.0.smithi060.stdout:removed './105/15' 2024-04-14T23:53:39.401 INFO:tasks.workunit.client.0.smithi060.stdout:removed './105/31' 2024-04-14T23:53:39.401 INFO:tasks.workunit.client.0.smithi060.stdout:removed './105/36' 2024-04-14T23:53:39.402 INFO:tasks.workunit.client.0.smithi060.stdout:removed './105/20' 2024-04-14T23:53:39.402 INFO:tasks.workunit.client.0.smithi060.stdout:removed './105/12' 2024-04-14T23:53:39.402 INFO:tasks.workunit.client.0.smithi060.stdout:removed './105/39' 2024-04-14T23:53:39.402 INFO:tasks.workunit.client.0.smithi060.stdout:removed './105/10' 2024-04-14T23:53:39.402 INFO:tasks.workunit.client.0.smithi060.stdout:removed './105/34' 2024-04-14T23:53:39.402 INFO:tasks.workunit.client.0.smithi060.stdout:removed './105/38' 2024-04-14T23:53:39.402 INFO:tasks.workunit.client.0.smithi060.stdout:removed './105/11' 2024-04-14T23:53:39.402 INFO:tasks.workunit.client.0.smithi060.stdout:removed './105/16' 2024-04-14T23:53:39.402 INFO:tasks.workunit.client.0.smithi060.stdout:removed './105/32' 2024-04-14T23:53:39.402 INFO:tasks.workunit.client.0.smithi060.stdout:removed './105/30' 2024-04-14T23:53:39.402 INFO:tasks.workunit.client.0.smithi060.stdout:removed './105/14' 2024-04-14T23:53:39.402 INFO:tasks.workunit.client.0.smithi060.stdout:removed './105/18' 2024-04-14T23:53:39.402 INFO:tasks.workunit.client.0.smithi060.stdout:removed './105/13' 2024-04-14T23:53:39.402 INFO:tasks.workunit.client.0.smithi060.stdout:removed './105/33' 2024-04-14T23:53:39.403 INFO:tasks.workunit.client.0.smithi060.stdout:removed './105/17' 2024-04-14T23:53:39.403 INFO:tasks.workunit.client.0.smithi060.stdout:removed './105/35' 2024-04-14T23:53:39.403 INFO:tasks.workunit.client.0.smithi060.stdout:removed './105/37' 2024-04-14T23:53:39.403 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './105' 2024-04-14T23:53:39.403 INFO:tasks.workunit.client.0.smithi060.stdout:removed './106/19' 2024-04-14T23:53:39.403 INFO:tasks.workunit.client.0.smithi060.stdout:removed './106/15' 2024-04-14T23:53:39.403 INFO:tasks.workunit.client.0.smithi060.stdout:removed './106/31' 2024-04-14T23:53:39.403 INFO:tasks.workunit.client.0.smithi060.stdout:removed './106/36' 2024-04-14T23:53:39.403 INFO:tasks.workunit.client.0.smithi060.stdout:removed './106/20' 2024-04-14T23:53:39.403 INFO:tasks.workunit.client.0.smithi060.stdout:removed './106/12' 2024-04-14T23:53:39.403 INFO:tasks.workunit.client.0.smithi060.stdout:removed './106/39' 2024-04-14T23:53:39.403 INFO:tasks.workunit.client.0.smithi060.stdout:removed './106/10' 2024-04-14T23:53:39.403 INFO:tasks.workunit.client.0.smithi060.stdout:removed './106/34' 2024-04-14T23:53:39.403 INFO:tasks.workunit.client.0.smithi060.stdout:removed './106/38' 2024-04-14T23:53:39.403 INFO:tasks.workunit.client.0.smithi060.stdout:removed './106/11' 2024-04-14T23:53:39.404 INFO:tasks.workunit.client.0.smithi060.stdout:removed './106/16' 2024-04-14T23:53:39.404 INFO:tasks.workunit.client.0.smithi060.stdout:removed './106/32' 2024-04-14T23:53:39.404 INFO:tasks.workunit.client.0.smithi060.stdout:removed './106/30' 2024-04-14T23:53:39.404 INFO:tasks.workunit.client.0.smithi060.stdout:removed './106/14' 2024-04-14T23:53:39.404 INFO:tasks.workunit.client.0.smithi060.stdout:removed './106/18' 2024-04-14T23:53:39.404 INFO:tasks.workunit.client.0.smithi060.stdout:removed './106/13' 2024-04-14T23:53:39.404 INFO:tasks.workunit.client.0.smithi060.stdout:removed './106/33' 2024-04-14T23:53:39.404 INFO:tasks.workunit.client.0.smithi060.stdout:removed './106/17' 2024-04-14T23:53:39.404 INFO:tasks.workunit.client.0.smithi060.stdout:removed './106/35' 2024-04-14T23:53:39.404 INFO:tasks.workunit.client.0.smithi060.stdout:removed './106/37' 2024-04-14T23:53:39.404 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './106' 2024-04-14T23:53:39.404 INFO:tasks.workunit.client.0.smithi060.stdout:removed './107/19' 2024-04-14T23:53:39.404 INFO:tasks.workunit.client.0.smithi060.stdout:removed './107/15' 2024-04-14T23:53:39.404 INFO:tasks.workunit.client.0.smithi060.stdout:removed './107/31' 2024-04-14T23:53:39.404 INFO:tasks.workunit.client.0.smithi060.stdout:removed './107/36' 2024-04-14T23:53:39.405 INFO:tasks.workunit.client.0.smithi060.stdout:removed './107/20' 2024-04-14T23:53:39.405 INFO:tasks.workunit.client.0.smithi060.stdout:removed './107/12' 2024-04-14T23:53:39.405 INFO:tasks.workunit.client.0.smithi060.stdout:removed './107/39' 2024-04-14T23:53:39.405 INFO:tasks.workunit.client.0.smithi060.stdout:removed './107/10' 2024-04-14T23:53:39.405 INFO:tasks.workunit.client.0.smithi060.stdout:removed './107/34' 2024-04-14T23:53:39.405 INFO:tasks.workunit.client.0.smithi060.stdout:removed './107/38' 2024-04-14T23:53:39.405 INFO:tasks.workunit.client.0.smithi060.stdout:removed './107/11' 2024-04-14T23:53:39.405 INFO:tasks.workunit.client.0.smithi060.stdout:removed './107/16' 2024-04-14T23:53:39.405 INFO:tasks.workunit.client.0.smithi060.stdout:removed './107/32' 2024-04-14T23:53:39.405 INFO:tasks.workunit.client.0.smithi060.stdout:removed './107/30' 2024-04-14T23:53:39.405 INFO:tasks.workunit.client.0.smithi060.stdout:removed './107/14' 2024-04-14T23:53:39.405 INFO:tasks.workunit.client.0.smithi060.stdout:removed './107/18' 2024-04-14T23:53:39.405 INFO:tasks.workunit.client.0.smithi060.stdout:removed './107/13' 2024-04-14T23:53:39.405 INFO:tasks.workunit.client.0.smithi060.stdout:removed './107/33' 2024-04-14T23:53:39.406 INFO:tasks.workunit.client.0.smithi060.stdout:removed './107/17' 2024-04-14T23:53:39.406 INFO:tasks.workunit.client.0.smithi060.stdout:removed './107/35' 2024-04-14T23:53:39.406 INFO:tasks.workunit.client.0.smithi060.stdout:removed './107/37' 2024-04-14T23:53:39.406 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './107' 2024-04-14T23:53:39.406 INFO:tasks.workunit.client.0.smithi060.stdout:removed './108/19' 2024-04-14T23:53:39.406 INFO:tasks.workunit.client.0.smithi060.stdout:removed './108/15' 2024-04-14T23:53:39.406 INFO:tasks.workunit.client.0.smithi060.stdout:removed './108/31' 2024-04-14T23:53:39.406 INFO:tasks.workunit.client.0.smithi060.stdout:removed './108/36' 2024-04-14T23:53:39.406 INFO:tasks.workunit.client.0.smithi060.stdout:removed './108/20' 2024-04-14T23:53:39.406 INFO:tasks.workunit.client.0.smithi060.stdout:removed './108/12' 2024-04-14T23:53:39.406 INFO:tasks.workunit.client.0.smithi060.stdout:removed './108/39' 2024-04-14T23:53:39.406 INFO:tasks.workunit.client.0.smithi060.stdout:removed './108/10' 2024-04-14T23:53:39.406 INFO:tasks.workunit.client.0.smithi060.stdout:removed './108/34' 2024-04-14T23:53:39.406 INFO:tasks.workunit.client.0.smithi060.stdout:removed './108/38' 2024-04-14T23:53:39.406 INFO:tasks.workunit.client.0.smithi060.stdout:removed './108/11' 2024-04-14T23:53:39.407 INFO:tasks.workunit.client.0.smithi060.stdout:removed './108/16' 2024-04-14T23:53:39.407 INFO:tasks.workunit.client.0.smithi060.stdout:removed './108/32' 2024-04-14T23:53:39.407 INFO:tasks.workunit.client.0.smithi060.stdout:removed './108/30' 2024-04-14T23:53:39.407 INFO:tasks.workunit.client.0.smithi060.stdout:removed './108/14' 2024-04-14T23:53:39.407 INFO:tasks.workunit.client.0.smithi060.stdout:removed './108/18' 2024-04-14T23:53:39.407 INFO:tasks.workunit.client.0.smithi060.stdout:removed './108/13' 2024-04-14T23:53:39.407 INFO:tasks.workunit.client.0.smithi060.stdout:removed './108/33' 2024-04-14T23:53:39.407 INFO:tasks.workunit.client.0.smithi060.stdout:removed './108/17' 2024-04-14T23:53:39.407 INFO:tasks.workunit.client.0.smithi060.stdout:removed './108/35' 2024-04-14T23:53:39.407 INFO:tasks.workunit.client.0.smithi060.stdout:removed './108/37' 2024-04-14T23:53:39.407 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './108' 2024-04-14T23:53:39.407 INFO:tasks.workunit.client.0.smithi060.stdout:removed './109/19' 2024-04-14T23:53:39.407 INFO:tasks.workunit.client.0.smithi060.stdout:removed './109/15' 2024-04-14T23:53:39.407 INFO:tasks.workunit.client.0.smithi060.stdout:removed './109/31' 2024-04-14T23:53:39.408 INFO:tasks.workunit.client.0.smithi060.stdout:removed './109/36' 2024-04-14T23:53:39.408 INFO:tasks.workunit.client.0.smithi060.stdout:removed './109/20' 2024-04-14T23:53:39.408 INFO:tasks.workunit.client.0.smithi060.stdout:removed './109/12' 2024-04-14T23:53:39.408 INFO:tasks.workunit.client.0.smithi060.stdout:removed './109/39' 2024-04-14T23:53:39.408 INFO:tasks.workunit.client.0.smithi060.stdout:removed './109/10' 2024-04-14T23:53:39.408 INFO:tasks.workunit.client.0.smithi060.stdout:removed './109/34' 2024-04-14T23:53:39.408 INFO:tasks.workunit.client.0.smithi060.stdout:removed './109/38' 2024-04-14T23:53:39.408 INFO:tasks.workunit.client.0.smithi060.stdout:removed './109/11' 2024-04-14T23:53:39.408 INFO:tasks.workunit.client.0.smithi060.stdout:removed './109/16' 2024-04-14T23:53:39.408 INFO:tasks.workunit.client.0.smithi060.stdout:removed './109/32' 2024-04-14T23:53:39.408 INFO:tasks.workunit.client.0.smithi060.stdout:removed './109/30' 2024-04-14T23:53:43.470 INFO:tasks.workunit.client.0.smithi060.stdout:removed './109/14' 2024-04-14T23:53:43.470 INFO:tasks.workunit.client.0.smithi060.stdout:removed './109/18' 2024-04-14T23:53:43.470 INFO:tasks.workunit.client.0.smithi060.stdout:removed './109/13' 2024-04-14T23:53:43.470 INFO:tasks.workunit.client.0.smithi060.stdout:removed './109/33' 2024-04-14T23:53:43.470 INFO:tasks.workunit.client.0.smithi060.stdout:removed './109/17' 2024-04-14T23:53:43.470 INFO:tasks.workunit.client.0.smithi060.stdout:removed './109/35' 2024-04-14T23:53:43.470 INFO:tasks.workunit.client.0.smithi060.stdout:removed './109/37' 2024-04-14T23:53:43.470 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './109' 2024-04-14T23:53:43.470 INFO:tasks.workunit.client.0.smithi060.stdout:removed './110/19' 2024-04-14T23:53:43.471 INFO:tasks.workunit.client.0.smithi060.stdout:removed './110/15' 2024-04-14T23:53:43.471 INFO:tasks.workunit.client.0.smithi060.stdout:removed './110/31' 2024-04-14T23:53:43.471 INFO:tasks.workunit.client.0.smithi060.stdout:removed './110/36' 2024-04-14T23:53:43.471 INFO:tasks.workunit.client.0.smithi060.stdout:removed './110/20' 2024-04-14T23:53:43.471 INFO:tasks.workunit.client.0.smithi060.stdout:removed './110/12' 2024-04-14T23:53:43.471 INFO:tasks.workunit.client.0.smithi060.stdout:removed './110/39' 2024-04-14T23:53:43.471 INFO:tasks.workunit.client.0.smithi060.stdout:removed './110/10' 2024-04-14T23:53:43.471 INFO:tasks.workunit.client.0.smithi060.stdout:removed './110/34' 2024-04-14T23:53:43.471 INFO:tasks.workunit.client.0.smithi060.stdout:removed './110/38' 2024-04-14T23:53:43.471 INFO:tasks.workunit.client.0.smithi060.stdout:removed './110/11' 2024-04-14T23:53:43.471 INFO:tasks.workunit.client.0.smithi060.stdout:removed './110/16' 2024-04-14T23:53:43.471 INFO:tasks.workunit.client.0.smithi060.stdout:removed './110/32' 2024-04-14T23:53:43.471 INFO:tasks.workunit.client.0.smithi060.stdout:removed './110/30' 2024-04-14T23:53:43.471 INFO:tasks.workunit.client.0.smithi060.stdout:removed './110/14' 2024-04-14T23:53:43.471 INFO:tasks.workunit.client.0.smithi060.stdout:removed './110/18' 2024-04-14T23:53:43.472 INFO:tasks.workunit.client.0.smithi060.stdout:removed './110/13' 2024-04-14T23:53:43.472 INFO:tasks.workunit.client.0.smithi060.stdout:removed './110/33' 2024-04-14T23:53:43.472 INFO:tasks.workunit.client.0.smithi060.stdout:removed './110/17' 2024-04-14T23:53:43.472 INFO:tasks.workunit.client.0.smithi060.stdout:removed './110/35' 2024-04-14T23:53:43.472 INFO:tasks.workunit.client.0.smithi060.stdout:removed './110/37' 2024-04-14T23:53:43.472 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './110' 2024-04-14T23:53:43.472 INFO:tasks.workunit.client.0.smithi060.stdout:removed './111/19' 2024-04-14T23:53:43.472 INFO:tasks.workunit.client.0.smithi060.stdout:removed './111/15' 2024-04-14T23:53:43.472 INFO:tasks.workunit.client.0.smithi060.stdout:removed './111/31' 2024-04-14T23:53:43.472 INFO:tasks.workunit.client.0.smithi060.stdout:removed './111/36' 2024-04-14T23:53:43.472 INFO:tasks.workunit.client.0.smithi060.stdout:removed './111/20' 2024-04-14T23:53:43.472 INFO:tasks.workunit.client.0.smithi060.stdout:removed './111/12' 2024-04-14T23:53:43.472 INFO:tasks.workunit.client.0.smithi060.stdout:removed './111/39' 2024-04-14T23:53:43.472 INFO:tasks.workunit.client.0.smithi060.stdout:removed './111/10' 2024-04-14T23:53:43.472 INFO:tasks.workunit.client.0.smithi060.stdout:removed './111/34' 2024-04-14T23:53:43.473 INFO:tasks.workunit.client.0.smithi060.stdout:removed './111/38' 2024-04-14T23:53:43.473 INFO:tasks.workunit.client.0.smithi060.stdout:removed './111/11' 2024-04-14T23:53:43.473 INFO:tasks.workunit.client.0.smithi060.stdout:removed './111/16' 2024-04-14T23:53:43.473 INFO:tasks.workunit.client.0.smithi060.stdout:removed './111/32' 2024-04-14T23:53:43.473 INFO:tasks.workunit.client.0.smithi060.stdout:removed './111/30' 2024-04-14T23:53:43.473 INFO:tasks.workunit.client.0.smithi060.stdout:removed './111/14' 2024-04-14T23:53:43.473 INFO:tasks.workunit.client.0.smithi060.stdout:removed './111/18' 2024-04-14T23:53:43.473 INFO:tasks.workunit.client.0.smithi060.stdout:removed './111/13' 2024-04-14T23:53:43.473 INFO:tasks.workunit.client.0.smithi060.stdout:removed './111/33' 2024-04-14T23:53:43.473 INFO:tasks.workunit.client.0.smithi060.stdout:removed './111/17' 2024-04-14T23:53:43.473 INFO:tasks.workunit.client.0.smithi060.stdout:removed './111/35' 2024-04-14T23:53:43.473 INFO:tasks.workunit.client.0.smithi060.stdout:removed './111/37' 2024-04-14T23:53:43.473 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './111' 2024-04-14T23:53:43.473 INFO:tasks.workunit.client.0.smithi060.stdout:removed './112/19' 2024-04-14T23:53:43.473 INFO:tasks.workunit.client.0.smithi060.stdout:removed './112/15' 2024-04-14T23:53:43.474 INFO:tasks.workunit.client.0.smithi060.stdout:removed './112/31' 2024-04-14T23:53:43.474 INFO:tasks.workunit.client.0.smithi060.stdout:removed './112/36' 2024-04-14T23:53:43.474 INFO:tasks.workunit.client.0.smithi060.stdout:removed './112/20' 2024-04-14T23:53:43.474 INFO:tasks.workunit.client.0.smithi060.stdout:removed './112/12' 2024-04-14T23:53:43.474 INFO:tasks.workunit.client.0.smithi060.stdout:removed './112/39' 2024-04-14T23:53:43.474 INFO:tasks.workunit.client.0.smithi060.stdout:removed './112/10' 2024-04-14T23:53:43.474 INFO:tasks.workunit.client.0.smithi060.stdout:removed './112/34' 2024-04-14T23:53:43.474 INFO:tasks.workunit.client.0.smithi060.stdout:removed './112/38' 2024-04-14T23:53:43.474 INFO:tasks.workunit.client.0.smithi060.stdout:removed './112/11' 2024-04-14T23:53:43.474 INFO:tasks.workunit.client.0.smithi060.stdout:removed './112/16' 2024-04-14T23:53:43.474 INFO:tasks.workunit.client.0.smithi060.stdout:removed './112/32' 2024-04-14T23:53:43.474 INFO:tasks.workunit.client.0.smithi060.stdout:removed './112/30' 2024-04-14T23:53:43.474 INFO:tasks.workunit.client.0.smithi060.stdout:removed './112/14' 2024-04-14T23:53:43.474 INFO:tasks.workunit.client.0.smithi060.stdout:removed './112/18' 2024-04-14T23:53:43.475 INFO:tasks.workunit.client.0.smithi060.stdout:removed './112/13' 2024-04-14T23:53:43.475 INFO:tasks.workunit.client.0.smithi060.stdout:removed './112/33' 2024-04-14T23:53:43.475 INFO:tasks.workunit.client.0.smithi060.stdout:removed './112/17' 2024-04-14T23:53:43.475 INFO:tasks.workunit.client.0.smithi060.stdout:removed './112/35' 2024-04-14T23:53:43.475 INFO:tasks.workunit.client.0.smithi060.stdout:removed './112/37' 2024-04-14T23:53:43.475 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './112' 2024-04-14T23:53:43.475 INFO:tasks.workunit.client.0.smithi060.stdout:removed './113/19' 2024-04-14T23:53:43.475 INFO:tasks.workunit.client.0.smithi060.stdout:removed './113/15' 2024-04-14T23:53:43.475 INFO:tasks.workunit.client.0.smithi060.stdout:removed './113/31' 2024-04-14T23:53:43.475 INFO:tasks.workunit.client.0.smithi060.stdout:removed './113/36' 2024-04-14T23:53:43.475 INFO:tasks.workunit.client.0.smithi060.stdout:removed './113/20' 2024-04-14T23:53:43.475 INFO:tasks.workunit.client.0.smithi060.stdout:removed './113/12' 2024-04-14T23:53:43.475 INFO:tasks.workunit.client.0.smithi060.stdout:removed './113/39' 2024-04-14T23:53:43.475 INFO:tasks.workunit.client.0.smithi060.stdout:removed './113/10' 2024-04-14T23:53:43.475 INFO:tasks.workunit.client.0.smithi060.stdout:removed './113/34' 2024-04-14T23:53:43.476 INFO:tasks.workunit.client.0.smithi060.stdout:removed './113/38' 2024-04-14T23:53:43.476 INFO:tasks.workunit.client.0.smithi060.stdout:removed './113/11' 2024-04-14T23:53:43.476 INFO:tasks.workunit.client.0.smithi060.stdout:removed './113/16' 2024-04-14T23:53:43.476 INFO:tasks.workunit.client.0.smithi060.stdout:removed './113/32' 2024-04-14T23:53:43.476 INFO:tasks.workunit.client.0.smithi060.stdout:removed './113/30' 2024-04-14T23:53:43.476 INFO:tasks.workunit.client.0.smithi060.stdout:removed './113/14' 2024-04-14T23:53:43.476 INFO:tasks.workunit.client.0.smithi060.stdout:removed './113/18' 2024-04-14T23:53:43.476 INFO:tasks.workunit.client.0.smithi060.stdout:removed './113/13' 2024-04-14T23:53:43.476 INFO:tasks.workunit.client.0.smithi060.stdout:removed './113/33' 2024-04-14T23:53:43.476 INFO:tasks.workunit.client.0.smithi060.stdout:removed './113/17' 2024-04-14T23:53:43.476 INFO:tasks.workunit.client.0.smithi060.stdout:removed './113/35' 2024-04-14T23:53:43.476 INFO:tasks.workunit.client.0.smithi060.stdout:removed './113/37' 2024-04-14T23:53:43.476 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './113' 2024-04-14T23:53:43.476 INFO:tasks.workunit.client.0.smithi060.stdout:removed './114/19' 2024-04-14T23:53:43.476 INFO:tasks.workunit.client.0.smithi060.stdout:removed './114/15' 2024-04-14T23:53:43.477 INFO:tasks.workunit.client.0.smithi060.stdout:removed './114/31' 2024-04-14T23:53:43.477 INFO:tasks.workunit.client.0.smithi060.stdout:removed './114/36' 2024-04-14T23:53:43.477 INFO:tasks.workunit.client.0.smithi060.stdout:removed './114/20' 2024-04-14T23:53:43.477 INFO:tasks.workunit.client.0.smithi060.stdout:removed './114/12' 2024-04-14T23:53:43.477 INFO:tasks.workunit.client.0.smithi060.stdout:removed './114/39' 2024-04-14T23:53:43.477 INFO:tasks.workunit.client.0.smithi060.stdout:removed './114/10' 2024-04-14T23:53:43.477 INFO:tasks.workunit.client.0.smithi060.stdout:removed './114/34' 2024-04-14T23:53:43.477 INFO:tasks.workunit.client.0.smithi060.stdout:removed './114/38' 2024-04-14T23:53:43.477 INFO:tasks.workunit.client.0.smithi060.stdout:removed './114/11' 2024-04-14T23:53:43.477 INFO:tasks.workunit.client.0.smithi060.stdout:removed './114/16' 2024-04-14T23:53:43.477 INFO:tasks.workunit.client.0.smithi060.stdout:removed './114/32' 2024-04-14T23:53:43.477 INFO:tasks.workunit.client.0.smithi060.stdout:removed './114/30' 2024-04-14T23:53:43.477 INFO:tasks.workunit.client.0.smithi060.stdout:removed './114/14' 2024-04-14T23:53:43.477 INFO:tasks.workunit.client.0.smithi060.stdout:removed './114/18' 2024-04-14T23:53:43.478 INFO:tasks.workunit.client.0.smithi060.stdout:removed './114/13' 2024-04-14T23:53:43.478 INFO:tasks.workunit.client.0.smithi060.stdout:removed './114/33' 2024-04-14T23:53:43.478 INFO:tasks.workunit.client.0.smithi060.stdout:removed './114/17' 2024-04-14T23:53:43.478 INFO:tasks.workunit.client.0.smithi060.stdout:removed './114/35' 2024-04-14T23:53:43.478 INFO:tasks.workunit.client.0.smithi060.stdout:removed './114/37' 2024-04-14T23:53:43.478 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './114' 2024-04-14T23:53:43.478 INFO:tasks.workunit.client.0.smithi060.stdout:removed './115/19' 2024-04-14T23:53:43.478 INFO:tasks.workunit.client.0.smithi060.stdout:removed './115/15' 2024-04-14T23:53:43.478 INFO:tasks.workunit.client.0.smithi060.stdout:removed './115/31' 2024-04-14T23:53:43.479 INFO:tasks.workunit.client.0.smithi060.stdout:removed './115/36' 2024-04-14T23:53:43.479 INFO:tasks.workunit.client.0.smithi060.stdout:removed './115/20' 2024-04-14T23:53:43.479 INFO:tasks.workunit.client.0.smithi060.stdout:removed './115/12' 2024-04-14T23:53:43.479 INFO:tasks.workunit.client.0.smithi060.stdout:removed './115/39' 2024-04-14T23:53:43.479 INFO:tasks.workunit.client.0.smithi060.stdout:removed './115/10' 2024-04-14T23:53:43.479 INFO:tasks.workunit.client.0.smithi060.stdout:removed './115/34' 2024-04-14T23:53:43.479 INFO:tasks.workunit.client.0.smithi060.stdout:removed './115/38' 2024-04-14T23:53:43.479 INFO:tasks.workunit.client.0.smithi060.stdout:removed './115/11' 2024-04-14T23:53:43.479 INFO:tasks.workunit.client.0.smithi060.stdout:removed './115/16' 2024-04-14T23:53:43.479 INFO:tasks.workunit.client.0.smithi060.stdout:removed './115/32' 2024-04-14T23:53:43.479 INFO:tasks.workunit.client.0.smithi060.stdout:removed './115/30' 2024-04-14T23:53:43.479 INFO:tasks.workunit.client.0.smithi060.stdout:removed './115/14' 2024-04-14T23:53:43.479 INFO:tasks.workunit.client.0.smithi060.stdout:removed './115/18' 2024-04-14T23:53:43.480 INFO:tasks.workunit.client.0.smithi060.stdout:removed './115/13' 2024-04-14T23:53:43.480 INFO:tasks.workunit.client.0.smithi060.stdout:removed './115/33' 2024-04-14T23:53:43.480 INFO:tasks.workunit.client.0.smithi060.stdout:removed './115/17' 2024-04-14T23:53:43.480 INFO:tasks.workunit.client.0.smithi060.stdout:removed './115/35' 2024-04-14T23:53:43.480 INFO:tasks.workunit.client.0.smithi060.stdout:removed './115/37' 2024-04-14T23:53:43.480 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './115' 2024-04-14T23:53:43.480 INFO:tasks.workunit.client.0.smithi060.stdout:removed './116/19' 2024-04-14T23:53:43.480 INFO:tasks.workunit.client.0.smithi060.stdout:removed './116/15' 2024-04-14T23:53:43.480 INFO:tasks.workunit.client.0.smithi060.stdout:removed './116/31' 2024-04-14T23:53:43.480 INFO:tasks.workunit.client.0.smithi060.stdout:removed './116/36' 2024-04-14T23:53:43.480 INFO:tasks.workunit.client.0.smithi060.stdout:removed './116/20' 2024-04-14T23:53:43.480 INFO:tasks.workunit.client.0.smithi060.stdout:removed './116/12' 2024-04-14T23:53:43.480 INFO:tasks.workunit.client.0.smithi060.stdout:removed './116/39' 2024-04-14T23:53:43.480 INFO:tasks.workunit.client.0.smithi060.stdout:removed './116/10' 2024-04-14T23:53:43.480 INFO:tasks.workunit.client.0.smithi060.stdout:removed './116/34' 2024-04-14T23:53:43.481 INFO:tasks.workunit.client.0.smithi060.stdout:removed './116/38' 2024-04-14T23:53:43.481 INFO:tasks.workunit.client.0.smithi060.stdout:removed './116/11' 2024-04-14T23:53:43.481 INFO:tasks.workunit.client.0.smithi060.stdout:removed './116/16' 2024-04-14T23:53:43.481 INFO:tasks.workunit.client.0.smithi060.stdout:removed './116/32' 2024-04-14T23:53:43.481 INFO:tasks.workunit.client.0.smithi060.stdout:removed './116/30' 2024-04-14T23:53:43.481 INFO:tasks.workunit.client.0.smithi060.stdout:removed './116/14' 2024-04-14T23:53:43.481 INFO:tasks.workunit.client.0.smithi060.stdout:removed './116/18' 2024-04-14T23:53:43.481 INFO:tasks.workunit.client.0.smithi060.stdout:removed './116/13' 2024-04-14T23:53:43.481 INFO:tasks.workunit.client.0.smithi060.stdout:removed './116/33' 2024-04-14T23:53:43.481 INFO:tasks.workunit.client.0.smithi060.stdout:removed './116/17' 2024-04-14T23:53:43.481 INFO:tasks.workunit.client.0.smithi060.stdout:removed './116/35' 2024-04-14T23:53:43.481 INFO:tasks.workunit.client.0.smithi060.stdout:removed './116/37' 2024-04-14T23:53:43.481 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './116' 2024-04-14T23:53:43.481 INFO:tasks.workunit.client.0.smithi060.stdout:removed './117/19' 2024-04-14T23:53:43.481 INFO:tasks.workunit.client.0.smithi060.stdout:removed './117/15' 2024-04-14T23:53:43.482 INFO:tasks.workunit.client.0.smithi060.stdout:removed './117/31' 2024-04-14T23:53:43.482 INFO:tasks.workunit.client.0.smithi060.stdout:removed './117/36' 2024-04-14T23:53:43.482 INFO:tasks.workunit.client.0.smithi060.stdout:removed './117/20' 2024-04-14T23:53:43.482 INFO:tasks.workunit.client.0.smithi060.stdout:removed './117/12' 2024-04-14T23:53:43.482 INFO:tasks.workunit.client.0.smithi060.stdout:removed './117/39' 2024-04-14T23:53:43.482 INFO:tasks.workunit.client.0.smithi060.stdout:removed './117/10' 2024-04-14T23:53:43.482 INFO:tasks.workunit.client.0.smithi060.stdout:removed './117/34' 2024-04-14T23:53:43.482 INFO:tasks.workunit.client.0.smithi060.stdout:removed './117/38' 2024-04-14T23:53:43.482 INFO:tasks.workunit.client.0.smithi060.stdout:removed './117/11' 2024-04-14T23:53:43.482 INFO:tasks.workunit.client.0.smithi060.stdout:removed './117/16' 2024-04-14T23:53:43.482 INFO:tasks.workunit.client.0.smithi060.stdout:removed './117/32' 2024-04-14T23:53:43.482 INFO:tasks.workunit.client.0.smithi060.stdout:removed './117/30' 2024-04-14T23:53:43.482 INFO:tasks.workunit.client.0.smithi060.stdout:removed './117/14' 2024-04-14T23:53:43.482 INFO:tasks.workunit.client.0.smithi060.stdout:removed './117/18' 2024-04-14T23:53:43.482 INFO:tasks.workunit.client.0.smithi060.stdout:removed './117/13' 2024-04-14T23:53:43.483 INFO:tasks.workunit.client.0.smithi060.stdout:removed './117/33' 2024-04-14T23:53:43.483 INFO:tasks.workunit.client.0.smithi060.stdout:removed './117/17' 2024-04-14T23:53:43.483 INFO:tasks.workunit.client.0.smithi060.stdout:removed './117/35' 2024-04-14T23:53:43.483 INFO:tasks.workunit.client.0.smithi060.stdout:removed './117/37' 2024-04-14T23:53:43.483 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './117' 2024-04-14T23:53:43.483 INFO:tasks.workunit.client.0.smithi060.stdout:removed './118/19' 2024-04-14T23:53:43.483 INFO:tasks.workunit.client.0.smithi060.stdout:removed './118/15' 2024-04-14T23:53:43.483 INFO:tasks.workunit.client.0.smithi060.stdout:removed './118/31' 2024-04-14T23:53:43.483 INFO:tasks.workunit.client.0.smithi060.stdout:removed './118/36' 2024-04-14T23:53:43.483 INFO:tasks.workunit.client.0.smithi060.stdout:removed './118/20' 2024-04-14T23:53:43.483 INFO:tasks.workunit.client.0.smithi060.stdout:removed './118/12' 2024-04-14T23:53:43.483 INFO:tasks.workunit.client.0.smithi060.stdout:removed './118/39' 2024-04-14T23:53:43.483 INFO:tasks.workunit.client.0.smithi060.stdout:removed './118/10' 2024-04-14T23:53:43.483 INFO:tasks.workunit.client.0.smithi060.stdout:removed './118/34' 2024-04-14T23:53:43.483 INFO:tasks.workunit.client.0.smithi060.stdout:removed './118/38' 2024-04-14T23:53:43.484 INFO:tasks.workunit.client.0.smithi060.stdout:removed './118/11' 2024-04-14T23:53:43.484 INFO:tasks.workunit.client.0.smithi060.stdout:removed './118/16' 2024-04-14T23:53:43.484 INFO:tasks.workunit.client.0.smithi060.stdout:removed './118/32' 2024-04-14T23:53:43.484 INFO:tasks.workunit.client.0.smithi060.stdout:removed './118/30' 2024-04-14T23:53:43.484 INFO:tasks.workunit.client.0.smithi060.stdout:removed './118/14' 2024-04-14T23:53:43.484 INFO:tasks.workunit.client.0.smithi060.stdout:removed './118/18' 2024-04-14T23:53:43.484 INFO:tasks.workunit.client.0.smithi060.stdout:removed './118/13' 2024-04-14T23:53:43.484 INFO:tasks.workunit.client.0.smithi060.stdout:removed './118/33' 2024-04-14T23:53:43.484 INFO:tasks.workunit.client.0.smithi060.stdout:removed './118/17' 2024-04-14T23:53:43.484 INFO:tasks.workunit.client.0.smithi060.stdout:removed './118/35' 2024-04-14T23:53:43.484 INFO:tasks.workunit.client.0.smithi060.stdout:removed './118/37' 2024-04-14T23:53:43.484 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './118' 2024-04-14T23:53:43.484 INFO:tasks.workunit.client.0.smithi060.stdout:removed './119/19' 2024-04-14T23:53:43.484 INFO:tasks.workunit.client.0.smithi060.stdout:removed './119/15' 2024-04-14T23:53:43.484 INFO:tasks.workunit.client.0.smithi060.stdout:removed './119/31' 2024-04-14T23:53:43.485 INFO:tasks.workunit.client.0.smithi060.stdout:removed './119/36' 2024-04-14T23:53:43.485 INFO:tasks.workunit.client.0.smithi060.stdout:removed './119/20' 2024-04-14T23:53:43.485 INFO:tasks.workunit.client.0.smithi060.stdout:removed './119/12' 2024-04-14T23:53:44.506 INFO:tasks.workunit.client.0.smithi060.stdout:removed './119/39' 2024-04-14T23:53:44.506 INFO:tasks.workunit.client.0.smithi060.stdout:removed './119/10' 2024-04-14T23:53:44.506 INFO:tasks.workunit.client.0.smithi060.stdout:removed './119/34' 2024-04-14T23:53:44.507 INFO:tasks.workunit.client.0.smithi060.stdout:removed './119/38' 2024-04-14T23:53:44.507 INFO:tasks.workunit.client.0.smithi060.stdout:removed './119/11' 2024-04-14T23:53:44.507 INFO:tasks.workunit.client.0.smithi060.stdout:removed './119/16' 2024-04-14T23:53:44.507 INFO:tasks.workunit.client.0.smithi060.stdout:removed './119/32' 2024-04-14T23:53:44.507 INFO:tasks.workunit.client.0.smithi060.stdout:removed './119/30' 2024-04-14T23:53:44.507 INFO:tasks.workunit.client.0.smithi060.stdout:removed './119/14' 2024-04-14T23:53:44.507 INFO:tasks.workunit.client.0.smithi060.stdout:removed './119/18' 2024-04-14T23:53:44.507 INFO:tasks.workunit.client.0.smithi060.stdout:removed './119/13' 2024-04-14T23:53:44.507 INFO:tasks.workunit.client.0.smithi060.stdout:removed './119/33' 2024-04-14T23:53:44.507 INFO:tasks.workunit.client.0.smithi060.stdout:removed './119/17' 2024-04-14T23:53:44.507 INFO:tasks.workunit.client.0.smithi060.stdout:removed './119/35' 2024-04-14T23:53:44.507 INFO:tasks.workunit.client.0.smithi060.stdout:removed './119/37' 2024-04-14T23:53:44.507 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './119' 2024-04-14T23:53:44.508 INFO:tasks.workunit.client.0.smithi060.stdout:removed './120/19' 2024-04-14T23:53:44.508 INFO:tasks.workunit.client.0.smithi060.stdout:removed './120/15' 2024-04-14T23:53:44.508 INFO:tasks.workunit.client.0.smithi060.stdout:removed './120/31' 2024-04-14T23:53:44.508 INFO:tasks.workunit.client.0.smithi060.stdout:removed './120/36' 2024-04-14T23:53:44.508 INFO:tasks.workunit.client.0.smithi060.stdout:removed './120/20' 2024-04-14T23:53:44.508 INFO:tasks.workunit.client.0.smithi060.stdout:removed './120/12' 2024-04-14T23:53:44.508 INFO:tasks.workunit.client.0.smithi060.stdout:removed './120/39' 2024-04-14T23:53:44.508 INFO:tasks.workunit.client.0.smithi060.stdout:removed './120/10' 2024-04-14T23:53:44.508 INFO:tasks.workunit.client.0.smithi060.stdout:removed './120/34' 2024-04-14T23:53:44.508 INFO:tasks.workunit.client.0.smithi060.stdout:removed './120/38' 2024-04-14T23:53:44.508 INFO:tasks.workunit.client.0.smithi060.stdout:removed './120/11' 2024-04-14T23:53:44.508 INFO:tasks.workunit.client.0.smithi060.stdout:removed './120/16' 2024-04-14T23:53:44.508 INFO:tasks.workunit.client.0.smithi060.stdout:removed './120/32' 2024-04-14T23:53:44.508 INFO:tasks.workunit.client.0.smithi060.stdout:removed './120/30' 2024-04-14T23:53:44.508 INFO:tasks.workunit.client.0.smithi060.stdout:removed './120/14' 2024-04-14T23:53:44.509 INFO:tasks.workunit.client.0.smithi060.stdout:removed './120/18' 2024-04-14T23:53:44.509 INFO:tasks.workunit.client.0.smithi060.stdout:removed './120/13' 2024-04-14T23:53:44.509 INFO:tasks.workunit.client.0.smithi060.stdout:removed './120/33' 2024-04-14T23:53:44.509 INFO:tasks.workunit.client.0.smithi060.stdout:removed './120/17' 2024-04-14T23:53:44.509 INFO:tasks.workunit.client.0.smithi060.stdout:removed './120/35' 2024-04-14T23:53:44.509 INFO:tasks.workunit.client.0.smithi060.stdout:removed './120/37' 2024-04-14T23:53:44.509 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './120' 2024-04-14T23:53:44.509 INFO:tasks.workunit.client.0.smithi060.stdout:removed './121/19' 2024-04-14T23:53:44.509 INFO:tasks.workunit.client.0.smithi060.stdout:removed './121/15' 2024-04-14T23:53:44.509 INFO:tasks.workunit.client.0.smithi060.stdout:removed './121/31' 2024-04-14T23:53:44.509 INFO:tasks.workunit.client.0.smithi060.stdout:removed './121/36' 2024-04-14T23:53:44.509 INFO:tasks.workunit.client.0.smithi060.stdout:removed './121/20' 2024-04-14T23:53:44.509 INFO:tasks.workunit.client.0.smithi060.stdout:removed './121/12' 2024-04-14T23:53:44.509 INFO:tasks.workunit.client.0.smithi060.stdout:removed './121/39' 2024-04-14T23:53:44.509 INFO:tasks.workunit.client.0.smithi060.stdout:removed './121/10' 2024-04-14T23:53:44.510 INFO:tasks.workunit.client.0.smithi060.stdout:removed './121/34' 2024-04-14T23:53:44.510 INFO:tasks.workunit.client.0.smithi060.stdout:removed './121/38' 2024-04-14T23:53:44.510 INFO:tasks.workunit.client.0.smithi060.stdout:removed './121/11' 2024-04-14T23:53:44.510 INFO:tasks.workunit.client.0.smithi060.stdout:removed './121/16' 2024-04-14T23:53:44.510 INFO:tasks.workunit.client.0.smithi060.stdout:removed './121/32' 2024-04-14T23:53:44.510 INFO:tasks.workunit.client.0.smithi060.stdout:removed './121/30' 2024-04-14T23:53:44.510 INFO:tasks.workunit.client.0.smithi060.stdout:removed './121/14' 2024-04-14T23:53:44.510 INFO:tasks.workunit.client.0.smithi060.stdout:removed './121/18' 2024-04-14T23:53:44.510 INFO:tasks.workunit.client.0.smithi060.stdout:removed './121/13' 2024-04-14T23:53:44.510 INFO:tasks.workunit.client.0.smithi060.stdout:removed './121/33' 2024-04-14T23:53:44.510 INFO:tasks.workunit.client.0.smithi060.stdout:removed './121/17' 2024-04-14T23:53:44.510 INFO:tasks.workunit.client.0.smithi060.stdout:removed './121/35' 2024-04-14T23:53:44.510 INFO:tasks.workunit.client.0.smithi060.stdout:removed './121/37' 2024-04-14T23:53:44.511 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './121' 2024-04-14T23:53:44.511 INFO:tasks.workunit.client.0.smithi060.stdout:removed './122/19' 2024-04-14T23:53:44.511 INFO:tasks.workunit.client.0.smithi060.stdout:removed './122/15' 2024-04-14T23:53:44.511 INFO:tasks.workunit.client.0.smithi060.stdout:removed './122/31' 2024-04-14T23:53:44.511 INFO:tasks.workunit.client.0.smithi060.stdout:removed './122/36' 2024-04-14T23:53:44.511 INFO:tasks.workunit.client.0.smithi060.stdout:removed './122/20' 2024-04-14T23:53:44.511 INFO:tasks.workunit.client.0.smithi060.stdout:removed './122/12' 2024-04-14T23:53:44.511 INFO:tasks.workunit.client.0.smithi060.stdout:removed './122/39' 2024-04-14T23:53:44.511 INFO:tasks.workunit.client.0.smithi060.stdout:removed './122/10' 2024-04-14T23:53:44.511 INFO:tasks.workunit.client.0.smithi060.stdout:removed './122/34' 2024-04-14T23:53:44.511 INFO:tasks.workunit.client.0.smithi060.stdout:removed './122/38' 2024-04-14T23:53:44.511 INFO:tasks.workunit.client.0.smithi060.stdout:removed './122/11' 2024-04-14T23:53:44.511 INFO:tasks.workunit.client.0.smithi060.stdout:removed './122/16' 2024-04-14T23:53:44.511 INFO:tasks.workunit.client.0.smithi060.stdout:removed './122/32' 2024-04-14T23:53:44.511 INFO:tasks.workunit.client.0.smithi060.stdout:removed './122/30' 2024-04-14T23:53:44.512 INFO:tasks.workunit.client.0.smithi060.stdout:removed './122/14' 2024-04-14T23:53:44.512 INFO:tasks.workunit.client.0.smithi060.stdout:removed './122/18' 2024-04-14T23:53:44.512 INFO:tasks.workunit.client.0.smithi060.stdout:removed './122/13' 2024-04-14T23:53:44.512 INFO:tasks.workunit.client.0.smithi060.stdout:removed './122/33' 2024-04-14T23:53:44.512 INFO:tasks.workunit.client.0.smithi060.stdout:removed './122/17' 2024-04-14T23:53:44.512 INFO:tasks.workunit.client.0.smithi060.stdout:removed './122/35' 2024-04-14T23:53:44.512 INFO:tasks.workunit.client.0.smithi060.stdout:removed './122/37' 2024-04-14T23:53:44.512 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './122' 2024-04-14T23:53:44.512 INFO:tasks.workunit.client.0.smithi060.stdout:removed './123/19' 2024-04-14T23:53:44.512 INFO:tasks.workunit.client.0.smithi060.stdout:removed './123/15' 2024-04-14T23:53:44.512 INFO:tasks.workunit.client.0.smithi060.stdout:removed './123/31' 2024-04-14T23:53:44.512 INFO:tasks.workunit.client.0.smithi060.stdout:removed './123/36' 2024-04-14T23:53:44.512 INFO:tasks.workunit.client.0.smithi060.stdout:removed './123/20' 2024-04-14T23:53:44.512 INFO:tasks.workunit.client.0.smithi060.stdout:removed './123/12' 2024-04-14T23:53:44.512 INFO:tasks.workunit.client.0.smithi060.stdout:removed './123/39' 2024-04-14T23:53:44.512 INFO:tasks.workunit.client.0.smithi060.stdout:removed './123/10' 2024-04-14T23:53:44.513 INFO:tasks.workunit.client.0.smithi060.stdout:removed './123/34' 2024-04-14T23:53:44.513 INFO:tasks.workunit.client.0.smithi060.stdout:removed './123/38' 2024-04-14T23:53:44.513 INFO:tasks.workunit.client.0.smithi060.stdout:removed './123/11' 2024-04-14T23:53:44.513 INFO:tasks.workunit.client.0.smithi060.stdout:removed './123/16' 2024-04-14T23:53:44.513 INFO:tasks.workunit.client.0.smithi060.stdout:removed './123/32' 2024-04-14T23:53:44.513 INFO:tasks.workunit.client.0.smithi060.stdout:removed './123/30' 2024-04-14T23:53:44.513 INFO:tasks.workunit.client.0.smithi060.stdout:removed './123/14' 2024-04-14T23:53:44.513 INFO:tasks.workunit.client.0.smithi060.stdout:removed './123/18' 2024-04-14T23:53:44.513 INFO:tasks.workunit.client.0.smithi060.stdout:removed './123/13' 2024-04-14T23:53:44.513 INFO:tasks.workunit.client.0.smithi060.stdout:removed './123/33' 2024-04-14T23:53:44.513 INFO:tasks.workunit.client.0.smithi060.stdout:removed './123/17' 2024-04-14T23:53:44.513 INFO:tasks.workunit.client.0.smithi060.stdout:removed './123/35' 2024-04-14T23:53:44.513 INFO:tasks.workunit.client.0.smithi060.stdout:removed './123/37' 2024-04-14T23:53:44.513 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './123' 2024-04-14T23:53:44.513 INFO:tasks.workunit.client.0.smithi060.stdout:removed './124/19' 2024-04-14T23:53:44.514 INFO:tasks.workunit.client.0.smithi060.stdout:removed './124/15' 2024-04-14T23:53:44.514 INFO:tasks.workunit.client.0.smithi060.stdout:removed './124/31' 2024-04-14T23:53:44.514 INFO:tasks.workunit.client.0.smithi060.stdout:removed './124/36' 2024-04-14T23:53:44.514 INFO:tasks.workunit.client.0.smithi060.stdout:removed './124/20' 2024-04-14T23:53:44.514 INFO:tasks.workunit.client.0.smithi060.stdout:removed './124/12' 2024-04-14T23:53:44.514 INFO:tasks.workunit.client.0.smithi060.stdout:removed './124/39' 2024-04-14T23:53:44.514 INFO:tasks.workunit.client.0.smithi060.stdout:removed './124/10' 2024-04-14T23:53:44.514 INFO:tasks.workunit.client.0.smithi060.stdout:removed './124/34' 2024-04-14T23:53:44.514 INFO:tasks.workunit.client.0.smithi060.stdout:removed './124/38' 2024-04-14T23:53:44.514 INFO:tasks.workunit.client.0.smithi060.stdout:removed './124/11' 2024-04-14T23:53:44.514 INFO:tasks.workunit.client.0.smithi060.stdout:removed './124/16' 2024-04-14T23:53:44.514 INFO:tasks.workunit.client.0.smithi060.stdout:removed './124/32' 2024-04-14T23:53:44.514 INFO:tasks.workunit.client.0.smithi060.stdout:removed './124/30' 2024-04-14T23:53:44.514 INFO:tasks.workunit.client.0.smithi060.stdout:removed './124/14' 2024-04-14T23:53:44.515 INFO:tasks.workunit.client.0.smithi060.stdout:removed './124/18' 2024-04-14T23:53:44.515 INFO:tasks.workunit.client.0.smithi060.stdout:removed './124/13' 2024-04-14T23:53:44.515 INFO:tasks.workunit.client.0.smithi060.stdout:removed './124/33' 2024-04-14T23:53:44.515 INFO:tasks.workunit.client.0.smithi060.stdout:removed './124/17' 2024-04-14T23:53:44.515 INFO:tasks.workunit.client.0.smithi060.stdout:removed './124/35' 2024-04-14T23:53:44.515 INFO:tasks.workunit.client.0.smithi060.stdout:removed './124/37' 2024-04-14T23:53:44.515 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './124' 2024-04-14T23:53:44.515 INFO:tasks.workunit.client.0.smithi060.stdout:removed './125/19' 2024-04-14T23:53:44.515 INFO:tasks.workunit.client.0.smithi060.stdout:removed './125/15' 2024-04-14T23:53:44.515 INFO:tasks.workunit.client.0.smithi060.stdout:removed './125/31' 2024-04-14T23:53:44.515 INFO:tasks.workunit.client.0.smithi060.stdout:removed './125/36' 2024-04-14T23:53:44.515 INFO:tasks.workunit.client.0.smithi060.stdout:removed './125/20' 2024-04-14T23:53:44.515 INFO:tasks.workunit.client.0.smithi060.stdout:removed './125/12' 2024-04-14T23:53:44.515 INFO:tasks.workunit.client.0.smithi060.stdout:removed './125/39' 2024-04-14T23:53:44.515 INFO:tasks.workunit.client.0.smithi060.stdout:removed './125/10' 2024-04-14T23:53:44.516 INFO:tasks.workunit.client.0.smithi060.stdout:removed './125/34' 2024-04-14T23:53:44.516 INFO:tasks.workunit.client.0.smithi060.stdout:removed './125/38' 2024-04-14T23:53:44.516 INFO:tasks.workunit.client.0.smithi060.stdout:removed './125/11' 2024-04-14T23:53:44.516 INFO:tasks.workunit.client.0.smithi060.stdout:removed './125/16' 2024-04-14T23:53:44.516 INFO:tasks.workunit.client.0.smithi060.stdout:removed './125/32' 2024-04-14T23:53:44.516 INFO:tasks.workunit.client.0.smithi060.stdout:removed './125/30' 2024-04-14T23:53:44.516 INFO:tasks.workunit.client.0.smithi060.stdout:removed './125/14' 2024-04-14T23:53:44.516 INFO:tasks.workunit.client.0.smithi060.stdout:removed './125/18' 2024-04-14T23:53:44.516 INFO:tasks.workunit.client.0.smithi060.stdout:removed './125/13' 2024-04-14T23:53:44.516 INFO:tasks.workunit.client.0.smithi060.stdout:removed './125/33' 2024-04-14T23:53:44.516 INFO:tasks.workunit.client.0.smithi060.stdout:removed './125/17' 2024-04-14T23:53:44.516 INFO:tasks.workunit.client.0.smithi060.stdout:removed './125/35' 2024-04-14T23:53:44.516 INFO:tasks.workunit.client.0.smithi060.stdout:removed './125/37' 2024-04-14T23:53:44.516 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './125' 2024-04-14T23:53:44.517 INFO:tasks.workunit.client.0.smithi060.stdout:removed './126/19' 2024-04-14T23:53:44.517 INFO:tasks.workunit.client.0.smithi060.stdout:removed './126/15' 2024-04-14T23:53:44.517 INFO:tasks.workunit.client.0.smithi060.stdout:removed './126/31' 2024-04-14T23:53:44.517 INFO:tasks.workunit.client.0.smithi060.stdout:removed './126/36' 2024-04-14T23:53:44.517 INFO:tasks.workunit.client.0.smithi060.stdout:removed './126/20' 2024-04-14T23:53:44.517 INFO:tasks.workunit.client.0.smithi060.stdout:removed './126/12' 2024-04-14T23:53:44.517 INFO:tasks.workunit.client.0.smithi060.stdout:removed './126/39' 2024-04-14T23:53:44.517 INFO:tasks.workunit.client.0.smithi060.stdout:removed './126/10' 2024-04-14T23:53:44.517 INFO:tasks.workunit.client.0.smithi060.stdout:removed './126/34' 2024-04-14T23:53:44.517 INFO:tasks.workunit.client.0.smithi060.stdout:removed './126/38' 2024-04-14T23:53:44.517 INFO:tasks.workunit.client.0.smithi060.stdout:removed './126/11' 2024-04-14T23:53:44.517 INFO:tasks.workunit.client.0.smithi060.stdout:removed './126/16' 2024-04-14T23:53:44.517 INFO:tasks.workunit.client.0.smithi060.stdout:removed './126/32' 2024-04-14T23:53:44.517 INFO:tasks.workunit.client.0.smithi060.stdout:removed './126/30' 2024-04-14T23:53:44.517 INFO:tasks.workunit.client.0.smithi060.stdout:removed './126/14' 2024-04-14T23:53:44.518 INFO:tasks.workunit.client.0.smithi060.stdout:removed './126/18' 2024-04-14T23:53:44.518 INFO:tasks.workunit.client.0.smithi060.stdout:removed './126/13' 2024-04-14T23:53:44.518 INFO:tasks.workunit.client.0.smithi060.stdout:removed './126/33' 2024-04-14T23:53:44.518 INFO:tasks.workunit.client.0.smithi060.stdout:removed './126/17' 2024-04-14T23:53:44.518 INFO:tasks.workunit.client.0.smithi060.stdout:removed './126/35' 2024-04-14T23:53:44.518 INFO:tasks.workunit.client.0.smithi060.stdout:removed './126/37' 2024-04-14T23:53:44.518 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './126' 2024-04-14T23:53:44.518 INFO:tasks.workunit.client.0.smithi060.stdout:removed './127/19' 2024-04-14T23:53:44.518 INFO:tasks.workunit.client.0.smithi060.stdout:removed './127/15' 2024-04-14T23:53:44.518 INFO:tasks.workunit.client.0.smithi060.stdout:removed './127/31' 2024-04-14T23:53:44.518 INFO:tasks.workunit.client.0.smithi060.stdout:removed './127/36' 2024-04-14T23:53:44.518 INFO:tasks.workunit.client.0.smithi060.stdout:removed './127/20' 2024-04-14T23:53:44.518 INFO:tasks.workunit.client.0.smithi060.stdout:removed './127/12' 2024-04-14T23:53:44.518 INFO:tasks.workunit.client.0.smithi060.stdout:removed './127/39' 2024-04-14T23:53:44.519 INFO:tasks.workunit.client.0.smithi060.stdout:removed './127/10' 2024-04-14T23:53:44.519 INFO:tasks.workunit.client.0.smithi060.stdout:removed './127/34' 2024-04-14T23:53:44.519 INFO:tasks.workunit.client.0.smithi060.stdout:removed './127/38' 2024-04-14T23:53:44.519 INFO:tasks.workunit.client.0.smithi060.stdout:removed './127/11' 2024-04-14T23:53:44.519 INFO:tasks.workunit.client.0.smithi060.stdout:removed './127/16' 2024-04-14T23:53:44.519 INFO:tasks.workunit.client.0.smithi060.stdout:removed './127/32' 2024-04-14T23:53:44.519 INFO:tasks.workunit.client.0.smithi060.stdout:removed './127/30' 2024-04-14T23:53:44.519 INFO:tasks.workunit.client.0.smithi060.stdout:removed './127/14' 2024-04-14T23:53:44.519 INFO:tasks.workunit.client.0.smithi060.stdout:removed './127/18' 2024-04-14T23:53:44.519 INFO:tasks.workunit.client.0.smithi060.stdout:removed './127/13' 2024-04-14T23:53:44.519 INFO:tasks.workunit.client.0.smithi060.stdout:removed './127/33' 2024-04-14T23:53:44.519 INFO:tasks.workunit.client.0.smithi060.stdout:removed './127/17' 2024-04-14T23:53:44.519 INFO:tasks.workunit.client.0.smithi060.stdout:removed './127/35' 2024-04-14T23:53:44.519 INFO:tasks.workunit.client.0.smithi060.stdout:removed './127/37' 2024-04-14T23:53:44.519 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './127' 2024-04-14T23:53:44.520 INFO:tasks.workunit.client.0.smithi060.stdout:removed './128/19' 2024-04-14T23:53:44.520 INFO:tasks.workunit.client.0.smithi060.stdout:removed './128/15' 2024-04-14T23:53:44.520 INFO:tasks.workunit.client.0.smithi060.stdout:removed './128/31' 2024-04-14T23:53:44.520 INFO:tasks.workunit.client.0.smithi060.stdout:removed './128/36' 2024-04-14T23:53:44.520 INFO:tasks.workunit.client.0.smithi060.stdout:removed './128/20' 2024-04-14T23:53:44.520 INFO:tasks.workunit.client.0.smithi060.stdout:removed './128/12' 2024-04-14T23:53:44.520 INFO:tasks.workunit.client.0.smithi060.stdout:removed './128/39' 2024-04-14T23:53:44.520 INFO:tasks.workunit.client.0.smithi060.stdout:removed './128/10' 2024-04-14T23:53:44.520 INFO:tasks.workunit.client.0.smithi060.stdout:removed './128/34' 2024-04-14T23:53:44.520 INFO:tasks.workunit.client.0.smithi060.stdout:removed './128/38' 2024-04-14T23:53:44.520 INFO:tasks.workunit.client.0.smithi060.stdout:removed './128/11' 2024-04-14T23:53:44.520 INFO:tasks.workunit.client.0.smithi060.stdout:removed './128/16' 2024-04-14T23:53:44.520 INFO:tasks.workunit.client.0.smithi060.stdout:removed './128/32' 2024-04-14T23:53:44.520 INFO:tasks.workunit.client.0.smithi060.stdout:removed './128/30' 2024-04-14T23:53:44.521 INFO:tasks.workunit.client.0.smithi060.stdout:removed './128/14' 2024-04-14T23:53:44.521 INFO:tasks.workunit.client.0.smithi060.stdout:removed './128/18' 2024-04-14T23:53:44.521 INFO:tasks.workunit.client.0.smithi060.stdout:removed './128/13' 2024-04-14T23:53:44.521 INFO:tasks.workunit.client.0.smithi060.stdout:removed './128/33' 2024-04-14T23:53:44.521 INFO:tasks.workunit.client.0.smithi060.stdout:removed './128/17' 2024-04-14T23:53:44.521 INFO:tasks.workunit.client.0.smithi060.stdout:removed './128/35' 2024-04-14T23:53:45.225 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-14T23:53:45.234 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-14T23:53:47.244 INFO:tasks.workunit.client.0.smithi060.stdout:removed './128/37' 2024-04-14T23:53:47.244 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './128' 2024-04-14T23:53:47.244 INFO:tasks.workunit.client.0.smithi060.stdout:removed './129/19' 2024-04-14T23:53:47.244 INFO:tasks.workunit.client.0.smithi060.stdout:removed './129/15' 2024-04-14T23:53:47.244 INFO:tasks.workunit.client.0.smithi060.stdout:removed './129/31' 2024-04-14T23:53:47.244 INFO:tasks.workunit.client.0.smithi060.stdout:removed './129/36' 2024-04-14T23:53:47.244 INFO:tasks.workunit.client.0.smithi060.stdout:removed './129/20' 2024-04-14T23:53:47.245 INFO:tasks.workunit.client.0.smithi060.stdout:removed './129/12' 2024-04-14T23:53:47.245 INFO:tasks.workunit.client.0.smithi060.stdout:removed './129/39' 2024-04-14T23:53:47.245 INFO:tasks.workunit.client.0.smithi060.stdout:removed './129/10' 2024-04-14T23:53:47.245 INFO:tasks.workunit.client.0.smithi060.stdout:removed './129/34' 2024-04-14T23:53:47.245 INFO:tasks.workunit.client.0.smithi060.stdout:removed './129/38' 2024-04-14T23:53:47.245 INFO:tasks.workunit.client.0.smithi060.stdout:removed './129/11' 2024-04-14T23:53:47.245 INFO:tasks.workunit.client.0.smithi060.stdout:removed './129/16' 2024-04-14T23:53:47.245 INFO:tasks.workunit.client.0.smithi060.stdout:removed './129/32' 2024-04-14T23:53:47.245 INFO:tasks.workunit.client.0.smithi060.stdout:removed './129/30' 2024-04-14T23:53:47.245 INFO:tasks.workunit.client.0.smithi060.stdout:removed './129/14' 2024-04-14T23:53:47.245 INFO:tasks.workunit.client.0.smithi060.stdout:removed './129/18' 2024-04-14T23:53:47.245 INFO:tasks.workunit.client.0.smithi060.stdout:removed './129/13' 2024-04-14T23:53:47.245 INFO:tasks.workunit.client.0.smithi060.stdout:removed './129/33' 2024-04-14T23:53:47.245 INFO:tasks.workunit.client.0.smithi060.stdout:removed './129/17' 2024-04-14T23:53:47.245 INFO:tasks.workunit.client.0.smithi060.stdout:removed './129/35' 2024-04-14T23:53:47.246 INFO:tasks.workunit.client.0.smithi060.stdout:removed './129/37' 2024-04-14T23:53:47.246 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './129' 2024-04-14T23:53:47.246 INFO:tasks.workunit.client.0.smithi060.stdout:removed './130/19' 2024-04-14T23:53:47.246 INFO:tasks.workunit.client.0.smithi060.stdout:removed './130/15' 2024-04-14T23:53:47.246 INFO:tasks.workunit.client.0.smithi060.stdout:removed './130/31' 2024-04-14T23:53:47.246 INFO:tasks.workunit.client.0.smithi060.stdout:removed './130/36' 2024-04-14T23:53:47.246 INFO:tasks.workunit.client.0.smithi060.stdout:removed './130/20' 2024-04-14T23:53:47.246 INFO:tasks.workunit.client.0.smithi060.stdout:removed './130/12' 2024-04-14T23:53:47.246 INFO:tasks.workunit.client.0.smithi060.stdout:removed './130/39' 2024-04-14T23:53:47.246 INFO:tasks.workunit.client.0.smithi060.stdout:removed './130/10' 2024-04-14T23:53:47.246 INFO:tasks.workunit.client.0.smithi060.stdout:removed './130/34' 2024-04-14T23:53:47.246 INFO:tasks.workunit.client.0.smithi060.stdout:removed './130/38' 2024-04-14T23:53:47.247 INFO:tasks.workunit.client.0.smithi060.stdout:removed './130/11' 2024-04-14T23:53:47.247 INFO:tasks.workunit.client.0.smithi060.stdout:removed './130/16' 2024-04-14T23:53:47.247 INFO:tasks.workunit.client.0.smithi060.stdout:removed './130/32' 2024-04-14T23:53:47.247 INFO:tasks.workunit.client.0.smithi060.stdout:removed './130/30' 2024-04-14T23:53:47.247 INFO:tasks.workunit.client.0.smithi060.stdout:removed './130/14' 2024-04-14T23:53:47.247 INFO:tasks.workunit.client.0.smithi060.stdout:removed './130/18' 2024-04-14T23:53:47.247 INFO:tasks.workunit.client.0.smithi060.stdout:removed './130/13' 2024-04-14T23:53:47.247 INFO:tasks.workunit.client.0.smithi060.stdout:removed './130/33' 2024-04-14T23:53:47.247 INFO:tasks.workunit.client.0.smithi060.stdout:removed './130/17' 2024-04-14T23:53:47.247 INFO:tasks.workunit.client.0.smithi060.stdout:removed './130/35' 2024-04-14T23:53:47.247 INFO:tasks.workunit.client.0.smithi060.stdout:removed './130/37' 2024-04-14T23:53:47.247 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './130' 2024-04-14T23:53:47.247 INFO:tasks.workunit.client.0.smithi060.stdout:removed './131/19' 2024-04-14T23:53:47.247 INFO:tasks.workunit.client.0.smithi060.stdout:removed './131/15' 2024-04-14T23:53:47.247 INFO:tasks.workunit.client.0.smithi060.stdout:removed './131/31' 2024-04-14T23:53:47.248 INFO:tasks.workunit.client.0.smithi060.stdout:removed './131/36' 2024-04-14T23:53:47.248 INFO:tasks.workunit.client.0.smithi060.stdout:removed './131/20' 2024-04-14T23:53:47.248 INFO:tasks.workunit.client.0.smithi060.stdout:removed './131/12' 2024-04-14T23:53:47.248 INFO:tasks.workunit.client.0.smithi060.stdout:removed './131/39' 2024-04-14T23:53:47.248 INFO:tasks.workunit.client.0.smithi060.stdout:removed './131/10' 2024-04-14T23:53:47.248 INFO:tasks.workunit.client.0.smithi060.stdout:removed './131/34' 2024-04-14T23:53:47.248 INFO:tasks.workunit.client.0.smithi060.stdout:removed './131/38' 2024-04-14T23:53:47.248 INFO:tasks.workunit.client.0.smithi060.stdout:removed './131/11' 2024-04-14T23:53:47.248 INFO:tasks.workunit.client.0.smithi060.stdout:removed './131/16' 2024-04-14T23:53:47.248 INFO:tasks.workunit.client.0.smithi060.stdout:removed './131/32' 2024-04-14T23:53:47.248 INFO:tasks.workunit.client.0.smithi060.stdout:removed './131/30' 2024-04-14T23:53:47.248 INFO:tasks.workunit.client.0.smithi060.stdout:removed './131/14' 2024-04-14T23:53:47.248 INFO:tasks.workunit.client.0.smithi060.stdout:removed './131/18' 2024-04-14T23:53:47.248 INFO:tasks.workunit.client.0.smithi060.stdout:removed './131/13' 2024-04-14T23:53:47.249 INFO:tasks.workunit.client.0.smithi060.stdout:removed './131/33' 2024-04-14T23:53:47.249 INFO:tasks.workunit.client.0.smithi060.stdout:removed './131/17' 2024-04-14T23:53:47.249 INFO:tasks.workunit.client.0.smithi060.stdout:removed './131/35' 2024-04-14T23:53:47.251 INFO:tasks.workunit.client.0.smithi060.stdout:removed './131/37' 2024-04-14T23:53:47.251 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './131' 2024-04-14T23:53:47.251 INFO:tasks.workunit.client.0.smithi060.stdout:removed './132/19' 2024-04-14T23:53:47.251 INFO:tasks.workunit.client.0.smithi060.stdout:removed './132/15' 2024-04-14T23:53:47.251 INFO:tasks.workunit.client.0.smithi060.stdout:removed './132/31' 2024-04-14T23:53:47.252 INFO:tasks.workunit.client.0.smithi060.stdout:removed './132/36' 2024-04-14T23:53:47.252 INFO:tasks.workunit.client.0.smithi060.stdout:removed './132/20' 2024-04-14T23:53:47.252 INFO:tasks.workunit.client.0.smithi060.stdout:removed './132/12' 2024-04-14T23:53:47.252 INFO:tasks.workunit.client.0.smithi060.stdout:removed './132/39' 2024-04-14T23:53:47.252 INFO:tasks.workunit.client.0.smithi060.stdout:removed './132/10' 2024-04-14T23:53:47.252 INFO:tasks.workunit.client.0.smithi060.stdout:removed './132/34' 2024-04-14T23:53:47.252 INFO:tasks.workunit.client.0.smithi060.stdout:removed './132/38' 2024-04-14T23:53:47.252 INFO:tasks.workunit.client.0.smithi060.stdout:removed './132/11' 2024-04-14T23:53:47.252 INFO:tasks.workunit.client.0.smithi060.stdout:removed './132/16' 2024-04-14T23:53:47.252 INFO:tasks.workunit.client.0.smithi060.stdout:removed './132/32' 2024-04-14T23:53:47.252 INFO:tasks.workunit.client.0.smithi060.stdout:removed './132/30' 2024-04-14T23:53:47.252 INFO:tasks.workunit.client.0.smithi060.stdout:removed './132/14' 2024-04-14T23:53:47.252 INFO:tasks.workunit.client.0.smithi060.stdout:removed './132/18' 2024-04-14T23:53:47.252 INFO:tasks.workunit.client.0.smithi060.stdout:removed './132/13' 2024-04-14T23:53:47.252 INFO:tasks.workunit.client.0.smithi060.stdout:removed './132/33' 2024-04-14T23:53:47.253 INFO:tasks.workunit.client.0.smithi060.stdout:removed './132/17' 2024-04-14T23:53:47.253 INFO:tasks.workunit.client.0.smithi060.stdout:removed './132/35' 2024-04-14T23:53:47.253 INFO:tasks.workunit.client.0.smithi060.stdout:removed './132/37' 2024-04-14T23:53:47.253 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './132' 2024-04-14T23:53:47.253 INFO:tasks.workunit.client.0.smithi060.stdout:removed './133/19' 2024-04-14T23:53:47.253 INFO:tasks.workunit.client.0.smithi060.stdout:removed './133/15' 2024-04-14T23:53:47.253 INFO:tasks.workunit.client.0.smithi060.stdout:removed './133/31' 2024-04-14T23:53:47.253 INFO:tasks.workunit.client.0.smithi060.stdout:removed './133/36' 2024-04-14T23:53:47.253 INFO:tasks.workunit.client.0.smithi060.stdout:removed './133/20' 2024-04-14T23:53:47.253 INFO:tasks.workunit.client.0.smithi060.stdout:removed './133/12' 2024-04-14T23:53:47.253 INFO:tasks.workunit.client.0.smithi060.stdout:removed './133/39' 2024-04-14T23:53:47.253 INFO:tasks.workunit.client.0.smithi060.stdout:removed './133/10' 2024-04-14T23:53:47.253 INFO:tasks.workunit.client.0.smithi060.stdout:removed './133/34' 2024-04-14T23:53:47.253 INFO:tasks.workunit.client.0.smithi060.stdout:removed './133/38' 2024-04-14T23:53:47.253 INFO:tasks.workunit.client.0.smithi060.stdout:removed './133/11' 2024-04-14T23:53:47.254 INFO:tasks.workunit.client.0.smithi060.stdout:removed './133/16' 2024-04-14T23:53:47.254 INFO:tasks.workunit.client.0.smithi060.stdout:removed './133/32' 2024-04-14T23:53:47.254 INFO:tasks.workunit.client.0.smithi060.stdout:removed './133/30' 2024-04-14T23:53:47.254 INFO:tasks.workunit.client.0.smithi060.stdout:removed './133/14' 2024-04-14T23:53:47.254 INFO:tasks.workunit.client.0.smithi060.stdout:removed './133/18' 2024-04-14T23:53:47.254 INFO:tasks.workunit.client.0.smithi060.stdout:removed './133/13' 2024-04-14T23:53:47.254 INFO:tasks.workunit.client.0.smithi060.stdout:removed './133/33' 2024-04-14T23:53:47.254 INFO:tasks.workunit.client.0.smithi060.stdout:removed './133/17' 2024-04-14T23:53:47.254 INFO:tasks.workunit.client.0.smithi060.stdout:removed './133/35' 2024-04-14T23:53:47.254 INFO:tasks.workunit.client.0.smithi060.stdout:removed './133/37' 2024-04-14T23:53:47.254 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './133' 2024-04-14T23:53:47.254 INFO:tasks.workunit.client.0.smithi060.stdout:removed './134/19' 2024-04-14T23:53:47.254 INFO:tasks.workunit.client.0.smithi060.stdout:removed './134/15' 2024-04-14T23:53:47.254 INFO:tasks.workunit.client.0.smithi060.stdout:removed './134/31' 2024-04-14T23:53:47.254 INFO:tasks.workunit.client.0.smithi060.stdout:removed './134/36' 2024-04-14T23:53:47.255 INFO:tasks.workunit.client.0.smithi060.stdout:removed './134/20' 2024-04-14T23:53:47.255 INFO:tasks.workunit.client.0.smithi060.stdout:removed './134/12' 2024-04-14T23:53:47.255 INFO:tasks.workunit.client.0.smithi060.stdout:removed './134/39' 2024-04-14T23:53:47.255 INFO:tasks.workunit.client.0.smithi060.stdout:removed './134/10' 2024-04-14T23:53:47.255 INFO:tasks.workunit.client.0.smithi060.stdout:removed './134/34' 2024-04-14T23:53:47.255 INFO:tasks.workunit.client.0.smithi060.stdout:removed './134/38' 2024-04-14T23:53:47.255 INFO:tasks.workunit.client.0.smithi060.stdout:removed './134/11' 2024-04-14T23:53:47.255 INFO:tasks.workunit.client.0.smithi060.stdout:removed './134/16' 2024-04-14T23:53:47.255 INFO:tasks.workunit.client.0.smithi060.stdout:removed './134/32' 2024-04-14T23:53:47.255 INFO:tasks.workunit.client.0.smithi060.stdout:removed './134/30' 2024-04-14T23:53:47.255 INFO:tasks.workunit.client.0.smithi060.stdout:removed './134/14' 2024-04-14T23:53:47.255 INFO:tasks.workunit.client.0.smithi060.stdout:removed './134/18' 2024-04-14T23:53:47.255 INFO:tasks.workunit.client.0.smithi060.stdout:removed './134/13' 2024-04-14T23:53:47.255 INFO:tasks.workunit.client.0.smithi060.stdout:removed './134/33' 2024-04-14T23:53:47.255 INFO:tasks.workunit.client.0.smithi060.stdout:removed './134/17' 2024-04-14T23:53:47.256 INFO:tasks.workunit.client.0.smithi060.stdout:removed './134/35' 2024-04-14T23:53:47.256 INFO:tasks.workunit.client.0.smithi060.stdout:removed './134/37' 2024-04-14T23:53:47.256 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './134' 2024-04-14T23:53:47.256 INFO:tasks.workunit.client.0.smithi060.stdout:removed './135/19' 2024-04-14T23:53:47.256 INFO:tasks.workunit.client.0.smithi060.stdout:removed './135/15' 2024-04-14T23:53:47.256 INFO:tasks.workunit.client.0.smithi060.stdout:removed './135/31' 2024-04-14T23:53:47.256 INFO:tasks.workunit.client.0.smithi060.stdout:removed './135/36' 2024-04-14T23:53:47.256 INFO:tasks.workunit.client.0.smithi060.stdout:removed './135/20' 2024-04-14T23:53:47.256 INFO:tasks.workunit.client.0.smithi060.stdout:removed './135/12' 2024-04-14T23:53:47.256 INFO:tasks.workunit.client.0.smithi060.stdout:removed './135/39' 2024-04-14T23:53:47.256 INFO:tasks.workunit.client.0.smithi060.stdout:removed './135/10' 2024-04-14T23:53:47.256 INFO:tasks.workunit.client.0.smithi060.stdout:removed './135/34' 2024-04-14T23:53:47.256 INFO:tasks.workunit.client.0.smithi060.stdout:removed './135/38' 2024-04-14T23:53:47.256 INFO:tasks.workunit.client.0.smithi060.stdout:removed './135/11' 2024-04-14T23:53:47.256 INFO:tasks.workunit.client.0.smithi060.stdout:removed './135/16' 2024-04-14T23:53:47.257 INFO:tasks.workunit.client.0.smithi060.stdout:removed './135/32' 2024-04-14T23:53:47.257 INFO:tasks.workunit.client.0.smithi060.stdout:removed './135/30' 2024-04-14T23:53:47.257 INFO:tasks.workunit.client.0.smithi060.stdout:removed './135/14' 2024-04-14T23:53:47.257 INFO:tasks.workunit.client.0.smithi060.stdout:removed './135/18' 2024-04-14T23:53:47.257 INFO:tasks.workunit.client.0.smithi060.stdout:removed './135/13' 2024-04-14T23:53:47.257 INFO:tasks.workunit.client.0.smithi060.stdout:removed './135/33' 2024-04-14T23:53:47.257 INFO:tasks.workunit.client.0.smithi060.stdout:removed './135/17' 2024-04-14T23:53:47.257 INFO:tasks.workunit.client.0.smithi060.stdout:removed './135/35' 2024-04-14T23:53:47.257 INFO:tasks.workunit.client.0.smithi060.stdout:removed './135/37' 2024-04-14T23:53:47.257 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './135' 2024-04-14T23:53:47.257 INFO:tasks.workunit.client.0.smithi060.stdout:removed './136/19' 2024-04-14T23:53:47.257 INFO:tasks.workunit.client.0.smithi060.stdout:removed './136/15' 2024-04-14T23:53:47.257 INFO:tasks.workunit.client.0.smithi060.stdout:removed './136/31' 2024-04-14T23:53:47.257 INFO:tasks.workunit.client.0.smithi060.stdout:removed './136/36' 2024-04-14T23:53:47.257 INFO:tasks.workunit.client.0.smithi060.stdout:removed './136/20' 2024-04-14T23:53:47.258 INFO:tasks.workunit.client.0.smithi060.stdout:removed './136/12' 2024-04-14T23:53:47.258 INFO:tasks.workunit.client.0.smithi060.stdout:removed './136/39' 2024-04-14T23:53:47.258 INFO:tasks.workunit.client.0.smithi060.stdout:removed './136/10' 2024-04-14T23:53:47.258 INFO:tasks.workunit.client.0.smithi060.stdout:removed './136/34' 2024-04-14T23:53:47.258 INFO:tasks.workunit.client.0.smithi060.stdout:removed './136/38' 2024-04-14T23:53:47.258 INFO:tasks.workunit.client.0.smithi060.stdout:removed './136/11' 2024-04-14T23:53:47.258 INFO:tasks.workunit.client.0.smithi060.stdout:removed './136/16' 2024-04-14T23:53:47.258 INFO:tasks.workunit.client.0.smithi060.stdout:removed './136/32' 2024-04-14T23:53:47.258 INFO:tasks.workunit.client.0.smithi060.stdout:removed './136/30' 2024-04-14T23:53:47.258 INFO:tasks.workunit.client.0.smithi060.stdout:removed './136/14' 2024-04-14T23:53:47.258 INFO:tasks.workunit.client.0.smithi060.stdout:removed './136/18' 2024-04-14T23:53:47.258 INFO:tasks.workunit.client.0.smithi060.stdout:removed './136/13' 2024-04-14T23:53:47.258 INFO:tasks.workunit.client.0.smithi060.stdout:removed './136/33' 2024-04-14T23:53:47.258 INFO:tasks.workunit.client.0.smithi060.stdout:removed './136/17' 2024-04-14T23:53:47.258 INFO:tasks.workunit.client.0.smithi060.stdout:removed './136/35' 2024-04-14T23:53:47.259 INFO:tasks.workunit.client.0.smithi060.stdout:removed './136/37' 2024-04-14T23:53:47.259 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './136' 2024-04-14T23:53:47.259 INFO:tasks.workunit.client.0.smithi060.stdout:removed './137/19' 2024-04-14T23:53:47.259 INFO:tasks.workunit.client.0.smithi060.stdout:removed './137/15' 2024-04-14T23:53:47.259 INFO:tasks.workunit.client.0.smithi060.stdout:removed './137/31' 2024-04-14T23:53:47.259 INFO:tasks.workunit.client.0.smithi060.stdout:removed './137/36' 2024-04-14T23:53:47.259 INFO:tasks.workunit.client.0.smithi060.stdout:removed './137/20' 2024-04-14T23:53:47.259 INFO:tasks.workunit.client.0.smithi060.stdout:removed './137/12' 2024-04-14T23:53:47.259 INFO:tasks.workunit.client.0.smithi060.stdout:removed './137/39' 2024-04-14T23:53:47.259 INFO:tasks.workunit.client.0.smithi060.stdout:removed './137/10' 2024-04-14T23:53:47.259 INFO:tasks.workunit.client.0.smithi060.stdout:removed './137/34' 2024-04-14T23:53:47.259 INFO:tasks.workunit.client.0.smithi060.stdout:removed './137/38' 2024-04-14T23:53:47.259 INFO:tasks.workunit.client.0.smithi060.stdout:removed './137/11' 2024-04-14T23:53:47.259 INFO:tasks.workunit.client.0.smithi060.stdout:removed './137/16' 2024-04-14T23:53:47.259 INFO:tasks.workunit.client.0.smithi060.stdout:removed './137/32' 2024-04-14T23:53:47.259 INFO:tasks.workunit.client.0.smithi060.stdout:removed './137/30' 2024-04-14T23:53:47.260 INFO:tasks.workunit.client.0.smithi060.stdout:removed './137/14' 2024-04-14T23:53:47.260 INFO:tasks.workunit.client.0.smithi060.stdout:removed './137/18' 2024-04-14T23:53:47.260 INFO:tasks.workunit.client.0.smithi060.stdout:removed './137/13' 2024-04-14T23:53:47.260 INFO:tasks.workunit.client.0.smithi060.stdout:removed './137/33' 2024-04-14T23:53:47.260 INFO:tasks.workunit.client.0.smithi060.stdout:removed './137/17' 2024-04-14T23:53:47.260 INFO:tasks.workunit.client.0.smithi060.stdout:removed './137/35' 2024-04-14T23:53:47.260 INFO:tasks.workunit.client.0.smithi060.stdout:removed './137/37' 2024-04-14T23:53:47.260 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './137' 2024-04-14T23:53:47.260 INFO:tasks.workunit.client.0.smithi060.stdout:removed './138/19' 2024-04-14T23:53:47.260 INFO:tasks.workunit.client.0.smithi060.stdout:removed './138/15' 2024-04-14T23:53:47.260 INFO:tasks.workunit.client.0.smithi060.stdout:removed './138/31' 2024-04-14T23:53:47.261 INFO:tasks.workunit.client.0.smithi060.stdout:removed './138/36' 2024-04-14T23:53:47.261 INFO:tasks.workunit.client.0.smithi060.stdout:removed './138/20' 2024-04-14T23:53:47.261 INFO:tasks.workunit.client.0.smithi060.stdout:removed './138/12' 2024-04-14T23:53:47.261 INFO:tasks.workunit.client.0.smithi060.stdout:removed './138/39' 2024-04-14T23:53:47.261 INFO:tasks.workunit.client.0.smithi060.stdout:removed './138/10' 2024-04-14T23:53:47.261 INFO:tasks.workunit.client.0.smithi060.stdout:removed './138/34' 2024-04-14T23:53:47.261 INFO:tasks.workunit.client.0.smithi060.stdout:removed './138/38' 2024-04-14T23:53:47.261 INFO:tasks.workunit.client.0.smithi060.stdout:removed './138/11' 2024-04-14T23:53:47.261 INFO:tasks.workunit.client.0.smithi060.stdout:removed './138/16' 2024-04-14T23:53:48.260 INFO:tasks.workunit.client.0.smithi060.stdout:removed './138/32' 2024-04-14T23:53:48.261 INFO:tasks.workunit.client.0.smithi060.stdout:removed './138/30' 2024-04-14T23:53:48.261 INFO:tasks.workunit.client.0.smithi060.stdout:removed './138/14' 2024-04-14T23:53:48.261 INFO:tasks.workunit.client.0.smithi060.stdout:removed './138/18' 2024-04-14T23:53:48.261 INFO:tasks.workunit.client.0.smithi060.stdout:removed './138/13' 2024-04-14T23:53:48.261 INFO:tasks.workunit.client.0.smithi060.stdout:removed './138/33' 2024-04-14T23:53:48.261 INFO:tasks.workunit.client.0.smithi060.stdout:removed './138/17' 2024-04-14T23:53:48.261 INFO:tasks.workunit.client.0.smithi060.stdout:removed './138/35' 2024-04-14T23:53:48.261 INFO:tasks.workunit.client.0.smithi060.stdout:removed './138/37' 2024-04-14T23:53:48.261 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './138' 2024-04-14T23:53:48.261 INFO:tasks.workunit.client.0.smithi060.stdout:removed './139/19' 2024-04-14T23:53:48.261 INFO:tasks.workunit.client.0.smithi060.stdout:removed './139/15' 2024-04-14T23:53:48.261 INFO:tasks.workunit.client.0.smithi060.stdout:removed './139/31' 2024-04-14T23:53:48.261 INFO:tasks.workunit.client.0.smithi060.stdout:removed './139/36' 2024-04-14T23:53:48.261 INFO:tasks.workunit.client.0.smithi060.stdout:removed './139/20' 2024-04-14T23:53:48.262 INFO:tasks.workunit.client.0.smithi060.stdout:removed './139/12' 2024-04-14T23:53:48.262 INFO:tasks.workunit.client.0.smithi060.stdout:removed './139/39' 2024-04-14T23:53:48.262 INFO:tasks.workunit.client.0.smithi060.stdout:removed './139/10' 2024-04-14T23:53:48.262 INFO:tasks.workunit.client.0.smithi060.stdout:removed './139/34' 2024-04-14T23:53:48.262 INFO:tasks.workunit.client.0.smithi060.stdout:removed './139/38' 2024-04-14T23:53:48.262 INFO:tasks.workunit.client.0.smithi060.stdout:removed './139/11' 2024-04-14T23:53:48.262 INFO:tasks.workunit.client.0.smithi060.stdout:removed './139/16' 2024-04-14T23:53:48.262 INFO:tasks.workunit.client.0.smithi060.stdout:removed './139/32' 2024-04-14T23:53:48.262 INFO:tasks.workunit.client.0.smithi060.stdout:removed './139/30' 2024-04-14T23:53:48.262 INFO:tasks.workunit.client.0.smithi060.stdout:removed './139/14' 2024-04-14T23:53:48.262 INFO:tasks.workunit.client.0.smithi060.stdout:removed './139/18' 2024-04-14T23:53:48.262 INFO:tasks.workunit.client.0.smithi060.stdout:removed './139/13' 2024-04-14T23:53:48.262 INFO:tasks.workunit.client.0.smithi060.stdout:removed './139/33' 2024-04-14T23:53:48.262 INFO:tasks.workunit.client.0.smithi060.stdout:removed './139/17' 2024-04-14T23:53:48.262 INFO:tasks.workunit.client.0.smithi060.stdout:removed './139/35' 2024-04-14T23:53:48.263 INFO:tasks.workunit.client.0.smithi060.stdout:removed './139/37' 2024-04-14T23:53:48.263 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './139' 2024-04-14T23:53:48.263 INFO:tasks.workunit.client.0.smithi060.stdout:removed './140/19' 2024-04-14T23:53:48.263 INFO:tasks.workunit.client.0.smithi060.stdout:removed './140/15' 2024-04-14T23:53:48.263 INFO:tasks.workunit.client.0.smithi060.stdout:removed './140/31' 2024-04-14T23:53:48.263 INFO:tasks.workunit.client.0.smithi060.stdout:removed './140/36' 2024-04-14T23:53:48.263 INFO:tasks.workunit.client.0.smithi060.stdout:removed './140/20' 2024-04-14T23:53:48.263 INFO:tasks.workunit.client.0.smithi060.stdout:removed './140/12' 2024-04-14T23:53:48.263 INFO:tasks.workunit.client.0.smithi060.stdout:removed './140/39' 2024-04-14T23:53:48.263 INFO:tasks.workunit.client.0.smithi060.stdout:removed './140/10' 2024-04-14T23:53:48.263 INFO:tasks.workunit.client.0.smithi060.stdout:removed './140/34' 2024-04-14T23:53:48.263 INFO:tasks.workunit.client.0.smithi060.stdout:removed './140/38' 2024-04-14T23:53:48.263 INFO:tasks.workunit.client.0.smithi060.stdout:removed './140/11' 2024-04-14T23:53:48.263 INFO:tasks.workunit.client.0.smithi060.stdout:removed './140/16' 2024-04-14T23:53:48.264 INFO:tasks.workunit.client.0.smithi060.stdout:removed './140/32' 2024-04-14T23:53:48.264 INFO:tasks.workunit.client.0.smithi060.stdout:removed './140/30' 2024-04-14T23:53:48.264 INFO:tasks.workunit.client.0.smithi060.stdout:removed './140/14' 2024-04-14T23:53:48.264 INFO:tasks.workunit.client.0.smithi060.stdout:removed './140/18' 2024-04-14T23:53:48.264 INFO:tasks.workunit.client.0.smithi060.stdout:removed './140/13' 2024-04-14T23:53:48.264 INFO:tasks.workunit.client.0.smithi060.stdout:removed './140/33' 2024-04-14T23:53:48.264 INFO:tasks.workunit.client.0.smithi060.stdout:removed './140/17' 2024-04-14T23:53:48.264 INFO:tasks.workunit.client.0.smithi060.stdout:removed './140/35' 2024-04-14T23:53:48.264 INFO:tasks.workunit.client.0.smithi060.stdout:removed './140/37' 2024-04-14T23:53:48.264 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './140' 2024-04-14T23:53:48.264 INFO:tasks.workunit.client.0.smithi060.stdout:removed './141/19' 2024-04-14T23:53:48.264 INFO:tasks.workunit.client.0.smithi060.stdout:removed './141/15' 2024-04-14T23:53:48.264 INFO:tasks.workunit.client.0.smithi060.stdout:removed './141/31' 2024-04-14T23:53:48.264 INFO:tasks.workunit.client.0.smithi060.stdout:removed './141/36' 2024-04-14T23:53:48.264 INFO:tasks.workunit.client.0.smithi060.stdout:removed './141/20' 2024-04-14T23:53:48.265 INFO:tasks.workunit.client.0.smithi060.stdout:removed './141/12' 2024-04-14T23:53:48.265 INFO:tasks.workunit.client.0.smithi060.stdout:removed './141/39' 2024-04-14T23:53:48.265 INFO:tasks.workunit.client.0.smithi060.stdout:removed './141/10' 2024-04-14T23:53:48.265 INFO:tasks.workunit.client.0.smithi060.stdout:removed './141/34' 2024-04-14T23:53:48.265 INFO:tasks.workunit.client.0.smithi060.stdout:removed './141/38' 2024-04-14T23:53:48.265 INFO:tasks.workunit.client.0.smithi060.stdout:removed './141/11' 2024-04-14T23:53:48.265 INFO:tasks.workunit.client.0.smithi060.stdout:removed './141/16' 2024-04-14T23:53:48.265 INFO:tasks.workunit.client.0.smithi060.stdout:removed './141/32' 2024-04-14T23:53:48.265 INFO:tasks.workunit.client.0.smithi060.stdout:removed './141/30' 2024-04-14T23:53:48.265 INFO:tasks.workunit.client.0.smithi060.stdout:removed './141/14' 2024-04-14T23:53:48.265 INFO:tasks.workunit.client.0.smithi060.stdout:removed './141/18' 2024-04-14T23:53:48.265 INFO:tasks.workunit.client.0.smithi060.stdout:removed './141/13' 2024-04-14T23:53:48.265 INFO:tasks.workunit.client.0.smithi060.stdout:removed './141/33' 2024-04-14T23:53:48.265 INFO:tasks.workunit.client.0.smithi060.stdout:removed './141/17' 2024-04-14T23:53:48.265 INFO:tasks.workunit.client.0.smithi060.stdout:removed './141/35' 2024-04-14T23:53:48.266 INFO:tasks.workunit.client.0.smithi060.stdout:removed './141/37' 2024-04-14T23:53:48.266 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './141' 2024-04-14T23:53:48.266 INFO:tasks.workunit.client.0.smithi060.stdout:removed './142/19' 2024-04-14T23:53:48.266 INFO:tasks.workunit.client.0.smithi060.stdout:removed './142/15' 2024-04-14T23:53:48.266 INFO:tasks.workunit.client.0.smithi060.stdout:removed './142/31' 2024-04-14T23:53:48.266 INFO:tasks.workunit.client.0.smithi060.stdout:removed './142/36' 2024-04-14T23:53:48.266 INFO:tasks.workunit.client.0.smithi060.stdout:removed './142/20' 2024-04-14T23:53:48.266 INFO:tasks.workunit.client.0.smithi060.stdout:removed './142/12' 2024-04-14T23:53:48.266 INFO:tasks.workunit.client.0.smithi060.stdout:removed './142/39' 2024-04-14T23:53:48.266 INFO:tasks.workunit.client.0.smithi060.stdout:removed './142/10' 2024-04-14T23:53:48.266 INFO:tasks.workunit.client.0.smithi060.stdout:removed './142/34' 2024-04-14T23:53:48.266 INFO:tasks.workunit.client.0.smithi060.stdout:removed './142/38' 2024-04-14T23:53:48.266 INFO:tasks.workunit.client.0.smithi060.stdout:removed './142/11' 2024-04-14T23:53:48.266 INFO:tasks.workunit.client.0.smithi060.stdout:removed './142/16' 2024-04-14T23:53:48.267 INFO:tasks.workunit.client.0.smithi060.stdout:removed './142/32' 2024-04-14T23:53:48.267 INFO:tasks.workunit.client.0.smithi060.stdout:removed './142/30' 2024-04-14T23:53:48.267 INFO:tasks.workunit.client.0.smithi060.stdout:removed './142/14' 2024-04-14T23:53:48.267 INFO:tasks.workunit.client.0.smithi060.stdout:removed './142/18' 2024-04-14T23:53:48.267 INFO:tasks.workunit.client.0.smithi060.stdout:removed './142/13' 2024-04-14T23:53:48.267 INFO:tasks.workunit.client.0.smithi060.stdout:removed './142/33' 2024-04-14T23:53:48.267 INFO:tasks.workunit.client.0.smithi060.stdout:removed './142/17' 2024-04-14T23:53:48.267 INFO:tasks.workunit.client.0.smithi060.stdout:removed './142/35' 2024-04-14T23:53:48.267 INFO:tasks.workunit.client.0.smithi060.stdout:removed './142/37' 2024-04-14T23:53:48.267 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './142' 2024-04-14T23:53:48.267 INFO:tasks.workunit.client.0.smithi060.stdout:removed './143/19' 2024-04-14T23:53:48.267 INFO:tasks.workunit.client.0.smithi060.stdout:removed './143/15' 2024-04-14T23:53:48.267 INFO:tasks.workunit.client.0.smithi060.stdout:removed './143/31' 2024-04-14T23:53:48.267 INFO:tasks.workunit.client.0.smithi060.stdout:removed './143/36' 2024-04-14T23:53:48.267 INFO:tasks.workunit.client.0.smithi060.stdout:removed './143/20' 2024-04-14T23:53:48.268 INFO:tasks.workunit.client.0.smithi060.stdout:removed './143/12' 2024-04-14T23:53:48.268 INFO:tasks.workunit.client.0.smithi060.stdout:removed './143/39' 2024-04-14T23:53:48.268 INFO:tasks.workunit.client.0.smithi060.stdout:removed './143/10' 2024-04-14T23:53:48.268 INFO:tasks.workunit.client.0.smithi060.stdout:removed './143/34' 2024-04-14T23:53:48.268 INFO:tasks.workunit.client.0.smithi060.stdout:removed './143/38' 2024-04-14T23:53:48.268 INFO:tasks.workunit.client.0.smithi060.stdout:removed './143/11' 2024-04-14T23:53:48.268 INFO:tasks.workunit.client.0.smithi060.stdout:removed './143/16' 2024-04-14T23:53:48.268 INFO:tasks.workunit.client.0.smithi060.stdout:removed './143/32' 2024-04-14T23:53:48.268 INFO:tasks.workunit.client.0.smithi060.stdout:removed './143/30' 2024-04-14T23:53:48.268 INFO:tasks.workunit.client.0.smithi060.stdout:removed './143/14' 2024-04-14T23:53:48.268 INFO:tasks.workunit.client.0.smithi060.stdout:removed './143/18' 2024-04-14T23:53:48.268 INFO:tasks.workunit.client.0.smithi060.stdout:removed './143/13' 2024-04-14T23:53:48.268 INFO:tasks.workunit.client.0.smithi060.stdout:removed './143/33' 2024-04-14T23:53:48.268 INFO:tasks.workunit.client.0.smithi060.stdout:removed './143/17' 2024-04-14T23:53:48.269 INFO:tasks.workunit.client.0.smithi060.stdout:removed './143/35' 2024-04-14T23:53:48.269 INFO:tasks.workunit.client.0.smithi060.stdout:removed './143/37' 2024-04-14T23:53:48.269 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './143' 2024-04-14T23:53:48.269 INFO:tasks.workunit.client.0.smithi060.stdout:removed './144/19' 2024-04-14T23:53:48.269 INFO:tasks.workunit.client.0.smithi060.stdout:removed './144/15' 2024-04-14T23:53:48.269 INFO:tasks.workunit.client.0.smithi060.stdout:removed './144/31' 2024-04-14T23:53:48.269 INFO:tasks.workunit.client.0.smithi060.stdout:removed './144/36' 2024-04-14T23:53:48.269 INFO:tasks.workunit.client.0.smithi060.stdout:removed './144/20' 2024-04-14T23:53:48.269 INFO:tasks.workunit.client.0.smithi060.stdout:removed './144/12' 2024-04-14T23:53:48.269 INFO:tasks.workunit.client.0.smithi060.stdout:removed './144/39' 2024-04-14T23:53:48.269 INFO:tasks.workunit.client.0.smithi060.stdout:removed './144/10' 2024-04-14T23:53:48.270 INFO:tasks.workunit.client.0.smithi060.stdout:removed './144/34' 2024-04-14T23:53:48.270 INFO:tasks.workunit.client.0.smithi060.stdout:removed './144/38' 2024-04-14T23:53:48.270 INFO:tasks.workunit.client.0.smithi060.stdout:removed './144/11' 2024-04-14T23:53:48.270 INFO:tasks.workunit.client.0.smithi060.stdout:removed './144/16' 2024-04-14T23:53:48.270 INFO:tasks.workunit.client.0.smithi060.stdout:removed './144/32' 2024-04-14T23:53:48.270 INFO:tasks.workunit.client.0.smithi060.stdout:removed './144/30' 2024-04-14T23:53:48.270 INFO:tasks.workunit.client.0.smithi060.stdout:removed './144/14' 2024-04-14T23:53:48.270 INFO:tasks.workunit.client.0.smithi060.stdout:removed './144/18' 2024-04-14T23:53:48.270 INFO:tasks.workunit.client.0.smithi060.stdout:removed './144/13' 2024-04-14T23:53:48.270 INFO:tasks.workunit.client.0.smithi060.stdout:removed './144/33' 2024-04-14T23:53:48.270 INFO:tasks.workunit.client.0.smithi060.stdout:removed './144/17' 2024-04-14T23:53:48.270 INFO:tasks.workunit.client.0.smithi060.stdout:removed './144/35' 2024-04-14T23:53:48.270 INFO:tasks.workunit.client.0.smithi060.stdout:removed './144/37' 2024-04-14T23:53:48.270 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './144' 2024-04-14T23:53:48.271 INFO:tasks.workunit.client.0.smithi060.stdout:removed './145/19' 2024-04-14T23:53:48.271 INFO:tasks.workunit.client.0.smithi060.stdout:removed './145/15' 2024-04-14T23:53:48.271 INFO:tasks.workunit.client.0.smithi060.stdout:removed './145/31' 2024-04-14T23:53:48.271 INFO:tasks.workunit.client.0.smithi060.stdout:removed './145/36' 2024-04-14T23:53:48.271 INFO:tasks.workunit.client.0.smithi060.stdout:removed './145/20' 2024-04-14T23:53:48.271 INFO:tasks.workunit.client.0.smithi060.stdout:removed './145/12' 2024-04-14T23:53:48.271 INFO:tasks.workunit.client.0.smithi060.stdout:removed './145/39' 2024-04-14T23:53:48.271 INFO:tasks.workunit.client.0.smithi060.stdout:removed './145/10' 2024-04-14T23:53:48.271 INFO:tasks.workunit.client.0.smithi060.stdout:removed './145/34' 2024-04-14T23:53:48.271 INFO:tasks.workunit.client.0.smithi060.stdout:removed './145/38' 2024-04-14T23:53:48.271 INFO:tasks.workunit.client.0.smithi060.stdout:removed './145/11' 2024-04-14T23:53:48.271 INFO:tasks.workunit.client.0.smithi060.stdout:removed './145/16' 2024-04-14T23:53:48.271 INFO:tasks.workunit.client.0.smithi060.stdout:removed './145/32' 2024-04-14T23:53:48.271 INFO:tasks.workunit.client.0.smithi060.stdout:removed './145/30' 2024-04-14T23:53:48.271 INFO:tasks.workunit.client.0.smithi060.stdout:removed './145/14' 2024-04-14T23:53:48.271 INFO:tasks.workunit.client.0.smithi060.stdout:removed './145/18' 2024-04-14T23:53:48.272 INFO:tasks.workunit.client.0.smithi060.stdout:removed './145/13' 2024-04-14T23:53:48.272 INFO:tasks.workunit.client.0.smithi060.stdout:removed './145/33' 2024-04-14T23:53:48.272 INFO:tasks.workunit.client.0.smithi060.stdout:removed './145/17' 2024-04-14T23:53:48.272 INFO:tasks.workunit.client.0.smithi060.stdout:removed './145/35' 2024-04-14T23:53:48.272 INFO:tasks.workunit.client.0.smithi060.stdout:removed './145/37' 2024-04-14T23:53:48.272 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './145' 2024-04-14T23:53:48.272 INFO:tasks.workunit.client.0.smithi060.stdout:removed './146/19' 2024-04-14T23:53:48.272 INFO:tasks.workunit.client.0.smithi060.stdout:removed './146/15' 2024-04-14T23:53:48.272 INFO:tasks.workunit.client.0.smithi060.stdout:removed './146/31' 2024-04-14T23:53:48.272 INFO:tasks.workunit.client.0.smithi060.stdout:removed './146/36' 2024-04-14T23:53:48.272 INFO:tasks.workunit.client.0.smithi060.stdout:removed './146/20' 2024-04-14T23:53:48.272 INFO:tasks.workunit.client.0.smithi060.stdout:removed './146/12' 2024-04-14T23:53:48.272 INFO:tasks.workunit.client.0.smithi060.stdout:removed './146/39' 2024-04-14T23:53:48.272 INFO:tasks.workunit.client.0.smithi060.stdout:removed './146/10' 2024-04-14T23:53:48.272 INFO:tasks.workunit.client.0.smithi060.stdout:removed './146/34' 2024-04-14T23:53:48.273 INFO:tasks.workunit.client.0.smithi060.stdout:removed './146/38' 2024-04-14T23:53:48.273 INFO:tasks.workunit.client.0.smithi060.stdout:removed './146/11' 2024-04-14T23:53:48.273 INFO:tasks.workunit.client.0.smithi060.stdout:removed './146/16' 2024-04-14T23:53:48.273 INFO:tasks.workunit.client.0.smithi060.stdout:removed './146/32' 2024-04-14T23:53:48.273 INFO:tasks.workunit.client.0.smithi060.stdout:removed './146/30' 2024-04-14T23:53:48.273 INFO:tasks.workunit.client.0.smithi060.stdout:removed './146/14' 2024-04-14T23:53:48.273 INFO:tasks.workunit.client.0.smithi060.stdout:removed './146/18' 2024-04-14T23:53:48.273 INFO:tasks.workunit.client.0.smithi060.stdout:removed './146/13' 2024-04-14T23:53:48.273 INFO:tasks.workunit.client.0.smithi060.stdout:removed './146/33' 2024-04-14T23:53:48.273 INFO:tasks.workunit.client.0.smithi060.stdout:removed './146/17' 2024-04-14T23:53:48.273 INFO:tasks.workunit.client.0.smithi060.stdout:removed './146/35' 2024-04-14T23:53:48.273 INFO:tasks.workunit.client.0.smithi060.stdout:removed './146/37' 2024-04-14T23:53:48.273 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './146' 2024-04-14T23:53:48.273 INFO:tasks.workunit.client.0.smithi060.stdout:removed './147/19' 2024-04-14T23:53:48.273 INFO:tasks.workunit.client.0.smithi060.stdout:removed './147/15' 2024-04-14T23:53:48.274 INFO:tasks.workunit.client.0.smithi060.stdout:removed './147/31' 2024-04-14T23:53:48.274 INFO:tasks.workunit.client.0.smithi060.stdout:removed './147/36' 2024-04-14T23:53:48.274 INFO:tasks.workunit.client.0.smithi060.stdout:removed './147/20' 2024-04-14T23:53:48.274 INFO:tasks.workunit.client.0.smithi060.stdout:removed './147/12' 2024-04-14T23:53:48.274 INFO:tasks.workunit.client.0.smithi060.stdout:removed './147/39' 2024-04-14T23:53:48.274 INFO:tasks.workunit.client.0.smithi060.stdout:removed './147/10' 2024-04-14T23:53:48.274 INFO:tasks.workunit.client.0.smithi060.stdout:removed './147/34' 2024-04-14T23:53:48.274 INFO:tasks.workunit.client.0.smithi060.stdout:removed './147/38' 2024-04-14T23:53:48.274 INFO:tasks.workunit.client.0.smithi060.stdout:removed './147/11' 2024-04-14T23:53:48.274 INFO:tasks.workunit.client.0.smithi060.stdout:removed './147/16' 2024-04-14T23:53:48.274 INFO:tasks.workunit.client.0.smithi060.stdout:removed './147/32' 2024-04-14T23:53:48.274 INFO:tasks.workunit.client.0.smithi060.stdout:removed './147/30' 2024-04-14T23:53:48.274 INFO:tasks.workunit.client.0.smithi060.stdout:removed './147/14' 2024-04-14T23:53:48.274 INFO:tasks.workunit.client.0.smithi060.stdout:removed './147/18' 2024-04-14T23:53:48.275 INFO:tasks.workunit.client.0.smithi060.stdout:removed './147/13' 2024-04-14T23:53:48.275 INFO:tasks.workunit.client.0.smithi060.stdout:removed './147/33' 2024-04-14T23:53:48.275 INFO:tasks.workunit.client.0.smithi060.stdout:removed './147/17' 2024-04-14T23:53:48.275 INFO:tasks.workunit.client.0.smithi060.stdout:removed './147/35' 2024-04-14T23:53:48.275 INFO:tasks.workunit.client.0.smithi060.stdout:removed './147/37' 2024-04-14T23:53:48.275 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './147' 2024-04-14T23:53:48.275 INFO:tasks.workunit.client.0.smithi060.stdout:removed './148/19' 2024-04-14T23:53:48.275 INFO:tasks.workunit.client.0.smithi060.stdout:removed './148/15' 2024-04-14T23:53:48.275 INFO:tasks.workunit.client.0.smithi060.stdout:removed './148/31' 2024-04-14T23:53:48.275 INFO:tasks.workunit.client.0.smithi060.stdout:removed './148/36' 2024-04-14T23:53:50.271 INFO:tasks.workunit.client.0.smithi060.stdout:removed './148/20' 2024-04-14T23:53:50.271 INFO:tasks.workunit.client.0.smithi060.stdout:removed './148/12' 2024-04-14T23:53:50.271 INFO:tasks.workunit.client.0.smithi060.stdout:removed './148/39' 2024-04-14T23:53:50.271 INFO:tasks.workunit.client.0.smithi060.stdout:removed './148/10' 2024-04-14T23:53:50.271 INFO:tasks.workunit.client.0.smithi060.stdout:removed './148/34' 2024-04-14T23:53:50.271 INFO:tasks.workunit.client.0.smithi060.stdout:removed './148/38' 2024-04-14T23:53:50.272 INFO:tasks.workunit.client.0.smithi060.stdout:removed './148/11' 2024-04-14T23:53:50.272 INFO:tasks.workunit.client.0.smithi060.stdout:removed './148/16' 2024-04-14T23:53:50.272 INFO:tasks.workunit.client.0.smithi060.stdout:removed './148/32' 2024-04-14T23:53:50.272 INFO:tasks.workunit.client.0.smithi060.stdout:removed './148/30' 2024-04-14T23:53:50.272 INFO:tasks.workunit.client.0.smithi060.stdout:removed './148/14' 2024-04-14T23:53:50.272 INFO:tasks.workunit.client.0.smithi060.stdout:removed './148/18' 2024-04-14T23:53:50.272 INFO:tasks.workunit.client.0.smithi060.stdout:removed './148/13' 2024-04-14T23:53:50.272 INFO:tasks.workunit.client.0.smithi060.stdout:removed './148/33' 2024-04-14T23:53:50.272 INFO:tasks.workunit.client.0.smithi060.stdout:removed './148/17' 2024-04-14T23:53:50.272 INFO:tasks.workunit.client.0.smithi060.stdout:removed './148/35' 2024-04-14T23:53:50.272 INFO:tasks.workunit.client.0.smithi060.stdout:removed './148/37' 2024-04-14T23:53:50.272 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './148' 2024-04-14T23:53:50.272 INFO:tasks.workunit.client.0.smithi060.stdout:removed './149/19' 2024-04-14T23:53:50.272 INFO:tasks.workunit.client.0.smithi060.stdout:removed './149/15' 2024-04-14T23:53:50.273 INFO:tasks.workunit.client.0.smithi060.stdout:removed './149/31' 2024-04-14T23:53:50.273 INFO:tasks.workunit.client.0.smithi060.stdout:removed './149/36' 2024-04-14T23:53:50.273 INFO:tasks.workunit.client.0.smithi060.stdout:removed './149/20' 2024-04-14T23:53:50.273 INFO:tasks.workunit.client.0.smithi060.stdout:removed './149/12' 2024-04-14T23:53:50.273 INFO:tasks.workunit.client.0.smithi060.stdout:removed './149/39' 2024-04-14T23:53:50.273 INFO:tasks.workunit.client.0.smithi060.stdout:removed './149/10' 2024-04-14T23:53:50.273 INFO:tasks.workunit.client.0.smithi060.stdout:removed './149/34' 2024-04-14T23:53:50.273 INFO:tasks.workunit.client.0.smithi060.stdout:removed './149/38' 2024-04-14T23:53:50.273 INFO:tasks.workunit.client.0.smithi060.stdout:removed './149/11' 2024-04-14T23:53:50.273 INFO:tasks.workunit.client.0.smithi060.stdout:removed './149/16' 2024-04-14T23:53:50.273 INFO:tasks.workunit.client.0.smithi060.stdout:removed './149/32' 2024-04-14T23:53:50.273 INFO:tasks.workunit.client.0.smithi060.stdout:removed './149/30' 2024-04-14T23:53:50.273 INFO:tasks.workunit.client.0.smithi060.stdout:removed './149/14' 2024-04-14T23:53:50.273 INFO:tasks.workunit.client.0.smithi060.stdout:removed './149/18' 2024-04-14T23:53:50.274 INFO:tasks.workunit.client.0.smithi060.stdout:removed './149/13' 2024-04-14T23:53:50.274 INFO:tasks.workunit.client.0.smithi060.stdout:removed './149/33' 2024-04-14T23:53:50.274 INFO:tasks.workunit.client.0.smithi060.stdout:removed './149/17' 2024-04-14T23:53:50.274 INFO:tasks.workunit.client.0.smithi060.stdout:removed './149/35' 2024-04-14T23:53:50.274 INFO:tasks.workunit.client.0.smithi060.stdout:removed './149/37' 2024-04-14T23:53:50.274 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './149' 2024-04-14T23:53:50.274 INFO:tasks.workunit.client.0.smithi060.stdout:removed './150/19' 2024-04-14T23:53:50.274 INFO:tasks.workunit.client.0.smithi060.stdout:removed './150/15' 2024-04-14T23:53:50.274 INFO:tasks.workunit.client.0.smithi060.stdout:removed './150/31' 2024-04-14T23:53:50.274 INFO:tasks.workunit.client.0.smithi060.stdout:removed './150/36' 2024-04-14T23:53:50.274 INFO:tasks.workunit.client.0.smithi060.stdout:removed './150/20' 2024-04-14T23:53:50.274 INFO:tasks.workunit.client.0.smithi060.stdout:removed './150/12' 2024-04-14T23:53:50.274 INFO:tasks.workunit.client.0.smithi060.stdout:removed './150/39' 2024-04-14T23:53:50.274 INFO:tasks.workunit.client.0.smithi060.stdout:removed './150/10' 2024-04-14T23:53:50.275 INFO:tasks.workunit.client.0.smithi060.stdout:removed './150/34' 2024-04-14T23:53:50.275 INFO:tasks.workunit.client.0.smithi060.stdout:removed './150/38' 2024-04-14T23:53:50.275 INFO:tasks.workunit.client.0.smithi060.stdout:removed './150/11' 2024-04-14T23:53:50.275 INFO:tasks.workunit.client.0.smithi060.stdout:removed './150/16' 2024-04-14T23:53:50.275 INFO:tasks.workunit.client.0.smithi060.stdout:removed './150/32' 2024-04-14T23:53:50.275 INFO:tasks.workunit.client.0.smithi060.stdout:removed './150/30' 2024-04-14T23:53:50.275 INFO:tasks.workunit.client.0.smithi060.stdout:removed './150/14' 2024-04-14T23:53:50.275 INFO:tasks.workunit.client.0.smithi060.stdout:removed './150/18' 2024-04-14T23:53:50.275 INFO:tasks.workunit.client.0.smithi060.stdout:removed './150/13' 2024-04-14T23:53:50.275 INFO:tasks.workunit.client.0.smithi060.stdout:removed './150/33' 2024-04-14T23:53:50.275 INFO:tasks.workunit.client.0.smithi060.stdout:removed './150/17' 2024-04-14T23:53:50.275 INFO:tasks.workunit.client.0.smithi060.stdout:removed './150/35' 2024-04-14T23:53:50.275 INFO:tasks.workunit.client.0.smithi060.stdout:removed './150/37' 2024-04-14T23:53:50.275 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './150' 2024-04-14T23:53:50.276 INFO:tasks.workunit.client.0.smithi060.stdout:removed './151/19' 2024-04-14T23:53:50.276 INFO:tasks.workunit.client.0.smithi060.stdout:removed './151/15' 2024-04-14T23:53:50.276 INFO:tasks.workunit.client.0.smithi060.stdout:removed './151/31' 2024-04-14T23:53:50.276 INFO:tasks.workunit.client.0.smithi060.stdout:removed './151/36' 2024-04-14T23:53:50.276 INFO:tasks.workunit.client.0.smithi060.stdout:removed './151/20' 2024-04-14T23:53:50.276 INFO:tasks.workunit.client.0.smithi060.stdout:removed './151/12' 2024-04-14T23:53:50.276 INFO:tasks.workunit.client.0.smithi060.stdout:removed './151/39' 2024-04-14T23:53:50.276 INFO:tasks.workunit.client.0.smithi060.stdout:removed './151/10' 2024-04-14T23:53:50.276 INFO:tasks.workunit.client.0.smithi060.stdout:removed './151/34' 2024-04-14T23:53:50.276 INFO:tasks.workunit.client.0.smithi060.stdout:removed './151/38' 2024-04-14T23:53:50.276 INFO:tasks.workunit.client.0.smithi060.stdout:removed './151/11' 2024-04-14T23:53:50.276 INFO:tasks.workunit.client.0.smithi060.stdout:removed './151/16' 2024-04-14T23:53:50.276 INFO:tasks.workunit.client.0.smithi060.stdout:removed './151/32' 2024-04-14T23:53:50.277 INFO:tasks.workunit.client.0.smithi060.stdout:removed './151/30' 2024-04-14T23:53:50.277 INFO:tasks.workunit.client.0.smithi060.stdout:removed './151/14' 2024-04-14T23:53:50.277 INFO:tasks.workunit.client.0.smithi060.stdout:removed './151/18' 2024-04-14T23:53:50.277 INFO:tasks.workunit.client.0.smithi060.stdout:removed './151/13' 2024-04-14T23:53:50.277 INFO:tasks.workunit.client.0.smithi060.stdout:removed './151/33' 2024-04-14T23:53:50.277 INFO:tasks.workunit.client.0.smithi060.stdout:removed './151/17' 2024-04-14T23:53:50.277 INFO:tasks.workunit.client.0.smithi060.stdout:removed './151/35' 2024-04-14T23:53:50.277 INFO:tasks.workunit.client.0.smithi060.stdout:removed './151/37' 2024-04-14T23:53:50.277 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './151' 2024-04-14T23:53:50.277 INFO:tasks.workunit.client.0.smithi060.stdout:removed './152/19' 2024-04-14T23:53:50.277 INFO:tasks.workunit.client.0.smithi060.stdout:removed './152/15' 2024-04-14T23:53:50.277 INFO:tasks.workunit.client.0.smithi060.stdout:removed './152/31' 2024-04-14T23:53:50.277 INFO:tasks.workunit.client.0.smithi060.stdout:removed './152/36' 2024-04-14T23:53:50.278 INFO:tasks.workunit.client.0.smithi060.stdout:removed './152/20' 2024-04-14T23:53:50.278 INFO:tasks.workunit.client.0.smithi060.stdout:removed './152/12' 2024-04-14T23:53:50.278 INFO:tasks.workunit.client.0.smithi060.stdout:removed './152/39' 2024-04-14T23:53:50.278 INFO:tasks.workunit.client.0.smithi060.stdout:removed './152/10' 2024-04-14T23:53:50.278 INFO:tasks.workunit.client.0.smithi060.stdout:removed './152/34' 2024-04-14T23:53:50.278 INFO:tasks.workunit.client.0.smithi060.stdout:removed './152/38' 2024-04-14T23:53:50.278 INFO:tasks.workunit.client.0.smithi060.stdout:removed './152/11' 2024-04-14T23:53:50.278 INFO:tasks.workunit.client.0.smithi060.stdout:removed './152/16' 2024-04-14T23:53:50.278 INFO:tasks.workunit.client.0.smithi060.stdout:removed './152/32' 2024-04-14T23:53:50.278 INFO:tasks.workunit.client.0.smithi060.stdout:removed './152/30' 2024-04-14T23:53:50.278 INFO:tasks.workunit.client.0.smithi060.stdout:removed './152/14' 2024-04-14T23:53:50.278 INFO:tasks.workunit.client.0.smithi060.stdout:removed './152/18' 2024-04-14T23:53:50.278 INFO:tasks.workunit.client.0.smithi060.stdout:removed './152/13' 2024-04-14T23:53:50.278 INFO:tasks.workunit.client.0.smithi060.stdout:removed './152/33' 2024-04-14T23:53:50.278 INFO:tasks.workunit.client.0.smithi060.stdout:removed './152/17' 2024-04-14T23:53:50.279 INFO:tasks.workunit.client.0.smithi060.stdout:removed './152/35' 2024-04-14T23:53:50.279 INFO:tasks.workunit.client.0.smithi060.stdout:removed './152/37' 2024-04-14T23:53:50.279 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './152' 2024-04-14T23:53:50.279 INFO:tasks.workunit.client.0.smithi060.stdout:removed './153/19' 2024-04-14T23:53:50.279 INFO:tasks.workunit.client.0.smithi060.stdout:removed './153/15' 2024-04-14T23:53:50.279 INFO:tasks.workunit.client.0.smithi060.stdout:removed './153/31' 2024-04-14T23:53:50.279 INFO:tasks.workunit.client.0.smithi060.stdout:removed './153/36' 2024-04-14T23:53:50.279 INFO:tasks.workunit.client.0.smithi060.stdout:removed './153/20' 2024-04-14T23:53:50.279 INFO:tasks.workunit.client.0.smithi060.stdout:removed './153/12' 2024-04-14T23:53:50.279 INFO:tasks.workunit.client.0.smithi060.stdout:removed './153/39' 2024-04-14T23:53:50.279 INFO:tasks.workunit.client.0.smithi060.stdout:removed './153/10' 2024-04-14T23:53:50.279 INFO:tasks.workunit.client.0.smithi060.stdout:removed './153/34' 2024-04-14T23:53:50.279 INFO:tasks.workunit.client.0.smithi060.stdout:removed './153/38' 2024-04-14T23:53:50.280 INFO:tasks.workunit.client.0.smithi060.stdout:removed './153/11' 2024-04-14T23:53:50.280 INFO:tasks.workunit.client.0.smithi060.stdout:removed './153/16' 2024-04-14T23:53:50.280 INFO:tasks.workunit.client.0.smithi060.stdout:removed './153/32' 2024-04-14T23:53:50.280 INFO:tasks.workunit.client.0.smithi060.stdout:removed './153/30' 2024-04-14T23:53:50.280 INFO:tasks.workunit.client.0.smithi060.stdout:removed './153/14' 2024-04-14T23:53:50.280 INFO:tasks.workunit.client.0.smithi060.stdout:removed './153/18' 2024-04-14T23:53:50.280 INFO:tasks.workunit.client.0.smithi060.stdout:removed './153/13' 2024-04-14T23:53:50.280 INFO:tasks.workunit.client.0.smithi060.stdout:removed './153/33' 2024-04-14T23:53:50.280 INFO:tasks.workunit.client.0.smithi060.stdout:removed './153/17' 2024-04-14T23:53:50.280 INFO:tasks.workunit.client.0.smithi060.stdout:removed './153/35' 2024-04-14T23:53:50.280 INFO:tasks.workunit.client.0.smithi060.stdout:removed './153/37' 2024-04-14T23:53:50.281 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './153' 2024-04-14T23:53:50.281 INFO:tasks.workunit.client.0.smithi060.stdout:removed './154/19' 2024-04-14T23:53:50.281 INFO:tasks.workunit.client.0.smithi060.stdout:removed './154/15' 2024-04-14T23:53:50.281 INFO:tasks.workunit.client.0.smithi060.stdout:removed './154/31' 2024-04-14T23:53:50.281 INFO:tasks.workunit.client.0.smithi060.stdout:removed './154/36' 2024-04-14T23:53:50.281 INFO:tasks.workunit.client.0.smithi060.stdout:removed './154/20' 2024-04-14T23:53:50.281 INFO:tasks.workunit.client.0.smithi060.stdout:removed './154/12' 2024-04-14T23:53:50.281 INFO:tasks.workunit.client.0.smithi060.stdout:removed './154/39' 2024-04-14T23:53:50.281 INFO:tasks.workunit.client.0.smithi060.stdout:removed './154/10' 2024-04-14T23:53:50.281 INFO:tasks.workunit.client.0.smithi060.stdout:removed './154/34' 2024-04-14T23:53:50.281 INFO:tasks.workunit.client.0.smithi060.stdout:removed './154/38' 2024-04-14T23:53:50.281 INFO:tasks.workunit.client.0.smithi060.stdout:removed './154/11' 2024-04-14T23:53:50.281 INFO:tasks.workunit.client.0.smithi060.stdout:removed './154/16' 2024-04-14T23:53:50.282 INFO:tasks.workunit.client.0.smithi060.stdout:removed './154/32' 2024-04-14T23:53:50.282 INFO:tasks.workunit.client.0.smithi060.stdout:removed './154/30' 2024-04-14T23:53:50.282 INFO:tasks.workunit.client.0.smithi060.stdout:removed './154/14' 2024-04-14T23:53:50.282 INFO:tasks.workunit.client.0.smithi060.stdout:removed './154/18' 2024-04-14T23:53:50.282 INFO:tasks.workunit.client.0.smithi060.stdout:removed './154/13' 2024-04-14T23:53:50.282 INFO:tasks.workunit.client.0.smithi060.stdout:removed './154/33' 2024-04-14T23:53:50.282 INFO:tasks.workunit.client.0.smithi060.stdout:removed './154/17' 2024-04-14T23:53:50.282 INFO:tasks.workunit.client.0.smithi060.stdout:removed './154/35' 2024-04-14T23:53:50.282 INFO:tasks.workunit.client.0.smithi060.stdout:removed './154/37' 2024-04-14T23:53:50.282 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './154' 2024-04-14T23:53:50.282 INFO:tasks.workunit.client.0.smithi060.stdout:removed './155/19' 2024-04-14T23:53:50.282 INFO:tasks.workunit.client.0.smithi060.stdout:removed './155/15' 2024-04-14T23:53:50.282 INFO:tasks.workunit.client.0.smithi060.stdout:removed './155/31' 2024-04-14T23:53:50.282 INFO:tasks.workunit.client.0.smithi060.stdout:removed './155/36' 2024-04-14T23:53:50.283 INFO:tasks.workunit.client.0.smithi060.stdout:removed './155/20' 2024-04-14T23:53:50.283 INFO:tasks.workunit.client.0.smithi060.stdout:removed './155/12' 2024-04-14T23:53:50.283 INFO:tasks.workunit.client.0.smithi060.stdout:removed './155/39' 2024-04-14T23:53:50.283 INFO:tasks.workunit.client.0.smithi060.stdout:removed './155/10' 2024-04-14T23:53:50.283 INFO:tasks.workunit.client.0.smithi060.stdout:removed './155/34' 2024-04-14T23:53:50.283 INFO:tasks.workunit.client.0.smithi060.stdout:removed './155/38' 2024-04-14T23:53:50.283 INFO:tasks.workunit.client.0.smithi060.stdout:removed './155/11' 2024-04-14T23:53:50.283 INFO:tasks.workunit.client.0.smithi060.stdout:removed './155/16' 2024-04-14T23:53:50.283 INFO:tasks.workunit.client.0.smithi060.stdout:removed './155/32' 2024-04-14T23:53:50.283 INFO:tasks.workunit.client.0.smithi060.stdout:removed './155/30' 2024-04-14T23:53:50.283 INFO:tasks.workunit.client.0.smithi060.stdout:removed './155/14' 2024-04-14T23:53:50.283 INFO:tasks.workunit.client.0.smithi060.stdout:removed './155/18' 2024-04-14T23:53:50.283 INFO:tasks.workunit.client.0.smithi060.stdout:removed './155/13' 2024-04-14T23:53:50.284 INFO:tasks.workunit.client.0.smithi060.stdout:removed './155/33' 2024-04-14T23:53:50.284 INFO:tasks.workunit.client.0.smithi060.stdout:removed './155/17' 2024-04-14T23:53:50.284 INFO:tasks.workunit.client.0.smithi060.stdout:removed './155/35' 2024-04-14T23:53:50.284 INFO:tasks.workunit.client.0.smithi060.stdout:removed './155/37' 2024-04-14T23:53:50.284 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './155' 2024-04-14T23:53:50.284 INFO:tasks.workunit.client.0.smithi060.stdout:removed './156/19' 2024-04-14T23:53:50.284 INFO:tasks.workunit.client.0.smithi060.stdout:removed './156/15' 2024-04-14T23:53:50.284 INFO:tasks.workunit.client.0.smithi060.stdout:removed './156/31' 2024-04-14T23:53:50.284 INFO:tasks.workunit.client.0.smithi060.stdout:removed './156/36' 2024-04-14T23:53:50.284 INFO:tasks.workunit.client.0.smithi060.stdout:removed './156/20' 2024-04-14T23:53:50.284 INFO:tasks.workunit.client.0.smithi060.stdout:removed './156/12' 2024-04-14T23:53:50.284 INFO:tasks.workunit.client.0.smithi060.stdout:removed './156/39' 2024-04-14T23:53:50.284 INFO:tasks.workunit.client.0.smithi060.stdout:removed './156/10' 2024-04-14T23:53:50.284 INFO:tasks.workunit.client.0.smithi060.stdout:removed './156/34' 2024-04-14T23:53:50.285 INFO:tasks.workunit.client.0.smithi060.stdout:removed './156/38' 2024-04-14T23:53:50.285 INFO:tasks.workunit.client.0.smithi060.stdout:removed './156/11' 2024-04-14T23:53:50.285 INFO:tasks.workunit.client.0.smithi060.stdout:removed './156/16' 2024-04-14T23:53:50.285 INFO:tasks.workunit.client.0.smithi060.stdout:removed './156/32' 2024-04-14T23:53:50.285 INFO:tasks.workunit.client.0.smithi060.stdout:removed './156/30' 2024-04-14T23:53:50.285 INFO:tasks.workunit.client.0.smithi060.stdout:removed './156/14' 2024-04-14T23:53:50.285 INFO:tasks.workunit.client.0.smithi060.stdout:removed './156/18' 2024-04-14T23:53:50.285 INFO:tasks.workunit.client.0.smithi060.stdout:removed './156/13' 2024-04-14T23:53:50.285 INFO:tasks.workunit.client.0.smithi060.stdout:removed './156/33' 2024-04-14T23:53:50.285 INFO:tasks.workunit.client.0.smithi060.stdout:removed './156/17' 2024-04-14T23:53:50.285 INFO:tasks.workunit.client.0.smithi060.stdout:removed './156/35' 2024-04-14T23:53:50.285 INFO:tasks.workunit.client.0.smithi060.stdout:removed './156/37' 2024-04-14T23:53:50.285 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './156' 2024-04-14T23:53:50.285 INFO:tasks.workunit.client.0.smithi060.stdout:removed './157/19' 2024-04-14T23:53:50.286 INFO:tasks.workunit.client.0.smithi060.stdout:removed './157/15' 2024-04-14T23:53:50.286 INFO:tasks.workunit.client.0.smithi060.stdout:removed './157/31' 2024-04-14T23:53:50.286 INFO:tasks.workunit.client.0.smithi060.stdout:removed './157/36' 2024-04-14T23:53:50.286 INFO:tasks.workunit.client.0.smithi060.stdout:removed './157/20' 2024-04-14T23:53:50.286 INFO:tasks.workunit.client.0.smithi060.stdout:removed './157/12' 2024-04-14T23:53:50.286 INFO:tasks.workunit.client.0.smithi060.stdout:removed './157/39' 2024-04-14T23:53:50.286 INFO:tasks.workunit.client.0.smithi060.stdout:removed './157/10' 2024-04-14T23:53:50.286 INFO:tasks.workunit.client.0.smithi060.stdout:removed './157/34' 2024-04-14T23:53:50.286 INFO:tasks.workunit.client.0.smithi060.stdout:removed './157/38' 2024-04-14T23:53:50.286 INFO:tasks.workunit.client.0.smithi060.stdout:removed './157/11' 2024-04-14T23:53:50.286 INFO:tasks.workunit.client.0.smithi060.stdout:removed './157/16' 2024-04-14T23:53:50.286 INFO:tasks.workunit.client.0.smithi060.stdout:removed './157/32' 2024-04-14T23:53:50.286 INFO:tasks.workunit.client.0.smithi060.stdout:removed './157/30' 2024-04-14T23:53:50.286 INFO:tasks.workunit.client.0.smithi060.stdout:removed './157/14' 2024-04-14T23:53:50.287 INFO:tasks.workunit.client.0.smithi060.stdout:removed './157/18' 2024-04-14T23:53:50.287 INFO:tasks.workunit.client.0.smithi060.stdout:removed './157/13' 2024-04-14T23:53:50.287 INFO:tasks.workunit.client.0.smithi060.stdout:removed './157/33' 2024-04-14T23:53:51.721 INFO:tasks.workunit.client.0.smithi060.stdout:removed './157/17' 2024-04-14T23:53:51.721 INFO:tasks.workunit.client.0.smithi060.stdout:removed './157/35' 2024-04-14T23:53:51.721 INFO:tasks.workunit.client.0.smithi060.stdout:removed './157/37' 2024-04-14T23:53:51.721 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './157' 2024-04-14T23:53:51.721 INFO:tasks.workunit.client.0.smithi060.stdout:removed './158/19' 2024-04-14T23:53:51.721 INFO:tasks.workunit.client.0.smithi060.stdout:removed './158/15' 2024-04-14T23:53:51.721 INFO:tasks.workunit.client.0.smithi060.stdout:removed './158/31' 2024-04-14T23:53:51.721 INFO:tasks.workunit.client.0.smithi060.stdout:removed './158/36' 2024-04-14T23:53:51.721 INFO:tasks.workunit.client.0.smithi060.stdout:removed './158/20' 2024-04-14T23:53:51.722 INFO:tasks.workunit.client.0.smithi060.stdout:removed './158/12' 2024-04-14T23:53:51.722 INFO:tasks.workunit.client.0.smithi060.stdout:removed './158/39' 2024-04-14T23:53:51.722 INFO:tasks.workunit.client.0.smithi060.stdout:removed './158/10' 2024-04-14T23:53:51.722 INFO:tasks.workunit.client.0.smithi060.stdout:removed './158/34' 2024-04-14T23:53:51.722 INFO:tasks.workunit.client.0.smithi060.stdout:removed './158/38' 2024-04-14T23:53:51.722 INFO:tasks.workunit.client.0.smithi060.stdout:removed './158/11' 2024-04-14T23:53:51.722 INFO:tasks.workunit.client.0.smithi060.stdout:removed './158/16' 2024-04-14T23:53:51.722 INFO:tasks.workunit.client.0.smithi060.stdout:removed './158/32' 2024-04-14T23:53:51.722 INFO:tasks.workunit.client.0.smithi060.stdout:removed './158/30' 2024-04-14T23:53:51.722 INFO:tasks.workunit.client.0.smithi060.stdout:removed './158/14' 2024-04-14T23:53:51.722 INFO:tasks.workunit.client.0.smithi060.stdout:removed './158/18' 2024-04-14T23:53:51.722 INFO:tasks.workunit.client.0.smithi060.stdout:removed './158/13' 2024-04-14T23:53:51.722 INFO:tasks.workunit.client.0.smithi060.stdout:removed './158/33' 2024-04-14T23:53:51.722 INFO:tasks.workunit.client.0.smithi060.stdout:removed './158/17' 2024-04-14T23:53:51.722 INFO:tasks.workunit.client.0.smithi060.stdout:removed './158/35' 2024-04-14T23:53:51.723 INFO:tasks.workunit.client.0.smithi060.stdout:removed './158/37' 2024-04-14T23:53:51.723 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './158' 2024-04-14T23:53:51.723 INFO:tasks.workunit.client.0.smithi060.stdout:removed './159/19' 2024-04-14T23:53:51.723 INFO:tasks.workunit.client.0.smithi060.stdout:removed './159/15' 2024-04-14T23:53:51.723 INFO:tasks.workunit.client.0.smithi060.stdout:removed './159/31' 2024-04-14T23:53:51.723 INFO:tasks.workunit.client.0.smithi060.stdout:removed './159/36' 2024-04-14T23:53:51.723 INFO:tasks.workunit.client.0.smithi060.stdout:removed './159/20' 2024-04-14T23:53:51.723 INFO:tasks.workunit.client.0.smithi060.stdout:removed './159/12' 2024-04-14T23:53:51.723 INFO:tasks.workunit.client.0.smithi060.stdout:removed './159/39' 2024-04-14T23:53:51.723 INFO:tasks.workunit.client.0.smithi060.stdout:removed './159/10' 2024-04-14T23:53:51.723 INFO:tasks.workunit.client.0.smithi060.stdout:removed './159/34' 2024-04-14T23:53:51.723 INFO:tasks.workunit.client.0.smithi060.stdout:removed './159/38' 2024-04-14T23:53:51.723 INFO:tasks.workunit.client.0.smithi060.stdout:removed './159/11' 2024-04-14T23:53:51.723 INFO:tasks.workunit.client.0.smithi060.stdout:removed './159/16' 2024-04-14T23:53:51.723 INFO:tasks.workunit.client.0.smithi060.stdout:removed './159/32' 2024-04-14T23:53:51.724 INFO:tasks.workunit.client.0.smithi060.stdout:removed './159/30' 2024-04-14T23:53:51.724 INFO:tasks.workunit.client.0.smithi060.stdout:removed './159/14' 2024-04-14T23:53:51.724 INFO:tasks.workunit.client.0.smithi060.stdout:removed './159/18' 2024-04-14T23:53:51.724 INFO:tasks.workunit.client.0.smithi060.stdout:removed './159/13' 2024-04-14T23:53:51.724 INFO:tasks.workunit.client.0.smithi060.stdout:removed './159/33' 2024-04-14T23:53:51.724 INFO:tasks.workunit.client.0.smithi060.stdout:removed './159/17' 2024-04-14T23:53:51.724 INFO:tasks.workunit.client.0.smithi060.stdout:removed './159/35' 2024-04-14T23:53:51.724 INFO:tasks.workunit.client.0.smithi060.stdout:removed './159/37' 2024-04-14T23:53:51.724 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './159' 2024-04-14T23:53:51.724 INFO:tasks.workunit.client.0.smithi060.stdout:removed './160/19' 2024-04-14T23:53:51.724 INFO:tasks.workunit.client.0.smithi060.stdout:removed './160/15' 2024-04-14T23:53:51.724 INFO:tasks.workunit.client.0.smithi060.stdout:removed './160/31' 2024-04-14T23:53:51.724 INFO:tasks.workunit.client.0.smithi060.stdout:removed './160/36' 2024-04-14T23:53:51.724 INFO:tasks.workunit.client.0.smithi060.stdout:removed './160/20' 2024-04-14T23:53:51.724 INFO:tasks.workunit.client.0.smithi060.stdout:removed './160/12' 2024-04-14T23:53:51.725 INFO:tasks.workunit.client.0.smithi060.stdout:removed './160/39' 2024-04-14T23:53:51.725 INFO:tasks.workunit.client.0.smithi060.stdout:removed './160/10' 2024-04-14T23:53:51.725 INFO:tasks.workunit.client.0.smithi060.stdout:removed './160/34' 2024-04-14T23:53:51.725 INFO:tasks.workunit.client.0.smithi060.stdout:removed './160/38' 2024-04-14T23:53:51.725 INFO:tasks.workunit.client.0.smithi060.stdout:removed './160/11' 2024-04-14T23:53:51.725 INFO:tasks.workunit.client.0.smithi060.stdout:removed './160/16' 2024-04-14T23:53:51.725 INFO:tasks.workunit.client.0.smithi060.stdout:removed './160/32' 2024-04-14T23:53:51.725 INFO:tasks.workunit.client.0.smithi060.stdout:removed './160/30' 2024-04-14T23:53:51.725 INFO:tasks.workunit.client.0.smithi060.stdout:removed './160/14' 2024-04-14T23:53:51.725 INFO:tasks.workunit.client.0.smithi060.stdout:removed './160/18' 2024-04-14T23:53:51.725 INFO:tasks.workunit.client.0.smithi060.stdout:removed './160/13' 2024-04-14T23:53:51.725 INFO:tasks.workunit.client.0.smithi060.stdout:removed './160/33' 2024-04-14T23:53:51.725 INFO:tasks.workunit.client.0.smithi060.stdout:removed './160/17' 2024-04-14T23:53:51.725 INFO:tasks.workunit.client.0.smithi060.stdout:removed './160/35' 2024-04-14T23:53:51.725 INFO:tasks.workunit.client.0.smithi060.stdout:removed './160/37' 2024-04-14T23:53:51.726 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './160' 2024-04-14T23:53:51.726 INFO:tasks.workunit.client.0.smithi060.stdout:removed './161/19' 2024-04-14T23:53:51.726 INFO:tasks.workunit.client.0.smithi060.stdout:removed './161/15' 2024-04-14T23:53:51.726 INFO:tasks.workunit.client.0.smithi060.stdout:removed './161/31' 2024-04-14T23:53:51.726 INFO:tasks.workunit.client.0.smithi060.stdout:removed './161/36' 2024-04-14T23:53:51.726 INFO:tasks.workunit.client.0.smithi060.stdout:removed './161/20' 2024-04-14T23:53:51.726 INFO:tasks.workunit.client.0.smithi060.stdout:removed './161/12' 2024-04-14T23:53:51.726 INFO:tasks.workunit.client.0.smithi060.stdout:removed './161/39' 2024-04-14T23:53:51.726 INFO:tasks.workunit.client.0.smithi060.stdout:removed './161/10' 2024-04-14T23:53:51.726 INFO:tasks.workunit.client.0.smithi060.stdout:removed './161/34' 2024-04-14T23:53:51.726 INFO:tasks.workunit.client.0.smithi060.stdout:removed './161/38' 2024-04-14T23:53:51.726 INFO:tasks.workunit.client.0.smithi060.stdout:removed './161/11' 2024-04-14T23:53:51.726 INFO:tasks.workunit.client.0.smithi060.stdout:removed './161/16' 2024-04-14T23:53:51.726 INFO:tasks.workunit.client.0.smithi060.stdout:removed './161/32' 2024-04-14T23:53:51.727 INFO:tasks.workunit.client.0.smithi060.stdout:removed './161/30' 2024-04-14T23:53:51.727 INFO:tasks.workunit.client.0.smithi060.stdout:removed './161/14' 2024-04-14T23:53:51.727 INFO:tasks.workunit.client.0.smithi060.stdout:removed './161/18' 2024-04-14T23:53:51.727 INFO:tasks.workunit.client.0.smithi060.stdout:removed './161/13' 2024-04-14T23:53:51.727 INFO:tasks.workunit.client.0.smithi060.stdout:removed './161/33' 2024-04-14T23:53:51.727 INFO:tasks.workunit.client.0.smithi060.stdout:removed './161/17' 2024-04-14T23:53:51.727 INFO:tasks.workunit.client.0.smithi060.stdout:removed './161/35' 2024-04-14T23:53:51.727 INFO:tasks.workunit.client.0.smithi060.stdout:removed './161/37' 2024-04-14T23:53:51.727 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './161' 2024-04-14T23:53:51.727 INFO:tasks.workunit.client.0.smithi060.stdout:removed './162/19' 2024-04-14T23:53:51.727 INFO:tasks.workunit.client.0.smithi060.stdout:removed './162/15' 2024-04-14T23:53:51.727 INFO:tasks.workunit.client.0.smithi060.stdout:removed './162/31' 2024-04-14T23:53:51.727 INFO:tasks.workunit.client.0.smithi060.stdout:removed './162/36' 2024-04-14T23:53:51.727 INFO:tasks.workunit.client.0.smithi060.stdout:removed './162/20' 2024-04-14T23:53:51.727 INFO:tasks.workunit.client.0.smithi060.stdout:removed './162/12' 2024-04-14T23:53:51.728 INFO:tasks.workunit.client.0.smithi060.stdout:removed './162/39' 2024-04-14T23:53:51.728 INFO:tasks.workunit.client.0.smithi060.stdout:removed './162/10' 2024-04-14T23:53:51.728 INFO:tasks.workunit.client.0.smithi060.stdout:removed './162/34' 2024-04-14T23:53:51.728 INFO:tasks.workunit.client.0.smithi060.stdout:removed './162/38' 2024-04-14T23:53:51.728 INFO:tasks.workunit.client.0.smithi060.stdout:removed './162/11' 2024-04-14T23:53:51.728 INFO:tasks.workunit.client.0.smithi060.stdout:removed './162/16' 2024-04-14T23:53:51.728 INFO:tasks.workunit.client.0.smithi060.stdout:removed './162/32' 2024-04-14T23:53:51.728 INFO:tasks.workunit.client.0.smithi060.stdout:removed './162/30' 2024-04-14T23:53:51.728 INFO:tasks.workunit.client.0.smithi060.stdout:removed './162/14' 2024-04-14T23:53:51.728 INFO:tasks.workunit.client.0.smithi060.stdout:removed './162/18' 2024-04-14T23:53:51.728 INFO:tasks.workunit.client.0.smithi060.stdout:removed './162/13' 2024-04-14T23:53:51.728 INFO:tasks.workunit.client.0.smithi060.stdout:removed './162/33' 2024-04-14T23:53:51.728 INFO:tasks.workunit.client.0.smithi060.stdout:removed './162/17' 2024-04-14T23:53:51.728 INFO:tasks.workunit.client.0.smithi060.stdout:removed './162/35' 2024-04-14T23:53:51.728 INFO:tasks.workunit.client.0.smithi060.stdout:removed './162/37' 2024-04-14T23:53:51.729 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './162' 2024-04-14T23:53:51.729 INFO:tasks.workunit.client.0.smithi060.stdout:removed './163/19' 2024-04-14T23:53:51.729 INFO:tasks.workunit.client.0.smithi060.stdout:removed './163/15' 2024-04-14T23:53:51.729 INFO:tasks.workunit.client.0.smithi060.stdout:removed './163/31' 2024-04-14T23:53:51.729 INFO:tasks.workunit.client.0.smithi060.stdout:removed './163/36' 2024-04-14T23:53:51.729 INFO:tasks.workunit.client.0.smithi060.stdout:removed './163/20' 2024-04-14T23:53:51.729 INFO:tasks.workunit.client.0.smithi060.stdout:removed './163/12' 2024-04-14T23:53:51.729 INFO:tasks.workunit.client.0.smithi060.stdout:removed './163/39' 2024-04-14T23:53:51.729 INFO:tasks.workunit.client.0.smithi060.stdout:removed './163/10' 2024-04-14T23:53:51.729 INFO:tasks.workunit.client.0.smithi060.stdout:removed './163/34' 2024-04-14T23:53:51.729 INFO:tasks.workunit.client.0.smithi060.stdout:removed './163/38' 2024-04-14T23:53:51.729 INFO:tasks.workunit.client.0.smithi060.stdout:removed './163/11' 2024-04-14T23:53:51.729 INFO:tasks.workunit.client.0.smithi060.stdout:removed './163/16' 2024-04-14T23:53:51.729 INFO:tasks.workunit.client.0.smithi060.stdout:removed './163/32' 2024-04-14T23:53:51.729 INFO:tasks.workunit.client.0.smithi060.stdout:removed './163/30' 2024-04-14T23:53:51.730 INFO:tasks.workunit.client.0.smithi060.stdout:removed './163/14' 2024-04-14T23:53:51.730 INFO:tasks.workunit.client.0.smithi060.stdout:removed './163/18' 2024-04-14T23:53:51.730 INFO:tasks.workunit.client.0.smithi060.stdout:removed './163/13' 2024-04-14T23:53:51.730 INFO:tasks.workunit.client.0.smithi060.stdout:removed './163/33' 2024-04-14T23:53:51.730 INFO:tasks.workunit.client.0.smithi060.stdout:removed './163/17' 2024-04-14T23:53:51.730 INFO:tasks.workunit.client.0.smithi060.stdout:removed './163/35' 2024-04-14T23:53:51.730 INFO:tasks.workunit.client.0.smithi060.stdout:removed './163/37' 2024-04-14T23:53:51.730 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './163' 2024-04-14T23:53:51.730 INFO:tasks.workunit.client.0.smithi060.stdout:removed './164/19' 2024-04-14T23:53:51.730 INFO:tasks.workunit.client.0.smithi060.stdout:removed './164/15' 2024-04-14T23:53:51.730 INFO:tasks.workunit.client.0.smithi060.stdout:removed './164/31' 2024-04-14T23:53:51.730 INFO:tasks.workunit.client.0.smithi060.stdout:removed './164/36' 2024-04-14T23:53:51.730 INFO:tasks.workunit.client.0.smithi060.stdout:removed './164/20' 2024-04-14T23:53:51.730 INFO:tasks.workunit.client.0.smithi060.stdout:removed './164/12' 2024-04-14T23:53:51.731 INFO:tasks.workunit.client.0.smithi060.stdout:removed './164/39' 2024-04-14T23:53:51.731 INFO:tasks.workunit.client.0.smithi060.stdout:removed './164/10' 2024-04-14T23:53:51.731 INFO:tasks.workunit.client.0.smithi060.stdout:removed './164/34' 2024-04-14T23:53:51.731 INFO:tasks.workunit.client.0.smithi060.stdout:removed './164/38' 2024-04-14T23:53:51.731 INFO:tasks.workunit.client.0.smithi060.stdout:removed './164/11' 2024-04-14T23:53:51.731 INFO:tasks.workunit.client.0.smithi060.stdout:removed './164/16' 2024-04-14T23:53:51.731 INFO:tasks.workunit.client.0.smithi060.stdout:removed './164/32' 2024-04-14T23:53:51.731 INFO:tasks.workunit.client.0.smithi060.stdout:removed './164/30' 2024-04-14T23:53:51.731 INFO:tasks.workunit.client.0.smithi060.stdout:removed './164/14' 2024-04-14T23:53:51.731 INFO:tasks.workunit.client.0.smithi060.stdout:removed './164/18' 2024-04-14T23:53:51.731 INFO:tasks.workunit.client.0.smithi060.stdout:removed './164/13' 2024-04-14T23:53:51.731 INFO:tasks.workunit.client.0.smithi060.stdout:removed './164/33' 2024-04-14T23:53:51.731 INFO:tasks.workunit.client.0.smithi060.stdout:removed './164/17' 2024-04-14T23:53:51.731 INFO:tasks.workunit.client.0.smithi060.stdout:removed './164/35' 2024-04-14T23:53:51.731 INFO:tasks.workunit.client.0.smithi060.stdout:removed './164/37' 2024-04-14T23:53:51.732 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './164' 2024-04-14T23:53:51.732 INFO:tasks.workunit.client.0.smithi060.stdout:removed './165/19' 2024-04-14T23:53:51.732 INFO:tasks.workunit.client.0.smithi060.stdout:removed './165/15' 2024-04-14T23:53:51.732 INFO:tasks.workunit.client.0.smithi060.stdout:removed './165/31' 2024-04-14T23:53:51.732 INFO:tasks.workunit.client.0.smithi060.stdout:removed './165/36' 2024-04-14T23:53:51.732 INFO:tasks.workunit.client.0.smithi060.stdout:removed './165/20' 2024-04-14T23:53:51.732 INFO:tasks.workunit.client.0.smithi060.stdout:removed './165/12' 2024-04-14T23:53:51.732 INFO:tasks.workunit.client.0.smithi060.stdout:removed './165/39' 2024-04-14T23:53:51.732 INFO:tasks.workunit.client.0.smithi060.stdout:removed './165/10' 2024-04-14T23:53:51.732 INFO:tasks.workunit.client.0.smithi060.stdout:removed './165/34' 2024-04-14T23:53:51.732 INFO:tasks.workunit.client.0.smithi060.stdout:removed './165/38' 2024-04-14T23:53:51.732 INFO:tasks.workunit.client.0.smithi060.stdout:removed './165/11' 2024-04-14T23:53:51.732 INFO:tasks.workunit.client.0.smithi060.stdout:removed './165/16' 2024-04-14T23:53:51.732 INFO:tasks.workunit.client.0.smithi060.stdout:removed './165/32' 2024-04-14T23:53:51.732 INFO:tasks.workunit.client.0.smithi060.stdout:removed './165/30' 2024-04-14T23:53:51.733 INFO:tasks.workunit.client.0.smithi060.stdout:removed './165/14' 2024-04-14T23:53:51.733 INFO:tasks.workunit.client.0.smithi060.stdout:removed './165/18' 2024-04-14T23:53:51.733 INFO:tasks.workunit.client.0.smithi060.stdout:removed './165/13' 2024-04-14T23:53:51.733 INFO:tasks.workunit.client.0.smithi060.stdout:removed './165/33' 2024-04-14T23:53:51.733 INFO:tasks.workunit.client.0.smithi060.stdout:removed './165/17' 2024-04-14T23:53:51.733 INFO:tasks.workunit.client.0.smithi060.stdout:removed './165/35' 2024-04-14T23:53:51.733 INFO:tasks.workunit.client.0.smithi060.stdout:removed './165/37' 2024-04-14T23:53:51.733 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './165' 2024-04-14T23:53:51.733 INFO:tasks.workunit.client.0.smithi060.stdout:removed './166/19' 2024-04-14T23:53:51.733 INFO:tasks.workunit.client.0.smithi060.stdout:removed './166/15' 2024-04-14T23:53:51.733 INFO:tasks.workunit.client.0.smithi060.stdout:removed './166/31' 2024-04-14T23:53:51.733 INFO:tasks.workunit.client.0.smithi060.stdout:removed './166/36' 2024-04-14T23:53:51.733 INFO:tasks.workunit.client.0.smithi060.stdout:removed './166/20' 2024-04-14T23:53:51.733 INFO:tasks.workunit.client.0.smithi060.stdout:removed './166/12' 2024-04-14T23:53:51.733 INFO:tasks.workunit.client.0.smithi060.stdout:removed './166/39' 2024-04-14T23:53:51.734 INFO:tasks.workunit.client.0.smithi060.stdout:removed './166/10' 2024-04-14T23:53:51.734 INFO:tasks.workunit.client.0.smithi060.stdout:removed './166/34' 2024-04-14T23:53:51.734 INFO:tasks.workunit.client.0.smithi060.stdout:removed './166/38' 2024-04-14T23:53:51.734 INFO:tasks.workunit.client.0.smithi060.stdout:removed './166/11' 2024-04-14T23:53:51.734 INFO:tasks.workunit.client.0.smithi060.stdout:removed './166/16' 2024-04-14T23:53:51.734 INFO:tasks.workunit.client.0.smithi060.stdout:removed './166/32' 2024-04-14T23:53:51.734 INFO:tasks.workunit.client.0.smithi060.stdout:removed './166/30' 2024-04-14T23:53:51.734 INFO:tasks.workunit.client.0.smithi060.stdout:removed './166/14' 2024-04-14T23:53:51.734 INFO:tasks.workunit.client.0.smithi060.stdout:removed './166/18' 2024-04-14T23:53:51.734 INFO:tasks.workunit.client.0.smithi060.stdout:removed './166/13' 2024-04-14T23:53:51.734 INFO:tasks.workunit.client.0.smithi060.stdout:removed './166/33' 2024-04-14T23:53:51.734 INFO:tasks.workunit.client.0.smithi060.stdout:removed './166/17' 2024-04-14T23:53:51.734 INFO:tasks.workunit.client.0.smithi060.stdout:removed './166/35' 2024-04-14T23:53:51.734 INFO:tasks.workunit.client.0.smithi060.stdout:removed './166/37' 2024-04-14T23:53:51.735 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './166' 2024-04-14T23:53:51.735 INFO:tasks.workunit.client.0.smithi060.stdout:removed './167/19' 2024-04-14T23:53:51.735 INFO:tasks.workunit.client.0.smithi060.stdout:removed './167/15' 2024-04-14T23:53:51.735 INFO:tasks.workunit.client.0.smithi060.stdout:removed './167/31' 2024-04-14T23:53:51.735 INFO:tasks.workunit.client.0.smithi060.stdout:removed './167/36' 2024-04-14T23:53:51.735 INFO:tasks.workunit.client.0.smithi060.stdout:removed './167/20' 2024-04-14T23:53:51.735 INFO:tasks.workunit.client.0.smithi060.stdout:removed './167/12' 2024-04-14T23:53:51.735 INFO:tasks.workunit.client.0.smithi060.stdout:removed './167/39' 2024-04-14T23:53:51.735 INFO:tasks.workunit.client.0.smithi060.stdout:removed './167/10' 2024-04-14T23:53:51.735 INFO:tasks.workunit.client.0.smithi060.stdout:removed './167/34' 2024-04-14T23:53:51.735 INFO:tasks.workunit.client.0.smithi060.stdout:removed './167/38' 2024-04-14T23:53:54.781 INFO:tasks.workunit.client.0.smithi060.stdout:removed './167/11' 2024-04-14T23:53:54.781 INFO:tasks.workunit.client.0.smithi060.stdout:removed './167/16' 2024-04-14T23:53:54.781 INFO:tasks.workunit.client.0.smithi060.stdout:removed './167/32' 2024-04-14T23:53:54.781 INFO:tasks.workunit.client.0.smithi060.stdout:removed './167/30' 2024-04-14T23:53:54.781 INFO:tasks.workunit.client.0.smithi060.stdout:removed './167/14' 2024-04-14T23:53:54.782 INFO:tasks.workunit.client.0.smithi060.stdout:removed './167/18' 2024-04-14T23:53:54.782 INFO:tasks.workunit.client.0.smithi060.stdout:removed './167/13' 2024-04-14T23:53:54.782 INFO:tasks.workunit.client.0.smithi060.stdout:removed './167/33' 2024-04-14T23:53:54.782 INFO:tasks.workunit.client.0.smithi060.stdout:removed './167/17' 2024-04-14T23:53:54.782 INFO:tasks.workunit.client.0.smithi060.stdout:removed './167/35' 2024-04-14T23:53:54.782 INFO:tasks.workunit.client.0.smithi060.stdout:removed './167/37' 2024-04-14T23:53:54.782 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './167' 2024-04-14T23:53:54.782 INFO:tasks.workunit.client.0.smithi060.stdout:removed './168/19' 2024-04-14T23:53:54.782 INFO:tasks.workunit.client.0.smithi060.stdout:removed './168/15' 2024-04-14T23:53:54.782 INFO:tasks.workunit.client.0.smithi060.stdout:removed './168/31' 2024-04-14T23:53:54.782 INFO:tasks.workunit.client.0.smithi060.stdout:removed './168/36' 2024-04-14T23:53:54.782 INFO:tasks.workunit.client.0.smithi060.stdout:removed './168/20' 2024-04-14T23:53:54.782 INFO:tasks.workunit.client.0.smithi060.stdout:removed './168/12' 2024-04-14T23:53:54.782 INFO:tasks.workunit.client.0.smithi060.stdout:removed './168/39' 2024-04-14T23:53:54.783 INFO:tasks.workunit.client.0.smithi060.stdout:removed './168/10' 2024-04-14T23:53:54.783 INFO:tasks.workunit.client.0.smithi060.stdout:removed './168/34' 2024-04-14T23:53:54.783 INFO:tasks.workunit.client.0.smithi060.stdout:removed './168/38' 2024-04-14T23:53:54.783 INFO:tasks.workunit.client.0.smithi060.stdout:removed './168/11' 2024-04-14T23:53:54.783 INFO:tasks.workunit.client.0.smithi060.stdout:removed './168/16' 2024-04-14T23:53:54.783 INFO:tasks.workunit.client.0.smithi060.stdout:removed './168/32' 2024-04-14T23:53:54.783 INFO:tasks.workunit.client.0.smithi060.stdout:removed './168/30' 2024-04-14T23:53:54.783 INFO:tasks.workunit.client.0.smithi060.stdout:removed './168/14' 2024-04-14T23:53:54.783 INFO:tasks.workunit.client.0.smithi060.stdout:removed './168/18' 2024-04-14T23:53:54.783 INFO:tasks.workunit.client.0.smithi060.stdout:removed './168/13' 2024-04-14T23:53:54.783 INFO:tasks.workunit.client.0.smithi060.stdout:removed './168/33' 2024-04-14T23:53:54.783 INFO:tasks.workunit.client.0.smithi060.stdout:removed './168/17' 2024-04-14T23:53:54.783 INFO:tasks.workunit.client.0.smithi060.stdout:removed './168/35' 2024-04-14T23:53:54.783 INFO:tasks.workunit.client.0.smithi060.stdout:removed './168/37' 2024-04-14T23:53:54.783 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './168' 2024-04-14T23:53:54.784 INFO:tasks.workunit.client.0.smithi060.stdout:removed './169/19' 2024-04-14T23:53:54.784 INFO:tasks.workunit.client.0.smithi060.stdout:removed './169/15' 2024-04-14T23:53:54.784 INFO:tasks.workunit.client.0.smithi060.stdout:removed './169/31' 2024-04-14T23:53:54.784 INFO:tasks.workunit.client.0.smithi060.stdout:removed './169/36' 2024-04-14T23:53:54.784 INFO:tasks.workunit.client.0.smithi060.stdout:removed './169/20' 2024-04-14T23:53:54.784 INFO:tasks.workunit.client.0.smithi060.stdout:removed './169/12' 2024-04-14T23:53:54.784 INFO:tasks.workunit.client.0.smithi060.stdout:removed './169/39' 2024-04-14T23:53:54.784 INFO:tasks.workunit.client.0.smithi060.stdout:removed './169/10' 2024-04-14T23:53:54.784 INFO:tasks.workunit.client.0.smithi060.stdout:removed './169/34' 2024-04-14T23:53:54.784 INFO:tasks.workunit.client.0.smithi060.stdout:removed './169/38' 2024-04-14T23:53:54.784 INFO:tasks.workunit.client.0.smithi060.stdout:removed './169/11' 2024-04-14T23:53:54.784 INFO:tasks.workunit.client.0.smithi060.stdout:removed './169/16' 2024-04-14T23:53:54.784 INFO:tasks.workunit.client.0.smithi060.stdout:removed './169/32' 2024-04-14T23:53:54.784 INFO:tasks.workunit.client.0.smithi060.stdout:removed './169/30' 2024-04-14T23:53:54.784 INFO:tasks.workunit.client.0.smithi060.stdout:removed './169/14' 2024-04-14T23:53:54.784 INFO:tasks.workunit.client.0.smithi060.stdout:removed './169/18' 2024-04-14T23:53:54.785 INFO:tasks.workunit.client.0.smithi060.stdout:removed './169/13' 2024-04-14T23:53:54.785 INFO:tasks.workunit.client.0.smithi060.stdout:removed './169/33' 2024-04-14T23:53:54.785 INFO:tasks.workunit.client.0.smithi060.stdout:removed './169/17' 2024-04-14T23:53:54.785 INFO:tasks.workunit.client.0.smithi060.stdout:removed './169/35' 2024-04-14T23:53:54.785 INFO:tasks.workunit.client.0.smithi060.stdout:removed './169/37' 2024-04-14T23:53:54.785 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './169' 2024-04-14T23:53:54.785 INFO:tasks.workunit.client.0.smithi060.stdout:removed './170/19' 2024-04-14T23:53:54.785 INFO:tasks.workunit.client.0.smithi060.stdout:removed './170/15' 2024-04-14T23:53:54.785 INFO:tasks.workunit.client.0.smithi060.stdout:removed './170/31' 2024-04-14T23:53:54.785 INFO:tasks.workunit.client.0.smithi060.stdout:removed './170/36' 2024-04-14T23:53:54.785 INFO:tasks.workunit.client.0.smithi060.stdout:removed './170/20' 2024-04-14T23:53:54.785 INFO:tasks.workunit.client.0.smithi060.stdout:removed './170/12' 2024-04-14T23:53:54.785 INFO:tasks.workunit.client.0.smithi060.stdout:removed './170/39' 2024-04-14T23:53:54.785 INFO:tasks.workunit.client.0.smithi060.stdout:removed './170/10' 2024-04-14T23:53:54.785 INFO:tasks.workunit.client.0.smithi060.stdout:removed './170/34' 2024-04-14T23:53:54.786 INFO:tasks.workunit.client.0.smithi060.stdout:removed './170/38' 2024-04-14T23:53:54.786 INFO:tasks.workunit.client.0.smithi060.stdout:removed './170/11' 2024-04-14T23:53:54.786 INFO:tasks.workunit.client.0.smithi060.stdout:removed './170/16' 2024-04-14T23:53:54.786 INFO:tasks.workunit.client.0.smithi060.stdout:removed './170/32' 2024-04-14T23:53:54.786 INFO:tasks.workunit.client.0.smithi060.stdout:removed './170/30' 2024-04-14T23:53:54.786 INFO:tasks.workunit.client.0.smithi060.stdout:removed './170/14' 2024-04-14T23:53:54.786 INFO:tasks.workunit.client.0.smithi060.stdout:removed './170/18' 2024-04-14T23:53:54.786 INFO:tasks.workunit.client.0.smithi060.stdout:removed './170/13' 2024-04-14T23:53:54.786 INFO:tasks.workunit.client.0.smithi060.stdout:removed './170/33' 2024-04-14T23:53:54.786 INFO:tasks.workunit.client.0.smithi060.stdout:removed './170/17' 2024-04-14T23:53:54.786 INFO:tasks.workunit.client.0.smithi060.stdout:removed './170/35' 2024-04-14T23:53:54.786 INFO:tasks.workunit.client.0.smithi060.stdout:removed './170/37' 2024-04-14T23:53:54.786 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './170' 2024-04-14T23:53:54.786 INFO:tasks.workunit.client.0.smithi060.stdout:removed './171/19' 2024-04-14T23:53:54.786 INFO:tasks.workunit.client.0.smithi060.stdout:removed './171/15' 2024-04-14T23:53:54.787 INFO:tasks.workunit.client.0.smithi060.stdout:removed './171/31' 2024-04-14T23:53:54.787 INFO:tasks.workunit.client.0.smithi060.stdout:removed './171/36' 2024-04-14T23:53:54.787 INFO:tasks.workunit.client.0.smithi060.stdout:removed './171/20' 2024-04-14T23:53:54.787 INFO:tasks.workunit.client.0.smithi060.stdout:removed './171/12' 2024-04-14T23:53:54.787 INFO:tasks.workunit.client.0.smithi060.stdout:removed './171/39' 2024-04-14T23:53:54.787 INFO:tasks.workunit.client.0.smithi060.stdout:removed './171/10' 2024-04-14T23:53:54.787 INFO:tasks.workunit.client.0.smithi060.stdout:removed './171/34' 2024-04-14T23:53:54.787 INFO:tasks.workunit.client.0.smithi060.stdout:removed './171/38' 2024-04-14T23:53:54.787 INFO:tasks.workunit.client.0.smithi060.stdout:removed './171/11' 2024-04-14T23:53:54.787 INFO:tasks.workunit.client.0.smithi060.stdout:removed './171/16' 2024-04-14T23:53:54.787 INFO:tasks.workunit.client.0.smithi060.stdout:removed './171/32' 2024-04-14T23:53:54.787 INFO:tasks.workunit.client.0.smithi060.stdout:removed './171/30' 2024-04-14T23:53:54.787 INFO:tasks.workunit.client.0.smithi060.stdout:removed './171/14' 2024-04-14T23:53:54.787 INFO:tasks.workunit.client.0.smithi060.stdout:removed './171/18' 2024-04-14T23:53:54.787 INFO:tasks.workunit.client.0.smithi060.stdout:removed './171/13' 2024-04-14T23:53:54.788 INFO:tasks.workunit.client.0.smithi060.stdout:removed './171/33' 2024-04-14T23:53:54.788 INFO:tasks.workunit.client.0.smithi060.stdout:removed './171/17' 2024-04-14T23:53:54.788 INFO:tasks.workunit.client.0.smithi060.stdout:removed './171/35' 2024-04-14T23:53:54.788 INFO:tasks.workunit.client.0.smithi060.stdout:removed './171/37' 2024-04-14T23:53:54.788 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './171' 2024-04-14T23:53:54.788 INFO:tasks.workunit.client.0.smithi060.stdout:removed './172/19' 2024-04-14T23:53:54.788 INFO:tasks.workunit.client.0.smithi060.stdout:removed './172/15' 2024-04-14T23:53:54.788 INFO:tasks.workunit.client.0.smithi060.stdout:removed './172/31' 2024-04-14T23:53:54.788 INFO:tasks.workunit.client.0.smithi060.stdout:removed './172/36' 2024-04-14T23:53:54.788 INFO:tasks.workunit.client.0.smithi060.stdout:removed './172/20' 2024-04-14T23:53:54.788 INFO:tasks.workunit.client.0.smithi060.stdout:removed './172/12' 2024-04-14T23:53:54.788 INFO:tasks.workunit.client.0.smithi060.stdout:removed './172/39' 2024-04-14T23:53:54.788 INFO:tasks.workunit.client.0.smithi060.stdout:removed './172/10' 2024-04-14T23:53:54.788 INFO:tasks.workunit.client.0.smithi060.stdout:removed './172/34' 2024-04-14T23:53:54.788 INFO:tasks.workunit.client.0.smithi060.stdout:removed './172/38' 2024-04-14T23:53:54.789 INFO:tasks.workunit.client.0.smithi060.stdout:removed './172/11' 2024-04-14T23:53:54.789 INFO:tasks.workunit.client.0.smithi060.stdout:removed './172/16' 2024-04-14T23:53:54.789 INFO:tasks.workunit.client.0.smithi060.stdout:removed './172/32' 2024-04-14T23:53:54.789 INFO:tasks.workunit.client.0.smithi060.stdout:removed './172/30' 2024-04-14T23:53:54.789 INFO:tasks.workunit.client.0.smithi060.stdout:removed './172/14' 2024-04-14T23:53:54.789 INFO:tasks.workunit.client.0.smithi060.stdout:removed './172/18' 2024-04-14T23:53:54.789 INFO:tasks.workunit.client.0.smithi060.stdout:removed './172/13' 2024-04-14T23:53:54.789 INFO:tasks.workunit.client.0.smithi060.stdout:removed './172/33' 2024-04-14T23:53:54.789 INFO:tasks.workunit.client.0.smithi060.stdout:removed './172/17' 2024-04-14T23:53:54.789 INFO:tasks.workunit.client.0.smithi060.stdout:removed './172/35' 2024-04-14T23:53:54.789 INFO:tasks.workunit.client.0.smithi060.stdout:removed './172/37' 2024-04-14T23:53:54.789 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './172' 2024-04-14T23:53:54.789 INFO:tasks.workunit.client.0.smithi060.stdout:removed './173/19' 2024-04-14T23:53:54.789 INFO:tasks.workunit.client.0.smithi060.stdout:removed './173/15' 2024-04-14T23:53:54.789 INFO:tasks.workunit.client.0.smithi060.stdout:removed './173/31' 2024-04-14T23:53:54.790 INFO:tasks.workunit.client.0.smithi060.stdout:removed './173/36' 2024-04-14T23:53:54.790 INFO:tasks.workunit.client.0.smithi060.stdout:removed './173/20' 2024-04-14T23:53:54.790 INFO:tasks.workunit.client.0.smithi060.stdout:removed './173/12' 2024-04-14T23:53:54.790 INFO:tasks.workunit.client.0.smithi060.stdout:removed './173/39' 2024-04-14T23:53:54.790 INFO:tasks.workunit.client.0.smithi060.stdout:removed './173/10' 2024-04-14T23:53:54.790 INFO:tasks.workunit.client.0.smithi060.stdout:removed './173/34' 2024-04-14T23:53:54.790 INFO:tasks.workunit.client.0.smithi060.stdout:removed './173/38' 2024-04-14T23:53:54.790 INFO:tasks.workunit.client.0.smithi060.stdout:removed './173/11' 2024-04-14T23:53:54.790 INFO:tasks.workunit.client.0.smithi060.stdout:removed './173/16' 2024-04-14T23:53:54.791 INFO:tasks.workunit.client.0.smithi060.stdout:removed './173/32' 2024-04-14T23:53:54.791 INFO:tasks.workunit.client.0.smithi060.stdout:removed './173/30' 2024-04-14T23:53:54.791 INFO:tasks.workunit.client.0.smithi060.stdout:removed './173/14' 2024-04-14T23:53:54.791 INFO:tasks.workunit.client.0.smithi060.stdout:removed './173/18' 2024-04-14T23:53:54.791 INFO:tasks.workunit.client.0.smithi060.stdout:removed './173/13' 2024-04-14T23:53:54.791 INFO:tasks.workunit.client.0.smithi060.stdout:removed './173/33' 2024-04-14T23:53:54.791 INFO:tasks.workunit.client.0.smithi060.stdout:removed './173/17' 2024-04-14T23:53:54.791 INFO:tasks.workunit.client.0.smithi060.stdout:removed './173/35' 2024-04-14T23:53:54.791 INFO:tasks.workunit.client.0.smithi060.stdout:removed './173/37' 2024-04-14T23:53:54.791 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './173' 2024-04-14T23:53:54.791 INFO:tasks.workunit.client.0.smithi060.stdout:removed './174/19' 2024-04-14T23:53:54.791 INFO:tasks.workunit.client.0.smithi060.stdout:removed './174/15' 2024-04-14T23:53:54.791 INFO:tasks.workunit.client.0.smithi060.stdout:removed './174/31' 2024-04-14T23:53:54.791 INFO:tasks.workunit.client.0.smithi060.stdout:removed './174/36' 2024-04-14T23:53:54.791 INFO:tasks.workunit.client.0.smithi060.stdout:removed './174/20' 2024-04-14T23:53:54.791 INFO:tasks.workunit.client.0.smithi060.stdout:removed './174/12' 2024-04-14T23:53:54.792 INFO:tasks.workunit.client.0.smithi060.stdout:removed './174/39' 2024-04-14T23:53:54.792 INFO:tasks.workunit.client.0.smithi060.stdout:removed './174/10' 2024-04-14T23:53:54.792 INFO:tasks.workunit.client.0.smithi060.stdout:removed './174/34' 2024-04-14T23:53:54.792 INFO:tasks.workunit.client.0.smithi060.stdout:removed './174/38' 2024-04-14T23:53:54.792 INFO:tasks.workunit.client.0.smithi060.stdout:removed './174/11' 2024-04-14T23:53:54.792 INFO:tasks.workunit.client.0.smithi060.stdout:removed './174/16' 2024-04-14T23:53:54.792 INFO:tasks.workunit.client.0.smithi060.stdout:removed './174/32' 2024-04-14T23:53:54.792 INFO:tasks.workunit.client.0.smithi060.stdout:removed './174/30' 2024-04-14T23:53:54.792 INFO:tasks.workunit.client.0.smithi060.stdout:removed './174/14' 2024-04-14T23:53:54.792 INFO:tasks.workunit.client.0.smithi060.stdout:removed './174/18' 2024-04-14T23:53:54.792 INFO:tasks.workunit.client.0.smithi060.stdout:removed './174/13' 2024-04-14T23:53:54.792 INFO:tasks.workunit.client.0.smithi060.stdout:removed './174/33' 2024-04-14T23:53:54.792 INFO:tasks.workunit.client.0.smithi060.stdout:removed './174/17' 2024-04-14T23:53:54.792 INFO:tasks.workunit.client.0.smithi060.stdout:removed './174/35' 2024-04-14T23:53:54.792 INFO:tasks.workunit.client.0.smithi060.stdout:removed './174/37' 2024-04-14T23:53:54.793 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './174' 2024-04-14T23:53:54.793 INFO:tasks.workunit.client.0.smithi060.stdout:removed './175/19' 2024-04-14T23:53:54.793 INFO:tasks.workunit.client.0.smithi060.stdout:removed './175/15' 2024-04-14T23:53:54.793 INFO:tasks.workunit.client.0.smithi060.stdout:removed './175/31' 2024-04-14T23:53:54.793 INFO:tasks.workunit.client.0.smithi060.stdout:removed './175/36' 2024-04-14T23:53:54.793 INFO:tasks.workunit.client.0.smithi060.stdout:removed './175/20' 2024-04-14T23:53:54.793 INFO:tasks.workunit.client.0.smithi060.stdout:removed './175/12' 2024-04-14T23:53:54.793 INFO:tasks.workunit.client.0.smithi060.stdout:removed './175/39' 2024-04-14T23:53:54.793 INFO:tasks.workunit.client.0.smithi060.stdout:removed './175/10' 2024-04-14T23:53:54.793 INFO:tasks.workunit.client.0.smithi060.stdout:removed './175/34' 2024-04-14T23:53:54.793 INFO:tasks.workunit.client.0.smithi060.stdout:removed './175/38' 2024-04-14T23:53:54.793 INFO:tasks.workunit.client.0.smithi060.stdout:removed './175/11' 2024-04-14T23:53:54.793 INFO:tasks.workunit.client.0.smithi060.stdout:removed './175/16' 2024-04-14T23:53:54.793 INFO:tasks.workunit.client.0.smithi060.stdout:removed './175/32' 2024-04-14T23:53:54.793 INFO:tasks.workunit.client.0.smithi060.stdout:removed './175/30' 2024-04-14T23:53:54.794 INFO:tasks.workunit.client.0.smithi060.stdout:removed './175/14' 2024-04-14T23:53:54.794 INFO:tasks.workunit.client.0.smithi060.stdout:removed './175/18' 2024-04-14T23:53:54.794 INFO:tasks.workunit.client.0.smithi060.stdout:removed './175/13' 2024-04-14T23:53:54.794 INFO:tasks.workunit.client.0.smithi060.stdout:removed './175/33' 2024-04-14T23:53:54.794 INFO:tasks.workunit.client.0.smithi060.stdout:removed './175/17' 2024-04-14T23:53:54.794 INFO:tasks.workunit.client.0.smithi060.stdout:removed './175/35' 2024-04-14T23:53:54.794 INFO:tasks.workunit.client.0.smithi060.stdout:removed './175/37' 2024-04-14T23:53:54.794 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './175' 2024-04-14T23:53:54.794 INFO:tasks.workunit.client.0.smithi060.stdout:removed './176/19' 2024-04-14T23:53:54.794 INFO:tasks.workunit.client.0.smithi060.stdout:removed './176/15' 2024-04-14T23:53:54.794 INFO:tasks.workunit.client.0.smithi060.stdout:removed './176/31' 2024-04-14T23:53:54.794 INFO:tasks.workunit.client.0.smithi060.stdout:removed './176/36' 2024-04-14T23:53:54.794 INFO:tasks.workunit.client.0.smithi060.stdout:removed './176/20' 2024-04-14T23:53:54.794 INFO:tasks.workunit.client.0.smithi060.stdout:removed './176/12' 2024-04-14T23:53:54.795 INFO:tasks.workunit.client.0.smithi060.stdout:removed './176/39' 2024-04-14T23:53:54.795 INFO:tasks.workunit.client.0.smithi060.stdout:removed './176/10' 2024-04-14T23:53:54.795 INFO:tasks.workunit.client.0.smithi060.stdout:removed './176/34' 2024-04-14T23:53:54.795 INFO:tasks.workunit.client.0.smithi060.stdout:removed './176/38' 2024-04-14T23:53:54.795 INFO:tasks.workunit.client.0.smithi060.stdout:removed './176/11' 2024-04-14T23:53:54.795 INFO:tasks.workunit.client.0.smithi060.stdout:removed './176/16' 2024-04-14T23:53:54.795 INFO:tasks.workunit.client.0.smithi060.stdout:removed './176/32' 2024-04-14T23:53:54.795 INFO:tasks.workunit.client.0.smithi060.stdout:removed './176/30' 2024-04-14T23:53:54.795 INFO:tasks.workunit.client.0.smithi060.stdout:removed './176/14' 2024-04-14T23:53:54.795 INFO:tasks.workunit.client.0.smithi060.stdout:removed './176/18' 2024-04-14T23:53:54.795 INFO:tasks.workunit.client.0.smithi060.stdout:removed './176/13' 2024-04-14T23:53:54.795 INFO:tasks.workunit.client.0.smithi060.stdout:removed './176/33' 2024-04-14T23:53:54.795 INFO:tasks.workunit.client.0.smithi060.stdout:removed './176/17' 2024-04-14T23:53:54.795 INFO:tasks.workunit.client.0.smithi060.stdout:removed './176/35' 2024-04-14T23:53:54.795 INFO:tasks.workunit.client.0.smithi060.stdout:removed './176/37' 2024-04-14T23:53:54.796 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './176' 2024-04-14T23:53:54.796 INFO:tasks.workunit.client.0.smithi060.stdout:removed './177/19' 2024-04-14T23:53:54.796 INFO:tasks.workunit.client.0.smithi060.stdout:removed './177/15' 2024-04-14T23:53:56.541 INFO:tasks.workunit.client.0.smithi060.stdout:removed './177/31' 2024-04-14T23:53:56.541 INFO:tasks.workunit.client.0.smithi060.stdout:removed './177/36' 2024-04-14T23:53:56.541 INFO:tasks.workunit.client.0.smithi060.stdout:removed './177/20' 2024-04-14T23:53:56.541 INFO:tasks.workunit.client.0.smithi060.stdout:removed './177/12' 2024-04-14T23:53:56.542 INFO:tasks.workunit.client.0.smithi060.stdout:removed './177/39' 2024-04-14T23:53:56.542 INFO:tasks.workunit.client.0.smithi060.stdout:removed './177/10' 2024-04-14T23:53:56.542 INFO:tasks.workunit.client.0.smithi060.stdout:removed './177/34' 2024-04-14T23:53:56.542 INFO:tasks.workunit.client.0.smithi060.stdout:removed './177/38' 2024-04-14T23:53:56.542 INFO:tasks.workunit.client.0.smithi060.stdout:removed './177/11' 2024-04-14T23:53:56.542 INFO:tasks.workunit.client.0.smithi060.stdout:removed './177/16' 2024-04-14T23:53:56.542 INFO:tasks.workunit.client.0.smithi060.stdout:removed './177/32' 2024-04-14T23:53:56.542 INFO:tasks.workunit.client.0.smithi060.stdout:removed './177/30' 2024-04-14T23:53:56.542 INFO:tasks.workunit.client.0.smithi060.stdout:removed './177/14' 2024-04-14T23:53:56.542 INFO:tasks.workunit.client.0.smithi060.stdout:removed './177/18' 2024-04-14T23:53:56.543 INFO:tasks.workunit.client.0.smithi060.stdout:removed './177/13' 2024-04-14T23:53:56.543 INFO:tasks.workunit.client.0.smithi060.stdout:removed './177/33' 2024-04-14T23:53:56.543 INFO:tasks.workunit.client.0.smithi060.stdout:removed './177/17' 2024-04-14T23:53:56.543 INFO:tasks.workunit.client.0.smithi060.stdout:removed './177/35' 2024-04-14T23:53:56.543 INFO:tasks.workunit.client.0.smithi060.stdout:removed './177/37' 2024-04-14T23:53:56.543 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './177' 2024-04-14T23:53:56.543 INFO:tasks.workunit.client.0.smithi060.stdout:removed './178/19' 2024-04-14T23:53:56.543 INFO:tasks.workunit.client.0.smithi060.stdout:removed './178/15' 2024-04-14T23:53:56.543 INFO:tasks.workunit.client.0.smithi060.stdout:removed './178/31' 2024-04-14T23:53:56.543 INFO:tasks.workunit.client.0.smithi060.stdout:removed './178/36' 2024-04-14T23:53:56.543 INFO:tasks.workunit.client.0.smithi060.stdout:removed './178/20' 2024-04-14T23:53:56.544 INFO:tasks.workunit.client.0.smithi060.stdout:removed './178/12' 2024-04-14T23:53:56.544 INFO:tasks.workunit.client.0.smithi060.stdout:removed './178/39' 2024-04-14T23:53:56.544 INFO:tasks.workunit.client.0.smithi060.stdout:removed './178/10' 2024-04-14T23:53:56.544 INFO:tasks.workunit.client.0.smithi060.stdout:removed './178/34' 2024-04-14T23:53:56.544 INFO:tasks.workunit.client.0.smithi060.stdout:removed './178/38' 2024-04-14T23:53:56.544 INFO:tasks.workunit.client.0.smithi060.stdout:removed './178/11' 2024-04-14T23:53:56.544 INFO:tasks.workunit.client.0.smithi060.stdout:removed './178/16' 2024-04-14T23:53:56.544 INFO:tasks.workunit.client.0.smithi060.stdout:removed './178/32' 2024-04-14T23:53:56.544 INFO:tasks.workunit.client.0.smithi060.stdout:removed './178/30' 2024-04-14T23:53:56.544 INFO:tasks.workunit.client.0.smithi060.stdout:removed './178/14' 2024-04-14T23:53:56.544 INFO:tasks.workunit.client.0.smithi060.stdout:removed './178/18' 2024-04-14T23:53:56.544 INFO:tasks.workunit.client.0.smithi060.stdout:removed './178/13' 2024-04-14T23:53:56.544 INFO:tasks.workunit.client.0.smithi060.stdout:removed './178/33' 2024-04-14T23:53:56.544 INFO:tasks.workunit.client.0.smithi060.stdout:removed './178/17' 2024-04-14T23:53:56.545 INFO:tasks.workunit.client.0.smithi060.stdout:removed './178/35' 2024-04-14T23:53:56.545 INFO:tasks.workunit.client.0.smithi060.stdout:removed './178/37' 2024-04-14T23:53:56.545 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './178' 2024-04-14T23:53:56.545 INFO:tasks.workunit.client.0.smithi060.stdout:removed './179/19' 2024-04-14T23:53:56.545 INFO:tasks.workunit.client.0.smithi060.stdout:removed './179/15' 2024-04-14T23:53:56.545 INFO:tasks.workunit.client.0.smithi060.stdout:removed './179/31' 2024-04-14T23:53:56.545 INFO:tasks.workunit.client.0.smithi060.stdout:removed './179/36' 2024-04-14T23:53:56.545 INFO:tasks.workunit.client.0.smithi060.stdout:removed './179/20' 2024-04-14T23:53:56.545 INFO:tasks.workunit.client.0.smithi060.stdout:removed './179/12' 2024-04-14T23:53:56.545 INFO:tasks.workunit.client.0.smithi060.stdout:removed './179/39' 2024-04-14T23:53:56.545 INFO:tasks.workunit.client.0.smithi060.stdout:removed './179/10' 2024-04-14T23:53:56.545 INFO:tasks.workunit.client.0.smithi060.stdout:removed './179/34' 2024-04-14T23:53:56.545 INFO:tasks.workunit.client.0.smithi060.stdout:removed './179/38' 2024-04-14T23:53:56.545 INFO:tasks.workunit.client.0.smithi060.stdout:removed './179/11' 2024-04-14T23:53:56.545 INFO:tasks.workunit.client.0.smithi060.stdout:removed './179/16' 2024-04-14T23:53:56.545 INFO:tasks.workunit.client.0.smithi060.stdout:removed './179/32' 2024-04-14T23:53:56.546 INFO:tasks.workunit.client.0.smithi060.stdout:removed './179/30' 2024-04-14T23:53:56.546 INFO:tasks.workunit.client.0.smithi060.stdout:removed './179/14' 2024-04-14T23:53:56.546 INFO:tasks.workunit.client.0.smithi060.stdout:removed './179/18' 2024-04-14T23:53:56.546 INFO:tasks.workunit.client.0.smithi060.stdout:removed './179/13' 2024-04-14T23:53:56.546 INFO:tasks.workunit.client.0.smithi060.stdout:removed './179/33' 2024-04-14T23:53:56.546 INFO:tasks.workunit.client.0.smithi060.stdout:removed './179/17' 2024-04-14T23:53:56.546 INFO:tasks.workunit.client.0.smithi060.stdout:removed './179/35' 2024-04-14T23:53:56.546 INFO:tasks.workunit.client.0.smithi060.stdout:removed './179/37' 2024-04-14T23:53:56.546 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './179' 2024-04-14T23:53:56.546 INFO:tasks.workunit.client.0.smithi060.stdout:removed './180/19' 2024-04-14T23:53:56.546 INFO:tasks.workunit.client.0.smithi060.stdout:removed './180/15' 2024-04-14T23:53:56.546 INFO:tasks.workunit.client.0.smithi060.stdout:removed './180/31' 2024-04-14T23:53:56.546 INFO:tasks.workunit.client.0.smithi060.stdout:removed './180/36' 2024-04-14T23:53:56.546 INFO:tasks.workunit.client.0.smithi060.stdout:removed './180/20' 2024-04-14T23:53:56.547 INFO:tasks.workunit.client.0.smithi060.stdout:removed './180/12' 2024-04-14T23:53:56.547 INFO:tasks.workunit.client.0.smithi060.stdout:removed './180/39' 2024-04-14T23:53:56.547 INFO:tasks.workunit.client.0.smithi060.stdout:removed './180/10' 2024-04-14T23:53:56.547 INFO:tasks.workunit.client.0.smithi060.stdout:removed './180/34' 2024-04-14T23:53:56.547 INFO:tasks.workunit.client.0.smithi060.stdout:removed './180/38' 2024-04-14T23:53:56.547 INFO:tasks.workunit.client.0.smithi060.stdout:removed './180/11' 2024-04-14T23:53:56.547 INFO:tasks.workunit.client.0.smithi060.stdout:removed './180/16' 2024-04-14T23:53:56.547 INFO:tasks.workunit.client.0.smithi060.stdout:removed './180/32' 2024-04-14T23:53:56.547 INFO:tasks.workunit.client.0.smithi060.stdout:removed './180/30' 2024-04-14T23:53:56.547 INFO:tasks.workunit.client.0.smithi060.stdout:removed './180/14' 2024-04-14T23:53:56.547 INFO:tasks.workunit.client.0.smithi060.stdout:removed './180/18' 2024-04-14T23:53:56.547 INFO:tasks.workunit.client.0.smithi060.stdout:removed './180/13' 2024-04-14T23:53:56.547 INFO:tasks.workunit.client.0.smithi060.stdout:removed './180/33' 2024-04-14T23:53:56.547 INFO:tasks.workunit.client.0.smithi060.stdout:removed './180/17' 2024-04-14T23:53:56.548 INFO:tasks.workunit.client.0.smithi060.stdout:removed './180/35' 2024-04-14T23:53:56.548 INFO:tasks.workunit.client.0.smithi060.stdout:removed './180/37' 2024-04-14T23:53:56.548 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './180' 2024-04-14T23:53:56.548 INFO:tasks.workunit.client.0.smithi060.stdout:removed './181/19' 2024-04-14T23:53:56.548 INFO:tasks.workunit.client.0.smithi060.stdout:removed './181/15' 2024-04-14T23:53:56.548 INFO:tasks.workunit.client.0.smithi060.stdout:removed './181/31' 2024-04-14T23:53:56.548 INFO:tasks.workunit.client.0.smithi060.stdout:removed './181/36' 2024-04-14T23:53:56.548 INFO:tasks.workunit.client.0.smithi060.stdout:removed './181/20' 2024-04-14T23:53:56.548 INFO:tasks.workunit.client.0.smithi060.stdout:removed './181/12' 2024-04-14T23:53:56.548 INFO:tasks.workunit.client.0.smithi060.stdout:removed './181/39' 2024-04-14T23:53:56.548 INFO:tasks.workunit.client.0.smithi060.stdout:removed './181/10' 2024-04-14T23:53:56.548 INFO:tasks.workunit.client.0.smithi060.stdout:removed './181/34' 2024-04-14T23:53:56.548 INFO:tasks.workunit.client.0.smithi060.stdout:removed './181/38' 2024-04-14T23:53:56.548 INFO:tasks.workunit.client.0.smithi060.stdout:removed './181/11' 2024-04-14T23:53:56.548 INFO:tasks.workunit.client.0.smithi060.stdout:removed './181/16' 2024-04-14T23:53:56.548 INFO:tasks.workunit.client.0.smithi060.stdout:removed './181/32' 2024-04-14T23:53:56.549 INFO:tasks.workunit.client.0.smithi060.stdout:removed './181/30' 2024-04-14T23:53:56.549 INFO:tasks.workunit.client.0.smithi060.stdout:removed './181/14' 2024-04-14T23:53:56.549 INFO:tasks.workunit.client.0.smithi060.stdout:removed './181/18' 2024-04-14T23:53:56.549 INFO:tasks.workunit.client.0.smithi060.stdout:removed './181/13' 2024-04-14T23:53:56.549 INFO:tasks.workunit.client.0.smithi060.stdout:removed './181/33' 2024-04-14T23:53:56.549 INFO:tasks.workunit.client.0.smithi060.stdout:removed './181/17' 2024-04-14T23:53:56.549 INFO:tasks.workunit.client.0.smithi060.stdout:removed './181/35' 2024-04-14T23:53:56.549 INFO:tasks.workunit.client.0.smithi060.stdout:removed './181/37' 2024-04-14T23:53:56.549 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './181' 2024-04-14T23:53:56.549 INFO:tasks.workunit.client.0.smithi060.stdout:removed './182/19' 2024-04-14T23:53:56.549 INFO:tasks.workunit.client.0.smithi060.stdout:removed './182/15' 2024-04-14T23:53:56.549 INFO:tasks.workunit.client.0.smithi060.stdout:removed './182/31' 2024-04-14T23:53:56.549 INFO:tasks.workunit.client.0.smithi060.stdout:removed './182/36' 2024-04-14T23:53:56.549 INFO:tasks.workunit.client.0.smithi060.stdout:removed './182/20' 2024-04-14T23:53:56.549 INFO:tasks.workunit.client.0.smithi060.stdout:removed './182/12' 2024-04-14T23:53:56.550 INFO:tasks.workunit.client.0.smithi060.stdout:removed './182/39' 2024-04-14T23:53:56.550 INFO:tasks.workunit.client.0.smithi060.stdout:removed './182/10' 2024-04-14T23:53:56.550 INFO:tasks.workunit.client.0.smithi060.stdout:removed './182/34' 2024-04-14T23:53:56.550 INFO:tasks.workunit.client.0.smithi060.stdout:removed './182/38' 2024-04-14T23:53:56.550 INFO:tasks.workunit.client.0.smithi060.stdout:removed './182/11' 2024-04-14T23:53:56.550 INFO:tasks.workunit.client.0.smithi060.stdout:removed './182/16' 2024-04-14T23:53:56.550 INFO:tasks.workunit.client.0.smithi060.stdout:removed './182/32' 2024-04-14T23:53:56.550 INFO:tasks.workunit.client.0.smithi060.stdout:removed './182/30' 2024-04-14T23:53:56.550 INFO:tasks.workunit.client.0.smithi060.stdout:removed './182/14' 2024-04-14T23:53:56.550 INFO:tasks.workunit.client.0.smithi060.stdout:removed './182/18' 2024-04-14T23:53:56.550 INFO:tasks.workunit.client.0.smithi060.stdout:removed './182/13' 2024-04-14T23:53:56.550 INFO:tasks.workunit.client.0.smithi060.stdout:removed './182/33' 2024-04-14T23:53:56.550 INFO:tasks.workunit.client.0.smithi060.stdout:removed './182/17' 2024-04-14T23:53:56.550 INFO:tasks.workunit.client.0.smithi060.stdout:removed './182/35' 2024-04-14T23:53:56.551 INFO:tasks.workunit.client.0.smithi060.stdout:removed './182/37' 2024-04-14T23:53:56.551 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './182' 2024-04-14T23:53:56.551 INFO:tasks.workunit.client.0.smithi060.stdout:removed './183/19' 2024-04-14T23:53:56.551 INFO:tasks.workunit.client.0.smithi060.stdout:removed './183/15' 2024-04-14T23:53:56.551 INFO:tasks.workunit.client.0.smithi060.stdout:removed './183/31' 2024-04-14T23:53:56.551 INFO:tasks.workunit.client.0.smithi060.stdout:removed './183/36' 2024-04-14T23:53:56.551 INFO:tasks.workunit.client.0.smithi060.stdout:removed './183/20' 2024-04-14T23:53:56.551 INFO:tasks.workunit.client.0.smithi060.stdout:removed './183/12' 2024-04-14T23:53:56.551 INFO:tasks.workunit.client.0.smithi060.stdout:removed './183/39' 2024-04-14T23:53:56.551 INFO:tasks.workunit.client.0.smithi060.stdout:removed './183/10' 2024-04-14T23:53:56.551 INFO:tasks.workunit.client.0.smithi060.stdout:removed './183/34' 2024-04-14T23:53:56.551 INFO:tasks.workunit.client.0.smithi060.stdout:removed './183/38' 2024-04-14T23:53:56.551 INFO:tasks.workunit.client.0.smithi060.stdout:removed './183/11' 2024-04-14T23:53:56.551 INFO:tasks.workunit.client.0.smithi060.stdout:removed './183/16' 2024-04-14T23:53:56.552 INFO:tasks.workunit.client.0.smithi060.stdout:removed './183/32' 2024-04-14T23:53:56.552 INFO:tasks.workunit.client.0.smithi060.stdout:removed './183/30' 2024-04-14T23:53:56.552 INFO:tasks.workunit.client.0.smithi060.stdout:removed './183/14' 2024-04-14T23:53:56.552 INFO:tasks.workunit.client.0.smithi060.stdout:removed './183/18' 2024-04-14T23:53:56.552 INFO:tasks.workunit.client.0.smithi060.stdout:removed './183/13' 2024-04-14T23:53:56.552 INFO:tasks.workunit.client.0.smithi060.stdout:removed './183/33' 2024-04-14T23:53:56.552 INFO:tasks.workunit.client.0.smithi060.stdout:removed './183/17' 2024-04-14T23:53:56.552 INFO:tasks.workunit.client.0.smithi060.stdout:removed './183/35' 2024-04-14T23:53:56.552 INFO:tasks.workunit.client.0.smithi060.stdout:removed './183/37' 2024-04-14T23:53:56.552 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './183' 2024-04-14T23:53:56.552 INFO:tasks.workunit.client.0.smithi060.stdout:removed './184/19' 2024-04-14T23:53:56.552 INFO:tasks.workunit.client.0.smithi060.stdout:removed './184/15' 2024-04-14T23:53:56.552 INFO:tasks.workunit.client.0.smithi060.stdout:removed './184/31' 2024-04-14T23:53:56.552 INFO:tasks.workunit.client.0.smithi060.stdout:removed './184/36' 2024-04-14T23:53:56.553 INFO:tasks.workunit.client.0.smithi060.stdout:removed './184/20' 2024-04-14T23:53:56.553 INFO:tasks.workunit.client.0.smithi060.stdout:removed './184/12' 2024-04-14T23:53:56.553 INFO:tasks.workunit.client.0.smithi060.stdout:removed './184/39' 2024-04-14T23:53:56.553 INFO:tasks.workunit.client.0.smithi060.stdout:removed './184/10' 2024-04-14T23:53:56.553 INFO:tasks.workunit.client.0.smithi060.stdout:removed './184/34' 2024-04-14T23:53:56.553 INFO:tasks.workunit.client.0.smithi060.stdout:removed './184/38' 2024-04-14T23:53:56.553 INFO:tasks.workunit.client.0.smithi060.stdout:removed './184/11' 2024-04-14T23:53:56.553 INFO:tasks.workunit.client.0.smithi060.stdout:removed './184/16' 2024-04-14T23:53:56.553 INFO:tasks.workunit.client.0.smithi060.stdout:removed './184/32' 2024-04-14T23:53:56.553 INFO:tasks.workunit.client.0.smithi060.stdout:removed './184/30' 2024-04-14T23:53:56.553 INFO:tasks.workunit.client.0.smithi060.stdout:removed './184/14' 2024-04-14T23:53:56.553 INFO:tasks.workunit.client.0.smithi060.stdout:removed './184/18' 2024-04-14T23:53:56.554 INFO:tasks.workunit.client.0.smithi060.stdout:removed './184/13' 2024-04-14T23:53:56.554 INFO:tasks.workunit.client.0.smithi060.stdout:removed './184/33' 2024-04-14T23:53:56.554 INFO:tasks.workunit.client.0.smithi060.stdout:removed './184/17' 2024-04-14T23:53:56.554 INFO:tasks.workunit.client.0.smithi060.stdout:removed './184/35' 2024-04-14T23:53:56.554 INFO:tasks.workunit.client.0.smithi060.stdout:removed './184/37' 2024-04-14T23:53:56.554 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './184' 2024-04-14T23:53:56.554 INFO:tasks.workunit.client.0.smithi060.stdout:removed './185/19' 2024-04-14T23:53:56.554 INFO:tasks.workunit.client.0.smithi060.stdout:removed './185/15' 2024-04-14T23:53:56.554 INFO:tasks.workunit.client.0.smithi060.stdout:removed './185/31' 2024-04-14T23:53:56.554 INFO:tasks.workunit.client.0.smithi060.stdout:removed './185/36' 2024-04-14T23:53:56.554 INFO:tasks.workunit.client.0.smithi060.stdout:removed './185/20' 2024-04-14T23:53:56.554 INFO:tasks.workunit.client.0.smithi060.stdout:removed './185/12' 2024-04-14T23:53:56.554 INFO:tasks.workunit.client.0.smithi060.stdout:removed './185/39' 2024-04-14T23:53:56.555 INFO:tasks.workunit.client.0.smithi060.stdout:removed './185/10' 2024-04-14T23:53:56.555 INFO:tasks.workunit.client.0.smithi060.stdout:removed './185/34' 2024-04-14T23:53:56.555 INFO:tasks.workunit.client.0.smithi060.stdout:removed './185/38' 2024-04-14T23:53:56.555 INFO:tasks.workunit.client.0.smithi060.stdout:removed './185/11' 2024-04-14T23:53:56.555 INFO:tasks.workunit.client.0.smithi060.stdout:removed './185/16' 2024-04-14T23:53:56.555 INFO:tasks.workunit.client.0.smithi060.stdout:removed './185/32' 2024-04-14T23:53:56.555 INFO:tasks.workunit.client.0.smithi060.stdout:removed './185/30' 2024-04-14T23:53:56.555 INFO:tasks.workunit.client.0.smithi060.stdout:removed './185/14' 2024-04-14T23:53:56.555 INFO:tasks.workunit.client.0.smithi060.stdout:removed './185/18' 2024-04-14T23:53:56.555 INFO:tasks.workunit.client.0.smithi060.stdout:removed './185/13' 2024-04-14T23:53:56.555 INFO:tasks.workunit.client.0.smithi060.stdout:removed './185/33' 2024-04-14T23:53:56.555 INFO:tasks.workunit.client.0.smithi060.stdout:removed './185/17' 2024-04-14T23:53:56.555 INFO:tasks.workunit.client.0.smithi060.stdout:removed './185/35' 2024-04-14T23:53:56.555 INFO:tasks.workunit.client.0.smithi060.stdout:removed './185/37' 2024-04-14T23:53:56.556 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './185' 2024-04-14T23:53:56.556 INFO:tasks.workunit.client.0.smithi060.stdout:removed './186/19' 2024-04-14T23:53:56.556 INFO:tasks.workunit.client.0.smithi060.stdout:removed './186/15' 2024-04-14T23:53:56.556 INFO:tasks.workunit.client.0.smithi060.stdout:removed './186/31' 2024-04-14T23:53:56.556 INFO:tasks.workunit.client.0.smithi060.stdout:removed './186/36' 2024-04-14T23:53:56.556 INFO:tasks.workunit.client.0.smithi060.stdout:removed './186/20' 2024-04-14T23:53:56.556 INFO:tasks.workunit.client.0.smithi060.stdout:removed './186/12' 2024-04-14T23:53:56.556 INFO:tasks.workunit.client.0.smithi060.stdout:removed './186/39' 2024-04-14T23:53:56.556 INFO:tasks.workunit.client.0.smithi060.stdout:removed './186/10' 2024-04-14T23:53:56.556 INFO:tasks.workunit.client.0.smithi060.stdout:removed './186/34' 2024-04-14T23:53:56.556 INFO:tasks.workunit.client.0.smithi060.stdout:removed './186/38' 2024-04-14T23:53:56.556 INFO:tasks.workunit.client.0.smithi060.stdout:removed './186/11' 2024-04-14T23:53:56.556 INFO:tasks.workunit.client.0.smithi060.stdout:removed './186/16' 2024-04-14T23:53:56.556 INFO:tasks.workunit.client.0.smithi060.stdout:removed './186/32' 2024-04-14T23:53:56.556 INFO:tasks.workunit.client.0.smithi060.stdout:removed './186/30' 2024-04-14T23:53:56.557 INFO:tasks.workunit.client.0.smithi060.stdout:removed './186/14' 2024-04-14T23:53:56.557 INFO:tasks.workunit.client.0.smithi060.stdout:removed './186/18' 2024-04-14T23:53:57.936 INFO:tasks.workunit.client.0.smithi060.stdout:removed './186/13' 2024-04-14T23:53:57.936 INFO:tasks.workunit.client.0.smithi060.stdout:removed './186/33' 2024-04-14T23:53:57.936 INFO:tasks.workunit.client.0.smithi060.stdout:removed './186/17' 2024-04-14T23:53:57.936 INFO:tasks.workunit.client.0.smithi060.stdout:removed './186/35' 2024-04-14T23:53:57.937 INFO:tasks.workunit.client.0.smithi060.stdout:removed './186/37' 2024-04-14T23:53:57.937 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './186' 2024-04-14T23:53:57.937 INFO:tasks.workunit.client.0.smithi060.stdout:removed './187/19' 2024-04-14T23:53:57.937 INFO:tasks.workunit.client.0.smithi060.stdout:removed './187/15' 2024-04-14T23:53:57.937 INFO:tasks.workunit.client.0.smithi060.stdout:removed './187/31' 2024-04-14T23:53:57.937 INFO:tasks.workunit.client.0.smithi060.stdout:removed './187/36' 2024-04-14T23:53:57.938 INFO:tasks.workunit.client.0.smithi060.stdout:removed './187/20' 2024-04-14T23:53:57.938 INFO:tasks.workunit.client.0.smithi060.stdout:removed './187/12' 2024-04-14T23:53:57.938 INFO:tasks.workunit.client.0.smithi060.stdout:removed './187/39' 2024-04-14T23:53:57.938 INFO:tasks.workunit.client.0.smithi060.stdout:removed './187/10' 2024-04-14T23:53:57.938 INFO:tasks.workunit.client.0.smithi060.stdout:removed './187/34' 2024-04-14T23:53:57.938 INFO:tasks.workunit.client.0.smithi060.stdout:removed './187/38' 2024-04-14T23:53:57.939 INFO:tasks.workunit.client.0.smithi060.stdout:removed './187/11' 2024-04-14T23:53:57.939 INFO:tasks.workunit.client.0.smithi060.stdout:removed './187/16' 2024-04-14T23:53:57.939 INFO:tasks.workunit.client.0.smithi060.stdout:removed './187/32' 2024-04-14T23:53:57.939 INFO:tasks.workunit.client.0.smithi060.stdout:removed './187/30' 2024-04-14T23:53:57.939 INFO:tasks.workunit.client.0.smithi060.stdout:removed './187/14' 2024-04-14T23:53:57.939 INFO:tasks.workunit.client.0.smithi060.stdout:removed './187/18' 2024-04-14T23:53:57.940 INFO:tasks.workunit.client.0.smithi060.stdout:removed './187/13' 2024-04-14T23:53:57.940 INFO:tasks.workunit.client.0.smithi060.stdout:removed './187/33' 2024-04-14T23:53:57.940 INFO:tasks.workunit.client.0.smithi060.stdout:removed './187/17' 2024-04-14T23:53:57.940 INFO:tasks.workunit.client.0.smithi060.stdout:removed './187/35' 2024-04-14T23:53:57.940 INFO:tasks.workunit.client.0.smithi060.stdout:removed './187/37' 2024-04-14T23:53:57.940 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './187' 2024-04-14T23:53:57.941 INFO:tasks.workunit.client.0.smithi060.stdout:removed './188/19' 2024-04-14T23:53:57.941 INFO:tasks.workunit.client.0.smithi060.stdout:removed './188/15' 2024-04-14T23:53:57.941 INFO:tasks.workunit.client.0.smithi060.stdout:removed './188/31' 2024-04-14T23:53:57.941 INFO:tasks.workunit.client.0.smithi060.stdout:removed './188/36' 2024-04-14T23:53:57.941 INFO:tasks.workunit.client.0.smithi060.stdout:removed './188/20' 2024-04-14T23:53:57.941 INFO:tasks.workunit.client.0.smithi060.stdout:removed './188/12' 2024-04-14T23:53:57.941 INFO:tasks.workunit.client.0.smithi060.stdout:removed './188/39' 2024-04-14T23:53:57.941 INFO:tasks.workunit.client.0.smithi060.stdout:removed './188/10' 2024-04-14T23:53:57.942 INFO:tasks.workunit.client.0.smithi060.stdout:removed './188/34' 2024-04-14T23:53:57.942 INFO:tasks.workunit.client.0.smithi060.stdout:removed './188/38' 2024-04-14T23:53:57.942 INFO:tasks.workunit.client.0.smithi060.stdout:removed './188/11' 2024-04-14T23:53:57.942 INFO:tasks.workunit.client.0.smithi060.stdout:removed './188/16' 2024-04-14T23:53:57.942 INFO:tasks.workunit.client.0.smithi060.stdout:removed './188/32' 2024-04-14T23:53:57.942 INFO:tasks.workunit.client.0.smithi060.stdout:removed './188/30' 2024-04-14T23:53:57.942 INFO:tasks.workunit.client.0.smithi060.stdout:removed './188/14' 2024-04-14T23:53:57.943 INFO:tasks.workunit.client.0.smithi060.stdout:removed './188/18' 2024-04-14T23:53:57.943 INFO:tasks.workunit.client.0.smithi060.stdout:removed './188/13' 2024-04-14T23:53:57.943 INFO:tasks.workunit.client.0.smithi060.stdout:removed './188/33' 2024-04-14T23:53:57.943 INFO:tasks.workunit.client.0.smithi060.stdout:removed './188/17' 2024-04-14T23:53:57.943 INFO:tasks.workunit.client.0.smithi060.stdout:removed './188/35' 2024-04-14T23:53:57.943 INFO:tasks.workunit.client.0.smithi060.stdout:removed './188/37' 2024-04-14T23:53:57.943 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './188' 2024-04-14T23:53:57.943 INFO:tasks.workunit.client.0.smithi060.stdout:removed './189/19' 2024-04-14T23:53:57.943 INFO:tasks.workunit.client.0.smithi060.stdout:removed './189/15' 2024-04-14T23:53:57.943 INFO:tasks.workunit.client.0.smithi060.stdout:removed './189/31' 2024-04-14T23:53:57.944 INFO:tasks.workunit.client.0.smithi060.stdout:removed './189/36' 2024-04-14T23:53:57.944 INFO:tasks.workunit.client.0.smithi060.stdout:removed './189/20' 2024-04-14T23:53:57.944 INFO:tasks.workunit.client.0.smithi060.stdout:removed './189/12' 2024-04-14T23:53:57.944 INFO:tasks.workunit.client.0.smithi060.stdout:removed './189/39' 2024-04-14T23:53:57.944 INFO:tasks.workunit.client.0.smithi060.stdout:removed './189/10' 2024-04-14T23:53:57.944 INFO:tasks.workunit.client.0.smithi060.stdout:removed './189/34' 2024-04-14T23:53:57.944 INFO:tasks.workunit.client.0.smithi060.stdout:removed './189/38' 2024-04-14T23:53:57.944 INFO:tasks.workunit.client.0.smithi060.stdout:removed './189/11' 2024-04-14T23:53:57.944 INFO:tasks.workunit.client.0.smithi060.stdout:removed './189/16' 2024-04-14T23:53:57.945 INFO:tasks.workunit.client.0.smithi060.stdout:removed './189/32' 2024-04-14T23:53:57.945 INFO:tasks.workunit.client.0.smithi060.stdout:removed './189/30' 2024-04-14T23:53:57.945 INFO:tasks.workunit.client.0.smithi060.stdout:removed './189/14' 2024-04-14T23:53:57.945 INFO:tasks.workunit.client.0.smithi060.stdout:removed './189/18' 2024-04-14T23:53:57.945 INFO:tasks.workunit.client.0.smithi060.stdout:removed './189/13' 2024-04-14T23:53:57.945 INFO:tasks.workunit.client.0.smithi060.stdout:removed './189/33' 2024-04-14T23:53:57.945 INFO:tasks.workunit.client.0.smithi060.stdout:removed './189/17' 2024-04-14T23:53:57.945 INFO:tasks.workunit.client.0.smithi060.stdout:removed './189/35' 2024-04-14T23:53:57.946 INFO:tasks.workunit.client.0.smithi060.stdout:removed './189/37' 2024-04-14T23:53:57.946 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './189' 2024-04-14T23:53:57.946 INFO:tasks.workunit.client.0.smithi060.stdout:removed './190/19' 2024-04-14T23:53:57.946 INFO:tasks.workunit.client.0.smithi060.stdout:removed './190/15' 2024-04-14T23:53:57.946 INFO:tasks.workunit.client.0.smithi060.stdout:removed './190/31' 2024-04-14T23:53:57.946 INFO:tasks.workunit.client.0.smithi060.stdout:removed './190/36' 2024-04-14T23:53:57.946 INFO:tasks.workunit.client.0.smithi060.stdout:removed './190/20' 2024-04-14T23:53:57.946 INFO:tasks.workunit.client.0.smithi060.stdout:removed './190/12' 2024-04-14T23:53:57.946 INFO:tasks.workunit.client.0.smithi060.stdout:removed './190/39' 2024-04-14T23:53:57.947 INFO:tasks.workunit.client.0.smithi060.stdout:removed './190/10' 2024-04-14T23:53:57.947 INFO:tasks.workunit.client.0.smithi060.stdout:removed './190/34' 2024-04-14T23:53:57.947 INFO:tasks.workunit.client.0.smithi060.stdout:removed './190/38' 2024-04-14T23:53:57.947 INFO:tasks.workunit.client.0.smithi060.stdout:removed './190/11' 2024-04-14T23:53:57.947 INFO:tasks.workunit.client.0.smithi060.stdout:removed './190/16' 2024-04-14T23:53:57.947 INFO:tasks.workunit.client.0.smithi060.stdout:removed './190/32' 2024-04-14T23:53:57.947 INFO:tasks.workunit.client.0.smithi060.stdout:removed './190/30' 2024-04-14T23:53:57.947 INFO:tasks.workunit.client.0.smithi060.stdout:removed './190/14' 2024-04-14T23:53:57.947 INFO:tasks.workunit.client.0.smithi060.stdout:removed './190/18' 2024-04-14T23:53:57.947 INFO:tasks.workunit.client.0.smithi060.stdout:removed './190/13' 2024-04-14T23:53:57.947 INFO:tasks.workunit.client.0.smithi060.stdout:removed './190/33' 2024-04-14T23:53:57.947 INFO:tasks.workunit.client.0.smithi060.stdout:removed './190/17' 2024-04-14T23:53:57.947 INFO:tasks.workunit.client.0.smithi060.stdout:removed './190/35' 2024-04-14T23:53:57.948 INFO:tasks.workunit.client.0.smithi060.stdout:removed './190/37' 2024-04-14T23:53:57.948 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './190' 2024-04-14T23:53:57.948 INFO:tasks.workunit.client.0.smithi060.stdout:removed './191/19' 2024-04-14T23:53:57.948 INFO:tasks.workunit.client.0.smithi060.stdout:removed './191/15' 2024-04-14T23:53:57.948 INFO:tasks.workunit.client.0.smithi060.stdout:removed './191/31' 2024-04-14T23:53:57.948 INFO:tasks.workunit.client.0.smithi060.stdout:removed './191/36' 2024-04-14T23:53:57.948 INFO:tasks.workunit.client.0.smithi060.stdout:removed './191/20' 2024-04-14T23:53:57.948 INFO:tasks.workunit.client.0.smithi060.stdout:removed './191/12' 2024-04-14T23:53:57.948 INFO:tasks.workunit.client.0.smithi060.stdout:removed './191/39' 2024-04-14T23:53:57.948 INFO:tasks.workunit.client.0.smithi060.stdout:removed './191/10' 2024-04-14T23:53:57.948 INFO:tasks.workunit.client.0.smithi060.stdout:removed './191/34' 2024-04-14T23:53:57.948 INFO:tasks.workunit.client.0.smithi060.stdout:removed './191/38' 2024-04-14T23:53:57.948 INFO:tasks.workunit.client.0.smithi060.stdout:removed './191/11' 2024-04-14T23:53:57.948 INFO:tasks.workunit.client.0.smithi060.stdout:removed './191/16' 2024-04-14T23:53:57.948 INFO:tasks.workunit.client.0.smithi060.stdout:removed './191/32' 2024-04-14T23:53:57.948 INFO:tasks.workunit.client.0.smithi060.stdout:removed './191/30' 2024-04-14T23:53:57.949 INFO:tasks.workunit.client.0.smithi060.stdout:removed './191/14' 2024-04-14T23:53:57.949 INFO:tasks.workunit.client.0.smithi060.stdout:removed './191/18' 2024-04-14T23:53:57.949 INFO:tasks.workunit.client.0.smithi060.stdout:removed './191/13' 2024-04-14T23:53:57.949 INFO:tasks.workunit.client.0.smithi060.stdout:removed './191/33' 2024-04-14T23:53:57.949 INFO:tasks.workunit.client.0.smithi060.stdout:removed './191/17' 2024-04-14T23:53:57.949 INFO:tasks.workunit.client.0.smithi060.stdout:removed './191/35' 2024-04-14T23:53:57.949 INFO:tasks.workunit.client.0.smithi060.stdout:removed './191/37' 2024-04-14T23:53:57.949 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './191' 2024-04-14T23:53:57.949 INFO:tasks.workunit.client.0.smithi060.stdout:removed './192/19' 2024-04-14T23:53:57.949 INFO:tasks.workunit.client.0.smithi060.stdout:removed './192/15' 2024-04-14T23:53:57.949 INFO:tasks.workunit.client.0.smithi060.stdout:removed './192/31' 2024-04-14T23:53:57.950 INFO:tasks.workunit.client.0.smithi060.stdout:removed './192/36' 2024-04-14T23:53:57.950 INFO:tasks.workunit.client.0.smithi060.stdout:removed './192/20' 2024-04-14T23:53:57.950 INFO:tasks.workunit.client.0.smithi060.stdout:removed './192/12' 2024-04-14T23:53:57.950 INFO:tasks.workunit.client.0.smithi060.stdout:removed './192/39' 2024-04-14T23:53:57.950 INFO:tasks.workunit.client.0.smithi060.stdout:removed './192/10' 2024-04-14T23:53:57.950 INFO:tasks.workunit.client.0.smithi060.stdout:removed './192/34' 2024-04-14T23:53:57.950 INFO:tasks.workunit.client.0.smithi060.stdout:removed './192/38' 2024-04-14T23:53:57.950 INFO:tasks.workunit.client.0.smithi060.stdout:removed './192/11' 2024-04-14T23:53:57.950 INFO:tasks.workunit.client.0.smithi060.stdout:removed './192/16' 2024-04-14T23:53:57.950 INFO:tasks.workunit.client.0.smithi060.stdout:removed './192/32' 2024-04-14T23:53:57.950 INFO:tasks.workunit.client.0.smithi060.stdout:removed './192/30' 2024-04-14T23:53:57.950 INFO:tasks.workunit.client.0.smithi060.stdout:removed './192/14' 2024-04-14T23:53:57.951 INFO:tasks.workunit.client.0.smithi060.stdout:removed './192/18' 2024-04-14T23:53:57.951 INFO:tasks.workunit.client.0.smithi060.stdout:removed './192/13' 2024-04-14T23:53:57.951 INFO:tasks.workunit.client.0.smithi060.stdout:removed './192/33' 2024-04-14T23:53:57.951 INFO:tasks.workunit.client.0.smithi060.stdout:removed './192/17' 2024-04-14T23:53:57.951 INFO:tasks.workunit.client.0.smithi060.stdout:removed './192/35' 2024-04-14T23:53:57.951 INFO:tasks.workunit.client.0.smithi060.stdout:removed './192/37' 2024-04-14T23:53:57.951 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './192' 2024-04-14T23:53:57.951 INFO:tasks.workunit.client.0.smithi060.stdout:removed './193/19' 2024-04-14T23:53:57.951 INFO:tasks.workunit.client.0.smithi060.stdout:removed './193/15' 2024-04-14T23:53:57.951 INFO:tasks.workunit.client.0.smithi060.stdout:removed './193/31' 2024-04-14T23:53:57.951 INFO:tasks.workunit.client.0.smithi060.stdout:removed './193/36' 2024-04-14T23:53:57.951 INFO:tasks.workunit.client.0.smithi060.stdout:removed './193/20' 2024-04-14T23:53:57.951 INFO:tasks.workunit.client.0.smithi060.stdout:removed './193/12' 2024-04-14T23:53:57.951 INFO:tasks.workunit.client.0.smithi060.stdout:removed './193/39' 2024-04-14T23:53:57.952 INFO:tasks.workunit.client.0.smithi060.stdout:removed './193/10' 2024-04-14T23:53:57.952 INFO:tasks.workunit.client.0.smithi060.stdout:removed './193/34' 2024-04-14T23:53:57.952 INFO:tasks.workunit.client.0.smithi060.stdout:removed './193/38' 2024-04-14T23:53:57.952 INFO:tasks.workunit.client.0.smithi060.stdout:removed './193/11' 2024-04-14T23:53:57.952 INFO:tasks.workunit.client.0.smithi060.stdout:removed './193/16' 2024-04-14T23:53:57.952 INFO:tasks.workunit.client.0.smithi060.stdout:removed './193/32' 2024-04-14T23:53:57.952 INFO:tasks.workunit.client.0.smithi060.stdout:removed './193/30' 2024-04-14T23:53:57.952 INFO:tasks.workunit.client.0.smithi060.stdout:removed './193/14' 2024-04-14T23:53:57.952 INFO:tasks.workunit.client.0.smithi060.stdout:removed './193/18' 2024-04-14T23:53:57.952 INFO:tasks.workunit.client.0.smithi060.stdout:removed './193/13' 2024-04-14T23:53:57.952 INFO:tasks.workunit.client.0.smithi060.stdout:removed './193/33' 2024-04-14T23:53:57.952 INFO:tasks.workunit.client.0.smithi060.stdout:removed './193/17' 2024-04-14T23:53:57.952 INFO:tasks.workunit.client.0.smithi060.stdout:removed './193/35' 2024-04-14T23:53:57.952 INFO:tasks.workunit.client.0.smithi060.stdout:removed './193/37' 2024-04-14T23:53:57.952 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './193' 2024-04-14T23:53:57.953 INFO:tasks.workunit.client.0.smithi060.stdout:removed './194/19' 2024-04-14T23:53:57.953 INFO:tasks.workunit.client.0.smithi060.stdout:removed './194/15' 2024-04-14T23:53:57.953 INFO:tasks.workunit.client.0.smithi060.stdout:removed './194/31' 2024-04-14T23:53:57.953 INFO:tasks.workunit.client.0.smithi060.stdout:removed './194/36' 2024-04-14T23:53:57.953 INFO:tasks.workunit.client.0.smithi060.stdout:removed './194/20' 2024-04-14T23:53:57.953 INFO:tasks.workunit.client.0.smithi060.stdout:removed './194/12' 2024-04-14T23:53:57.953 INFO:tasks.workunit.client.0.smithi060.stdout:removed './194/39' 2024-04-14T23:53:57.953 INFO:tasks.workunit.client.0.smithi060.stdout:removed './194/10' 2024-04-14T23:53:57.953 INFO:tasks.workunit.client.0.smithi060.stdout:removed './194/34' 2024-04-14T23:53:57.953 INFO:tasks.workunit.client.0.smithi060.stdout:removed './194/38' 2024-04-14T23:53:57.953 INFO:tasks.workunit.client.0.smithi060.stdout:removed './194/11' 2024-04-14T23:53:57.953 INFO:tasks.workunit.client.0.smithi060.stdout:removed './194/16' 2024-04-14T23:53:57.953 INFO:tasks.workunit.client.0.smithi060.stdout:removed './194/32' 2024-04-14T23:53:57.954 INFO:tasks.workunit.client.0.smithi060.stdout:removed './194/30' 2024-04-14T23:53:57.954 INFO:tasks.workunit.client.0.smithi060.stdout:removed './194/14' 2024-04-14T23:53:57.954 INFO:tasks.workunit.client.0.smithi060.stdout:removed './194/18' 2024-04-14T23:53:57.954 INFO:tasks.workunit.client.0.smithi060.stdout:removed './194/13' 2024-04-14T23:53:57.954 INFO:tasks.workunit.client.0.smithi060.stdout:removed './194/33' 2024-04-14T23:53:57.954 INFO:tasks.workunit.client.0.smithi060.stdout:removed './194/17' 2024-04-14T23:53:57.954 INFO:tasks.workunit.client.0.smithi060.stdout:removed './194/35' 2024-04-14T23:53:57.954 INFO:tasks.workunit.client.0.smithi060.stdout:removed './194/37' 2024-04-14T23:53:57.954 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './194' 2024-04-14T23:53:57.954 INFO:tasks.workunit.client.0.smithi060.stdout:removed './195/19' 2024-04-14T23:53:57.954 INFO:tasks.workunit.client.0.smithi060.stdout:removed './195/15' 2024-04-14T23:53:57.954 INFO:tasks.workunit.client.0.smithi060.stdout:removed './195/31' 2024-04-14T23:53:57.954 INFO:tasks.workunit.client.0.smithi060.stdout:removed './195/36' 2024-04-14T23:53:57.955 INFO:tasks.workunit.client.0.smithi060.stdout:removed './195/20' 2024-04-14T23:53:57.955 INFO:tasks.workunit.client.0.smithi060.stdout:removed './195/12' 2024-04-14T23:53:57.955 INFO:tasks.workunit.client.0.smithi060.stdout:removed './195/39' 2024-04-14T23:53:57.955 INFO:tasks.workunit.client.0.smithi060.stdout:removed './195/10' 2024-04-14T23:53:57.955 INFO:tasks.workunit.client.0.smithi060.stdout:removed './195/34' 2024-04-14T23:53:57.955 INFO:tasks.workunit.client.0.smithi060.stdout:removed './195/38' 2024-04-14T23:53:57.955 INFO:tasks.workunit.client.0.smithi060.stdout:removed './195/11' 2024-04-14T23:53:57.955 INFO:tasks.workunit.client.0.smithi060.stdout:removed './195/16' 2024-04-14T23:53:57.955 INFO:tasks.workunit.client.0.smithi060.stdout:removed './195/32' 2024-04-14T23:53:57.955 INFO:tasks.workunit.client.0.smithi060.stdout:removed './195/30' 2024-04-14T23:53:57.955 INFO:tasks.workunit.client.0.smithi060.stdout:removed './195/14' 2024-04-14T23:53:57.955 INFO:tasks.workunit.client.0.smithi060.stdout:removed './195/18' 2024-04-14T23:53:57.955 INFO:tasks.workunit.client.0.smithi060.stdout:removed './195/13' 2024-04-14T23:53:57.955 INFO:tasks.workunit.client.0.smithi060.stdout:removed './195/33' 2024-04-14T23:53:57.955 INFO:tasks.workunit.client.0.smithi060.stdout:removed './195/17' 2024-04-14T23:53:57.956 INFO:tasks.workunit.client.0.smithi060.stdout:removed './195/35' 2024-04-14T23:53:57.956 INFO:tasks.workunit.client.0.smithi060.stdout:removed './195/37' 2024-04-14T23:53:57.956 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './195' 2024-04-14T23:53:57.956 INFO:tasks.workunit.client.0.smithi060.stdout:removed './196/19' 2024-04-14T23:53:57.956 INFO:tasks.workunit.client.0.smithi060.stdout:removed './196/15' 2024-04-14T23:53:57.956 INFO:tasks.workunit.client.0.smithi060.stdout:removed './196/31' 2024-04-14T23:53:57.956 INFO:tasks.workunit.client.0.smithi060.stdout:removed './196/36' 2024-04-14T23:53:57.956 INFO:tasks.workunit.client.0.smithi060.stdout:removed './196/20' 2024-04-14T23:53:57.956 INFO:tasks.workunit.client.0.smithi060.stdout:removed './196/12' 2024-04-14T23:53:57.956 INFO:tasks.workunit.client.0.smithi060.stdout:removed './196/39' 2024-04-14T23:53:57.956 INFO:tasks.workunit.client.0.smithi060.stdout:removed './196/10' 2024-04-14T23:53:59.654 INFO:tasks.workunit.client.0.smithi060.stdout:removed './196/34' 2024-04-14T23:53:59.654 INFO:tasks.workunit.client.0.smithi060.stdout:removed './196/38' 2024-04-14T23:53:59.654 INFO:tasks.workunit.client.0.smithi060.stdout:removed './196/11' 2024-04-14T23:53:59.654 INFO:tasks.workunit.client.0.smithi060.stdout:removed './196/16' 2024-04-14T23:53:59.654 INFO:tasks.workunit.client.0.smithi060.stdout:removed './196/32' 2024-04-14T23:53:59.654 INFO:tasks.workunit.client.0.smithi060.stdout:removed './196/30' 2024-04-14T23:53:59.654 INFO:tasks.workunit.client.0.smithi060.stdout:removed './196/14' 2024-04-14T23:53:59.655 INFO:tasks.workunit.client.0.smithi060.stdout:removed './196/18' 2024-04-14T23:53:59.655 INFO:tasks.workunit.client.0.smithi060.stdout:removed './196/13' 2024-04-14T23:53:59.655 INFO:tasks.workunit.client.0.smithi060.stdout:removed './196/33' 2024-04-14T23:53:59.655 INFO:tasks.workunit.client.0.smithi060.stdout:removed './196/17' 2024-04-14T23:53:59.655 INFO:tasks.workunit.client.0.smithi060.stdout:removed './196/35' 2024-04-14T23:53:59.655 INFO:tasks.workunit.client.0.smithi060.stdout:removed './196/37' 2024-04-14T23:53:59.655 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './196' 2024-04-14T23:53:59.655 INFO:tasks.workunit.client.0.smithi060.stdout:removed './197/19' 2024-04-14T23:53:59.655 INFO:tasks.workunit.client.0.smithi060.stdout:removed './197/15' 2024-04-14T23:53:59.655 INFO:tasks.workunit.client.0.smithi060.stdout:removed './197/31' 2024-04-14T23:53:59.655 INFO:tasks.workunit.client.0.smithi060.stdout:removed './197/36' 2024-04-14T23:53:59.655 INFO:tasks.workunit.client.0.smithi060.stdout:removed './197/20' 2024-04-14T23:53:59.655 INFO:tasks.workunit.client.0.smithi060.stdout:removed './197/12' 2024-04-14T23:53:59.655 INFO:tasks.workunit.client.0.smithi060.stdout:removed './197/39' 2024-04-14T23:53:59.656 INFO:tasks.workunit.client.0.smithi060.stdout:removed './197/10' 2024-04-14T23:53:59.656 INFO:tasks.workunit.client.0.smithi060.stdout:removed './197/34' 2024-04-14T23:53:59.656 INFO:tasks.workunit.client.0.smithi060.stdout:removed './197/38' 2024-04-14T23:53:59.656 INFO:tasks.workunit.client.0.smithi060.stdout:removed './197/11' 2024-04-14T23:53:59.656 INFO:tasks.workunit.client.0.smithi060.stdout:removed './197/16' 2024-04-14T23:53:59.656 INFO:tasks.workunit.client.0.smithi060.stdout:removed './197/32' 2024-04-14T23:53:59.656 INFO:tasks.workunit.client.0.smithi060.stdout:removed './197/30' 2024-04-14T23:53:59.656 INFO:tasks.workunit.client.0.smithi060.stdout:removed './197/14' 2024-04-14T23:53:59.656 INFO:tasks.workunit.client.0.smithi060.stdout:removed './197/18' 2024-04-14T23:53:59.656 INFO:tasks.workunit.client.0.smithi060.stdout:removed './197/13' 2024-04-14T23:53:59.656 INFO:tasks.workunit.client.0.smithi060.stdout:removed './197/33' 2024-04-14T23:53:59.656 INFO:tasks.workunit.client.0.smithi060.stdout:removed './197/17' 2024-04-14T23:53:59.656 INFO:tasks.workunit.client.0.smithi060.stdout:removed './197/35' 2024-04-14T23:53:59.657 INFO:tasks.workunit.client.0.smithi060.stdout:removed './197/37' 2024-04-14T23:53:59.657 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './197' 2024-04-14T23:53:59.657 INFO:tasks.workunit.client.0.smithi060.stdout:removed './198/19' 2024-04-14T23:53:59.657 INFO:tasks.workunit.client.0.smithi060.stdout:removed './198/15' 2024-04-14T23:53:59.657 INFO:tasks.workunit.client.0.smithi060.stdout:removed './198/31' 2024-04-14T23:53:59.657 INFO:tasks.workunit.client.0.smithi060.stdout:removed './198/36' 2024-04-14T23:53:59.657 INFO:tasks.workunit.client.0.smithi060.stdout:removed './198/20' 2024-04-14T23:53:59.657 INFO:tasks.workunit.client.0.smithi060.stdout:removed './198/12' 2024-04-14T23:53:59.657 INFO:tasks.workunit.client.0.smithi060.stdout:removed './198/39' 2024-04-14T23:53:59.657 INFO:tasks.workunit.client.0.smithi060.stdout:removed './198/10' 2024-04-14T23:53:59.657 INFO:tasks.workunit.client.0.smithi060.stdout:removed './198/34' 2024-04-14T23:53:59.657 INFO:tasks.workunit.client.0.smithi060.stdout:removed './198/38' 2024-04-14T23:53:59.657 INFO:tasks.workunit.client.0.smithi060.stdout:removed './198/11' 2024-04-14T23:53:59.657 INFO:tasks.workunit.client.0.smithi060.stdout:removed './198/16' 2024-04-14T23:53:59.657 INFO:tasks.workunit.client.0.smithi060.stdout:removed './198/32' 2024-04-14T23:53:59.657 INFO:tasks.workunit.client.0.smithi060.stdout:removed './198/30' 2024-04-14T23:53:59.658 INFO:tasks.workunit.client.0.smithi060.stdout:removed './198/14' 2024-04-14T23:53:59.658 INFO:tasks.workunit.client.0.smithi060.stdout:removed './198/18' 2024-04-14T23:53:59.658 INFO:tasks.workunit.client.0.smithi060.stdout:removed './198/13' 2024-04-14T23:53:59.658 INFO:tasks.workunit.client.0.smithi060.stdout:removed './198/33' 2024-04-14T23:53:59.658 INFO:tasks.workunit.client.0.smithi060.stdout:removed './198/17' 2024-04-14T23:53:59.658 INFO:tasks.workunit.client.0.smithi060.stdout:removed './198/35' 2024-04-14T23:53:59.658 INFO:tasks.workunit.client.0.smithi060.stdout:removed './198/37' 2024-04-14T23:53:59.658 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './198' 2024-04-14T23:53:59.658 INFO:tasks.workunit.client.0.smithi060.stdout:removed './199/19' 2024-04-14T23:53:59.658 INFO:tasks.workunit.client.0.smithi060.stdout:removed './199/15' 2024-04-14T23:53:59.658 INFO:tasks.workunit.client.0.smithi060.stdout:removed './199/31' 2024-04-14T23:53:59.658 INFO:tasks.workunit.client.0.smithi060.stdout:removed './199/36' 2024-04-14T23:53:59.658 INFO:tasks.workunit.client.0.smithi060.stdout:removed './199/20' 2024-04-14T23:53:59.659 INFO:tasks.workunit.client.0.smithi060.stdout:removed './199/12' 2024-04-14T23:53:59.659 INFO:tasks.workunit.client.0.smithi060.stdout:removed './199/39' 2024-04-14T23:53:59.659 INFO:tasks.workunit.client.0.smithi060.stdout:removed './199/10' 2024-04-14T23:53:59.659 INFO:tasks.workunit.client.0.smithi060.stdout:removed './199/34' 2024-04-14T23:53:59.659 INFO:tasks.workunit.client.0.smithi060.stdout:removed './199/38' 2024-04-14T23:53:59.659 INFO:tasks.workunit.client.0.smithi060.stdout:removed './199/11' 2024-04-14T23:53:59.659 INFO:tasks.workunit.client.0.smithi060.stdout:removed './199/16' 2024-04-14T23:53:59.659 INFO:tasks.workunit.client.0.smithi060.stdout:removed './199/32' 2024-04-14T23:53:59.659 INFO:tasks.workunit.client.0.smithi060.stdout:removed './199/30' 2024-04-14T23:53:59.659 INFO:tasks.workunit.client.0.smithi060.stdout:removed './199/14' 2024-04-14T23:53:59.659 INFO:tasks.workunit.client.0.smithi060.stdout:removed './199/18' 2024-04-14T23:53:59.659 INFO:tasks.workunit.client.0.smithi060.stdout:removed './199/13' 2024-04-14T23:53:59.659 INFO:tasks.workunit.client.0.smithi060.stdout:removed './199/33' 2024-04-14T23:53:59.659 INFO:tasks.workunit.client.0.smithi060.stdout:removed './199/17' 2024-04-14T23:53:59.659 INFO:tasks.workunit.client.0.smithi060.stdout:removed './199/35' 2024-04-14T23:53:59.660 INFO:tasks.workunit.client.0.smithi060.stdout:removed './199/37' 2024-04-14T23:53:59.660 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './199' 2024-04-14T23:53:59.660 INFO:tasks.workunit.client.0.smithi060.stdout:removed './200/21' 2024-04-14T23:53:59.660 INFO:tasks.workunit.client.0.smithi060.stdout:removed './200/27' 2024-04-14T23:53:59.660 INFO:tasks.workunit.client.0.smithi060.stdout:removed './200/31' 2024-04-14T23:53:59.660 INFO:tasks.workunit.client.0.smithi060.stdout:removed './200/36' 2024-04-14T23:53:59.660 INFO:tasks.workunit.client.0.smithi060.stdout:removed './200/20' 2024-04-14T23:53:59.660 INFO:tasks.workunit.client.0.smithi060.stdout:removed './200/39' 2024-04-14T23:53:59.660 INFO:tasks.workunit.client.0.smithi060.stdout:removed './200/34' 2024-04-14T23:53:59.660 INFO:tasks.workunit.client.0.smithi060.stdout:removed './200/38' 2024-04-14T23:53:59.660 INFO:tasks.workunit.client.0.smithi060.stdout:removed './200/28' 2024-04-14T23:53:59.660 INFO:tasks.workunit.client.0.smithi060.stdout:removed './200/24' 2024-04-14T23:53:59.660 INFO:tasks.workunit.client.0.smithi060.stdout:removed './200/29' 2024-04-14T23:53:59.660 INFO:tasks.workunit.client.0.smithi060.stdout:removed './200/32' 2024-04-14T23:53:59.661 INFO:tasks.workunit.client.0.smithi060.stdout:removed './200/22' 2024-04-14T23:53:59.661 INFO:tasks.workunit.client.0.smithi060.stdout:removed './200/25' 2024-04-14T23:53:59.661 INFO:tasks.workunit.client.0.smithi060.stdout:removed './200/30' 2024-04-14T23:53:59.661 INFO:tasks.workunit.client.0.smithi060.stdout:removed './200/33' 2024-04-14T23:53:59.661 INFO:tasks.workunit.client.0.smithi060.stdout:removed './200/23' 2024-04-14T23:53:59.661 INFO:tasks.workunit.client.0.smithi060.stdout:removed './200/35' 2024-04-14T23:53:59.661 INFO:tasks.workunit.client.0.smithi060.stdout:removed './200/26' 2024-04-14T23:53:59.661 INFO:tasks.workunit.client.0.smithi060.stdout:removed './200/37' 2024-04-14T23:53:59.661 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './200' 2024-04-14T23:53:59.661 INFO:tasks.workunit.client.0.smithi060.stdout:removed './201/21' 2024-04-14T23:53:59.661 INFO:tasks.workunit.client.0.smithi060.stdout:removed './201/27' 2024-04-14T23:53:59.661 INFO:tasks.workunit.client.0.smithi060.stdout:removed './201/31' 2024-04-14T23:53:59.661 INFO:tasks.workunit.client.0.smithi060.stdout:removed './201/36' 2024-04-14T23:53:59.661 INFO:tasks.workunit.client.0.smithi060.stdout:removed './201/20' 2024-04-14T23:53:59.661 INFO:tasks.workunit.client.0.smithi060.stdout:removed './201/39' 2024-04-14T23:53:59.661 INFO:tasks.workunit.client.0.smithi060.stdout:removed './201/34' 2024-04-14T23:53:59.662 INFO:tasks.workunit.client.0.smithi060.stdout:removed './201/38' 2024-04-14T23:53:59.662 INFO:tasks.workunit.client.0.smithi060.stdout:removed './201/28' 2024-04-14T23:53:59.662 INFO:tasks.workunit.client.0.smithi060.stdout:removed './201/24' 2024-04-14T23:53:59.662 INFO:tasks.workunit.client.0.smithi060.stdout:removed './201/29' 2024-04-14T23:53:59.662 INFO:tasks.workunit.client.0.smithi060.stdout:removed './201/32' 2024-04-14T23:53:59.662 INFO:tasks.workunit.client.0.smithi060.stdout:removed './201/22' 2024-04-14T23:53:59.662 INFO:tasks.workunit.client.0.smithi060.stdout:removed './201/25' 2024-04-14T23:53:59.662 INFO:tasks.workunit.client.0.smithi060.stdout:removed './201/30' 2024-04-14T23:53:59.662 INFO:tasks.workunit.client.0.smithi060.stdout:removed './201/33' 2024-04-14T23:53:59.662 INFO:tasks.workunit.client.0.smithi060.stdout:removed './201/23' 2024-04-14T23:53:59.662 INFO:tasks.workunit.client.0.smithi060.stdout:removed './201/35' 2024-04-14T23:53:59.662 INFO:tasks.workunit.client.0.smithi060.stdout:removed './201/26' 2024-04-14T23:53:59.662 INFO:tasks.workunit.client.0.smithi060.stdout:removed './201/37' 2024-04-14T23:53:59.662 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './201' 2024-04-14T23:53:59.662 INFO:tasks.workunit.client.0.smithi060.stdout:removed './202/21' 2024-04-14T23:53:59.663 INFO:tasks.workunit.client.0.smithi060.stdout:removed './202/27' 2024-04-14T23:53:59.663 INFO:tasks.workunit.client.0.smithi060.stdout:removed './202/31' 2024-04-14T23:53:59.663 INFO:tasks.workunit.client.0.smithi060.stdout:removed './202/36' 2024-04-14T23:53:59.663 INFO:tasks.workunit.client.0.smithi060.stdout:removed './202/20' 2024-04-14T23:53:59.663 INFO:tasks.workunit.client.0.smithi060.stdout:removed './202/39' 2024-04-14T23:53:59.663 INFO:tasks.workunit.client.0.smithi060.stdout:removed './202/34' 2024-04-14T23:53:59.663 INFO:tasks.workunit.client.0.smithi060.stdout:removed './202/38' 2024-04-14T23:53:59.663 INFO:tasks.workunit.client.0.smithi060.stdout:removed './202/28' 2024-04-14T23:53:59.663 INFO:tasks.workunit.client.0.smithi060.stdout:removed './202/24' 2024-04-14T23:53:59.663 INFO:tasks.workunit.client.0.smithi060.stdout:removed './202/29' 2024-04-14T23:53:59.663 INFO:tasks.workunit.client.0.smithi060.stdout:removed './202/32' 2024-04-14T23:53:59.663 INFO:tasks.workunit.client.0.smithi060.stdout:removed './202/22' 2024-04-14T23:53:59.663 INFO:tasks.workunit.client.0.smithi060.stdout:removed './202/25' 2024-04-14T23:53:59.663 INFO:tasks.workunit.client.0.smithi060.stdout:removed './202/30' 2024-04-14T23:53:59.664 INFO:tasks.workunit.client.0.smithi060.stdout:removed './202/33' 2024-04-14T23:53:59.664 INFO:tasks.workunit.client.0.smithi060.stdout:removed './202/23' 2024-04-14T23:53:59.664 INFO:tasks.workunit.client.0.smithi060.stdout:removed './202/35' 2024-04-14T23:53:59.664 INFO:tasks.workunit.client.0.smithi060.stdout:removed './202/26' 2024-04-14T23:53:59.664 INFO:tasks.workunit.client.0.smithi060.stdout:removed './202/37' 2024-04-14T23:53:59.664 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './202' 2024-04-14T23:53:59.664 INFO:tasks.workunit.client.0.smithi060.stdout:removed './203/21' 2024-04-14T23:53:59.664 INFO:tasks.workunit.client.0.smithi060.stdout:removed './203/27' 2024-04-14T23:53:59.664 INFO:tasks.workunit.client.0.smithi060.stdout:removed './203/31' 2024-04-14T23:53:59.664 INFO:tasks.workunit.client.0.smithi060.stdout:removed './203/36' 2024-04-14T23:53:59.664 INFO:tasks.workunit.client.0.smithi060.stdout:removed './203/20' 2024-04-14T23:53:59.664 INFO:tasks.workunit.client.0.smithi060.stdout:removed './203/39' 2024-04-14T23:53:59.664 INFO:tasks.workunit.client.0.smithi060.stdout:removed './203/34' 2024-04-14T23:53:59.664 INFO:tasks.workunit.client.0.smithi060.stdout:removed './203/38' 2024-04-14T23:53:59.664 INFO:tasks.workunit.client.0.smithi060.stdout:removed './203/28' 2024-04-14T23:53:59.665 INFO:tasks.workunit.client.0.smithi060.stdout:removed './203/24' 2024-04-14T23:53:59.665 INFO:tasks.workunit.client.0.smithi060.stdout:removed './203/29' 2024-04-14T23:53:59.665 INFO:tasks.workunit.client.0.smithi060.stdout:removed './203/32' 2024-04-14T23:53:59.665 INFO:tasks.workunit.client.0.smithi060.stdout:removed './203/22' 2024-04-14T23:53:59.665 INFO:tasks.workunit.client.0.smithi060.stdout:removed './203/25' 2024-04-14T23:53:59.665 INFO:tasks.workunit.client.0.smithi060.stdout:removed './203/30' 2024-04-14T23:53:59.665 INFO:tasks.workunit.client.0.smithi060.stdout:removed './203/33' 2024-04-14T23:53:59.665 INFO:tasks.workunit.client.0.smithi060.stdout:removed './203/23' 2024-04-14T23:53:59.665 INFO:tasks.workunit.client.0.smithi060.stdout:removed './203/35' 2024-04-14T23:53:59.665 INFO:tasks.workunit.client.0.smithi060.stdout:removed './203/26' 2024-04-14T23:53:59.665 INFO:tasks.workunit.client.0.smithi060.stdout:removed './203/37' 2024-04-14T23:53:59.665 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './203' 2024-04-14T23:53:59.665 INFO:tasks.workunit.client.0.smithi060.stdout:removed './204/21' 2024-04-14T23:53:59.665 INFO:tasks.workunit.client.0.smithi060.stdout:removed './204/27' 2024-04-14T23:53:59.665 INFO:tasks.workunit.client.0.smithi060.stdout:removed './204/31' 2024-04-14T23:53:59.666 INFO:tasks.workunit.client.0.smithi060.stdout:removed './204/36' 2024-04-14T23:53:59.666 INFO:tasks.workunit.client.0.smithi060.stdout:removed './204/20' 2024-04-14T23:53:59.666 INFO:tasks.workunit.client.0.smithi060.stdout:removed './204/39' 2024-04-14T23:53:59.666 INFO:tasks.workunit.client.0.smithi060.stdout:removed './204/34' 2024-04-14T23:53:59.666 INFO:tasks.workunit.client.0.smithi060.stdout:removed './204/38' 2024-04-14T23:53:59.666 INFO:tasks.workunit.client.0.smithi060.stdout:removed './204/28' 2024-04-14T23:53:59.666 INFO:tasks.workunit.client.0.smithi060.stdout:removed './204/24' 2024-04-14T23:53:59.666 INFO:tasks.workunit.client.0.smithi060.stdout:removed './204/29' 2024-04-14T23:53:59.666 INFO:tasks.workunit.client.0.smithi060.stdout:removed './204/32' 2024-04-14T23:53:59.666 INFO:tasks.workunit.client.0.smithi060.stdout:removed './204/22' 2024-04-14T23:53:59.666 INFO:tasks.workunit.client.0.smithi060.stdout:removed './204/25' 2024-04-14T23:53:59.666 INFO:tasks.workunit.client.0.smithi060.stdout:removed './204/30' 2024-04-14T23:53:59.666 INFO:tasks.workunit.client.0.smithi060.stdout:removed './204/33' 2024-04-14T23:53:59.666 INFO:tasks.workunit.client.0.smithi060.stdout:removed './204/23' 2024-04-14T23:53:59.666 INFO:tasks.workunit.client.0.smithi060.stdout:removed './204/35' 2024-04-14T23:53:59.667 INFO:tasks.workunit.client.0.smithi060.stdout:removed './204/26' 2024-04-14T23:53:59.667 INFO:tasks.workunit.client.0.smithi060.stdout:removed './204/37' 2024-04-14T23:53:59.667 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './204' 2024-04-14T23:53:59.667 INFO:tasks.workunit.client.0.smithi060.stdout:removed './205/21' 2024-04-14T23:53:59.667 INFO:tasks.workunit.client.0.smithi060.stdout:removed './205/27' 2024-04-14T23:53:59.667 INFO:tasks.workunit.client.0.smithi060.stdout:removed './205/31' 2024-04-14T23:53:59.667 INFO:tasks.workunit.client.0.smithi060.stdout:removed './205/36' 2024-04-14T23:53:59.667 INFO:tasks.workunit.client.0.smithi060.stdout:removed './205/20' 2024-04-14T23:53:59.667 INFO:tasks.workunit.client.0.smithi060.stdout:removed './205/39' 2024-04-14T23:53:59.667 INFO:tasks.workunit.client.0.smithi060.stdout:removed './205/34' 2024-04-14T23:53:59.667 INFO:tasks.workunit.client.0.smithi060.stdout:removed './205/38' 2024-04-14T23:53:59.667 INFO:tasks.workunit.client.0.smithi060.stdout:removed './205/28' 2024-04-14T23:53:59.667 INFO:tasks.workunit.client.0.smithi060.stdout:removed './205/24' 2024-04-14T23:53:59.667 INFO:tasks.workunit.client.0.smithi060.stdout:removed './205/29' 2024-04-14T23:53:59.668 INFO:tasks.workunit.client.0.smithi060.stdout:removed './205/32' 2024-04-14T23:53:59.668 INFO:tasks.workunit.client.0.smithi060.stdout:removed './205/22' 2024-04-14T23:53:59.668 INFO:tasks.workunit.client.0.smithi060.stdout:removed './205/25' 2024-04-14T23:53:59.668 INFO:tasks.workunit.client.0.smithi060.stdout:removed './205/30' 2024-04-14T23:53:59.668 INFO:tasks.workunit.client.0.smithi060.stdout:removed './205/33' 2024-04-14T23:53:59.668 INFO:tasks.workunit.client.0.smithi060.stdout:removed './205/23' 2024-04-14T23:53:59.668 INFO:tasks.workunit.client.0.smithi060.stdout:removed './205/35' 2024-04-14T23:53:59.668 INFO:tasks.workunit.client.0.smithi060.stdout:removed './205/26' 2024-04-14T23:53:59.668 INFO:tasks.workunit.client.0.smithi060.stdout:removed './205/37' 2024-04-14T23:53:59.668 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './205' 2024-04-14T23:53:59.668 INFO:tasks.workunit.client.0.smithi060.stdout:removed './206/21' 2024-04-14T23:53:59.668 INFO:tasks.workunit.client.0.smithi060.stdout:removed './206/27' 2024-04-14T23:53:59.668 INFO:tasks.workunit.client.0.smithi060.stdout:removed './206/31' 2024-04-14T23:53:59.668 INFO:tasks.workunit.client.0.smithi060.stdout:removed './206/36' 2024-04-14T23:53:59.668 INFO:tasks.workunit.client.0.smithi060.stdout:removed './206/20' 2024-04-14T23:53:59.668 INFO:tasks.workunit.client.0.smithi060.stdout:removed './206/39' 2024-04-14T23:54:01.904 INFO:tasks.workunit.client.0.smithi060.stdout:removed './206/34' 2024-04-14T23:54:01.904 INFO:tasks.workunit.client.0.smithi060.stdout:removed './206/38' 2024-04-14T23:54:01.904 INFO:tasks.workunit.client.0.smithi060.stdout:removed './206/28' 2024-04-14T23:54:01.904 INFO:tasks.workunit.client.0.smithi060.stdout:removed './206/24' 2024-04-14T23:54:01.904 INFO:tasks.workunit.client.0.smithi060.stdout:removed './206/29' 2024-04-14T23:54:01.904 INFO:tasks.workunit.client.0.smithi060.stdout:removed './206/32' 2024-04-14T23:54:01.904 INFO:tasks.workunit.client.0.smithi060.stdout:removed './206/22' 2024-04-14T23:54:01.904 INFO:tasks.workunit.client.0.smithi060.stdout:removed './206/25' 2024-04-14T23:54:01.904 INFO:tasks.workunit.client.0.smithi060.stdout:removed './206/30' 2024-04-14T23:54:01.905 INFO:tasks.workunit.client.0.smithi060.stdout:removed './206/33' 2024-04-14T23:54:01.905 INFO:tasks.workunit.client.0.smithi060.stdout:removed './206/23' 2024-04-14T23:54:01.905 INFO:tasks.workunit.client.0.smithi060.stdout:removed './206/35' 2024-04-14T23:54:01.905 INFO:tasks.workunit.client.0.smithi060.stdout:removed './206/26' 2024-04-14T23:54:01.905 INFO:tasks.workunit.client.0.smithi060.stdout:removed './206/37' 2024-04-14T23:54:01.905 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './206' 2024-04-14T23:54:01.905 INFO:tasks.workunit.client.0.smithi060.stdout:removed './207/21' 2024-04-14T23:54:01.905 INFO:tasks.workunit.client.0.smithi060.stdout:removed './207/27' 2024-04-14T23:54:01.905 INFO:tasks.workunit.client.0.smithi060.stdout:removed './207/31' 2024-04-14T23:54:01.905 INFO:tasks.workunit.client.0.smithi060.stdout:removed './207/36' 2024-04-14T23:54:01.905 INFO:tasks.workunit.client.0.smithi060.stdout:removed './207/20' 2024-04-14T23:54:01.905 INFO:tasks.workunit.client.0.smithi060.stdout:removed './207/39' 2024-04-14T23:54:01.905 INFO:tasks.workunit.client.0.smithi060.stdout:removed './207/34' 2024-04-14T23:54:01.905 INFO:tasks.workunit.client.0.smithi060.stdout:removed './207/38' 2024-04-14T23:54:01.906 INFO:tasks.workunit.client.0.smithi060.stdout:removed './207/28' 2024-04-14T23:54:01.906 INFO:tasks.workunit.client.0.smithi060.stdout:removed './207/24' 2024-04-14T23:54:01.906 INFO:tasks.workunit.client.0.smithi060.stdout:removed './207/29' 2024-04-14T23:54:01.906 INFO:tasks.workunit.client.0.smithi060.stdout:removed './207/32' 2024-04-14T23:54:01.906 INFO:tasks.workunit.client.0.smithi060.stdout:removed './207/22' 2024-04-14T23:54:01.906 INFO:tasks.workunit.client.0.smithi060.stdout:removed './207/25' 2024-04-14T23:54:01.906 INFO:tasks.workunit.client.0.smithi060.stdout:removed './207/30' 2024-04-14T23:54:01.906 INFO:tasks.workunit.client.0.smithi060.stdout:removed './207/33' 2024-04-14T23:54:01.906 INFO:tasks.workunit.client.0.smithi060.stdout:removed './207/23' 2024-04-14T23:54:01.906 INFO:tasks.workunit.client.0.smithi060.stdout:removed './207/35' 2024-04-14T23:54:01.906 INFO:tasks.workunit.client.0.smithi060.stdout:removed './207/26' 2024-04-14T23:54:01.906 INFO:tasks.workunit.client.0.smithi060.stdout:removed './207/37' 2024-04-14T23:54:01.906 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './207' 2024-04-14T23:54:01.907 INFO:tasks.workunit.client.0.smithi060.stdout:removed './208/21' 2024-04-14T23:54:01.907 INFO:tasks.workunit.client.0.smithi060.stdout:removed './208/27' 2024-04-14T23:54:01.907 INFO:tasks.workunit.client.0.smithi060.stdout:removed './208/31' 2024-04-14T23:54:01.907 INFO:tasks.workunit.client.0.smithi060.stdout:removed './208/36' 2024-04-14T23:54:01.907 INFO:tasks.workunit.client.0.smithi060.stdout:removed './208/20' 2024-04-14T23:54:01.907 INFO:tasks.workunit.client.0.smithi060.stdout:removed './208/39' 2024-04-14T23:54:01.907 INFO:tasks.workunit.client.0.smithi060.stdout:removed './208/34' 2024-04-14T23:54:01.907 INFO:tasks.workunit.client.0.smithi060.stdout:removed './208/38' 2024-04-14T23:54:01.907 INFO:tasks.workunit.client.0.smithi060.stdout:removed './208/28' 2024-04-14T23:54:01.907 INFO:tasks.workunit.client.0.smithi060.stdout:removed './208/24' 2024-04-14T23:54:01.907 INFO:tasks.workunit.client.0.smithi060.stdout:removed './208/29' 2024-04-14T23:54:01.907 INFO:tasks.workunit.client.0.smithi060.stdout:removed './208/32' 2024-04-14T23:54:01.907 INFO:tasks.workunit.client.0.smithi060.stdout:removed './208/22' 2024-04-14T23:54:01.907 INFO:tasks.workunit.client.0.smithi060.stdout:removed './208/25' 2024-04-14T23:54:01.907 INFO:tasks.workunit.client.0.smithi060.stdout:removed './208/30' 2024-04-14T23:54:01.908 INFO:tasks.workunit.client.0.smithi060.stdout:removed './208/33' 2024-04-14T23:54:01.908 INFO:tasks.workunit.client.0.smithi060.stdout:removed './208/23' 2024-04-14T23:54:01.908 INFO:tasks.workunit.client.0.smithi060.stdout:removed './208/35' 2024-04-14T23:54:01.908 INFO:tasks.workunit.client.0.smithi060.stdout:removed './208/26' 2024-04-14T23:54:01.908 INFO:tasks.workunit.client.0.smithi060.stdout:removed './208/37' 2024-04-14T23:54:01.908 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './208' 2024-04-14T23:54:01.908 INFO:tasks.workunit.client.0.smithi060.stdout:removed './209/21' 2024-04-14T23:54:01.908 INFO:tasks.workunit.client.0.smithi060.stdout:removed './209/27' 2024-04-14T23:54:01.908 INFO:tasks.workunit.client.0.smithi060.stdout:removed './209/31' 2024-04-14T23:54:01.908 INFO:tasks.workunit.client.0.smithi060.stdout:removed './209/36' 2024-04-14T23:54:01.908 INFO:tasks.workunit.client.0.smithi060.stdout:removed './209/20' 2024-04-14T23:54:01.908 INFO:tasks.workunit.client.0.smithi060.stdout:removed './209/39' 2024-04-14T23:54:01.908 INFO:tasks.workunit.client.0.smithi060.stdout:removed './209/34' 2024-04-14T23:54:01.908 INFO:tasks.workunit.client.0.smithi060.stdout:removed './209/38' 2024-04-14T23:54:01.908 INFO:tasks.workunit.client.0.smithi060.stdout:removed './209/28' 2024-04-14T23:54:01.909 INFO:tasks.workunit.client.0.smithi060.stdout:removed './209/24' 2024-04-14T23:54:01.909 INFO:tasks.workunit.client.0.smithi060.stdout:removed './209/29' 2024-04-14T23:54:01.909 INFO:tasks.workunit.client.0.smithi060.stdout:removed './209/32' 2024-04-14T23:54:01.909 INFO:tasks.workunit.client.0.smithi060.stdout:removed './209/22' 2024-04-14T23:54:01.909 INFO:tasks.workunit.client.0.smithi060.stdout:removed './209/25' 2024-04-14T23:54:01.909 INFO:tasks.workunit.client.0.smithi060.stdout:removed './209/30' 2024-04-14T23:54:01.909 INFO:tasks.workunit.client.0.smithi060.stdout:removed './209/33' 2024-04-14T23:54:01.909 INFO:tasks.workunit.client.0.smithi060.stdout:removed './209/23' 2024-04-14T23:54:01.909 INFO:tasks.workunit.client.0.smithi060.stdout:removed './209/35' 2024-04-14T23:54:01.909 INFO:tasks.workunit.client.0.smithi060.stdout:removed './209/26' 2024-04-14T23:54:01.909 INFO:tasks.workunit.client.0.smithi060.stdout:removed './209/37' 2024-04-14T23:54:01.909 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './209' 2024-04-14T23:54:01.909 INFO:tasks.workunit.client.0.smithi060.stdout:removed './210/21' 2024-04-14T23:54:01.909 INFO:tasks.workunit.client.0.smithi060.stdout:removed './210/27' 2024-04-14T23:54:01.910 INFO:tasks.workunit.client.0.smithi060.stdout:removed './210/31' 2024-04-14T23:54:01.910 INFO:tasks.workunit.client.0.smithi060.stdout:removed './210/36' 2024-04-14T23:54:01.910 INFO:tasks.workunit.client.0.smithi060.stdout:removed './210/20' 2024-04-14T23:54:01.910 INFO:tasks.workunit.client.0.smithi060.stdout:removed './210/39' 2024-04-14T23:54:01.910 INFO:tasks.workunit.client.0.smithi060.stdout:removed './210/34' 2024-04-14T23:54:01.910 INFO:tasks.workunit.client.0.smithi060.stdout:removed './210/38' 2024-04-14T23:54:01.910 INFO:tasks.workunit.client.0.smithi060.stdout:removed './210/28' 2024-04-14T23:54:01.910 INFO:tasks.workunit.client.0.smithi060.stdout:removed './210/24' 2024-04-14T23:54:01.910 INFO:tasks.workunit.client.0.smithi060.stdout:removed './210/29' 2024-04-14T23:54:01.910 INFO:tasks.workunit.client.0.smithi060.stdout:removed './210/32' 2024-04-14T23:54:01.910 INFO:tasks.workunit.client.0.smithi060.stdout:removed './210/22' 2024-04-14T23:54:01.910 INFO:tasks.workunit.client.0.smithi060.stdout:removed './210/25' 2024-04-14T23:54:01.910 INFO:tasks.workunit.client.0.smithi060.stdout:removed './210/30' 2024-04-14T23:54:01.910 INFO:tasks.workunit.client.0.smithi060.stdout:removed './210/33' 2024-04-14T23:54:01.910 INFO:tasks.workunit.client.0.smithi060.stdout:removed './210/23' 2024-04-14T23:54:01.911 INFO:tasks.workunit.client.0.smithi060.stdout:removed './210/35' 2024-04-14T23:54:01.911 INFO:tasks.workunit.client.0.smithi060.stdout:removed './210/26' 2024-04-14T23:54:01.911 INFO:tasks.workunit.client.0.smithi060.stdout:removed './210/37' 2024-04-14T23:54:01.911 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './210' 2024-04-14T23:54:01.911 INFO:tasks.workunit.client.0.smithi060.stdout:removed './211/21' 2024-04-14T23:54:01.911 INFO:tasks.workunit.client.0.smithi060.stdout:removed './211/27' 2024-04-14T23:54:01.911 INFO:tasks.workunit.client.0.smithi060.stdout:removed './211/31' 2024-04-14T23:54:01.911 INFO:tasks.workunit.client.0.smithi060.stdout:removed './211/36' 2024-04-14T23:54:01.911 INFO:tasks.workunit.client.0.smithi060.stdout:removed './211/20' 2024-04-14T23:54:01.911 INFO:tasks.workunit.client.0.smithi060.stdout:removed './211/39' 2024-04-14T23:54:01.911 INFO:tasks.workunit.client.0.smithi060.stdout:removed './211/34' 2024-04-14T23:54:01.911 INFO:tasks.workunit.client.0.smithi060.stdout:removed './211/38' 2024-04-14T23:54:01.911 INFO:tasks.workunit.client.0.smithi060.stdout:removed './211/28' 2024-04-14T23:54:01.911 INFO:tasks.workunit.client.0.smithi060.stdout:removed './211/24' 2024-04-14T23:54:01.911 INFO:tasks.workunit.client.0.smithi060.stdout:removed './211/29' 2024-04-14T23:54:01.912 INFO:tasks.workunit.client.0.smithi060.stdout:removed './211/32' 2024-04-14T23:54:01.912 INFO:tasks.workunit.client.0.smithi060.stdout:removed './211/22' 2024-04-14T23:54:01.912 INFO:tasks.workunit.client.0.smithi060.stdout:removed './211/25' 2024-04-14T23:54:01.912 INFO:tasks.workunit.client.0.smithi060.stdout:removed './211/30' 2024-04-14T23:54:01.912 INFO:tasks.workunit.client.0.smithi060.stdout:removed './211/33' 2024-04-14T23:54:01.912 INFO:tasks.workunit.client.0.smithi060.stdout:removed './211/23' 2024-04-14T23:54:01.912 INFO:tasks.workunit.client.0.smithi060.stdout:removed './211/35' 2024-04-14T23:54:01.912 INFO:tasks.workunit.client.0.smithi060.stdout:removed './211/26' 2024-04-14T23:54:01.912 INFO:tasks.workunit.client.0.smithi060.stdout:removed './211/37' 2024-04-14T23:54:01.912 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './211' 2024-04-14T23:54:01.912 INFO:tasks.workunit.client.0.smithi060.stdout:removed './212/21' 2024-04-14T23:54:01.912 INFO:tasks.workunit.client.0.smithi060.stdout:removed './212/27' 2024-04-14T23:54:01.912 INFO:tasks.workunit.client.0.smithi060.stdout:removed './212/31' 2024-04-14T23:54:01.912 INFO:tasks.workunit.client.0.smithi060.stdout:removed './212/36' 2024-04-14T23:54:01.912 INFO:tasks.workunit.client.0.smithi060.stdout:removed './212/20' 2024-04-14T23:54:01.913 INFO:tasks.workunit.client.0.smithi060.stdout:removed './212/39' 2024-04-14T23:54:01.913 INFO:tasks.workunit.client.0.smithi060.stdout:removed './212/34' 2024-04-14T23:54:01.913 INFO:tasks.workunit.client.0.smithi060.stdout:removed './212/38' 2024-04-14T23:54:01.913 INFO:tasks.workunit.client.0.smithi060.stdout:removed './212/28' 2024-04-14T23:54:01.913 INFO:tasks.workunit.client.0.smithi060.stdout:removed './212/24' 2024-04-14T23:54:01.913 INFO:tasks.workunit.client.0.smithi060.stdout:removed './212/29' 2024-04-14T23:54:01.913 INFO:tasks.workunit.client.0.smithi060.stdout:removed './212/32' 2024-04-14T23:54:01.913 INFO:tasks.workunit.client.0.smithi060.stdout:removed './212/22' 2024-04-14T23:54:01.913 INFO:tasks.workunit.client.0.smithi060.stdout:removed './212/25' 2024-04-14T23:54:01.913 INFO:tasks.workunit.client.0.smithi060.stdout:removed './212/30' 2024-04-14T23:54:01.913 INFO:tasks.workunit.client.0.smithi060.stdout:removed './212/33' 2024-04-14T23:54:01.913 INFO:tasks.workunit.client.0.smithi060.stdout:removed './212/23' 2024-04-14T23:54:01.914 INFO:tasks.workunit.client.0.smithi060.stdout:removed './212/35' 2024-04-14T23:54:01.914 INFO:tasks.workunit.client.0.smithi060.stdout:removed './212/26' 2024-04-14T23:54:01.914 INFO:tasks.workunit.client.0.smithi060.stdout:removed './212/37' 2024-04-14T23:54:01.914 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './212' 2024-04-14T23:54:01.914 INFO:tasks.workunit.client.0.smithi060.stdout:removed './213/21' 2024-04-14T23:54:01.914 INFO:tasks.workunit.client.0.smithi060.stdout:removed './213/27' 2024-04-14T23:54:01.914 INFO:tasks.workunit.client.0.smithi060.stdout:removed './213/31' 2024-04-14T23:54:01.914 INFO:tasks.workunit.client.0.smithi060.stdout:removed './213/36' 2024-04-14T23:54:01.914 INFO:tasks.workunit.client.0.smithi060.stdout:removed './213/20' 2024-04-14T23:54:01.914 INFO:tasks.workunit.client.0.smithi060.stdout:removed './213/39' 2024-04-14T23:54:01.914 INFO:tasks.workunit.client.0.smithi060.stdout:removed './213/34' 2024-04-14T23:54:01.914 INFO:tasks.workunit.client.0.smithi060.stdout:removed './213/38' 2024-04-14T23:54:01.914 INFO:tasks.workunit.client.0.smithi060.stdout:removed './213/28' 2024-04-14T23:54:01.914 INFO:tasks.workunit.client.0.smithi060.stdout:removed './213/24' 2024-04-14T23:54:01.914 INFO:tasks.workunit.client.0.smithi060.stdout:removed './213/29' 2024-04-14T23:54:01.915 INFO:tasks.workunit.client.0.smithi060.stdout:removed './213/32' 2024-04-14T23:54:01.915 INFO:tasks.workunit.client.0.smithi060.stdout:removed './213/22' 2024-04-14T23:54:01.915 INFO:tasks.workunit.client.0.smithi060.stdout:removed './213/25' 2024-04-14T23:54:01.915 INFO:tasks.workunit.client.0.smithi060.stdout:removed './213/30' 2024-04-14T23:54:01.915 INFO:tasks.workunit.client.0.smithi060.stdout:removed './213/33' 2024-04-14T23:54:01.915 INFO:tasks.workunit.client.0.smithi060.stdout:removed './213/23' 2024-04-14T23:54:01.915 INFO:tasks.workunit.client.0.smithi060.stdout:removed './213/35' 2024-04-14T23:54:01.915 INFO:tasks.workunit.client.0.smithi060.stdout:removed './213/26' 2024-04-14T23:54:01.915 INFO:tasks.workunit.client.0.smithi060.stdout:removed './213/37' 2024-04-14T23:54:01.915 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './213' 2024-04-14T23:54:01.915 INFO:tasks.workunit.client.0.smithi060.stdout:removed './214/21' 2024-04-14T23:54:01.915 INFO:tasks.workunit.client.0.smithi060.stdout:removed './214/27' 2024-04-14T23:54:01.915 INFO:tasks.workunit.client.0.smithi060.stdout:removed './214/31' 2024-04-14T23:54:01.915 INFO:tasks.workunit.client.0.smithi060.stdout:removed './214/36' 2024-04-14T23:54:01.916 INFO:tasks.workunit.client.0.smithi060.stdout:removed './214/20' 2024-04-14T23:54:01.916 INFO:tasks.workunit.client.0.smithi060.stdout:removed './214/39' 2024-04-14T23:54:01.916 INFO:tasks.workunit.client.0.smithi060.stdout:removed './214/34' 2024-04-14T23:54:01.916 INFO:tasks.workunit.client.0.smithi060.stdout:removed './214/38' 2024-04-14T23:54:01.916 INFO:tasks.workunit.client.0.smithi060.stdout:removed './214/28' 2024-04-14T23:54:01.916 INFO:tasks.workunit.client.0.smithi060.stdout:removed './214/24' 2024-04-14T23:54:01.916 INFO:tasks.workunit.client.0.smithi060.stdout:removed './214/29' 2024-04-14T23:54:01.916 INFO:tasks.workunit.client.0.smithi060.stdout:removed './214/32' 2024-04-14T23:54:01.916 INFO:tasks.workunit.client.0.smithi060.stdout:removed './214/22' 2024-04-14T23:54:01.916 INFO:tasks.workunit.client.0.smithi060.stdout:removed './214/25' 2024-04-14T23:54:01.916 INFO:tasks.workunit.client.0.smithi060.stdout:removed './214/30' 2024-04-14T23:54:01.916 INFO:tasks.workunit.client.0.smithi060.stdout:removed './214/33' 2024-04-14T23:54:01.916 INFO:tasks.workunit.client.0.smithi060.stdout:removed './214/23' 2024-04-14T23:54:01.916 INFO:tasks.workunit.client.0.smithi060.stdout:removed './214/35' 2024-04-14T23:54:01.917 INFO:tasks.workunit.client.0.smithi060.stdout:removed './214/26' 2024-04-14T23:54:01.917 INFO:tasks.workunit.client.0.smithi060.stdout:removed './214/37' 2024-04-14T23:54:01.917 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './214' 2024-04-14T23:54:01.917 INFO:tasks.workunit.client.0.smithi060.stdout:removed './215/21' 2024-04-14T23:54:01.917 INFO:tasks.workunit.client.0.smithi060.stdout:removed './215/27' 2024-04-14T23:54:01.917 INFO:tasks.workunit.client.0.smithi060.stdout:removed './215/31' 2024-04-14T23:54:01.917 INFO:tasks.workunit.client.0.smithi060.stdout:removed './215/36' 2024-04-14T23:54:01.917 INFO:tasks.workunit.client.0.smithi060.stdout:removed './215/20' 2024-04-14T23:54:01.917 INFO:tasks.workunit.client.0.smithi060.stdout:removed './215/39' 2024-04-14T23:54:01.917 INFO:tasks.workunit.client.0.smithi060.stdout:removed './215/34' 2024-04-14T23:54:01.917 INFO:tasks.workunit.client.0.smithi060.stdout:removed './215/38' 2024-04-14T23:54:01.917 INFO:tasks.workunit.client.0.smithi060.stdout:removed './215/28' 2024-04-14T23:54:01.917 INFO:tasks.workunit.client.0.smithi060.stdout:removed './215/24' 2024-04-14T23:54:01.917 INFO:tasks.workunit.client.0.smithi060.stdout:removed './215/29' 2024-04-14T23:54:01.917 INFO:tasks.workunit.client.0.smithi060.stdout:removed './215/32' 2024-04-14T23:54:01.917 INFO:tasks.workunit.client.0.smithi060.stdout:removed './215/22' 2024-04-14T23:54:01.918 INFO:tasks.workunit.client.0.smithi060.stdout:removed './215/25' 2024-04-14T23:54:01.918 INFO:tasks.workunit.client.0.smithi060.stdout:removed './215/30' 2024-04-14T23:54:01.918 INFO:tasks.workunit.client.0.smithi060.stdout:removed './215/33' 2024-04-14T23:54:01.918 INFO:tasks.workunit.client.0.smithi060.stdout:removed './215/23' 2024-04-14T23:54:01.918 INFO:tasks.workunit.client.0.smithi060.stdout:removed './215/35' 2024-04-14T23:54:01.918 INFO:tasks.workunit.client.0.smithi060.stdout:removed './215/26' 2024-04-14T23:54:01.918 INFO:tasks.workunit.client.0.smithi060.stdout:removed './215/37' 2024-04-14T23:54:01.918 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './215' 2024-04-14T23:54:01.918 INFO:tasks.workunit.client.0.smithi060.stdout:removed './216/21' 2024-04-14T23:54:01.918 INFO:tasks.workunit.client.0.smithi060.stdout:removed './216/27' 2024-04-14T23:54:01.918 INFO:tasks.workunit.client.0.smithi060.stdout:removed './216/31' 2024-04-14T23:54:01.918 INFO:tasks.workunit.client.0.smithi060.stdout:removed './216/36' 2024-04-14T23:54:01.918 INFO:tasks.workunit.client.0.smithi060.stdout:removed './216/20' 2024-04-14T23:54:01.918 INFO:tasks.workunit.client.0.smithi060.stdout:removed './216/39' 2024-04-14T23:54:01.919 INFO:tasks.workunit.client.0.smithi060.stdout:removed './216/34' 2024-04-14T23:54:01.919 INFO:tasks.workunit.client.0.smithi060.stdout:removed './216/38' 2024-04-14T23:54:04.982 INFO:tasks.workunit.client.0.smithi060.stdout:removed './216/28' 2024-04-14T23:54:04.982 INFO:tasks.workunit.client.0.smithi060.stdout:removed './216/24' 2024-04-14T23:54:04.982 INFO:tasks.workunit.client.0.smithi060.stdout:removed './216/29' 2024-04-14T23:54:04.982 INFO:tasks.workunit.client.0.smithi060.stdout:removed './216/32' 2024-04-14T23:54:04.982 INFO:tasks.workunit.client.0.smithi060.stdout:removed './216/22' 2024-04-14T23:54:04.982 INFO:tasks.workunit.client.0.smithi060.stdout:removed './216/25' 2024-04-14T23:54:04.983 INFO:tasks.workunit.client.0.smithi060.stdout:removed './216/30' 2024-04-14T23:54:04.983 INFO:tasks.workunit.client.0.smithi060.stdout:removed './216/33' 2024-04-14T23:54:04.983 INFO:tasks.workunit.client.0.smithi060.stdout:removed './216/23' 2024-04-14T23:54:04.983 INFO:tasks.workunit.client.0.smithi060.stdout:removed './216/35' 2024-04-14T23:54:04.983 INFO:tasks.workunit.client.0.smithi060.stdout:removed './216/26' 2024-04-14T23:54:04.983 INFO:tasks.workunit.client.0.smithi060.stdout:removed './216/37' 2024-04-14T23:54:04.983 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './216' 2024-04-14T23:54:04.983 INFO:tasks.workunit.client.0.smithi060.stdout:removed './217/21' 2024-04-14T23:54:04.983 INFO:tasks.workunit.client.0.smithi060.stdout:removed './217/27' 2024-04-14T23:54:04.983 INFO:tasks.workunit.client.0.smithi060.stdout:removed './217/31' 2024-04-14T23:54:04.983 INFO:tasks.workunit.client.0.smithi060.stdout:removed './217/36' 2024-04-14T23:54:04.983 INFO:tasks.workunit.client.0.smithi060.stdout:removed './217/20' 2024-04-14T23:54:04.983 INFO:tasks.workunit.client.0.smithi060.stdout:removed './217/39' 2024-04-14T23:54:04.983 INFO:tasks.workunit.client.0.smithi060.stdout:removed './217/34' 2024-04-14T23:54:04.983 INFO:tasks.workunit.client.0.smithi060.stdout:removed './217/38' 2024-04-14T23:54:04.984 INFO:tasks.workunit.client.0.smithi060.stdout:removed './217/28' 2024-04-14T23:54:04.984 INFO:tasks.workunit.client.0.smithi060.stdout:removed './217/24' 2024-04-14T23:54:04.984 INFO:tasks.workunit.client.0.smithi060.stdout:removed './217/29' 2024-04-14T23:54:04.984 INFO:tasks.workunit.client.0.smithi060.stdout:removed './217/32' 2024-04-14T23:54:04.984 INFO:tasks.workunit.client.0.smithi060.stdout:removed './217/22' 2024-04-14T23:54:04.984 INFO:tasks.workunit.client.0.smithi060.stdout:removed './217/25' 2024-04-14T23:54:04.984 INFO:tasks.workunit.client.0.smithi060.stdout:removed './217/30' 2024-04-14T23:54:04.984 INFO:tasks.workunit.client.0.smithi060.stdout:removed './217/33' 2024-04-14T23:54:04.984 INFO:tasks.workunit.client.0.smithi060.stdout:removed './217/23' 2024-04-14T23:54:04.984 INFO:tasks.workunit.client.0.smithi060.stdout:removed './217/35' 2024-04-14T23:54:04.984 INFO:tasks.workunit.client.0.smithi060.stdout:removed './217/26' 2024-04-14T23:54:04.984 INFO:tasks.workunit.client.0.smithi060.stdout:removed './217/37' 2024-04-14T23:54:04.984 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './217' 2024-04-14T23:54:04.984 INFO:tasks.workunit.client.0.smithi060.stdout:removed './218/21' 2024-04-14T23:54:04.984 INFO:tasks.workunit.client.0.smithi060.stdout:removed './218/27' 2024-04-14T23:54:04.984 INFO:tasks.workunit.client.0.smithi060.stdout:removed './218/31' 2024-04-14T23:54:04.985 INFO:tasks.workunit.client.0.smithi060.stdout:removed './218/36' 2024-04-14T23:54:04.985 INFO:tasks.workunit.client.0.smithi060.stdout:removed './218/20' 2024-04-14T23:54:04.985 INFO:tasks.workunit.client.0.smithi060.stdout:removed './218/39' 2024-04-14T23:54:04.985 INFO:tasks.workunit.client.0.smithi060.stdout:removed './218/34' 2024-04-14T23:54:04.985 INFO:tasks.workunit.client.0.smithi060.stdout:removed './218/38' 2024-04-14T23:54:04.985 INFO:tasks.workunit.client.0.smithi060.stdout:removed './218/28' 2024-04-14T23:54:04.985 INFO:tasks.workunit.client.0.smithi060.stdout:removed './218/24' 2024-04-14T23:54:04.985 INFO:tasks.workunit.client.0.smithi060.stdout:removed './218/29' 2024-04-14T23:54:04.985 INFO:tasks.workunit.client.0.smithi060.stdout:removed './218/32' 2024-04-14T23:54:04.985 INFO:tasks.workunit.client.0.smithi060.stdout:removed './218/22' 2024-04-14T23:54:04.985 INFO:tasks.workunit.client.0.smithi060.stdout:removed './218/25' 2024-04-14T23:54:04.985 INFO:tasks.workunit.client.0.smithi060.stdout:removed './218/30' 2024-04-14T23:54:04.985 INFO:tasks.workunit.client.0.smithi060.stdout:removed './218/33' 2024-04-14T23:54:04.985 INFO:tasks.workunit.client.0.smithi060.stdout:removed './218/23' 2024-04-14T23:54:04.985 INFO:tasks.workunit.client.0.smithi060.stdout:removed './218/35' 2024-04-14T23:54:04.986 INFO:tasks.workunit.client.0.smithi060.stdout:removed './218/26' 2024-04-14T23:54:04.986 INFO:tasks.workunit.client.0.smithi060.stdout:removed './218/37' 2024-04-14T23:54:04.986 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './218' 2024-04-14T23:54:04.986 INFO:tasks.workunit.client.0.smithi060.stdout:removed './219/21' 2024-04-14T23:54:04.986 INFO:tasks.workunit.client.0.smithi060.stdout:removed './219/27' 2024-04-14T23:54:04.986 INFO:tasks.workunit.client.0.smithi060.stdout:removed './219/31' 2024-04-14T23:54:04.986 INFO:tasks.workunit.client.0.smithi060.stdout:removed './219/36' 2024-04-14T23:54:04.986 INFO:tasks.workunit.client.0.smithi060.stdout:removed './219/20' 2024-04-14T23:54:04.986 INFO:tasks.workunit.client.0.smithi060.stdout:removed './219/39' 2024-04-14T23:54:04.986 INFO:tasks.workunit.client.0.smithi060.stdout:removed './219/34' 2024-04-14T23:54:04.986 INFO:tasks.workunit.client.0.smithi060.stdout:removed './219/38' 2024-04-14T23:54:04.986 INFO:tasks.workunit.client.0.smithi060.stdout:removed './219/28' 2024-04-14T23:54:04.986 INFO:tasks.workunit.client.0.smithi060.stdout:removed './219/24' 2024-04-14T23:54:04.986 INFO:tasks.workunit.client.0.smithi060.stdout:removed './219/29' 2024-04-14T23:54:04.987 INFO:tasks.workunit.client.0.smithi060.stdout:removed './219/32' 2024-04-14T23:54:04.987 INFO:tasks.workunit.client.0.smithi060.stdout:removed './219/22' 2024-04-14T23:54:04.987 INFO:tasks.workunit.client.0.smithi060.stdout:removed './219/25' 2024-04-14T23:54:04.987 INFO:tasks.workunit.client.0.smithi060.stdout:removed './219/30' 2024-04-14T23:54:04.987 INFO:tasks.workunit.client.0.smithi060.stdout:removed './219/33' 2024-04-14T23:54:04.987 INFO:tasks.workunit.client.0.smithi060.stdout:removed './219/23' 2024-04-14T23:54:04.987 INFO:tasks.workunit.client.0.smithi060.stdout:removed './219/35' 2024-04-14T23:54:04.987 INFO:tasks.workunit.client.0.smithi060.stdout:removed './219/26' 2024-04-14T23:54:04.987 INFO:tasks.workunit.client.0.smithi060.stdout:removed './219/37' 2024-04-14T23:54:04.987 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './219' 2024-04-14T23:54:04.987 INFO:tasks.workunit.client.0.smithi060.stdout:removed './220/21' 2024-04-14T23:54:04.987 INFO:tasks.workunit.client.0.smithi060.stdout:removed './220/27' 2024-04-14T23:54:04.987 INFO:tasks.workunit.client.0.smithi060.stdout:removed './220/31' 2024-04-14T23:54:04.987 INFO:tasks.workunit.client.0.smithi060.stdout:removed './220/36' 2024-04-14T23:54:04.988 INFO:tasks.workunit.client.0.smithi060.stdout:removed './220/20' 2024-04-14T23:54:04.988 INFO:tasks.workunit.client.0.smithi060.stdout:removed './220/39' 2024-04-14T23:54:04.988 INFO:tasks.workunit.client.0.smithi060.stdout:removed './220/34' 2024-04-14T23:54:04.988 INFO:tasks.workunit.client.0.smithi060.stdout:removed './220/38' 2024-04-14T23:54:04.988 INFO:tasks.workunit.client.0.smithi060.stdout:removed './220/28' 2024-04-14T23:54:04.988 INFO:tasks.workunit.client.0.smithi060.stdout:removed './220/24' 2024-04-14T23:54:04.988 INFO:tasks.workunit.client.0.smithi060.stdout:removed './220/29' 2024-04-14T23:54:04.988 INFO:tasks.workunit.client.0.smithi060.stdout:removed './220/32' 2024-04-14T23:54:04.988 INFO:tasks.workunit.client.0.smithi060.stdout:removed './220/22' 2024-04-14T23:54:04.988 INFO:tasks.workunit.client.0.smithi060.stdout:removed './220/25' 2024-04-14T23:54:04.988 INFO:tasks.workunit.client.0.smithi060.stdout:removed './220/30' 2024-04-14T23:54:04.988 INFO:tasks.workunit.client.0.smithi060.stdout:removed './220/33' 2024-04-14T23:54:04.988 INFO:tasks.workunit.client.0.smithi060.stdout:removed './220/23' 2024-04-14T23:54:04.988 INFO:tasks.workunit.client.0.smithi060.stdout:removed './220/35' 2024-04-14T23:54:04.989 INFO:tasks.workunit.client.0.smithi060.stdout:removed './220/26' 2024-04-14T23:54:04.989 INFO:tasks.workunit.client.0.smithi060.stdout:removed './220/37' 2024-04-14T23:54:04.989 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './220' 2024-04-14T23:54:04.989 INFO:tasks.workunit.client.0.smithi060.stdout:removed './221/21' 2024-04-14T23:54:04.989 INFO:tasks.workunit.client.0.smithi060.stdout:removed './221/27' 2024-04-14T23:54:04.989 INFO:tasks.workunit.client.0.smithi060.stdout:removed './221/31' 2024-04-14T23:54:04.989 INFO:tasks.workunit.client.0.smithi060.stdout:removed './221/36' 2024-04-14T23:54:04.989 INFO:tasks.workunit.client.0.smithi060.stdout:removed './221/20' 2024-04-14T23:54:04.989 INFO:tasks.workunit.client.0.smithi060.stdout:removed './221/39' 2024-04-14T23:54:04.989 INFO:tasks.workunit.client.0.smithi060.stdout:removed './221/34' 2024-04-14T23:54:04.989 INFO:tasks.workunit.client.0.smithi060.stdout:removed './221/38' 2024-04-14T23:54:04.990 INFO:tasks.workunit.client.0.smithi060.stdout:removed './221/28' 2024-04-14T23:54:04.990 INFO:tasks.workunit.client.0.smithi060.stdout:removed './221/24' 2024-04-14T23:54:04.990 INFO:tasks.workunit.client.0.smithi060.stdout:removed './221/29' 2024-04-14T23:54:04.990 INFO:tasks.workunit.client.0.smithi060.stdout:removed './221/32' 2024-04-14T23:54:04.990 INFO:tasks.workunit.client.0.smithi060.stdout:removed './221/22' 2024-04-14T23:54:04.990 INFO:tasks.workunit.client.0.smithi060.stdout:removed './221/25' 2024-04-14T23:54:04.990 INFO:tasks.workunit.client.0.smithi060.stdout:removed './221/30' 2024-04-14T23:54:04.990 INFO:tasks.workunit.client.0.smithi060.stdout:removed './221/33' 2024-04-14T23:54:04.990 INFO:tasks.workunit.client.0.smithi060.stdout:removed './221/23' 2024-04-14T23:54:04.990 INFO:tasks.workunit.client.0.smithi060.stdout:removed './221/35' 2024-04-14T23:54:04.990 INFO:tasks.workunit.client.0.smithi060.stdout:removed './221/26' 2024-04-14T23:54:04.990 INFO:tasks.workunit.client.0.smithi060.stdout:removed './221/37' 2024-04-14T23:54:04.990 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './221' 2024-04-14T23:54:04.991 INFO:tasks.workunit.client.0.smithi060.stdout:removed './222/21' 2024-04-14T23:54:04.991 INFO:tasks.workunit.client.0.smithi060.stdout:removed './222/27' 2024-04-14T23:54:04.991 INFO:tasks.workunit.client.0.smithi060.stdout:removed './222/31' 2024-04-14T23:54:04.991 INFO:tasks.workunit.client.0.smithi060.stdout:removed './222/36' 2024-04-14T23:54:04.991 INFO:tasks.workunit.client.0.smithi060.stdout:removed './222/20' 2024-04-14T23:54:04.991 INFO:tasks.workunit.client.0.smithi060.stdout:removed './222/39' 2024-04-14T23:54:04.991 INFO:tasks.workunit.client.0.smithi060.stdout:removed './222/34' 2024-04-14T23:54:04.991 INFO:tasks.workunit.client.0.smithi060.stdout:removed './222/38' 2024-04-14T23:54:04.991 INFO:tasks.workunit.client.0.smithi060.stdout:removed './222/28' 2024-04-14T23:54:04.992 INFO:tasks.workunit.client.0.smithi060.stdout:removed './222/24' 2024-04-14T23:54:04.992 INFO:tasks.workunit.client.0.smithi060.stdout:removed './222/29' 2024-04-14T23:54:04.992 INFO:tasks.workunit.client.0.smithi060.stdout:removed './222/32' 2024-04-14T23:54:04.992 INFO:tasks.workunit.client.0.smithi060.stdout:removed './222/22' 2024-04-14T23:54:04.992 INFO:tasks.workunit.client.0.smithi060.stdout:removed './222/25' 2024-04-14T23:54:04.992 INFO:tasks.workunit.client.0.smithi060.stdout:removed './222/30' 2024-04-14T23:54:04.992 INFO:tasks.workunit.client.0.smithi060.stdout:removed './222/33' 2024-04-14T23:54:04.992 INFO:tasks.workunit.client.0.smithi060.stdout:removed './222/23' 2024-04-14T23:54:04.992 INFO:tasks.workunit.client.0.smithi060.stdout:removed './222/35' 2024-04-14T23:54:04.992 INFO:tasks.workunit.client.0.smithi060.stdout:removed './222/26' 2024-04-14T23:54:04.992 INFO:tasks.workunit.client.0.smithi060.stdout:removed './222/37' 2024-04-14T23:54:04.992 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './222' 2024-04-14T23:54:04.992 INFO:tasks.workunit.client.0.smithi060.stdout:removed './223/21' 2024-04-14T23:54:04.993 INFO:tasks.workunit.client.0.smithi060.stdout:removed './223/27' 2024-04-14T23:54:04.993 INFO:tasks.workunit.client.0.smithi060.stdout:removed './223/31' 2024-04-14T23:54:04.993 INFO:tasks.workunit.client.0.smithi060.stdout:removed './223/36' 2024-04-14T23:54:04.993 INFO:tasks.workunit.client.0.smithi060.stdout:removed './223/20' 2024-04-14T23:54:04.993 INFO:tasks.workunit.client.0.smithi060.stdout:removed './223/39' 2024-04-14T23:54:04.993 INFO:tasks.workunit.client.0.smithi060.stdout:removed './223/34' 2024-04-14T23:54:04.993 INFO:tasks.workunit.client.0.smithi060.stdout:removed './223/38' 2024-04-14T23:54:04.993 INFO:tasks.workunit.client.0.smithi060.stdout:removed './223/28' 2024-04-14T23:54:04.993 INFO:tasks.workunit.client.0.smithi060.stdout:removed './223/24' 2024-04-14T23:54:04.993 INFO:tasks.workunit.client.0.smithi060.stdout:removed './223/29' 2024-04-14T23:54:04.993 INFO:tasks.workunit.client.0.smithi060.stdout:removed './223/32' 2024-04-14T23:54:04.993 INFO:tasks.workunit.client.0.smithi060.stdout:removed './223/22' 2024-04-14T23:54:04.993 INFO:tasks.workunit.client.0.smithi060.stdout:removed './223/25' 2024-04-14T23:54:04.993 INFO:tasks.workunit.client.0.smithi060.stdout:removed './223/30' 2024-04-14T23:54:04.994 INFO:tasks.workunit.client.0.smithi060.stdout:removed './223/33' 2024-04-14T23:54:04.994 INFO:tasks.workunit.client.0.smithi060.stdout:removed './223/23' 2024-04-14T23:54:04.994 INFO:tasks.workunit.client.0.smithi060.stdout:removed './223/35' 2024-04-14T23:54:04.994 INFO:tasks.workunit.client.0.smithi060.stdout:removed './223/26' 2024-04-14T23:54:04.994 INFO:tasks.workunit.client.0.smithi060.stdout:removed './223/37' 2024-04-14T23:54:04.994 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './223' 2024-04-14T23:54:04.994 INFO:tasks.workunit.client.0.smithi060.stdout:removed './224/21' 2024-04-14T23:54:04.994 INFO:tasks.workunit.client.0.smithi060.stdout:removed './224/27' 2024-04-14T23:54:04.994 INFO:tasks.workunit.client.0.smithi060.stdout:removed './224/31' 2024-04-14T23:54:04.994 INFO:tasks.workunit.client.0.smithi060.stdout:removed './224/36' 2024-04-14T23:54:04.994 INFO:tasks.workunit.client.0.smithi060.stdout:removed './224/20' 2024-04-14T23:54:04.994 INFO:tasks.workunit.client.0.smithi060.stdout:removed './224/39' 2024-04-14T23:54:04.994 INFO:tasks.workunit.client.0.smithi060.stdout:removed './224/34' 2024-04-14T23:54:04.994 INFO:tasks.workunit.client.0.smithi060.stdout:removed './224/38' 2024-04-14T23:54:04.995 INFO:tasks.workunit.client.0.smithi060.stdout:removed './224/28' 2024-04-14T23:54:04.995 INFO:tasks.workunit.client.0.smithi060.stdout:removed './224/24' 2024-04-14T23:54:04.995 INFO:tasks.workunit.client.0.smithi060.stdout:removed './224/29' 2024-04-14T23:54:04.995 INFO:tasks.workunit.client.0.smithi060.stdout:removed './224/32' 2024-04-14T23:54:04.995 INFO:tasks.workunit.client.0.smithi060.stdout:removed './224/22' 2024-04-14T23:54:04.995 INFO:tasks.workunit.client.0.smithi060.stdout:removed './224/25' 2024-04-14T23:54:04.995 INFO:tasks.workunit.client.0.smithi060.stdout:removed './224/30' 2024-04-14T23:54:04.995 INFO:tasks.workunit.client.0.smithi060.stdout:removed './224/33' 2024-04-14T23:54:04.995 INFO:tasks.workunit.client.0.smithi060.stdout:removed './224/23' 2024-04-14T23:54:04.995 INFO:tasks.workunit.client.0.smithi060.stdout:removed './224/35' 2024-04-14T23:54:04.996 INFO:tasks.workunit.client.0.smithi060.stdout:removed './224/26' 2024-04-14T23:54:04.996 INFO:tasks.workunit.client.0.smithi060.stdout:removed './224/37' 2024-04-14T23:54:04.996 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './224' 2024-04-14T23:54:04.996 INFO:tasks.workunit.client.0.smithi060.stdout:removed './225/21' 2024-04-14T23:54:04.996 INFO:tasks.workunit.client.0.smithi060.stdout:removed './225/27' 2024-04-14T23:54:04.996 INFO:tasks.workunit.client.0.smithi060.stdout:removed './225/31' 2024-04-14T23:54:04.996 INFO:tasks.workunit.client.0.smithi060.stdout:removed './225/36' 2024-04-14T23:54:04.996 INFO:tasks.workunit.client.0.smithi060.stdout:removed './225/20' 2024-04-14T23:54:04.996 INFO:tasks.workunit.client.0.smithi060.stdout:removed './225/39' 2024-04-14T23:54:04.996 INFO:tasks.workunit.client.0.smithi060.stdout:removed './225/34' 2024-04-14T23:54:04.996 INFO:tasks.workunit.client.0.smithi060.stdout:removed './225/38' 2024-04-14T23:54:04.996 INFO:tasks.workunit.client.0.smithi060.stdout:removed './225/28' 2024-04-14T23:54:04.996 INFO:tasks.workunit.client.0.smithi060.stdout:removed './225/24' 2024-04-14T23:54:04.996 INFO:tasks.workunit.client.0.smithi060.stdout:removed './225/29' 2024-04-14T23:54:04.997 INFO:tasks.workunit.client.0.smithi060.stdout:removed './225/32' 2024-04-14T23:54:04.997 INFO:tasks.workunit.client.0.smithi060.stdout:removed './225/22' 2024-04-14T23:54:04.997 INFO:tasks.workunit.client.0.smithi060.stdout:removed './225/25' 2024-04-14T23:54:04.997 INFO:tasks.workunit.client.0.smithi060.stdout:removed './225/30' 2024-04-14T23:54:04.997 INFO:tasks.workunit.client.0.smithi060.stdout:removed './225/33' 2024-04-14T23:54:04.997 INFO:tasks.workunit.client.0.smithi060.stdout:removed './225/23' 2024-04-14T23:54:04.997 INFO:tasks.workunit.client.0.smithi060.stdout:removed './225/35' 2024-04-14T23:54:04.997 INFO:tasks.workunit.client.0.smithi060.stdout:removed './225/26' 2024-04-14T23:54:04.997 INFO:tasks.workunit.client.0.smithi060.stdout:removed './225/37' 2024-04-14T23:54:04.997 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './225' 2024-04-14T23:54:04.997 INFO:tasks.workunit.client.0.smithi060.stdout:removed './226/21' 2024-04-14T23:54:04.997 INFO:tasks.workunit.client.0.smithi060.stdout:removed './226/27' 2024-04-14T23:54:04.997 INFO:tasks.workunit.client.0.smithi060.stdout:removed './226/31' 2024-04-14T23:54:04.997 INFO:tasks.workunit.client.0.smithi060.stdout:removed './226/36' 2024-04-14T23:54:04.998 INFO:tasks.workunit.client.0.smithi060.stdout:removed './226/20' 2024-04-14T23:54:04.998 INFO:tasks.workunit.client.0.smithi060.stdout:removed './226/39' 2024-04-14T23:54:04.998 INFO:tasks.workunit.client.0.smithi060.stdout:removed './226/34' 2024-04-14T23:54:04.998 INFO:tasks.workunit.client.0.smithi060.stdout:removed './226/38' 2024-04-14T23:54:04.998 INFO:tasks.workunit.client.0.smithi060.stdout:removed './226/28' 2024-04-14T23:54:04.998 INFO:tasks.workunit.client.0.smithi060.stdout:removed './226/24' 2024-04-14T23:54:08.065 INFO:tasks.workunit.client.0.smithi060.stdout:removed './226/29' 2024-04-14T23:54:08.065 INFO:tasks.workunit.client.0.smithi060.stdout:removed './226/32' 2024-04-14T23:54:08.065 INFO:tasks.workunit.client.0.smithi060.stdout:removed './226/22' 2024-04-14T23:54:08.065 INFO:tasks.workunit.client.0.smithi060.stdout:removed './226/25' 2024-04-14T23:54:08.065 INFO:tasks.workunit.client.0.smithi060.stdout:removed './226/30' 2024-04-14T23:54:08.065 INFO:tasks.workunit.client.0.smithi060.stdout:removed './226/33' 2024-04-14T23:54:08.065 INFO:tasks.workunit.client.0.smithi060.stdout:removed './226/23' 2024-04-14T23:54:08.065 INFO:tasks.workunit.client.0.smithi060.stdout:removed './226/35' 2024-04-14T23:54:08.066 INFO:tasks.workunit.client.0.smithi060.stdout:removed './226/26' 2024-04-14T23:54:08.066 INFO:tasks.workunit.client.0.smithi060.stdout:removed './226/37' 2024-04-14T23:54:08.066 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './226' 2024-04-14T23:54:08.066 INFO:tasks.workunit.client.0.smithi060.stdout:removed './227/21' 2024-04-14T23:54:08.066 INFO:tasks.workunit.client.0.smithi060.stdout:removed './227/27' 2024-04-14T23:54:08.066 INFO:tasks.workunit.client.0.smithi060.stdout:removed './227/31' 2024-04-14T23:54:08.066 INFO:tasks.workunit.client.0.smithi060.stdout:removed './227/36' 2024-04-14T23:54:08.066 INFO:tasks.workunit.client.0.smithi060.stdout:removed './227/20' 2024-04-14T23:54:08.066 INFO:tasks.workunit.client.0.smithi060.stdout:removed './227/39' 2024-04-14T23:54:08.066 INFO:tasks.workunit.client.0.smithi060.stdout:removed './227/34' 2024-04-14T23:54:08.066 INFO:tasks.workunit.client.0.smithi060.stdout:removed './227/38' 2024-04-14T23:54:08.066 INFO:tasks.workunit.client.0.smithi060.stdout:removed './227/28' 2024-04-14T23:54:08.066 INFO:tasks.workunit.client.0.smithi060.stdout:removed './227/24' 2024-04-14T23:54:08.066 INFO:tasks.workunit.client.0.smithi060.stdout:removed './227/29' 2024-04-14T23:54:08.066 INFO:tasks.workunit.client.0.smithi060.stdout:removed './227/32' 2024-04-14T23:54:08.067 INFO:tasks.workunit.client.0.smithi060.stdout:removed './227/22' 2024-04-14T23:54:08.067 INFO:tasks.workunit.client.0.smithi060.stdout:removed './227/25' 2024-04-14T23:54:08.067 INFO:tasks.workunit.client.0.smithi060.stdout:removed './227/30' 2024-04-14T23:54:08.067 INFO:tasks.workunit.client.0.smithi060.stdout:removed './227/33' 2024-04-14T23:54:08.067 INFO:tasks.workunit.client.0.smithi060.stdout:removed './227/23' 2024-04-14T23:54:08.067 INFO:tasks.workunit.client.0.smithi060.stdout:removed './227/35' 2024-04-14T23:54:08.067 INFO:tasks.workunit.client.0.smithi060.stdout:removed './227/26' 2024-04-14T23:54:08.067 INFO:tasks.workunit.client.0.smithi060.stdout:removed './227/37' 2024-04-14T23:54:08.067 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './227' 2024-04-14T23:54:08.067 INFO:tasks.workunit.client.0.smithi060.stdout:removed './228/21' 2024-04-14T23:54:08.067 INFO:tasks.workunit.client.0.smithi060.stdout:removed './228/27' 2024-04-14T23:54:08.067 INFO:tasks.workunit.client.0.smithi060.stdout:removed './228/31' 2024-04-14T23:54:08.067 INFO:tasks.workunit.client.0.smithi060.stdout:removed './228/36' 2024-04-14T23:54:08.067 INFO:tasks.workunit.client.0.smithi060.stdout:removed './228/20' 2024-04-14T23:54:08.067 INFO:tasks.workunit.client.0.smithi060.stdout:removed './228/39' 2024-04-14T23:54:08.068 INFO:tasks.workunit.client.0.smithi060.stdout:removed './228/34' 2024-04-14T23:54:08.068 INFO:tasks.workunit.client.0.smithi060.stdout:removed './228/38' 2024-04-14T23:54:08.068 INFO:tasks.workunit.client.0.smithi060.stdout:removed './228/28' 2024-04-14T23:54:08.068 INFO:tasks.workunit.client.0.smithi060.stdout:removed './228/24' 2024-04-14T23:54:08.068 INFO:tasks.workunit.client.0.smithi060.stdout:removed './228/29' 2024-04-14T23:54:08.068 INFO:tasks.workunit.client.0.smithi060.stdout:removed './228/32' 2024-04-14T23:54:08.068 INFO:tasks.workunit.client.0.smithi060.stdout:removed './228/22' 2024-04-14T23:54:08.068 INFO:tasks.workunit.client.0.smithi060.stdout:removed './228/25' 2024-04-14T23:54:08.068 INFO:tasks.workunit.client.0.smithi060.stdout:removed './228/30' 2024-04-14T23:54:08.068 INFO:tasks.workunit.client.0.smithi060.stdout:removed './228/33' 2024-04-14T23:54:08.068 INFO:tasks.workunit.client.0.smithi060.stdout:removed './228/23' 2024-04-14T23:54:08.068 INFO:tasks.workunit.client.0.smithi060.stdout:removed './228/35' 2024-04-14T23:54:08.068 INFO:tasks.workunit.client.0.smithi060.stdout:removed './228/26' 2024-04-14T23:54:08.068 INFO:tasks.workunit.client.0.smithi060.stdout:removed './228/37' 2024-04-14T23:54:08.068 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './228' 2024-04-14T23:54:08.069 INFO:tasks.workunit.client.0.smithi060.stdout:removed './229/21' 2024-04-14T23:54:08.069 INFO:tasks.workunit.client.0.smithi060.stdout:removed './229/27' 2024-04-14T23:54:08.069 INFO:tasks.workunit.client.0.smithi060.stdout:removed './229/31' 2024-04-14T23:54:08.069 INFO:tasks.workunit.client.0.smithi060.stdout:removed './229/36' 2024-04-14T23:54:08.069 INFO:tasks.workunit.client.0.smithi060.stdout:removed './229/20' 2024-04-14T23:54:08.069 INFO:tasks.workunit.client.0.smithi060.stdout:removed './229/39' 2024-04-14T23:54:08.069 INFO:tasks.workunit.client.0.smithi060.stdout:removed './229/34' 2024-04-14T23:54:08.069 INFO:tasks.workunit.client.0.smithi060.stdout:removed './229/38' 2024-04-14T23:54:08.069 INFO:tasks.workunit.client.0.smithi060.stdout:removed './229/28' 2024-04-14T23:54:08.069 INFO:tasks.workunit.client.0.smithi060.stdout:removed './229/24' 2024-04-14T23:54:08.069 INFO:tasks.workunit.client.0.smithi060.stdout:removed './229/29' 2024-04-14T23:54:08.069 INFO:tasks.workunit.client.0.smithi060.stdout:removed './229/32' 2024-04-14T23:54:08.070 INFO:tasks.workunit.client.0.smithi060.stdout:removed './229/22' 2024-04-14T23:54:08.070 INFO:tasks.workunit.client.0.smithi060.stdout:removed './229/25' 2024-04-14T23:54:08.070 INFO:tasks.workunit.client.0.smithi060.stdout:removed './229/30' 2024-04-14T23:54:08.070 INFO:tasks.workunit.client.0.smithi060.stdout:removed './229/33' 2024-04-14T23:54:08.070 INFO:tasks.workunit.client.0.smithi060.stdout:removed './229/23' 2024-04-14T23:54:08.070 INFO:tasks.workunit.client.0.smithi060.stdout:removed './229/35' 2024-04-14T23:54:08.070 INFO:tasks.workunit.client.0.smithi060.stdout:removed './229/26' 2024-04-14T23:54:08.070 INFO:tasks.workunit.client.0.smithi060.stdout:removed './229/37' 2024-04-14T23:54:08.070 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './229' 2024-04-14T23:54:08.070 INFO:tasks.workunit.client.0.smithi060.stdout:removed './230/21' 2024-04-14T23:54:08.070 INFO:tasks.workunit.client.0.smithi060.stdout:removed './230/27' 2024-04-14T23:54:08.070 INFO:tasks.workunit.client.0.smithi060.stdout:removed './230/31' 2024-04-14T23:54:08.070 INFO:tasks.workunit.client.0.smithi060.stdout:removed './230/36' 2024-04-14T23:54:08.070 INFO:tasks.workunit.client.0.smithi060.stdout:removed './230/20' 2024-04-14T23:54:08.071 INFO:tasks.workunit.client.0.smithi060.stdout:removed './230/39' 2024-04-14T23:54:08.071 INFO:tasks.workunit.client.0.smithi060.stdout:removed './230/34' 2024-04-14T23:54:08.071 INFO:tasks.workunit.client.0.smithi060.stdout:removed './230/38' 2024-04-14T23:54:08.071 INFO:tasks.workunit.client.0.smithi060.stdout:removed './230/28' 2024-04-14T23:54:08.071 INFO:tasks.workunit.client.0.smithi060.stdout:removed './230/24' 2024-04-14T23:54:08.071 INFO:tasks.workunit.client.0.smithi060.stdout:removed './230/29' 2024-04-14T23:54:08.071 INFO:tasks.workunit.client.0.smithi060.stdout:removed './230/32' 2024-04-14T23:54:08.071 INFO:tasks.workunit.client.0.smithi060.stdout:removed './230/22' 2024-04-14T23:54:08.071 INFO:tasks.workunit.client.0.smithi060.stdout:removed './230/25' 2024-04-14T23:54:08.071 INFO:tasks.workunit.client.0.smithi060.stdout:removed './230/30' 2024-04-14T23:54:08.071 INFO:tasks.workunit.client.0.smithi060.stdout:removed './230/33' 2024-04-14T23:54:08.071 INFO:tasks.workunit.client.0.smithi060.stdout:removed './230/23' 2024-04-14T23:54:08.071 INFO:tasks.workunit.client.0.smithi060.stdout:removed './230/35' 2024-04-14T23:54:08.071 INFO:tasks.workunit.client.0.smithi060.stdout:removed './230/26' 2024-04-14T23:54:08.071 INFO:tasks.workunit.client.0.smithi060.stdout:removed './230/37' 2024-04-14T23:54:08.072 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './230' 2024-04-14T23:54:08.072 INFO:tasks.workunit.client.0.smithi060.stdout:removed './231/21' 2024-04-14T23:54:08.072 INFO:tasks.workunit.client.0.smithi060.stdout:removed './231/27' 2024-04-14T23:54:08.072 INFO:tasks.workunit.client.0.smithi060.stdout:removed './231/31' 2024-04-14T23:54:08.072 INFO:tasks.workunit.client.0.smithi060.stdout:removed './231/36' 2024-04-14T23:54:08.072 INFO:tasks.workunit.client.0.smithi060.stdout:removed './231/20' 2024-04-14T23:54:08.072 INFO:tasks.workunit.client.0.smithi060.stdout:removed './231/39' 2024-04-14T23:54:08.072 INFO:tasks.workunit.client.0.smithi060.stdout:removed './231/34' 2024-04-14T23:54:08.072 INFO:tasks.workunit.client.0.smithi060.stdout:removed './231/38' 2024-04-14T23:54:08.072 INFO:tasks.workunit.client.0.smithi060.stdout:removed './231/28' 2024-04-14T23:54:08.072 INFO:tasks.workunit.client.0.smithi060.stdout:removed './231/24' 2024-04-14T23:54:08.072 INFO:tasks.workunit.client.0.smithi060.stdout:removed './231/29' 2024-04-14T23:54:08.072 INFO:tasks.workunit.client.0.smithi060.stdout:removed './231/32' 2024-04-14T23:54:08.072 INFO:tasks.workunit.client.0.smithi060.stdout:removed './231/22' 2024-04-14T23:54:08.072 INFO:tasks.workunit.client.0.smithi060.stdout:removed './231/25' 2024-04-14T23:54:08.073 INFO:tasks.workunit.client.0.smithi060.stdout:removed './231/30' 2024-04-14T23:54:08.073 INFO:tasks.workunit.client.0.smithi060.stdout:removed './231/33' 2024-04-14T23:54:08.073 INFO:tasks.workunit.client.0.smithi060.stdout:removed './231/23' 2024-04-14T23:54:08.073 INFO:tasks.workunit.client.0.smithi060.stdout:removed './231/35' 2024-04-14T23:54:08.073 INFO:tasks.workunit.client.0.smithi060.stdout:removed './231/26' 2024-04-14T23:54:08.073 INFO:tasks.workunit.client.0.smithi060.stdout:removed './231/37' 2024-04-14T23:54:08.073 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './231' 2024-04-14T23:54:08.073 INFO:tasks.workunit.client.0.smithi060.stdout:removed './232/21' 2024-04-14T23:54:08.073 INFO:tasks.workunit.client.0.smithi060.stdout:removed './232/27' 2024-04-14T23:54:08.073 INFO:tasks.workunit.client.0.smithi060.stdout:removed './232/31' 2024-04-14T23:54:08.073 INFO:tasks.workunit.client.0.smithi060.stdout:removed './232/36' 2024-04-14T23:54:08.073 INFO:tasks.workunit.client.0.smithi060.stdout:removed './232/20' 2024-04-14T23:54:08.073 INFO:tasks.workunit.client.0.smithi060.stdout:removed './232/39' 2024-04-14T23:54:08.073 INFO:tasks.workunit.client.0.smithi060.stdout:removed './232/34' 2024-04-14T23:54:08.073 INFO:tasks.workunit.client.0.smithi060.stdout:removed './232/38' 2024-04-14T23:54:08.074 INFO:tasks.workunit.client.0.smithi060.stdout:removed './232/28' 2024-04-14T23:54:08.074 INFO:tasks.workunit.client.0.smithi060.stdout:removed './232/24' 2024-04-14T23:54:08.074 INFO:tasks.workunit.client.0.smithi060.stdout:removed './232/29' 2024-04-14T23:54:08.074 INFO:tasks.workunit.client.0.smithi060.stdout:removed './232/32' 2024-04-14T23:54:08.074 INFO:tasks.workunit.client.0.smithi060.stdout:removed './232/22' 2024-04-14T23:54:08.074 INFO:tasks.workunit.client.0.smithi060.stdout:removed './232/25' 2024-04-14T23:54:08.074 INFO:tasks.workunit.client.0.smithi060.stdout:removed './232/30' 2024-04-14T23:54:08.074 INFO:tasks.workunit.client.0.smithi060.stdout:removed './232/33' 2024-04-14T23:54:08.074 INFO:tasks.workunit.client.0.smithi060.stdout:removed './232/23' 2024-04-14T23:54:08.074 INFO:tasks.workunit.client.0.smithi060.stdout:removed './232/35' 2024-04-14T23:54:08.074 INFO:tasks.workunit.client.0.smithi060.stdout:removed './232/26' 2024-04-14T23:54:08.074 INFO:tasks.workunit.client.0.smithi060.stdout:removed './232/37' 2024-04-14T23:54:08.074 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './232' 2024-04-14T23:54:08.074 INFO:tasks.workunit.client.0.smithi060.stdout:removed './233/21' 2024-04-14T23:54:08.074 INFO:tasks.workunit.client.0.smithi060.stdout:removed './233/27' 2024-04-14T23:54:08.075 INFO:tasks.workunit.client.0.smithi060.stdout:removed './233/31' 2024-04-14T23:54:08.075 INFO:tasks.workunit.client.0.smithi060.stdout:removed './233/36' 2024-04-14T23:54:08.075 INFO:tasks.workunit.client.0.smithi060.stdout:removed './233/20' 2024-04-14T23:54:08.075 INFO:tasks.workunit.client.0.smithi060.stdout:removed './233/39' 2024-04-14T23:54:08.075 INFO:tasks.workunit.client.0.smithi060.stdout:removed './233/34' 2024-04-14T23:54:08.075 INFO:tasks.workunit.client.0.smithi060.stdout:removed './233/38' 2024-04-14T23:54:08.075 INFO:tasks.workunit.client.0.smithi060.stdout:removed './233/28' 2024-04-14T23:54:08.075 INFO:tasks.workunit.client.0.smithi060.stdout:removed './233/24' 2024-04-14T23:54:08.075 INFO:tasks.workunit.client.0.smithi060.stdout:removed './233/29' 2024-04-14T23:54:08.075 INFO:tasks.workunit.client.0.smithi060.stdout:removed './233/32' 2024-04-14T23:54:08.075 INFO:tasks.workunit.client.0.smithi060.stdout:removed './233/22' 2024-04-14T23:54:08.075 INFO:tasks.workunit.client.0.smithi060.stdout:removed './233/25' 2024-04-14T23:54:08.075 INFO:tasks.workunit.client.0.smithi060.stdout:removed './233/30' 2024-04-14T23:54:08.075 INFO:tasks.workunit.client.0.smithi060.stdout:removed './233/33' 2024-04-14T23:54:08.075 INFO:tasks.workunit.client.0.smithi060.stdout:removed './233/23' 2024-04-14T23:54:08.076 INFO:tasks.workunit.client.0.smithi060.stdout:removed './233/35' 2024-04-14T23:54:08.076 INFO:tasks.workunit.client.0.smithi060.stdout:removed './233/26' 2024-04-14T23:54:08.076 INFO:tasks.workunit.client.0.smithi060.stdout:removed './233/37' 2024-04-14T23:54:08.076 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './233' 2024-04-14T23:54:08.076 INFO:tasks.workunit.client.0.smithi060.stdout:removed './234/21' 2024-04-14T23:54:08.076 INFO:tasks.workunit.client.0.smithi060.stdout:removed './234/27' 2024-04-14T23:54:08.076 INFO:tasks.workunit.client.0.smithi060.stdout:removed './234/31' 2024-04-14T23:54:08.076 INFO:tasks.workunit.client.0.smithi060.stdout:removed './234/36' 2024-04-14T23:54:08.076 INFO:tasks.workunit.client.0.smithi060.stdout:removed './234/20' 2024-04-14T23:54:08.076 INFO:tasks.workunit.client.0.smithi060.stdout:removed './234/39' 2024-04-14T23:54:08.076 INFO:tasks.workunit.client.0.smithi060.stdout:removed './234/34' 2024-04-14T23:54:08.076 INFO:tasks.workunit.client.0.smithi060.stdout:removed './234/38' 2024-04-14T23:54:08.076 INFO:tasks.workunit.client.0.smithi060.stdout:removed './234/28' 2024-04-14T23:54:08.076 INFO:tasks.workunit.client.0.smithi060.stdout:removed './234/24' 2024-04-14T23:54:08.076 INFO:tasks.workunit.client.0.smithi060.stdout:removed './234/29' 2024-04-14T23:54:08.077 INFO:tasks.workunit.client.0.smithi060.stdout:removed './234/32' 2024-04-14T23:54:08.077 INFO:tasks.workunit.client.0.smithi060.stdout:removed './234/22' 2024-04-14T23:54:08.077 INFO:tasks.workunit.client.0.smithi060.stdout:removed './234/25' 2024-04-14T23:54:08.077 INFO:tasks.workunit.client.0.smithi060.stdout:removed './234/30' 2024-04-14T23:54:08.077 INFO:tasks.workunit.client.0.smithi060.stdout:removed './234/33' 2024-04-14T23:54:08.077 INFO:tasks.workunit.client.0.smithi060.stdout:removed './234/23' 2024-04-14T23:54:08.077 INFO:tasks.workunit.client.0.smithi060.stdout:removed './234/35' 2024-04-14T23:54:08.077 INFO:tasks.workunit.client.0.smithi060.stdout:removed './234/26' 2024-04-14T23:54:08.077 INFO:tasks.workunit.client.0.smithi060.stdout:removed './234/37' 2024-04-14T23:54:08.077 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './234' 2024-04-14T23:54:08.077 INFO:tasks.workunit.client.0.smithi060.stdout:removed './235/21' 2024-04-14T23:54:08.077 INFO:tasks.workunit.client.0.smithi060.stdout:removed './235/27' 2024-04-14T23:54:08.077 INFO:tasks.workunit.client.0.smithi060.stdout:removed './235/31' 2024-04-14T23:54:08.077 INFO:tasks.workunit.client.0.smithi060.stdout:removed './235/36' 2024-04-14T23:54:08.077 INFO:tasks.workunit.client.0.smithi060.stdout:removed './235/20' 2024-04-14T23:54:08.077 INFO:tasks.workunit.client.0.smithi060.stdout:removed './235/39' 2024-04-14T23:54:08.078 INFO:tasks.workunit.client.0.smithi060.stdout:removed './235/34' 2024-04-14T23:54:08.078 INFO:tasks.workunit.client.0.smithi060.stdout:removed './235/38' 2024-04-14T23:54:08.078 INFO:tasks.workunit.client.0.smithi060.stdout:removed './235/28' 2024-04-14T23:54:08.078 INFO:tasks.workunit.client.0.smithi060.stdout:removed './235/24' 2024-04-14T23:54:08.078 INFO:tasks.workunit.client.0.smithi060.stdout:removed './235/29' 2024-04-14T23:54:08.078 INFO:tasks.workunit.client.0.smithi060.stdout:removed './235/32' 2024-04-14T23:54:08.078 INFO:tasks.workunit.client.0.smithi060.stdout:removed './235/22' 2024-04-14T23:54:08.078 INFO:tasks.workunit.client.0.smithi060.stdout:removed './235/25' 2024-04-14T23:54:08.078 INFO:tasks.workunit.client.0.smithi060.stdout:removed './235/30' 2024-04-14T23:54:08.078 INFO:tasks.workunit.client.0.smithi060.stdout:removed './235/33' 2024-04-14T23:54:08.078 INFO:tasks.workunit.client.0.smithi060.stdout:removed './235/23' 2024-04-14T23:54:08.078 INFO:tasks.workunit.client.0.smithi060.stdout:removed './235/35' 2024-04-14T23:54:08.078 INFO:tasks.workunit.client.0.smithi060.stdout:removed './235/26' 2024-04-14T23:54:08.079 INFO:tasks.workunit.client.0.smithi060.stdout:removed './235/37' 2024-04-14T23:54:08.079 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './235' 2024-04-14T23:54:08.079 INFO:tasks.workunit.client.0.smithi060.stdout:removed './236/21' 2024-04-14T23:54:08.079 INFO:tasks.workunit.client.0.smithi060.stdout:removed './236/27' 2024-04-14T23:54:08.079 INFO:tasks.workunit.client.0.smithi060.stdout:removed './236/31' 2024-04-14T23:54:08.079 INFO:tasks.workunit.client.0.smithi060.stdout:removed './236/36' 2024-04-14T23:54:08.079 INFO:tasks.workunit.client.0.smithi060.stdout:removed './236/20' 2024-04-14T23:54:08.079 INFO:tasks.workunit.client.0.smithi060.stdout:removed './236/39' 2024-04-14T23:54:08.079 INFO:tasks.workunit.client.0.smithi060.stdout:removed './236/34' 2024-04-14T23:54:08.079 INFO:tasks.workunit.client.0.smithi060.stdout:removed './236/38' 2024-04-14T23:54:08.079 INFO:tasks.workunit.client.0.smithi060.stdout:removed './236/28' 2024-04-14T23:54:08.079 INFO:tasks.workunit.client.0.smithi060.stdout:removed './236/24' 2024-04-14T23:54:08.079 INFO:tasks.workunit.client.0.smithi060.stdout:removed './236/29' 2024-04-14T23:54:08.080 INFO:tasks.workunit.client.0.smithi060.stdout:removed './236/32' 2024-04-14T23:54:12.203 INFO:tasks.workunit.client.0.smithi060.stdout:removed './236/22' 2024-04-14T23:54:12.203 INFO:tasks.workunit.client.0.smithi060.stdout:removed './236/25' 2024-04-14T23:54:12.203 INFO:tasks.workunit.client.0.smithi060.stdout:removed './236/30' 2024-04-14T23:54:12.204 INFO:tasks.workunit.client.0.smithi060.stdout:removed './236/33' 2024-04-14T23:54:12.204 INFO:tasks.workunit.client.0.smithi060.stdout:removed './236/23' 2024-04-14T23:54:12.204 INFO:tasks.workunit.client.0.smithi060.stdout:removed './236/35' 2024-04-14T23:54:12.204 INFO:tasks.workunit.client.0.smithi060.stdout:removed './236/26' 2024-04-14T23:54:12.204 INFO:tasks.workunit.client.0.smithi060.stdout:removed './236/37' 2024-04-14T23:54:12.204 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './236' 2024-04-14T23:54:12.204 INFO:tasks.workunit.client.0.smithi060.stdout:removed './237/21' 2024-04-14T23:54:12.204 INFO:tasks.workunit.client.0.smithi060.stdout:removed './237/27' 2024-04-14T23:54:12.205 INFO:tasks.workunit.client.0.smithi060.stdout:removed './237/31' 2024-04-14T23:54:12.205 INFO:tasks.workunit.client.0.smithi060.stdout:removed './237/36' 2024-04-14T23:54:12.205 INFO:tasks.workunit.client.0.smithi060.stdout:removed './237/20' 2024-04-14T23:54:12.205 INFO:tasks.workunit.client.0.smithi060.stdout:removed './237/39' 2024-04-14T23:54:12.205 INFO:tasks.workunit.client.0.smithi060.stdout:removed './237/34' 2024-04-14T23:54:12.205 INFO:tasks.workunit.client.0.smithi060.stdout:removed './237/38' 2024-04-14T23:54:12.205 INFO:tasks.workunit.client.0.smithi060.stdout:removed './237/28' 2024-04-14T23:54:12.205 INFO:tasks.workunit.client.0.smithi060.stdout:removed './237/24' 2024-04-14T23:54:12.205 INFO:tasks.workunit.client.0.smithi060.stdout:removed './237/29' 2024-04-14T23:54:12.205 INFO:tasks.workunit.client.0.smithi060.stdout:removed './237/32' 2024-04-14T23:54:12.206 INFO:tasks.workunit.client.0.smithi060.stdout:removed './237/22' 2024-04-14T23:54:12.206 INFO:tasks.workunit.client.0.smithi060.stdout:removed './237/25' 2024-04-14T23:54:12.206 INFO:tasks.workunit.client.0.smithi060.stdout:removed './237/30' 2024-04-14T23:54:12.206 INFO:tasks.workunit.client.0.smithi060.stdout:removed './237/33' 2024-04-14T23:54:12.206 INFO:tasks.workunit.client.0.smithi060.stdout:removed './237/23' 2024-04-14T23:54:12.206 INFO:tasks.workunit.client.0.smithi060.stdout:removed './237/35' 2024-04-14T23:54:12.206 INFO:tasks.workunit.client.0.smithi060.stdout:removed './237/26' 2024-04-14T23:54:12.206 INFO:tasks.workunit.client.0.smithi060.stdout:removed './237/37' 2024-04-14T23:54:12.206 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './237' 2024-04-14T23:54:12.206 INFO:tasks.workunit.client.0.smithi060.stdout:removed './238/21' 2024-04-14T23:54:12.206 INFO:tasks.workunit.client.0.smithi060.stdout:removed './238/27' 2024-04-14T23:54:12.206 INFO:tasks.workunit.client.0.smithi060.stdout:removed './238/31' 2024-04-14T23:54:12.207 INFO:tasks.workunit.client.0.smithi060.stdout:removed './238/36' 2024-04-14T23:54:12.207 INFO:tasks.workunit.client.0.smithi060.stdout:removed './238/20' 2024-04-14T23:54:12.207 INFO:tasks.workunit.client.0.smithi060.stdout:removed './238/39' 2024-04-14T23:54:12.207 INFO:tasks.workunit.client.0.smithi060.stdout:removed './238/34' 2024-04-14T23:54:12.207 INFO:tasks.workunit.client.0.smithi060.stdout:removed './238/38' 2024-04-14T23:54:12.207 INFO:tasks.workunit.client.0.smithi060.stdout:removed './238/28' 2024-04-14T23:54:12.207 INFO:tasks.workunit.client.0.smithi060.stdout:removed './238/24' 2024-04-14T23:54:12.207 INFO:tasks.workunit.client.0.smithi060.stdout:removed './238/29' 2024-04-14T23:54:12.207 INFO:tasks.workunit.client.0.smithi060.stdout:removed './238/32' 2024-04-14T23:54:12.207 INFO:tasks.workunit.client.0.smithi060.stdout:removed './238/22' 2024-04-14T23:54:12.207 INFO:tasks.workunit.client.0.smithi060.stdout:removed './238/25' 2024-04-14T23:54:12.207 INFO:tasks.workunit.client.0.smithi060.stdout:removed './238/30' 2024-04-14T23:54:12.207 INFO:tasks.workunit.client.0.smithi060.stdout:removed './238/33' 2024-04-14T23:54:12.207 INFO:tasks.workunit.client.0.smithi060.stdout:removed './238/23' 2024-04-14T23:54:12.208 INFO:tasks.workunit.client.0.smithi060.stdout:removed './238/35' 2024-04-14T23:54:12.208 INFO:tasks.workunit.client.0.smithi060.stdout:removed './238/26' 2024-04-14T23:54:12.208 INFO:tasks.workunit.client.0.smithi060.stdout:removed './238/37' 2024-04-14T23:54:12.208 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './238' 2024-04-14T23:54:12.208 INFO:tasks.workunit.client.0.smithi060.stdout:removed './239/21' 2024-04-14T23:54:12.208 INFO:tasks.workunit.client.0.smithi060.stdout:removed './239/27' 2024-04-14T23:54:12.208 INFO:tasks.workunit.client.0.smithi060.stdout:removed './239/31' 2024-04-14T23:54:12.208 INFO:tasks.workunit.client.0.smithi060.stdout:removed './239/36' 2024-04-14T23:54:12.208 INFO:tasks.workunit.client.0.smithi060.stdout:removed './239/20' 2024-04-14T23:54:12.208 INFO:tasks.workunit.client.0.smithi060.stdout:removed './239/39' 2024-04-14T23:54:12.208 INFO:tasks.workunit.client.0.smithi060.stdout:removed './239/34' 2024-04-14T23:54:12.208 INFO:tasks.workunit.client.0.smithi060.stdout:removed './239/38' 2024-04-14T23:54:12.208 INFO:tasks.workunit.client.0.smithi060.stdout:removed './239/28' 2024-04-14T23:54:12.208 INFO:tasks.workunit.client.0.smithi060.stdout:removed './239/24' 2024-04-14T23:54:12.209 INFO:tasks.workunit.client.0.smithi060.stdout:removed './239/29' 2024-04-14T23:54:12.209 INFO:tasks.workunit.client.0.smithi060.stdout:removed './239/32' 2024-04-14T23:54:12.209 INFO:tasks.workunit.client.0.smithi060.stdout:removed './239/22' 2024-04-14T23:54:12.209 INFO:tasks.workunit.client.0.smithi060.stdout:removed './239/25' 2024-04-14T23:54:12.209 INFO:tasks.workunit.client.0.smithi060.stdout:removed './239/30' 2024-04-14T23:54:12.209 INFO:tasks.workunit.client.0.smithi060.stdout:removed './239/33' 2024-04-14T23:54:12.209 INFO:tasks.workunit.client.0.smithi060.stdout:removed './239/23' 2024-04-14T23:54:12.209 INFO:tasks.workunit.client.0.smithi060.stdout:removed './239/35' 2024-04-14T23:54:12.209 INFO:tasks.workunit.client.0.smithi060.stdout:removed './239/26' 2024-04-14T23:54:12.209 INFO:tasks.workunit.client.0.smithi060.stdout:removed './239/37' 2024-04-14T23:54:12.209 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './239' 2024-04-14T23:54:12.209 INFO:tasks.workunit.client.0.smithi060.stdout:removed './240/21' 2024-04-14T23:54:12.209 INFO:tasks.workunit.client.0.smithi060.stdout:removed './240/27' 2024-04-14T23:54:12.209 INFO:tasks.workunit.client.0.smithi060.stdout:removed './240/31' 2024-04-14T23:54:12.209 INFO:tasks.workunit.client.0.smithi060.stdout:removed './240/36' 2024-04-14T23:54:12.210 INFO:tasks.workunit.client.0.smithi060.stdout:removed './240/20' 2024-04-14T23:54:12.210 INFO:tasks.workunit.client.0.smithi060.stdout:removed './240/39' 2024-04-14T23:54:12.210 INFO:tasks.workunit.client.0.smithi060.stdout:removed './240/34' 2024-04-14T23:54:12.210 INFO:tasks.workunit.client.0.smithi060.stdout:removed './240/38' 2024-04-14T23:54:12.210 INFO:tasks.workunit.client.0.smithi060.stdout:removed './240/28' 2024-04-14T23:54:12.210 INFO:tasks.workunit.client.0.smithi060.stdout:removed './240/24' 2024-04-14T23:54:12.210 INFO:tasks.workunit.client.0.smithi060.stdout:removed './240/29' 2024-04-14T23:54:12.210 INFO:tasks.workunit.client.0.smithi060.stdout:removed './240/32' 2024-04-14T23:54:12.210 INFO:tasks.workunit.client.0.smithi060.stdout:removed './240/22' 2024-04-14T23:54:12.210 INFO:tasks.workunit.client.0.smithi060.stdout:removed './240/25' 2024-04-14T23:54:12.210 INFO:tasks.workunit.client.0.smithi060.stdout:removed './240/30' 2024-04-14T23:54:12.210 INFO:tasks.workunit.client.0.smithi060.stdout:removed './240/33' 2024-04-14T23:54:12.210 INFO:tasks.workunit.client.0.smithi060.stdout:removed './240/23' 2024-04-14T23:54:12.211 INFO:tasks.workunit.client.0.smithi060.stdout:removed './240/35' 2024-04-14T23:54:12.211 INFO:tasks.workunit.client.0.smithi060.stdout:removed './240/26' 2024-04-14T23:54:12.211 INFO:tasks.workunit.client.0.smithi060.stdout:removed './240/37' 2024-04-14T23:54:12.211 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './240' 2024-04-14T23:54:12.211 INFO:tasks.workunit.client.0.smithi060.stdout:removed './241/21' 2024-04-14T23:54:12.211 INFO:tasks.workunit.client.0.smithi060.stdout:removed './241/27' 2024-04-14T23:54:12.211 INFO:tasks.workunit.client.0.smithi060.stdout:removed './241/31' 2024-04-14T23:54:12.211 INFO:tasks.workunit.client.0.smithi060.stdout:removed './241/36' 2024-04-14T23:54:12.211 INFO:tasks.workunit.client.0.smithi060.stdout:removed './241/20' 2024-04-14T23:54:12.211 INFO:tasks.workunit.client.0.smithi060.stdout:removed './241/39' 2024-04-14T23:54:12.211 INFO:tasks.workunit.client.0.smithi060.stdout:removed './241/34' 2024-04-14T23:54:12.211 INFO:tasks.workunit.client.0.smithi060.stdout:removed './241/38' 2024-04-14T23:54:12.211 INFO:tasks.workunit.client.0.smithi060.stdout:removed './241/28' 2024-04-14T23:54:12.211 INFO:tasks.workunit.client.0.smithi060.stdout:removed './241/24' 2024-04-14T23:54:12.212 INFO:tasks.workunit.client.0.smithi060.stdout:removed './241/29' 2024-04-14T23:54:12.212 INFO:tasks.workunit.client.0.smithi060.stdout:removed './241/32' 2024-04-14T23:54:12.212 INFO:tasks.workunit.client.0.smithi060.stdout:removed './241/22' 2024-04-14T23:54:12.212 INFO:tasks.workunit.client.0.smithi060.stdout:removed './241/25' 2024-04-14T23:54:12.212 INFO:tasks.workunit.client.0.smithi060.stdout:removed './241/30' 2024-04-14T23:54:12.212 INFO:tasks.workunit.client.0.smithi060.stdout:removed './241/33' 2024-04-14T23:54:12.212 INFO:tasks.workunit.client.0.smithi060.stdout:removed './241/23' 2024-04-14T23:54:12.212 INFO:tasks.workunit.client.0.smithi060.stdout:removed './241/35' 2024-04-14T23:54:12.212 INFO:tasks.workunit.client.0.smithi060.stdout:removed './241/26' 2024-04-14T23:54:12.212 INFO:tasks.workunit.client.0.smithi060.stdout:removed './241/37' 2024-04-14T23:54:12.212 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './241' 2024-04-14T23:54:12.212 INFO:tasks.workunit.client.0.smithi060.stdout:removed './242/21' 2024-04-14T23:54:12.212 INFO:tasks.workunit.client.0.smithi060.stdout:removed './242/27' 2024-04-14T23:54:12.213 INFO:tasks.workunit.client.0.smithi060.stdout:removed './242/31' 2024-04-14T23:54:12.213 INFO:tasks.workunit.client.0.smithi060.stdout:removed './242/36' 2024-04-14T23:54:12.213 INFO:tasks.workunit.client.0.smithi060.stdout:removed './242/20' 2024-04-14T23:54:12.213 INFO:tasks.workunit.client.0.smithi060.stdout:removed './242/39' 2024-04-14T23:54:12.213 INFO:tasks.workunit.client.0.smithi060.stdout:removed './242/34' 2024-04-14T23:54:12.213 INFO:tasks.workunit.client.0.smithi060.stdout:removed './242/38' 2024-04-14T23:54:12.213 INFO:tasks.workunit.client.0.smithi060.stdout:removed './242/28' 2024-04-14T23:54:12.213 INFO:tasks.workunit.client.0.smithi060.stdout:removed './242/24' 2024-04-14T23:54:12.213 INFO:tasks.workunit.client.0.smithi060.stdout:removed './242/29' 2024-04-14T23:54:12.213 INFO:tasks.workunit.client.0.smithi060.stdout:removed './242/32' 2024-04-14T23:54:12.213 INFO:tasks.workunit.client.0.smithi060.stdout:removed './242/22' 2024-04-14T23:54:12.213 INFO:tasks.workunit.client.0.smithi060.stdout:removed './242/25' 2024-04-14T23:54:12.213 INFO:tasks.workunit.client.0.smithi060.stdout:removed './242/30' 2024-04-14T23:54:12.213 INFO:tasks.workunit.client.0.smithi060.stdout:removed './242/33' 2024-04-14T23:54:12.213 INFO:tasks.workunit.client.0.smithi060.stdout:removed './242/23' 2024-04-14T23:54:12.214 INFO:tasks.workunit.client.0.smithi060.stdout:removed './242/35' 2024-04-14T23:54:12.214 INFO:tasks.workunit.client.0.smithi060.stdout:removed './242/26' 2024-04-14T23:54:12.214 INFO:tasks.workunit.client.0.smithi060.stdout:removed './242/37' 2024-04-14T23:54:12.214 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './242' 2024-04-14T23:54:12.214 INFO:tasks.workunit.client.0.smithi060.stdout:removed './243/21' 2024-04-14T23:54:12.214 INFO:tasks.workunit.client.0.smithi060.stdout:removed './243/27' 2024-04-14T23:54:12.214 INFO:tasks.workunit.client.0.smithi060.stdout:removed './243/31' 2024-04-14T23:54:12.214 INFO:tasks.workunit.client.0.smithi060.stdout:removed './243/36' 2024-04-14T23:54:12.214 INFO:tasks.workunit.client.0.smithi060.stdout:removed './243/20' 2024-04-14T23:54:12.214 INFO:tasks.workunit.client.0.smithi060.stdout:removed './243/39' 2024-04-14T23:54:12.214 INFO:tasks.workunit.client.0.smithi060.stdout:removed './243/34' 2024-04-14T23:54:12.214 INFO:tasks.workunit.client.0.smithi060.stdout:removed './243/38' 2024-04-14T23:54:12.214 INFO:tasks.workunit.client.0.smithi060.stdout:removed './243/28' 2024-04-14T23:54:12.215 INFO:tasks.workunit.client.0.smithi060.stdout:removed './243/24' 2024-04-14T23:54:12.215 INFO:tasks.workunit.client.0.smithi060.stdout:removed './243/29' 2024-04-14T23:54:12.215 INFO:tasks.workunit.client.0.smithi060.stdout:removed './243/32' 2024-04-14T23:54:12.215 INFO:tasks.workunit.client.0.smithi060.stdout:removed './243/22' 2024-04-14T23:54:12.215 INFO:tasks.workunit.client.0.smithi060.stdout:removed './243/25' 2024-04-14T23:54:12.215 INFO:tasks.workunit.client.0.smithi060.stdout:removed './243/30' 2024-04-14T23:54:12.215 INFO:tasks.workunit.client.0.smithi060.stdout:removed './243/33' 2024-04-14T23:54:12.215 INFO:tasks.workunit.client.0.smithi060.stdout:removed './243/23' 2024-04-14T23:54:12.215 INFO:tasks.workunit.client.0.smithi060.stdout:removed './243/35' 2024-04-14T23:54:12.216 INFO:tasks.workunit.client.0.smithi060.stdout:removed './243/26' 2024-04-14T23:54:12.216 INFO:tasks.workunit.client.0.smithi060.stdout:removed './243/37' 2024-04-14T23:54:12.216 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './243' 2024-04-14T23:54:12.216 INFO:tasks.workunit.client.0.smithi060.stdout:removed './244/21' 2024-04-14T23:54:12.216 INFO:tasks.workunit.client.0.smithi060.stdout:removed './244/27' 2024-04-14T23:54:12.216 INFO:tasks.workunit.client.0.smithi060.stdout:removed './244/31' 2024-04-14T23:54:12.216 INFO:tasks.workunit.client.0.smithi060.stdout:removed './244/36' 2024-04-14T23:54:12.216 INFO:tasks.workunit.client.0.smithi060.stdout:removed './244/20' 2024-04-14T23:54:12.216 INFO:tasks.workunit.client.0.smithi060.stdout:removed './244/39' 2024-04-14T23:54:12.216 INFO:tasks.workunit.client.0.smithi060.stdout:removed './244/34' 2024-04-14T23:54:12.217 INFO:tasks.workunit.client.0.smithi060.stdout:removed './244/38' 2024-04-14T23:54:12.217 INFO:tasks.workunit.client.0.smithi060.stdout:removed './244/28' 2024-04-14T23:54:12.217 INFO:tasks.workunit.client.0.smithi060.stdout:removed './244/24' 2024-04-14T23:54:12.217 INFO:tasks.workunit.client.0.smithi060.stdout:removed './244/29' 2024-04-14T23:54:12.217 INFO:tasks.workunit.client.0.smithi060.stdout:removed './244/32' 2024-04-14T23:54:12.217 INFO:tasks.workunit.client.0.smithi060.stdout:removed './244/22' 2024-04-14T23:54:12.217 INFO:tasks.workunit.client.0.smithi060.stdout:removed './244/25' 2024-04-14T23:54:12.217 INFO:tasks.workunit.client.0.smithi060.stdout:removed './244/30' 2024-04-14T23:54:12.217 INFO:tasks.workunit.client.0.smithi060.stdout:removed './244/33' 2024-04-14T23:54:12.217 INFO:tasks.workunit.client.0.smithi060.stdout:removed './244/23' 2024-04-14T23:54:12.217 INFO:tasks.workunit.client.0.smithi060.stdout:removed './244/35' 2024-04-14T23:54:12.217 INFO:tasks.workunit.client.0.smithi060.stdout:removed './244/26' 2024-04-14T23:54:12.217 INFO:tasks.workunit.client.0.smithi060.stdout:removed './244/37' 2024-04-14T23:54:12.217 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './244' 2024-04-14T23:54:12.217 INFO:tasks.workunit.client.0.smithi060.stdout:removed './245/21' 2024-04-14T23:54:12.218 INFO:tasks.workunit.client.0.smithi060.stdout:removed './245/27' 2024-04-14T23:54:12.218 INFO:tasks.workunit.client.0.smithi060.stdout:removed './245/31' 2024-04-14T23:54:12.218 INFO:tasks.workunit.client.0.smithi060.stdout:removed './245/36' 2024-04-14T23:54:12.218 INFO:tasks.workunit.client.0.smithi060.stdout:removed './245/20' 2024-04-14T23:54:12.218 INFO:tasks.workunit.client.0.smithi060.stdout:removed './245/39' 2024-04-14T23:54:12.218 INFO:tasks.workunit.client.0.smithi060.stdout:removed './245/34' 2024-04-14T23:54:12.218 INFO:tasks.workunit.client.0.smithi060.stdout:removed './245/38' 2024-04-14T23:54:12.218 INFO:tasks.workunit.client.0.smithi060.stdout:removed './245/28' 2024-04-14T23:54:12.218 INFO:tasks.workunit.client.0.smithi060.stdout:removed './245/24' 2024-04-14T23:54:12.218 INFO:tasks.workunit.client.0.smithi060.stdout:removed './245/29' 2024-04-14T23:54:12.218 INFO:tasks.workunit.client.0.smithi060.stdout:removed './245/32' 2024-04-14T23:54:12.218 INFO:tasks.workunit.client.0.smithi060.stdout:removed './245/22' 2024-04-14T23:54:12.219 INFO:tasks.workunit.client.0.smithi060.stdout:removed './245/25' 2024-04-14T23:54:12.219 INFO:tasks.workunit.client.0.smithi060.stdout:removed './245/30' 2024-04-14T23:54:12.219 INFO:tasks.workunit.client.0.smithi060.stdout:removed './245/33' 2024-04-14T23:54:12.219 INFO:tasks.workunit.client.0.smithi060.stdout:removed './245/23' 2024-04-14T23:54:12.219 INFO:tasks.workunit.client.0.smithi060.stdout:removed './245/35' 2024-04-14T23:54:12.219 INFO:tasks.workunit.client.0.smithi060.stdout:removed './245/26' 2024-04-14T23:54:12.219 INFO:tasks.workunit.client.0.smithi060.stdout:removed './245/37' 2024-04-14T23:54:12.219 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './245' 2024-04-14T23:54:12.219 INFO:tasks.workunit.client.0.smithi060.stdout:removed './246/21' 2024-04-14T23:54:12.219 INFO:tasks.workunit.client.0.smithi060.stdout:removed './246/27' 2024-04-14T23:54:12.219 INFO:tasks.workunit.client.0.smithi060.stdout:removed './246/31' 2024-04-14T23:54:12.219 INFO:tasks.workunit.client.0.smithi060.stdout:removed './246/36' 2024-04-14T23:54:12.219 INFO:tasks.workunit.client.0.smithi060.stdout:removed './246/20' 2024-04-14T23:54:12.219 INFO:tasks.workunit.client.0.smithi060.stdout:removed './246/39' 2024-04-14T23:54:12.219 INFO:tasks.workunit.client.0.smithi060.stdout:removed './246/34' 2024-04-14T23:54:12.219 INFO:tasks.workunit.client.0.smithi060.stdout:removed './246/38' 2024-04-14T23:54:12.219 INFO:tasks.workunit.client.0.smithi060.stdout:removed './246/28' 2024-04-14T23:54:12.220 INFO:tasks.workunit.client.0.smithi060.stdout:removed './246/24' 2024-04-14T23:54:12.220 INFO:tasks.workunit.client.0.smithi060.stdout:removed './246/29' 2024-04-14T23:54:12.220 INFO:tasks.workunit.client.0.smithi060.stdout:removed './246/32' 2024-04-14T23:54:12.220 INFO:tasks.workunit.client.0.smithi060.stdout:removed './246/22' 2024-04-14T23:54:13.410 INFO:tasks.workunit.client.0.smithi060.stdout:removed './246/25' 2024-04-14T23:54:13.410 INFO:tasks.workunit.client.0.smithi060.stdout:removed './246/30' 2024-04-14T23:54:13.410 INFO:tasks.workunit.client.0.smithi060.stdout:removed './246/33' 2024-04-14T23:54:13.410 INFO:tasks.workunit.client.0.smithi060.stdout:removed './246/23' 2024-04-14T23:54:13.410 INFO:tasks.workunit.client.0.smithi060.stdout:removed './246/35' 2024-04-14T23:54:13.410 INFO:tasks.workunit.client.0.smithi060.stdout:removed './246/26' 2024-04-14T23:54:13.410 INFO:tasks.workunit.client.0.smithi060.stdout:removed './246/37' 2024-04-14T23:54:13.410 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './246' 2024-04-14T23:54:13.410 INFO:tasks.workunit.client.0.smithi060.stdout:removed './247/21' 2024-04-14T23:54:13.411 INFO:tasks.workunit.client.0.smithi060.stdout:removed './247/27' 2024-04-14T23:54:13.411 INFO:tasks.workunit.client.0.smithi060.stdout:removed './247/31' 2024-04-14T23:54:13.411 INFO:tasks.workunit.client.0.smithi060.stdout:removed './247/36' 2024-04-14T23:54:13.411 INFO:tasks.workunit.client.0.smithi060.stdout:removed './247/20' 2024-04-14T23:54:13.411 INFO:tasks.workunit.client.0.smithi060.stdout:removed './247/39' 2024-04-14T23:54:13.411 INFO:tasks.workunit.client.0.smithi060.stdout:removed './247/34' 2024-04-14T23:54:13.411 INFO:tasks.workunit.client.0.smithi060.stdout:removed './247/38' 2024-04-14T23:54:13.411 INFO:tasks.workunit.client.0.smithi060.stdout:removed './247/28' 2024-04-14T23:54:13.411 INFO:tasks.workunit.client.0.smithi060.stdout:removed './247/24' 2024-04-14T23:54:13.411 INFO:tasks.workunit.client.0.smithi060.stdout:removed './247/29' 2024-04-14T23:54:13.411 INFO:tasks.workunit.client.0.smithi060.stdout:removed './247/32' 2024-04-14T23:54:13.411 INFO:tasks.workunit.client.0.smithi060.stdout:removed './247/22' 2024-04-14T23:54:13.411 INFO:tasks.workunit.client.0.smithi060.stdout:removed './247/25' 2024-04-14T23:54:13.411 INFO:tasks.workunit.client.0.smithi060.stdout:removed './247/30' 2024-04-14T23:54:13.411 INFO:tasks.workunit.client.0.smithi060.stdout:removed './247/33' 2024-04-14T23:54:13.412 INFO:tasks.workunit.client.0.smithi060.stdout:removed './247/23' 2024-04-14T23:54:13.412 INFO:tasks.workunit.client.0.smithi060.stdout:removed './247/35' 2024-04-14T23:54:13.412 INFO:tasks.workunit.client.0.smithi060.stdout:removed './247/26' 2024-04-14T23:54:13.412 INFO:tasks.workunit.client.0.smithi060.stdout:removed './247/37' 2024-04-14T23:54:13.412 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './247' 2024-04-14T23:54:13.412 INFO:tasks.workunit.client.0.smithi060.stdout:removed './248/21' 2024-04-14T23:54:13.412 INFO:tasks.workunit.client.0.smithi060.stdout:removed './248/27' 2024-04-14T23:54:13.412 INFO:tasks.workunit.client.0.smithi060.stdout:removed './248/31' 2024-04-14T23:54:13.412 INFO:tasks.workunit.client.0.smithi060.stdout:removed './248/36' 2024-04-14T23:54:13.412 INFO:tasks.workunit.client.0.smithi060.stdout:removed './248/20' 2024-04-14T23:54:13.412 INFO:tasks.workunit.client.0.smithi060.stdout:removed './248/39' 2024-04-14T23:54:13.412 INFO:tasks.workunit.client.0.smithi060.stdout:removed './248/34' 2024-04-14T23:54:13.412 INFO:tasks.workunit.client.0.smithi060.stdout:removed './248/38' 2024-04-14T23:54:13.412 INFO:tasks.workunit.client.0.smithi060.stdout:removed './248/28' 2024-04-14T23:54:13.412 INFO:tasks.workunit.client.0.smithi060.stdout:removed './248/24' 2024-04-14T23:54:13.413 INFO:tasks.workunit.client.0.smithi060.stdout:removed './248/29' 2024-04-14T23:54:13.413 INFO:tasks.workunit.client.0.smithi060.stdout:removed './248/32' 2024-04-14T23:54:13.413 INFO:tasks.workunit.client.0.smithi060.stdout:removed './248/22' 2024-04-14T23:54:13.413 INFO:tasks.workunit.client.0.smithi060.stdout:removed './248/25' 2024-04-14T23:54:13.413 INFO:tasks.workunit.client.0.smithi060.stdout:removed './248/30' 2024-04-14T23:54:13.413 INFO:tasks.workunit.client.0.smithi060.stdout:removed './248/33' 2024-04-14T23:54:13.413 INFO:tasks.workunit.client.0.smithi060.stdout:removed './248/23' 2024-04-14T23:54:13.413 INFO:tasks.workunit.client.0.smithi060.stdout:removed './248/35' 2024-04-14T23:54:13.413 INFO:tasks.workunit.client.0.smithi060.stdout:removed './248/26' 2024-04-14T23:54:13.413 INFO:tasks.workunit.client.0.smithi060.stdout:removed './248/37' 2024-04-14T23:54:13.413 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './248' 2024-04-14T23:54:13.413 INFO:tasks.workunit.client.0.smithi060.stdout:removed './249/21' 2024-04-14T23:54:13.413 INFO:tasks.workunit.client.0.smithi060.stdout:removed './249/27' 2024-04-14T23:54:13.413 INFO:tasks.workunit.client.0.smithi060.stdout:removed './249/31' 2024-04-14T23:54:13.413 INFO:tasks.workunit.client.0.smithi060.stdout:removed './249/36' 2024-04-14T23:54:13.414 INFO:tasks.workunit.client.0.smithi060.stdout:removed './249/20' 2024-04-14T23:54:13.414 INFO:tasks.workunit.client.0.smithi060.stdout:removed './249/39' 2024-04-14T23:54:13.414 INFO:tasks.workunit.client.0.smithi060.stdout:removed './249/34' 2024-04-14T23:54:13.414 INFO:tasks.workunit.client.0.smithi060.stdout:removed './249/38' 2024-04-14T23:54:13.414 INFO:tasks.workunit.client.0.smithi060.stdout:removed './249/28' 2024-04-14T23:54:13.414 INFO:tasks.workunit.client.0.smithi060.stdout:removed './249/24' 2024-04-14T23:54:13.414 INFO:tasks.workunit.client.0.smithi060.stdout:removed './249/29' 2024-04-14T23:54:13.414 INFO:tasks.workunit.client.0.smithi060.stdout:removed './249/32' 2024-04-14T23:54:13.414 INFO:tasks.workunit.client.0.smithi060.stdout:removed './249/22' 2024-04-14T23:54:13.414 INFO:tasks.workunit.client.0.smithi060.stdout:removed './249/25' 2024-04-14T23:54:13.414 INFO:tasks.workunit.client.0.smithi060.stdout:removed './249/30' 2024-04-14T23:54:13.414 INFO:tasks.workunit.client.0.smithi060.stdout:removed './249/33' 2024-04-14T23:54:13.414 INFO:tasks.workunit.client.0.smithi060.stdout:removed './249/23' 2024-04-14T23:54:13.414 INFO:tasks.workunit.client.0.smithi060.stdout:removed './249/35' 2024-04-14T23:54:13.415 INFO:tasks.workunit.client.0.smithi060.stdout:removed './249/26' 2024-04-14T23:54:13.415 INFO:tasks.workunit.client.0.smithi060.stdout:removed './249/37' 2024-04-14T23:54:13.415 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './249' 2024-04-14T23:54:13.415 INFO:tasks.workunit.client.0.smithi060.stdout:removed './250/21' 2024-04-14T23:54:13.415 INFO:tasks.workunit.client.0.smithi060.stdout:removed './250/27' 2024-04-14T23:54:13.415 INFO:tasks.workunit.client.0.smithi060.stdout:removed './250/31' 2024-04-14T23:54:13.415 INFO:tasks.workunit.client.0.smithi060.stdout:removed './250/36' 2024-04-14T23:54:13.415 INFO:tasks.workunit.client.0.smithi060.stdout:removed './250/20' 2024-04-14T23:54:13.415 INFO:tasks.workunit.client.0.smithi060.stdout:removed './250/39' 2024-04-14T23:54:13.415 INFO:tasks.workunit.client.0.smithi060.stdout:removed './250/34' 2024-04-14T23:54:13.415 INFO:tasks.workunit.client.0.smithi060.stdout:removed './250/38' 2024-04-14T23:54:13.415 INFO:tasks.workunit.client.0.smithi060.stdout:removed './250/28' 2024-04-14T23:54:13.415 INFO:tasks.workunit.client.0.smithi060.stdout:removed './250/24' 2024-04-14T23:54:13.415 INFO:tasks.workunit.client.0.smithi060.stdout:removed './250/29' 2024-04-14T23:54:13.415 INFO:tasks.workunit.client.0.smithi060.stdout:removed './250/32' 2024-04-14T23:54:13.416 INFO:tasks.workunit.client.0.smithi060.stdout:removed './250/22' 2024-04-14T23:54:13.416 INFO:tasks.workunit.client.0.smithi060.stdout:removed './250/25' 2024-04-14T23:54:13.416 INFO:tasks.workunit.client.0.smithi060.stdout:removed './250/30' 2024-04-14T23:54:13.416 INFO:tasks.workunit.client.0.smithi060.stdout:removed './250/33' 2024-04-14T23:54:13.416 INFO:tasks.workunit.client.0.smithi060.stdout:removed './250/23' 2024-04-14T23:54:13.416 INFO:tasks.workunit.client.0.smithi060.stdout:removed './250/35' 2024-04-14T23:54:13.416 INFO:tasks.workunit.client.0.smithi060.stdout:removed './250/26' 2024-04-14T23:54:13.416 INFO:tasks.workunit.client.0.smithi060.stdout:removed './250/37' 2024-04-14T23:54:13.416 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './250' 2024-04-14T23:54:13.416 INFO:tasks.workunit.client.0.smithi060.stdout:removed './251/21' 2024-04-14T23:54:13.416 INFO:tasks.workunit.client.0.smithi060.stdout:removed './251/27' 2024-04-14T23:54:13.416 INFO:tasks.workunit.client.0.smithi060.stdout:removed './251/31' 2024-04-14T23:54:13.416 INFO:tasks.workunit.client.0.smithi060.stdout:removed './251/36' 2024-04-14T23:54:13.416 INFO:tasks.workunit.client.0.smithi060.stdout:removed './251/20' 2024-04-14T23:54:13.416 INFO:tasks.workunit.client.0.smithi060.stdout:removed './251/39' 2024-04-14T23:54:13.417 INFO:tasks.workunit.client.0.smithi060.stdout:removed './251/34' 2024-04-14T23:54:13.417 INFO:tasks.workunit.client.0.smithi060.stdout:removed './251/38' 2024-04-14T23:54:13.417 INFO:tasks.workunit.client.0.smithi060.stdout:removed './251/28' 2024-04-14T23:54:13.417 INFO:tasks.workunit.client.0.smithi060.stdout:removed './251/24' 2024-04-14T23:54:13.417 INFO:tasks.workunit.client.0.smithi060.stdout:removed './251/29' 2024-04-14T23:54:13.417 INFO:tasks.workunit.client.0.smithi060.stdout:removed './251/32' 2024-04-14T23:54:13.417 INFO:tasks.workunit.client.0.smithi060.stdout:removed './251/22' 2024-04-14T23:54:13.417 INFO:tasks.workunit.client.0.smithi060.stdout:removed './251/25' 2024-04-14T23:54:13.417 INFO:tasks.workunit.client.0.smithi060.stdout:removed './251/30' 2024-04-14T23:54:13.417 INFO:tasks.workunit.client.0.smithi060.stdout:removed './251/33' 2024-04-14T23:54:13.417 INFO:tasks.workunit.client.0.smithi060.stdout:removed './251/23' 2024-04-14T23:54:13.417 INFO:tasks.workunit.client.0.smithi060.stdout:removed './251/35' 2024-04-14T23:54:13.417 INFO:tasks.workunit.client.0.smithi060.stdout:removed './251/26' 2024-04-14T23:54:13.417 INFO:tasks.workunit.client.0.smithi060.stdout:removed './251/37' 2024-04-14T23:54:13.417 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './251' 2024-04-14T23:54:13.418 INFO:tasks.workunit.client.0.smithi060.stdout:removed './252/21' 2024-04-14T23:54:13.418 INFO:tasks.workunit.client.0.smithi060.stdout:removed './252/27' 2024-04-14T23:54:13.418 INFO:tasks.workunit.client.0.smithi060.stdout:removed './252/31' 2024-04-14T23:54:13.418 INFO:tasks.workunit.client.0.smithi060.stdout:removed './252/36' 2024-04-14T23:54:13.418 INFO:tasks.workunit.client.0.smithi060.stdout:removed './252/20' 2024-04-14T23:54:13.418 INFO:tasks.workunit.client.0.smithi060.stdout:removed './252/39' 2024-04-14T23:54:13.418 INFO:tasks.workunit.client.0.smithi060.stdout:removed './252/34' 2024-04-14T23:54:13.418 INFO:tasks.workunit.client.0.smithi060.stdout:removed './252/38' 2024-04-14T23:54:13.418 INFO:tasks.workunit.client.0.smithi060.stdout:removed './252/28' 2024-04-14T23:54:13.418 INFO:tasks.workunit.client.0.smithi060.stdout:removed './252/24' 2024-04-14T23:54:13.418 INFO:tasks.workunit.client.0.smithi060.stdout:removed './252/29' 2024-04-14T23:54:13.418 INFO:tasks.workunit.client.0.smithi060.stdout:removed './252/32' 2024-04-14T23:54:13.418 INFO:tasks.workunit.client.0.smithi060.stdout:removed './252/22' 2024-04-14T23:54:13.418 INFO:tasks.workunit.client.0.smithi060.stdout:removed './252/25' 2024-04-14T23:54:13.419 INFO:tasks.workunit.client.0.smithi060.stdout:removed './252/30' 2024-04-14T23:54:13.419 INFO:tasks.workunit.client.0.smithi060.stdout:removed './252/33' 2024-04-14T23:54:13.419 INFO:tasks.workunit.client.0.smithi060.stdout:removed './252/23' 2024-04-14T23:54:13.419 INFO:tasks.workunit.client.0.smithi060.stdout:removed './252/35' 2024-04-14T23:54:13.419 INFO:tasks.workunit.client.0.smithi060.stdout:removed './252/26' 2024-04-14T23:54:13.419 INFO:tasks.workunit.client.0.smithi060.stdout:removed './252/37' 2024-04-14T23:54:13.419 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './252' 2024-04-14T23:54:13.419 INFO:tasks.workunit.client.0.smithi060.stdout:removed './253/21' 2024-04-14T23:54:13.419 INFO:tasks.workunit.client.0.smithi060.stdout:removed './253/27' 2024-04-14T23:54:13.419 INFO:tasks.workunit.client.0.smithi060.stdout:removed './253/31' 2024-04-14T23:54:13.419 INFO:tasks.workunit.client.0.smithi060.stdout:removed './253/36' 2024-04-14T23:54:13.419 INFO:tasks.workunit.client.0.smithi060.stdout:removed './253/20' 2024-04-14T23:54:13.419 INFO:tasks.workunit.client.0.smithi060.stdout:removed './253/39' 2024-04-14T23:54:13.419 INFO:tasks.workunit.client.0.smithi060.stdout:removed './253/34' 2024-04-14T23:54:13.419 INFO:tasks.workunit.client.0.smithi060.stdout:removed './253/38' 2024-04-14T23:54:13.419 INFO:tasks.workunit.client.0.smithi060.stdout:removed './253/28' 2024-04-14T23:54:13.420 INFO:tasks.workunit.client.0.smithi060.stdout:removed './253/24' 2024-04-14T23:54:13.420 INFO:tasks.workunit.client.0.smithi060.stdout:removed './253/29' 2024-04-14T23:54:13.420 INFO:tasks.workunit.client.0.smithi060.stdout:removed './253/32' 2024-04-14T23:54:13.420 INFO:tasks.workunit.client.0.smithi060.stdout:removed './253/22' 2024-04-14T23:54:13.420 INFO:tasks.workunit.client.0.smithi060.stdout:removed './253/25' 2024-04-14T23:54:13.420 INFO:tasks.workunit.client.0.smithi060.stdout:removed './253/30' 2024-04-14T23:54:13.420 INFO:tasks.workunit.client.0.smithi060.stdout:removed './253/33' 2024-04-14T23:54:13.420 INFO:tasks.workunit.client.0.smithi060.stdout:removed './253/23' 2024-04-14T23:54:13.420 INFO:tasks.workunit.client.0.smithi060.stdout:removed './253/35' 2024-04-14T23:54:13.420 INFO:tasks.workunit.client.0.smithi060.stdout:removed './253/26' 2024-04-14T23:54:13.420 INFO:tasks.workunit.client.0.smithi060.stdout:removed './253/37' 2024-04-14T23:54:13.420 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './253' 2024-04-14T23:54:13.420 INFO:tasks.workunit.client.0.smithi060.stdout:removed './254/21' 2024-04-14T23:54:13.420 INFO:tasks.workunit.client.0.smithi060.stdout:removed './254/27' 2024-04-14T23:54:13.420 INFO:tasks.workunit.client.0.smithi060.stdout:removed './254/31' 2024-04-14T23:54:13.421 INFO:tasks.workunit.client.0.smithi060.stdout:removed './254/36' 2024-04-14T23:54:13.421 INFO:tasks.workunit.client.0.smithi060.stdout:removed './254/20' 2024-04-14T23:54:13.421 INFO:tasks.workunit.client.0.smithi060.stdout:removed './254/39' 2024-04-14T23:54:13.421 INFO:tasks.workunit.client.0.smithi060.stdout:removed './254/34' 2024-04-14T23:54:13.421 INFO:tasks.workunit.client.0.smithi060.stdout:removed './254/38' 2024-04-14T23:54:13.421 INFO:tasks.workunit.client.0.smithi060.stdout:removed './254/28' 2024-04-14T23:54:13.421 INFO:tasks.workunit.client.0.smithi060.stdout:removed './254/24' 2024-04-14T23:54:13.421 INFO:tasks.workunit.client.0.smithi060.stdout:removed './254/29' 2024-04-14T23:54:13.421 INFO:tasks.workunit.client.0.smithi060.stdout:removed './254/32' 2024-04-14T23:54:13.421 INFO:tasks.workunit.client.0.smithi060.stdout:removed './254/22' 2024-04-14T23:54:13.421 INFO:tasks.workunit.client.0.smithi060.stdout:removed './254/25' 2024-04-14T23:54:13.421 INFO:tasks.workunit.client.0.smithi060.stdout:removed './254/30' 2024-04-14T23:54:13.421 INFO:tasks.workunit.client.0.smithi060.stdout:removed './254/33' 2024-04-14T23:54:13.421 INFO:tasks.workunit.client.0.smithi060.stdout:removed './254/23' 2024-04-14T23:54:13.421 INFO:tasks.workunit.client.0.smithi060.stdout:removed './254/35' 2024-04-14T23:54:13.421 INFO:tasks.workunit.client.0.smithi060.stdout:removed './254/26' 2024-04-14T23:54:13.421 INFO:tasks.workunit.client.0.smithi060.stdout:removed './254/37' 2024-04-14T23:54:13.421 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './254' 2024-04-14T23:54:13.422 INFO:tasks.workunit.client.0.smithi060.stdout:removed './255/21' 2024-04-14T23:54:13.422 INFO:tasks.workunit.client.0.smithi060.stdout:removed './255/27' 2024-04-14T23:54:13.422 INFO:tasks.workunit.client.0.smithi060.stdout:removed './255/31' 2024-04-14T23:54:13.422 INFO:tasks.workunit.client.0.smithi060.stdout:removed './255/36' 2024-04-14T23:54:13.422 INFO:tasks.workunit.client.0.smithi060.stdout:removed './255/20' 2024-04-14T23:54:13.422 INFO:tasks.workunit.client.0.smithi060.stdout:removed './255/39' 2024-04-14T23:54:13.422 INFO:tasks.workunit.client.0.smithi060.stdout:removed './255/34' 2024-04-14T23:54:13.422 INFO:tasks.workunit.client.0.smithi060.stdout:removed './255/38' 2024-04-14T23:54:13.422 INFO:tasks.workunit.client.0.smithi060.stdout:removed './255/28' 2024-04-14T23:54:13.422 INFO:tasks.workunit.client.0.smithi060.stdout:removed './255/24' 2024-04-14T23:54:13.422 INFO:tasks.workunit.client.0.smithi060.stdout:removed './255/29' 2024-04-14T23:54:13.422 INFO:tasks.workunit.client.0.smithi060.stdout:removed './255/32' 2024-04-14T23:54:13.422 INFO:tasks.workunit.client.0.smithi060.stdout:removed './255/22' 2024-04-14T23:54:13.422 INFO:tasks.workunit.client.0.smithi060.stdout:removed './255/25' 2024-04-14T23:54:13.422 INFO:tasks.workunit.client.0.smithi060.stdout:removed './255/30' 2024-04-14T23:54:13.423 INFO:tasks.workunit.client.0.smithi060.stdout:removed './255/33' 2024-04-14T23:54:13.423 INFO:tasks.workunit.client.0.smithi060.stdout:removed './255/23' 2024-04-14T23:54:13.423 INFO:tasks.workunit.client.0.smithi060.stdout:removed './255/35' 2024-04-14T23:54:13.423 INFO:tasks.workunit.client.0.smithi060.stdout:removed './255/26' 2024-04-14T23:54:13.423 INFO:tasks.workunit.client.0.smithi060.stdout:removed './255/37' 2024-04-14T23:54:13.423 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './255' 2024-04-14T23:54:13.423 INFO:tasks.workunit.client.0.smithi060.stdout:removed './256/21' 2024-04-14T23:54:13.423 INFO:tasks.workunit.client.0.smithi060.stdout:removed './256/27' 2024-04-14T23:54:13.423 INFO:tasks.workunit.client.0.smithi060.stdout:removed './256/31' 2024-04-14T23:54:13.423 INFO:tasks.workunit.client.0.smithi060.stdout:removed './256/36' 2024-04-14T23:54:13.423 INFO:tasks.workunit.client.0.smithi060.stdout:removed './256/20' 2024-04-14T23:54:13.423 INFO:tasks.workunit.client.0.smithi060.stdout:removed './256/39' 2024-04-14T23:54:13.423 INFO:tasks.workunit.client.0.smithi060.stdout:removed './256/34' 2024-04-14T23:54:13.423 INFO:tasks.workunit.client.0.smithi060.stdout:removed './256/38' 2024-04-14T23:54:13.423 INFO:tasks.workunit.client.0.smithi060.stdout:removed './256/28' 2024-04-14T23:54:13.423 INFO:tasks.workunit.client.0.smithi060.stdout:removed './256/24' 2024-04-14T23:54:13.423 INFO:tasks.workunit.client.0.smithi060.stdout:removed './256/29' 2024-04-14T23:54:13.424 INFO:tasks.workunit.client.0.smithi060.stdout:removed './256/32' 2024-04-14T23:54:13.424 INFO:tasks.workunit.client.0.smithi060.stdout:removed './256/22' 2024-04-14T23:54:13.424 INFO:tasks.workunit.client.0.smithi060.stdout:removed './256/25' 2024-04-14T23:54:13.424 INFO:tasks.workunit.client.0.smithi060.stdout:removed './256/30' 2024-04-14T23:54:14.474 INFO:tasks.workunit.client.0.smithi060.stdout:removed './256/33' 2024-04-14T23:54:14.474 INFO:tasks.workunit.client.0.smithi060.stdout:removed './256/23' 2024-04-14T23:54:14.474 INFO:tasks.workunit.client.0.smithi060.stdout:removed './256/35' 2024-04-14T23:54:14.474 INFO:tasks.workunit.client.0.smithi060.stdout:removed './256/26' 2024-04-14T23:54:14.475 INFO:tasks.workunit.client.0.smithi060.stdout:removed './256/37' 2024-04-14T23:54:14.475 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './256' 2024-04-14T23:54:14.475 INFO:tasks.workunit.client.0.smithi060.stdout:removed './257/21' 2024-04-14T23:54:14.475 INFO:tasks.workunit.client.0.smithi060.stdout:removed './257/27' 2024-04-14T23:54:14.475 INFO:tasks.workunit.client.0.smithi060.stdout:removed './257/31' 2024-04-14T23:54:14.475 INFO:tasks.workunit.client.0.smithi060.stdout:removed './257/36' 2024-04-14T23:54:14.475 INFO:tasks.workunit.client.0.smithi060.stdout:removed './257/20' 2024-04-14T23:54:14.475 INFO:tasks.workunit.client.0.smithi060.stdout:removed './257/39' 2024-04-14T23:54:14.475 INFO:tasks.workunit.client.0.smithi060.stdout:removed './257/34' 2024-04-14T23:54:14.475 INFO:tasks.workunit.client.0.smithi060.stdout:removed './257/38' 2024-04-14T23:54:14.475 INFO:tasks.workunit.client.0.smithi060.stdout:removed './257/28' 2024-04-14T23:54:14.475 INFO:tasks.workunit.client.0.smithi060.stdout:removed './257/24' 2024-04-14T23:54:14.475 INFO:tasks.workunit.client.0.smithi060.stdout:removed './257/29' 2024-04-14T23:54:14.476 INFO:tasks.workunit.client.0.smithi060.stdout:removed './257/32' 2024-04-14T23:54:14.476 INFO:tasks.workunit.client.0.smithi060.stdout:removed './257/22' 2024-04-14T23:54:14.476 INFO:tasks.workunit.client.0.smithi060.stdout:removed './257/25' 2024-04-14T23:54:14.476 INFO:tasks.workunit.client.0.smithi060.stdout:removed './257/30' 2024-04-14T23:54:14.476 INFO:tasks.workunit.client.0.smithi060.stdout:removed './257/33' 2024-04-14T23:54:14.476 INFO:tasks.workunit.client.0.smithi060.stdout:removed './257/23' 2024-04-14T23:54:14.476 INFO:tasks.workunit.client.0.smithi060.stdout:removed './257/35' 2024-04-14T23:54:14.476 INFO:tasks.workunit.client.0.smithi060.stdout:removed './257/26' 2024-04-14T23:54:14.476 INFO:tasks.workunit.client.0.smithi060.stdout:removed './257/37' 2024-04-14T23:54:14.476 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './257' 2024-04-14T23:54:14.476 INFO:tasks.workunit.client.0.smithi060.stdout:removed './258/21' 2024-04-14T23:54:14.476 INFO:tasks.workunit.client.0.smithi060.stdout:removed './258/27' 2024-04-14T23:54:14.476 INFO:tasks.workunit.client.0.smithi060.stdout:removed './258/31' 2024-04-14T23:54:14.476 INFO:tasks.workunit.client.0.smithi060.stdout:removed './258/36' 2024-04-14T23:54:14.476 INFO:tasks.workunit.client.0.smithi060.stdout:removed './258/20' 2024-04-14T23:54:14.477 INFO:tasks.workunit.client.0.smithi060.stdout:removed './258/39' 2024-04-14T23:54:14.477 INFO:tasks.workunit.client.0.smithi060.stdout:removed './258/34' 2024-04-14T23:54:14.477 INFO:tasks.workunit.client.0.smithi060.stdout:removed './258/38' 2024-04-14T23:54:14.477 INFO:tasks.workunit.client.0.smithi060.stdout:removed './258/28' 2024-04-14T23:54:14.477 INFO:tasks.workunit.client.0.smithi060.stdout:removed './258/24' 2024-04-14T23:54:14.477 INFO:tasks.workunit.client.0.smithi060.stdout:removed './258/29' 2024-04-14T23:54:14.477 INFO:tasks.workunit.client.0.smithi060.stdout:removed './258/32' 2024-04-14T23:54:14.477 INFO:tasks.workunit.client.0.smithi060.stdout:removed './258/22' 2024-04-14T23:54:14.477 INFO:tasks.workunit.client.0.smithi060.stdout:removed './258/25' 2024-04-14T23:54:14.477 INFO:tasks.workunit.client.0.smithi060.stdout:removed './258/30' 2024-04-14T23:54:14.477 INFO:tasks.workunit.client.0.smithi060.stdout:removed './258/33' 2024-04-14T23:54:14.477 INFO:tasks.workunit.client.0.smithi060.stdout:removed './258/23' 2024-04-14T23:54:14.477 INFO:tasks.workunit.client.0.smithi060.stdout:removed './258/35' 2024-04-14T23:54:14.477 INFO:tasks.workunit.client.0.smithi060.stdout:removed './258/26' 2024-04-14T23:54:14.477 INFO:tasks.workunit.client.0.smithi060.stdout:removed './258/37' 2024-04-14T23:54:14.478 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './258' 2024-04-14T23:54:14.478 INFO:tasks.workunit.client.0.smithi060.stdout:removed './259/21' 2024-04-14T23:54:14.478 INFO:tasks.workunit.client.0.smithi060.stdout:removed './259/27' 2024-04-14T23:54:14.478 INFO:tasks.workunit.client.0.smithi060.stdout:removed './259/31' 2024-04-14T23:54:14.478 INFO:tasks.workunit.client.0.smithi060.stdout:removed './259/36' 2024-04-14T23:54:14.478 INFO:tasks.workunit.client.0.smithi060.stdout:removed './259/20' 2024-04-14T23:54:14.478 INFO:tasks.workunit.client.0.smithi060.stdout:removed './259/39' 2024-04-14T23:54:14.478 INFO:tasks.workunit.client.0.smithi060.stdout:removed './259/34' 2024-04-14T23:54:14.478 INFO:tasks.workunit.client.0.smithi060.stdout:removed './259/38' 2024-04-14T23:54:14.478 INFO:tasks.workunit.client.0.smithi060.stdout:removed './259/28' 2024-04-14T23:54:14.478 INFO:tasks.workunit.client.0.smithi060.stdout:removed './259/24' 2024-04-14T23:54:14.478 INFO:tasks.workunit.client.0.smithi060.stdout:removed './259/29' 2024-04-14T23:54:14.478 INFO:tasks.workunit.client.0.smithi060.stdout:removed './259/32' 2024-04-14T23:54:14.478 INFO:tasks.workunit.client.0.smithi060.stdout:removed './259/22' 2024-04-14T23:54:14.478 INFO:tasks.workunit.client.0.smithi060.stdout:removed './259/25' 2024-04-14T23:54:14.479 INFO:tasks.workunit.client.0.smithi060.stdout:removed './259/30' 2024-04-14T23:54:14.479 INFO:tasks.workunit.client.0.smithi060.stdout:removed './259/33' 2024-04-14T23:54:14.479 INFO:tasks.workunit.client.0.smithi060.stdout:removed './259/23' 2024-04-14T23:54:14.479 INFO:tasks.workunit.client.0.smithi060.stdout:removed './259/35' 2024-04-14T23:54:14.479 INFO:tasks.workunit.client.0.smithi060.stdout:removed './259/26' 2024-04-14T23:54:14.479 INFO:tasks.workunit.client.0.smithi060.stdout:removed './259/37' 2024-04-14T23:54:14.479 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './259' 2024-04-14T23:54:14.479 INFO:tasks.workunit.client.0.smithi060.stdout:removed './260/21' 2024-04-14T23:54:14.479 INFO:tasks.workunit.client.0.smithi060.stdout:removed './260/27' 2024-04-14T23:54:14.479 INFO:tasks.workunit.client.0.smithi060.stdout:removed './260/31' 2024-04-14T23:54:14.479 INFO:tasks.workunit.client.0.smithi060.stdout:removed './260/36' 2024-04-14T23:54:14.479 INFO:tasks.workunit.client.0.smithi060.stdout:removed './260/20' 2024-04-14T23:54:14.479 INFO:tasks.workunit.client.0.smithi060.stdout:removed './260/39' 2024-04-14T23:54:14.479 INFO:tasks.workunit.client.0.smithi060.stdout:removed './260/34' 2024-04-14T23:54:14.479 INFO:tasks.workunit.client.0.smithi060.stdout:removed './260/38' 2024-04-14T23:54:14.480 INFO:tasks.workunit.client.0.smithi060.stdout:removed './260/28' 2024-04-14T23:54:14.480 INFO:tasks.workunit.client.0.smithi060.stdout:removed './260/24' 2024-04-14T23:54:14.480 INFO:tasks.workunit.client.0.smithi060.stdout:removed './260/29' 2024-04-14T23:54:14.480 INFO:tasks.workunit.client.0.smithi060.stdout:removed './260/32' 2024-04-14T23:54:14.480 INFO:tasks.workunit.client.0.smithi060.stdout:removed './260/22' 2024-04-14T23:54:14.480 INFO:tasks.workunit.client.0.smithi060.stdout:removed './260/25' 2024-04-14T23:54:14.480 INFO:tasks.workunit.client.0.smithi060.stdout:removed './260/30' 2024-04-14T23:54:14.480 INFO:tasks.workunit.client.0.smithi060.stdout:removed './260/33' 2024-04-14T23:54:14.480 INFO:tasks.workunit.client.0.smithi060.stdout:removed './260/23' 2024-04-14T23:54:14.480 INFO:tasks.workunit.client.0.smithi060.stdout:removed './260/35' 2024-04-14T23:54:14.480 INFO:tasks.workunit.client.0.smithi060.stdout:removed './260/26' 2024-04-14T23:54:14.480 INFO:tasks.workunit.client.0.smithi060.stdout:removed './260/37' 2024-04-14T23:54:14.480 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './260' 2024-04-14T23:54:14.480 INFO:tasks.workunit.client.0.smithi060.stdout:removed './261/21' 2024-04-14T23:54:14.480 INFO:tasks.workunit.client.0.smithi060.stdout:removed './261/27' 2024-04-14T23:54:14.481 INFO:tasks.workunit.client.0.smithi060.stdout:removed './261/31' 2024-04-14T23:54:14.481 INFO:tasks.workunit.client.0.smithi060.stdout:removed './261/36' 2024-04-14T23:54:14.481 INFO:tasks.workunit.client.0.smithi060.stdout:removed './261/20' 2024-04-14T23:54:14.481 INFO:tasks.workunit.client.0.smithi060.stdout:removed './261/39' 2024-04-14T23:54:14.481 INFO:tasks.workunit.client.0.smithi060.stdout:removed './261/34' 2024-04-14T23:54:14.481 INFO:tasks.workunit.client.0.smithi060.stdout:removed './261/38' 2024-04-14T23:54:14.481 INFO:tasks.workunit.client.0.smithi060.stdout:removed './261/28' 2024-04-14T23:54:14.481 INFO:tasks.workunit.client.0.smithi060.stdout:removed './261/24' 2024-04-14T23:54:14.481 INFO:tasks.workunit.client.0.smithi060.stdout:removed './261/29' 2024-04-14T23:54:14.481 INFO:tasks.workunit.client.0.smithi060.stdout:removed './261/32' 2024-04-14T23:54:14.481 INFO:tasks.workunit.client.0.smithi060.stdout:removed './261/22' 2024-04-14T23:54:14.481 INFO:tasks.workunit.client.0.smithi060.stdout:removed './261/25' 2024-04-14T23:54:14.481 INFO:tasks.workunit.client.0.smithi060.stdout:removed './261/30' 2024-04-14T23:54:14.481 INFO:tasks.workunit.client.0.smithi060.stdout:removed './261/33' 2024-04-14T23:54:14.481 INFO:tasks.workunit.client.0.smithi060.stdout:removed './261/23' 2024-04-14T23:54:14.482 INFO:tasks.workunit.client.0.smithi060.stdout:removed './261/35' 2024-04-14T23:54:14.482 INFO:tasks.workunit.client.0.smithi060.stdout:removed './261/26' 2024-04-14T23:54:14.482 INFO:tasks.workunit.client.0.smithi060.stdout:removed './261/37' 2024-04-14T23:54:14.482 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './261' 2024-04-14T23:54:14.482 INFO:tasks.workunit.client.0.smithi060.stdout:removed './262/21' 2024-04-14T23:54:14.482 INFO:tasks.workunit.client.0.smithi060.stdout:removed './262/27' 2024-04-14T23:54:14.482 INFO:tasks.workunit.client.0.smithi060.stdout:removed './262/31' 2024-04-14T23:54:14.482 INFO:tasks.workunit.client.0.smithi060.stdout:removed './262/36' 2024-04-14T23:54:14.482 INFO:tasks.workunit.client.0.smithi060.stdout:removed './262/20' 2024-04-14T23:54:14.482 INFO:tasks.workunit.client.0.smithi060.stdout:removed './262/39' 2024-04-14T23:54:14.482 INFO:tasks.workunit.client.0.smithi060.stdout:removed './262/34' 2024-04-14T23:54:14.482 INFO:tasks.workunit.client.0.smithi060.stdout:removed './262/38' 2024-04-14T23:54:14.482 INFO:tasks.workunit.client.0.smithi060.stdout:removed './262/28' 2024-04-14T23:54:14.482 INFO:tasks.workunit.client.0.smithi060.stdout:removed './262/24' 2024-04-14T23:54:14.483 INFO:tasks.workunit.client.0.smithi060.stdout:removed './262/29' 2024-04-14T23:54:14.483 INFO:tasks.workunit.client.0.smithi060.stdout:removed './262/32' 2024-04-14T23:54:14.483 INFO:tasks.workunit.client.0.smithi060.stdout:removed './262/22' 2024-04-14T23:54:14.483 INFO:tasks.workunit.client.0.smithi060.stdout:removed './262/25' 2024-04-14T23:54:14.483 INFO:tasks.workunit.client.0.smithi060.stdout:removed './262/30' 2024-04-14T23:54:14.483 INFO:tasks.workunit.client.0.smithi060.stdout:removed './262/33' 2024-04-14T23:54:14.483 INFO:tasks.workunit.client.0.smithi060.stdout:removed './262/23' 2024-04-14T23:54:14.483 INFO:tasks.workunit.client.0.smithi060.stdout:removed './262/35' 2024-04-14T23:54:14.483 INFO:tasks.workunit.client.0.smithi060.stdout:removed './262/26' 2024-04-14T23:54:14.483 INFO:tasks.workunit.client.0.smithi060.stdout:removed './262/37' 2024-04-14T23:54:14.483 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './262' 2024-04-14T23:54:14.483 INFO:tasks.workunit.client.0.smithi060.stdout:removed './263/21' 2024-04-14T23:54:14.483 INFO:tasks.workunit.client.0.smithi060.stdout:removed './263/27' 2024-04-14T23:54:14.483 INFO:tasks.workunit.client.0.smithi060.stdout:removed './263/31' 2024-04-14T23:54:14.483 INFO:tasks.workunit.client.0.smithi060.stdout:removed './263/36' 2024-04-14T23:54:14.484 INFO:tasks.workunit.client.0.smithi060.stdout:removed './263/20' 2024-04-14T23:54:14.484 INFO:tasks.workunit.client.0.smithi060.stdout:removed './263/39' 2024-04-14T23:54:14.484 INFO:tasks.workunit.client.0.smithi060.stdout:removed './263/34' 2024-04-14T23:54:14.484 INFO:tasks.workunit.client.0.smithi060.stdout:removed './263/38' 2024-04-14T23:54:14.484 INFO:tasks.workunit.client.0.smithi060.stdout:removed './263/28' 2024-04-14T23:54:14.484 INFO:tasks.workunit.client.0.smithi060.stdout:removed './263/24' 2024-04-14T23:54:14.484 INFO:tasks.workunit.client.0.smithi060.stdout:removed './263/29' 2024-04-14T23:54:14.484 INFO:tasks.workunit.client.0.smithi060.stdout:removed './263/32' 2024-04-14T23:54:14.484 INFO:tasks.workunit.client.0.smithi060.stdout:removed './263/22' 2024-04-14T23:54:14.484 INFO:tasks.workunit.client.0.smithi060.stdout:removed './263/25' 2024-04-14T23:54:14.484 INFO:tasks.workunit.client.0.smithi060.stdout:removed './263/30' 2024-04-14T23:54:14.484 INFO:tasks.workunit.client.0.smithi060.stdout:removed './263/33' 2024-04-14T23:54:14.484 INFO:tasks.workunit.client.0.smithi060.stdout:removed './263/23' 2024-04-14T23:54:14.484 INFO:tasks.workunit.client.0.smithi060.stdout:removed './263/35' 2024-04-14T23:54:14.484 INFO:tasks.workunit.client.0.smithi060.stdout:removed './263/26' 2024-04-14T23:54:14.484 INFO:tasks.workunit.client.0.smithi060.stdout:removed './263/37' 2024-04-14T23:54:14.485 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './263' 2024-04-14T23:54:14.485 INFO:tasks.workunit.client.0.smithi060.stdout:removed './264/21' 2024-04-14T23:54:14.485 INFO:tasks.workunit.client.0.smithi060.stdout:removed './264/27' 2024-04-14T23:54:14.485 INFO:tasks.workunit.client.0.smithi060.stdout:removed './264/31' 2024-04-14T23:54:14.485 INFO:tasks.workunit.client.0.smithi060.stdout:removed './264/36' 2024-04-14T23:54:14.485 INFO:tasks.workunit.client.0.smithi060.stdout:removed './264/20' 2024-04-14T23:54:14.485 INFO:tasks.workunit.client.0.smithi060.stdout:removed './264/39' 2024-04-14T23:54:14.485 INFO:tasks.workunit.client.0.smithi060.stdout:removed './264/34' 2024-04-14T23:54:14.485 INFO:tasks.workunit.client.0.smithi060.stdout:removed './264/38' 2024-04-14T23:54:14.485 INFO:tasks.workunit.client.0.smithi060.stdout:removed './264/28' 2024-04-14T23:54:14.485 INFO:tasks.workunit.client.0.smithi060.stdout:removed './264/24' 2024-04-14T23:54:14.485 INFO:tasks.workunit.client.0.smithi060.stdout:removed './264/29' 2024-04-14T23:54:14.485 INFO:tasks.workunit.client.0.smithi060.stdout:removed './264/32' 2024-04-14T23:54:14.485 INFO:tasks.workunit.client.0.smithi060.stdout:removed './264/22' 2024-04-14T23:54:14.486 INFO:tasks.workunit.client.0.smithi060.stdout:removed './264/25' 2024-04-14T23:54:14.486 INFO:tasks.workunit.client.0.smithi060.stdout:removed './264/30' 2024-04-14T23:54:14.486 INFO:tasks.workunit.client.0.smithi060.stdout:removed './264/33' 2024-04-14T23:54:14.486 INFO:tasks.workunit.client.0.smithi060.stdout:removed './264/23' 2024-04-14T23:54:14.486 INFO:tasks.workunit.client.0.smithi060.stdout:removed './264/35' 2024-04-14T23:54:14.486 INFO:tasks.workunit.client.0.smithi060.stdout:removed './264/26' 2024-04-14T23:54:14.486 INFO:tasks.workunit.client.0.smithi060.stdout:removed './264/37' 2024-04-14T23:54:14.486 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './264' 2024-04-14T23:54:14.486 INFO:tasks.workunit.client.0.smithi060.stdout:removed './265/21' 2024-04-14T23:54:14.486 INFO:tasks.workunit.client.0.smithi060.stdout:removed './265/27' 2024-04-14T23:54:14.486 INFO:tasks.workunit.client.0.smithi060.stdout:removed './265/31' 2024-04-14T23:54:14.486 INFO:tasks.workunit.client.0.smithi060.stdout:removed './265/36' 2024-04-14T23:54:14.486 INFO:tasks.workunit.client.0.smithi060.stdout:removed './265/20' 2024-04-14T23:54:14.486 INFO:tasks.workunit.client.0.smithi060.stdout:removed './265/39' 2024-04-14T23:54:14.486 INFO:tasks.workunit.client.0.smithi060.stdout:removed './265/34' 2024-04-14T23:54:14.487 INFO:tasks.workunit.client.0.smithi060.stdout:removed './265/38' 2024-04-14T23:54:14.487 INFO:tasks.workunit.client.0.smithi060.stdout:removed './265/28' 2024-04-14T23:54:14.487 INFO:tasks.workunit.client.0.smithi060.stdout:removed './265/24' 2024-04-14T23:54:14.487 INFO:tasks.workunit.client.0.smithi060.stdout:removed './265/29' 2024-04-14T23:54:14.487 INFO:tasks.workunit.client.0.smithi060.stdout:removed './265/32' 2024-04-14T23:54:14.487 INFO:tasks.workunit.client.0.smithi060.stdout:removed './265/22' 2024-04-14T23:54:14.487 INFO:tasks.workunit.client.0.smithi060.stdout:removed './265/25' 2024-04-14T23:54:14.487 INFO:tasks.workunit.client.0.smithi060.stdout:removed './265/30' 2024-04-14T23:54:14.487 INFO:tasks.workunit.client.0.smithi060.stdout:removed './265/33' 2024-04-14T23:54:14.487 INFO:tasks.workunit.client.0.smithi060.stdout:removed './265/23' 2024-04-14T23:54:14.487 INFO:tasks.workunit.client.0.smithi060.stdout:removed './265/35' 2024-04-14T23:54:14.487 INFO:tasks.workunit.client.0.smithi060.stdout:removed './265/26' 2024-04-14T23:54:14.487 INFO:tasks.workunit.client.0.smithi060.stdout:removed './265/37' 2024-04-14T23:54:14.487 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './265' 2024-04-14T23:54:14.488 INFO:tasks.workunit.client.0.smithi060.stdout:removed './266/21' 2024-04-14T23:54:14.488 INFO:tasks.workunit.client.0.smithi060.stdout:removed './266/27' 2024-04-14T23:54:14.488 INFO:tasks.workunit.client.0.smithi060.stdout:removed './266/31' 2024-04-14T23:54:14.488 INFO:tasks.workunit.client.0.smithi060.stdout:removed './266/36' 2024-04-14T23:54:14.488 INFO:tasks.workunit.client.0.smithi060.stdout:removed './266/20' 2024-04-14T23:54:14.488 INFO:tasks.workunit.client.0.smithi060.stdout:removed './266/39' 2024-04-14T23:54:14.488 INFO:tasks.workunit.client.0.smithi060.stdout:removed './266/34' 2024-04-14T23:54:14.488 INFO:tasks.workunit.client.0.smithi060.stdout:removed './266/38' 2024-04-14T23:54:14.488 INFO:tasks.workunit.client.0.smithi060.stdout:removed './266/28' 2024-04-14T23:54:14.488 INFO:tasks.workunit.client.0.smithi060.stdout:removed './266/24' 2024-04-14T23:54:14.488 INFO:tasks.workunit.client.0.smithi060.stdout:removed './266/29' 2024-04-14T23:54:14.488 INFO:tasks.workunit.client.0.smithi060.stdout:removed './266/32' 2024-04-14T23:54:14.488 INFO:tasks.workunit.client.0.smithi060.stdout:removed './266/22' 2024-04-14T23:54:14.488 INFO:tasks.workunit.client.0.smithi060.stdout:removed './266/25' 2024-04-14T23:54:14.488 INFO:tasks.workunit.client.0.smithi060.stdout:removed './266/30' 2024-04-14T23:54:14.489 INFO:tasks.workunit.client.0.smithi060.stdout:removed './266/33' 2024-04-14T23:54:14.489 INFO:tasks.workunit.client.0.smithi060.stdout:removed './266/23' 2024-04-14T23:54:15.373 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-14T23:54:15.385 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-14T23:54:17.069 INFO:tasks.workunit.client.0.smithi060.stdout:removed './266/35' 2024-04-14T23:54:17.069 INFO:tasks.workunit.client.0.smithi060.stdout:removed './266/26' 2024-04-14T23:54:17.069 INFO:tasks.workunit.client.0.smithi060.stdout:removed './266/37' 2024-04-14T23:54:17.069 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './266' 2024-04-14T23:54:17.069 INFO:tasks.workunit.client.0.smithi060.stdout:removed './267/21' 2024-04-14T23:54:17.069 INFO:tasks.workunit.client.0.smithi060.stdout:removed './267/27' 2024-04-14T23:54:17.070 INFO:tasks.workunit.client.0.smithi060.stdout:removed './267/31' 2024-04-14T23:54:17.070 INFO:tasks.workunit.client.0.smithi060.stdout:removed './267/36' 2024-04-14T23:54:17.070 INFO:tasks.workunit.client.0.smithi060.stdout:removed './267/20' 2024-04-14T23:54:17.070 INFO:tasks.workunit.client.0.smithi060.stdout:removed './267/39' 2024-04-14T23:54:17.070 INFO:tasks.workunit.client.0.smithi060.stdout:removed './267/34' 2024-04-14T23:54:17.070 INFO:tasks.workunit.client.0.smithi060.stdout:removed './267/38' 2024-04-14T23:54:17.070 INFO:tasks.workunit.client.0.smithi060.stdout:removed './267/28' 2024-04-14T23:54:17.070 INFO:tasks.workunit.client.0.smithi060.stdout:removed './267/24' 2024-04-14T23:54:17.070 INFO:tasks.workunit.client.0.smithi060.stdout:removed './267/29' 2024-04-14T23:54:17.070 INFO:tasks.workunit.client.0.smithi060.stdout:removed './267/32' 2024-04-14T23:54:17.070 INFO:tasks.workunit.client.0.smithi060.stdout:removed './267/22' 2024-04-14T23:54:17.070 INFO:tasks.workunit.client.0.smithi060.stdout:removed './267/25' 2024-04-14T23:54:17.070 INFO:tasks.workunit.client.0.smithi060.stdout:removed './267/30' 2024-04-14T23:54:17.070 INFO:tasks.workunit.client.0.smithi060.stdout:removed './267/33' 2024-04-14T23:54:17.071 INFO:tasks.workunit.client.0.smithi060.stdout:removed './267/23' 2024-04-14T23:54:17.071 INFO:tasks.workunit.client.0.smithi060.stdout:removed './267/35' 2024-04-14T23:54:17.071 INFO:tasks.workunit.client.0.smithi060.stdout:removed './267/26' 2024-04-14T23:54:17.071 INFO:tasks.workunit.client.0.smithi060.stdout:removed './267/37' 2024-04-14T23:54:17.071 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './267' 2024-04-14T23:54:17.071 INFO:tasks.workunit.client.0.smithi060.stdout:removed './268/21' 2024-04-14T23:54:17.071 INFO:tasks.workunit.client.0.smithi060.stdout:removed './268/27' 2024-04-14T23:54:17.071 INFO:tasks.workunit.client.0.smithi060.stdout:removed './268/31' 2024-04-14T23:54:17.071 INFO:tasks.workunit.client.0.smithi060.stdout:removed './268/36' 2024-04-14T23:54:17.071 INFO:tasks.workunit.client.0.smithi060.stdout:removed './268/20' 2024-04-14T23:54:17.071 INFO:tasks.workunit.client.0.smithi060.stdout:removed './268/39' 2024-04-14T23:54:17.071 INFO:tasks.workunit.client.0.smithi060.stdout:removed './268/34' 2024-04-14T23:54:17.071 INFO:tasks.workunit.client.0.smithi060.stdout:removed './268/38' 2024-04-14T23:54:17.071 INFO:tasks.workunit.client.0.smithi060.stdout:removed './268/28' 2024-04-14T23:54:17.072 INFO:tasks.workunit.client.0.smithi060.stdout:removed './268/24' 2024-04-14T23:54:17.072 INFO:tasks.workunit.client.0.smithi060.stdout:removed './268/29' 2024-04-14T23:54:17.072 INFO:tasks.workunit.client.0.smithi060.stdout:removed './268/32' 2024-04-14T23:54:17.072 INFO:tasks.workunit.client.0.smithi060.stdout:removed './268/22' 2024-04-14T23:54:17.072 INFO:tasks.workunit.client.0.smithi060.stdout:removed './268/25' 2024-04-14T23:54:17.072 INFO:tasks.workunit.client.0.smithi060.stdout:removed './268/30' 2024-04-14T23:54:17.072 INFO:tasks.workunit.client.0.smithi060.stdout:removed './268/33' 2024-04-14T23:54:17.072 INFO:tasks.workunit.client.0.smithi060.stdout:removed './268/23' 2024-04-14T23:54:17.072 INFO:tasks.workunit.client.0.smithi060.stdout:removed './268/35' 2024-04-14T23:54:17.072 INFO:tasks.workunit.client.0.smithi060.stdout:removed './268/26' 2024-04-14T23:54:17.072 INFO:tasks.workunit.client.0.smithi060.stdout:removed './268/37' 2024-04-14T23:54:17.072 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './268' 2024-04-14T23:54:17.072 INFO:tasks.workunit.client.0.smithi060.stdout:removed './269/21' 2024-04-14T23:54:17.072 INFO:tasks.workunit.client.0.smithi060.stdout:removed './269/27' 2024-04-14T23:54:17.073 INFO:tasks.workunit.client.0.smithi060.stdout:removed './269/31' 2024-04-14T23:54:17.073 INFO:tasks.workunit.client.0.smithi060.stdout:removed './269/36' 2024-04-14T23:54:17.073 INFO:tasks.workunit.client.0.smithi060.stdout:removed './269/20' 2024-04-14T23:54:17.073 INFO:tasks.workunit.client.0.smithi060.stdout:removed './269/39' 2024-04-14T23:54:17.073 INFO:tasks.workunit.client.0.smithi060.stdout:removed './269/34' 2024-04-14T23:54:17.073 INFO:tasks.workunit.client.0.smithi060.stdout:removed './269/38' 2024-04-14T23:54:17.073 INFO:tasks.workunit.client.0.smithi060.stdout:removed './269/28' 2024-04-14T23:54:17.073 INFO:tasks.workunit.client.0.smithi060.stdout:removed './269/24' 2024-04-14T23:54:17.073 INFO:tasks.workunit.client.0.smithi060.stdout:removed './269/29' 2024-04-14T23:54:17.073 INFO:tasks.workunit.client.0.smithi060.stdout:removed './269/32' 2024-04-14T23:54:17.073 INFO:tasks.workunit.client.0.smithi060.stdout:removed './269/22' 2024-04-14T23:54:17.073 INFO:tasks.workunit.client.0.smithi060.stdout:removed './269/25' 2024-04-14T23:54:17.073 INFO:tasks.workunit.client.0.smithi060.stdout:removed './269/30' 2024-04-14T23:54:17.073 INFO:tasks.workunit.client.0.smithi060.stdout:removed './269/33' 2024-04-14T23:54:17.073 INFO:tasks.workunit.client.0.smithi060.stdout:removed './269/23' 2024-04-14T23:54:17.074 INFO:tasks.workunit.client.0.smithi060.stdout:removed './269/35' 2024-04-14T23:54:17.074 INFO:tasks.workunit.client.0.smithi060.stdout:removed './269/26' 2024-04-14T23:54:17.074 INFO:tasks.workunit.client.0.smithi060.stdout:removed './269/37' 2024-04-14T23:54:17.074 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './269' 2024-04-14T23:54:17.074 INFO:tasks.workunit.client.0.smithi060.stdout:removed './270/21' 2024-04-14T23:54:17.074 INFO:tasks.workunit.client.0.smithi060.stdout:removed './270/27' 2024-04-14T23:54:17.074 INFO:tasks.workunit.client.0.smithi060.stdout:removed './270/31' 2024-04-14T23:54:17.074 INFO:tasks.workunit.client.0.smithi060.stdout:removed './270/36' 2024-04-14T23:54:17.074 INFO:tasks.workunit.client.0.smithi060.stdout:removed './270/20' 2024-04-14T23:54:17.074 INFO:tasks.workunit.client.0.smithi060.stdout:removed './270/39' 2024-04-14T23:54:17.074 INFO:tasks.workunit.client.0.smithi060.stdout:removed './270/34' 2024-04-14T23:54:17.074 INFO:tasks.workunit.client.0.smithi060.stdout:removed './270/38' 2024-04-14T23:54:17.074 INFO:tasks.workunit.client.0.smithi060.stdout:removed './270/28' 2024-04-14T23:54:17.074 INFO:tasks.workunit.client.0.smithi060.stdout:removed './270/24' 2024-04-14T23:54:17.075 INFO:tasks.workunit.client.0.smithi060.stdout:removed './270/29' 2024-04-14T23:54:17.075 INFO:tasks.workunit.client.0.smithi060.stdout:removed './270/32' 2024-04-14T23:54:17.075 INFO:tasks.workunit.client.0.smithi060.stdout:removed './270/22' 2024-04-14T23:54:17.075 INFO:tasks.workunit.client.0.smithi060.stdout:removed './270/25' 2024-04-14T23:54:17.075 INFO:tasks.workunit.client.0.smithi060.stdout:removed './270/30' 2024-04-14T23:54:17.075 INFO:tasks.workunit.client.0.smithi060.stdout:removed './270/33' 2024-04-14T23:54:17.075 INFO:tasks.workunit.client.0.smithi060.stdout:removed './270/23' 2024-04-14T23:54:17.075 INFO:tasks.workunit.client.0.smithi060.stdout:removed './270/35' 2024-04-14T23:54:17.075 INFO:tasks.workunit.client.0.smithi060.stdout:removed './270/26' 2024-04-14T23:54:17.075 INFO:tasks.workunit.client.0.smithi060.stdout:removed './270/37' 2024-04-14T23:54:17.075 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './270' 2024-04-14T23:54:17.075 INFO:tasks.workunit.client.0.smithi060.stdout:removed './271/21' 2024-04-14T23:54:17.075 INFO:tasks.workunit.client.0.smithi060.stdout:removed './271/27' 2024-04-14T23:54:17.075 INFO:tasks.workunit.client.0.smithi060.stdout:removed './271/31' 2024-04-14T23:54:17.075 INFO:tasks.workunit.client.0.smithi060.stdout:removed './271/36' 2024-04-14T23:54:17.076 INFO:tasks.workunit.client.0.smithi060.stdout:removed './271/20' 2024-04-14T23:54:17.076 INFO:tasks.workunit.client.0.smithi060.stdout:removed './271/39' 2024-04-14T23:54:17.076 INFO:tasks.workunit.client.0.smithi060.stdout:removed './271/34' 2024-04-14T23:54:17.076 INFO:tasks.workunit.client.0.smithi060.stdout:removed './271/38' 2024-04-14T23:54:17.076 INFO:tasks.workunit.client.0.smithi060.stdout:removed './271/28' 2024-04-14T23:54:17.076 INFO:tasks.workunit.client.0.smithi060.stdout:removed './271/24' 2024-04-14T23:54:17.076 INFO:tasks.workunit.client.0.smithi060.stdout:removed './271/29' 2024-04-14T23:54:17.076 INFO:tasks.workunit.client.0.smithi060.stdout:removed './271/32' 2024-04-14T23:54:17.076 INFO:tasks.workunit.client.0.smithi060.stdout:removed './271/22' 2024-04-14T23:54:17.076 INFO:tasks.workunit.client.0.smithi060.stdout:removed './271/25' 2024-04-14T23:54:17.076 INFO:tasks.workunit.client.0.smithi060.stdout:removed './271/30' 2024-04-14T23:54:17.076 INFO:tasks.workunit.client.0.smithi060.stdout:removed './271/33' 2024-04-14T23:54:17.076 INFO:tasks.workunit.client.0.smithi060.stdout:removed './271/23' 2024-04-14T23:54:17.076 INFO:tasks.workunit.client.0.smithi060.stdout:removed './271/35' 2024-04-14T23:54:17.076 INFO:tasks.workunit.client.0.smithi060.stdout:removed './271/26' 2024-04-14T23:54:17.077 INFO:tasks.workunit.client.0.smithi060.stdout:removed './271/37' 2024-04-14T23:54:17.077 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './271' 2024-04-14T23:54:17.077 INFO:tasks.workunit.client.0.smithi060.stdout:removed './272/21' 2024-04-14T23:54:17.077 INFO:tasks.workunit.client.0.smithi060.stdout:removed './272/27' 2024-04-14T23:54:17.077 INFO:tasks.workunit.client.0.smithi060.stdout:removed './272/31' 2024-04-14T23:54:17.077 INFO:tasks.workunit.client.0.smithi060.stdout:removed './272/36' 2024-04-14T23:54:17.077 INFO:tasks.workunit.client.0.smithi060.stdout:removed './272/20' 2024-04-14T23:54:17.077 INFO:tasks.workunit.client.0.smithi060.stdout:removed './272/39' 2024-04-14T23:54:17.077 INFO:tasks.workunit.client.0.smithi060.stdout:removed './272/34' 2024-04-14T23:54:17.077 INFO:tasks.workunit.client.0.smithi060.stdout:removed './272/38' 2024-04-14T23:54:17.077 INFO:tasks.workunit.client.0.smithi060.stdout:removed './272/28' 2024-04-14T23:54:17.077 INFO:tasks.workunit.client.0.smithi060.stdout:removed './272/24' 2024-04-14T23:54:17.077 INFO:tasks.workunit.client.0.smithi060.stdout:removed './272/29' 2024-04-14T23:54:17.077 INFO:tasks.workunit.client.0.smithi060.stdout:removed './272/32' 2024-04-14T23:54:17.077 INFO:tasks.workunit.client.0.smithi060.stdout:removed './272/22' 2024-04-14T23:54:17.078 INFO:tasks.workunit.client.0.smithi060.stdout:removed './272/25' 2024-04-14T23:54:17.078 INFO:tasks.workunit.client.0.smithi060.stdout:removed './272/30' 2024-04-14T23:54:17.078 INFO:tasks.workunit.client.0.smithi060.stdout:removed './272/33' 2024-04-14T23:54:17.078 INFO:tasks.workunit.client.0.smithi060.stdout:removed './272/23' 2024-04-14T23:54:17.078 INFO:tasks.workunit.client.0.smithi060.stdout:removed './272/35' 2024-04-14T23:54:17.078 INFO:tasks.workunit.client.0.smithi060.stdout:removed './272/26' 2024-04-14T23:54:17.078 INFO:tasks.workunit.client.0.smithi060.stdout:removed './272/37' 2024-04-14T23:54:17.078 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './272' 2024-04-14T23:54:17.078 INFO:tasks.workunit.client.0.smithi060.stdout:removed './273/21' 2024-04-14T23:54:17.078 INFO:tasks.workunit.client.0.smithi060.stdout:removed './273/27' 2024-04-14T23:54:17.078 INFO:tasks.workunit.client.0.smithi060.stdout:removed './273/31' 2024-04-14T23:54:17.078 INFO:tasks.workunit.client.0.smithi060.stdout:removed './273/36' 2024-04-14T23:54:17.078 INFO:tasks.workunit.client.0.smithi060.stdout:removed './273/20' 2024-04-14T23:54:17.078 INFO:tasks.workunit.client.0.smithi060.stdout:removed './273/39' 2024-04-14T23:54:17.079 INFO:tasks.workunit.client.0.smithi060.stdout:removed './273/34' 2024-04-14T23:54:17.079 INFO:tasks.workunit.client.0.smithi060.stdout:removed './273/38' 2024-04-14T23:54:17.079 INFO:tasks.workunit.client.0.smithi060.stdout:removed './273/28' 2024-04-14T23:54:17.079 INFO:tasks.workunit.client.0.smithi060.stdout:removed './273/24' 2024-04-14T23:54:17.079 INFO:tasks.workunit.client.0.smithi060.stdout:removed './273/29' 2024-04-14T23:54:17.079 INFO:tasks.workunit.client.0.smithi060.stdout:removed './273/32' 2024-04-14T23:54:17.079 INFO:tasks.workunit.client.0.smithi060.stdout:removed './273/22' 2024-04-14T23:54:17.079 INFO:tasks.workunit.client.0.smithi060.stdout:removed './273/25' 2024-04-14T23:54:17.079 INFO:tasks.workunit.client.0.smithi060.stdout:removed './273/30' 2024-04-14T23:54:17.079 INFO:tasks.workunit.client.0.smithi060.stdout:removed './273/33' 2024-04-14T23:54:17.079 INFO:tasks.workunit.client.0.smithi060.stdout:removed './273/23' 2024-04-14T23:54:17.079 INFO:tasks.workunit.client.0.smithi060.stdout:removed './273/35' 2024-04-14T23:54:17.079 INFO:tasks.workunit.client.0.smithi060.stdout:removed './273/26' 2024-04-14T23:54:17.079 INFO:tasks.workunit.client.0.smithi060.stdout:removed './273/37' 2024-04-14T23:54:17.079 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './273' 2024-04-14T23:54:17.080 INFO:tasks.workunit.client.0.smithi060.stdout:removed './274/21' 2024-04-14T23:54:17.080 INFO:tasks.workunit.client.0.smithi060.stdout:removed './274/27' 2024-04-14T23:54:17.080 INFO:tasks.workunit.client.0.smithi060.stdout:removed './274/31' 2024-04-14T23:54:17.080 INFO:tasks.workunit.client.0.smithi060.stdout:removed './274/36' 2024-04-14T23:54:17.080 INFO:tasks.workunit.client.0.smithi060.stdout:removed './274/20' 2024-04-14T23:54:17.080 INFO:tasks.workunit.client.0.smithi060.stdout:removed './274/39' 2024-04-14T23:54:17.080 INFO:tasks.workunit.client.0.smithi060.stdout:removed './274/34' 2024-04-14T23:54:17.080 INFO:tasks.workunit.client.0.smithi060.stdout:removed './274/38' 2024-04-14T23:54:17.080 INFO:tasks.workunit.client.0.smithi060.stdout:removed './274/28' 2024-04-14T23:54:17.080 INFO:tasks.workunit.client.0.smithi060.stdout:removed './274/24' 2024-04-14T23:54:17.080 INFO:tasks.workunit.client.0.smithi060.stdout:removed './274/29' 2024-04-14T23:54:17.080 INFO:tasks.workunit.client.0.smithi060.stdout:removed './274/32' 2024-04-14T23:54:17.080 INFO:tasks.workunit.client.0.smithi060.stdout:removed './274/22' 2024-04-14T23:54:17.080 INFO:tasks.workunit.client.0.smithi060.stdout:removed './274/25' 2024-04-14T23:54:17.080 INFO:tasks.workunit.client.0.smithi060.stdout:removed './274/30' 2024-04-14T23:54:17.081 INFO:tasks.workunit.client.0.smithi060.stdout:removed './274/33' 2024-04-14T23:54:17.081 INFO:tasks.workunit.client.0.smithi060.stdout:removed './274/23' 2024-04-14T23:54:17.081 INFO:tasks.workunit.client.0.smithi060.stdout:removed './274/35' 2024-04-14T23:54:17.081 INFO:tasks.workunit.client.0.smithi060.stdout:removed './274/26' 2024-04-14T23:54:17.081 INFO:tasks.workunit.client.0.smithi060.stdout:removed './274/37' 2024-04-14T23:54:17.081 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './274' 2024-04-14T23:54:17.081 INFO:tasks.workunit.client.0.smithi060.stdout:removed './275/21' 2024-04-14T23:54:17.081 INFO:tasks.workunit.client.0.smithi060.stdout:removed './275/27' 2024-04-14T23:54:17.081 INFO:tasks.workunit.client.0.smithi060.stdout:removed './275/31' 2024-04-14T23:54:17.081 INFO:tasks.workunit.client.0.smithi060.stdout:removed './275/36' 2024-04-14T23:54:17.081 INFO:tasks.workunit.client.0.smithi060.stdout:removed './275/20' 2024-04-14T23:54:17.081 INFO:tasks.workunit.client.0.smithi060.stdout:removed './275/39' 2024-04-14T23:54:17.081 INFO:tasks.workunit.client.0.smithi060.stdout:removed './275/34' 2024-04-14T23:54:17.081 INFO:tasks.workunit.client.0.smithi060.stdout:removed './275/38' 2024-04-14T23:54:17.081 INFO:tasks.workunit.client.0.smithi060.stdout:removed './275/28' 2024-04-14T23:54:17.082 INFO:tasks.workunit.client.0.smithi060.stdout:removed './275/24' 2024-04-14T23:54:17.082 INFO:tasks.workunit.client.0.smithi060.stdout:removed './275/29' 2024-04-14T23:54:17.082 INFO:tasks.workunit.client.0.smithi060.stdout:removed './275/32' 2024-04-14T23:54:17.082 INFO:tasks.workunit.client.0.smithi060.stdout:removed './275/22' 2024-04-14T23:54:17.082 INFO:tasks.workunit.client.0.smithi060.stdout:removed './275/25' 2024-04-14T23:54:17.082 INFO:tasks.workunit.client.0.smithi060.stdout:removed './275/30' 2024-04-14T23:54:17.082 INFO:tasks.workunit.client.0.smithi060.stdout:removed './275/33' 2024-04-14T23:54:17.082 INFO:tasks.workunit.client.0.smithi060.stdout:removed './275/23' 2024-04-14T23:54:17.082 INFO:tasks.workunit.client.0.smithi060.stdout:removed './275/35' 2024-04-14T23:54:17.082 INFO:tasks.workunit.client.0.smithi060.stdout:removed './275/26' 2024-04-14T23:54:17.082 INFO:tasks.workunit.client.0.smithi060.stdout:removed './275/37' 2024-04-14T23:54:17.082 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './275' 2024-04-14T23:54:17.082 INFO:tasks.workunit.client.0.smithi060.stdout:removed './276/21' 2024-04-14T23:54:17.082 INFO:tasks.workunit.client.0.smithi060.stdout:removed './276/27' 2024-04-14T23:54:17.083 INFO:tasks.workunit.client.0.smithi060.stdout:removed './276/31' 2024-04-14T23:54:17.083 INFO:tasks.workunit.client.0.smithi060.stdout:removed './276/36' 2024-04-14T23:54:17.083 INFO:tasks.workunit.client.0.smithi060.stdout:removed './276/20' 2024-04-14T23:54:17.083 INFO:tasks.workunit.client.0.smithi060.stdout:removed './276/39' 2024-04-14T23:54:17.083 INFO:tasks.workunit.client.0.smithi060.stdout:removed './276/34' 2024-04-14T23:54:17.083 INFO:tasks.workunit.client.0.smithi060.stdout:removed './276/38' 2024-04-14T23:54:17.083 INFO:tasks.workunit.client.0.smithi060.stdout:removed './276/28' 2024-04-14T23:54:17.083 INFO:tasks.workunit.client.0.smithi060.stdout:removed './276/24' 2024-04-14T23:54:17.083 INFO:tasks.workunit.client.0.smithi060.stdout:removed './276/29' 2024-04-14T23:54:17.083 INFO:tasks.workunit.client.0.smithi060.stdout:removed './276/32' 2024-04-14T23:54:17.083 INFO:tasks.workunit.client.0.smithi060.stdout:removed './276/22' 2024-04-14T23:54:17.083 INFO:tasks.workunit.client.0.smithi060.stdout:removed './276/25' 2024-04-14T23:54:17.083 INFO:tasks.workunit.client.0.smithi060.stdout:removed './276/30' 2024-04-14T23:54:17.083 INFO:tasks.workunit.client.0.smithi060.stdout:removed './276/33' 2024-04-14T23:54:17.083 INFO:tasks.workunit.client.0.smithi060.stdout:removed './276/23' 2024-04-14T23:54:17.084 INFO:tasks.workunit.client.0.smithi060.stdout:removed './276/35' 2024-04-14T23:54:17.084 INFO:tasks.workunit.client.0.smithi060.stdout:removed './276/26' 2024-04-14T23:54:22.461 INFO:tasks.workunit.client.0.smithi060.stdout:removed './276/37' 2024-04-14T23:54:22.461 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './276' 2024-04-14T23:54:22.461 INFO:tasks.workunit.client.0.smithi060.stdout:removed './277/21' 2024-04-14T23:54:22.461 INFO:tasks.workunit.client.0.smithi060.stdout:removed './277/27' 2024-04-14T23:54:22.461 INFO:tasks.workunit.client.0.smithi060.stdout:removed './277/31' 2024-04-14T23:54:22.461 INFO:tasks.workunit.client.0.smithi060.stdout:removed './277/36' 2024-04-14T23:54:22.461 INFO:tasks.workunit.client.0.smithi060.stdout:removed './277/20' 2024-04-14T23:54:22.461 INFO:tasks.workunit.client.0.smithi060.stdout:removed './277/39' 2024-04-14T23:54:22.462 INFO:tasks.workunit.client.0.smithi060.stdout:removed './277/34' 2024-04-14T23:54:22.462 INFO:tasks.workunit.client.0.smithi060.stdout:removed './277/38' 2024-04-14T23:54:22.462 INFO:tasks.workunit.client.0.smithi060.stdout:removed './277/28' 2024-04-14T23:54:22.462 INFO:tasks.workunit.client.0.smithi060.stdout:removed './277/24' 2024-04-14T23:54:22.462 INFO:tasks.workunit.client.0.smithi060.stdout:removed './277/29' 2024-04-14T23:54:22.462 INFO:tasks.workunit.client.0.smithi060.stdout:removed './277/32' 2024-04-14T23:54:22.462 INFO:tasks.workunit.client.0.smithi060.stdout:removed './277/22' 2024-04-14T23:54:22.462 INFO:tasks.workunit.client.0.smithi060.stdout:removed './277/25' 2024-04-14T23:54:22.462 INFO:tasks.workunit.client.0.smithi060.stdout:removed './277/30' 2024-04-14T23:54:22.462 INFO:tasks.workunit.client.0.smithi060.stdout:removed './277/33' 2024-04-14T23:54:22.462 INFO:tasks.workunit.client.0.smithi060.stdout:removed './277/23' 2024-04-14T23:54:22.462 INFO:tasks.workunit.client.0.smithi060.stdout:removed './277/35' 2024-04-14T23:54:22.462 INFO:tasks.workunit.client.0.smithi060.stdout:removed './277/26' 2024-04-14T23:54:22.462 INFO:tasks.workunit.client.0.smithi060.stdout:removed './277/37' 2024-04-14T23:54:22.463 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './277' 2024-04-14T23:54:22.463 INFO:tasks.workunit.client.0.smithi060.stdout:removed './278/21' 2024-04-14T23:54:22.463 INFO:tasks.workunit.client.0.smithi060.stdout:removed './278/27' 2024-04-14T23:54:22.463 INFO:tasks.workunit.client.0.smithi060.stdout:removed './278/31' 2024-04-14T23:54:22.463 INFO:tasks.workunit.client.0.smithi060.stdout:removed './278/36' 2024-04-14T23:54:22.463 INFO:tasks.workunit.client.0.smithi060.stdout:removed './278/20' 2024-04-14T23:54:22.463 INFO:tasks.workunit.client.0.smithi060.stdout:removed './278/39' 2024-04-14T23:54:22.463 INFO:tasks.workunit.client.0.smithi060.stdout:removed './278/34' 2024-04-14T23:54:22.463 INFO:tasks.workunit.client.0.smithi060.stdout:removed './278/38' 2024-04-14T23:54:22.463 INFO:tasks.workunit.client.0.smithi060.stdout:removed './278/28' 2024-04-14T23:54:22.463 INFO:tasks.workunit.client.0.smithi060.stdout:removed './278/24' 2024-04-14T23:54:22.463 INFO:tasks.workunit.client.0.smithi060.stdout:removed './278/29' 2024-04-14T23:54:22.463 INFO:tasks.workunit.client.0.smithi060.stdout:removed './278/32' 2024-04-14T23:54:22.463 INFO:tasks.workunit.client.0.smithi060.stdout:removed './278/22' 2024-04-14T23:54:22.464 INFO:tasks.workunit.client.0.smithi060.stdout:removed './278/25' 2024-04-14T23:54:22.464 INFO:tasks.workunit.client.0.smithi060.stdout:removed './278/30' 2024-04-14T23:54:22.464 INFO:tasks.workunit.client.0.smithi060.stdout:removed './278/33' 2024-04-14T23:54:22.464 INFO:tasks.workunit.client.0.smithi060.stdout:removed './278/23' 2024-04-14T23:54:22.464 INFO:tasks.workunit.client.0.smithi060.stdout:removed './278/35' 2024-04-14T23:54:22.464 INFO:tasks.workunit.client.0.smithi060.stdout:removed './278/26' 2024-04-14T23:54:22.464 INFO:tasks.workunit.client.0.smithi060.stdout:removed './278/37' 2024-04-14T23:54:22.464 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './278' 2024-04-14T23:54:22.464 INFO:tasks.workunit.client.0.smithi060.stdout:removed './279/21' 2024-04-14T23:54:22.464 INFO:tasks.workunit.client.0.smithi060.stdout:removed './279/27' 2024-04-14T23:54:22.464 INFO:tasks.workunit.client.0.smithi060.stdout:removed './279/31' 2024-04-14T23:54:22.464 INFO:tasks.workunit.client.0.smithi060.stdout:removed './279/36' 2024-04-14T23:54:22.464 INFO:tasks.workunit.client.0.smithi060.stdout:removed './279/20' 2024-04-14T23:54:22.464 INFO:tasks.workunit.client.0.smithi060.stdout:removed './279/39' 2024-04-14T23:54:22.464 INFO:tasks.workunit.client.0.smithi060.stdout:removed './279/34' 2024-04-14T23:54:22.464 INFO:tasks.workunit.client.0.smithi060.stdout:removed './279/38' 2024-04-14T23:54:22.465 INFO:tasks.workunit.client.0.smithi060.stdout:removed './279/28' 2024-04-14T23:54:22.465 INFO:tasks.workunit.client.0.smithi060.stdout:removed './279/24' 2024-04-14T23:54:22.465 INFO:tasks.workunit.client.0.smithi060.stdout:removed './279/29' 2024-04-14T23:54:22.465 INFO:tasks.workunit.client.0.smithi060.stdout:removed './279/32' 2024-04-14T23:54:22.465 INFO:tasks.workunit.client.0.smithi060.stdout:removed './279/22' 2024-04-14T23:54:22.465 INFO:tasks.workunit.client.0.smithi060.stdout:removed './279/25' 2024-04-14T23:54:22.465 INFO:tasks.workunit.client.0.smithi060.stdout:removed './279/30' 2024-04-14T23:54:22.465 INFO:tasks.workunit.client.0.smithi060.stdout:removed './279/33' 2024-04-14T23:54:22.465 INFO:tasks.workunit.client.0.smithi060.stdout:removed './279/23' 2024-04-14T23:54:22.465 INFO:tasks.workunit.client.0.smithi060.stdout:removed './279/35' 2024-04-14T23:54:22.465 INFO:tasks.workunit.client.0.smithi060.stdout:removed './279/26' 2024-04-14T23:54:22.465 INFO:tasks.workunit.client.0.smithi060.stdout:removed './279/37' 2024-04-14T23:54:22.465 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './279' 2024-04-14T23:54:22.465 INFO:tasks.workunit.client.0.smithi060.stdout:removed './280/21' 2024-04-14T23:54:22.465 INFO:tasks.workunit.client.0.smithi060.stdout:removed './280/27' 2024-04-14T23:54:22.466 INFO:tasks.workunit.client.0.smithi060.stdout:removed './280/31' 2024-04-14T23:54:22.466 INFO:tasks.workunit.client.0.smithi060.stdout:removed './280/36' 2024-04-14T23:54:22.466 INFO:tasks.workunit.client.0.smithi060.stdout:removed './280/20' 2024-04-14T23:54:22.466 INFO:tasks.workunit.client.0.smithi060.stdout:removed './280/39' 2024-04-14T23:54:22.466 INFO:tasks.workunit.client.0.smithi060.stdout:removed './280/34' 2024-04-14T23:54:22.466 INFO:tasks.workunit.client.0.smithi060.stdout:removed './280/38' 2024-04-14T23:54:22.466 INFO:tasks.workunit.client.0.smithi060.stdout:removed './280/28' 2024-04-14T23:54:22.466 INFO:tasks.workunit.client.0.smithi060.stdout:removed './280/24' 2024-04-14T23:54:22.466 INFO:tasks.workunit.client.0.smithi060.stdout:removed './280/29' 2024-04-14T23:54:22.466 INFO:tasks.workunit.client.0.smithi060.stdout:removed './280/32' 2024-04-14T23:54:22.466 INFO:tasks.workunit.client.0.smithi060.stdout:removed './280/22' 2024-04-14T23:54:22.466 INFO:tasks.workunit.client.0.smithi060.stdout:removed './280/25' 2024-04-14T23:54:22.466 INFO:tasks.workunit.client.0.smithi060.stdout:removed './280/30' 2024-04-14T23:54:22.466 INFO:tasks.workunit.client.0.smithi060.stdout:removed './280/33' 2024-04-14T23:54:22.467 INFO:tasks.workunit.client.0.smithi060.stdout:removed './280/23' 2024-04-14T23:54:22.467 INFO:tasks.workunit.client.0.smithi060.stdout:removed './280/35' 2024-04-14T23:54:22.467 INFO:tasks.workunit.client.0.smithi060.stdout:removed './280/26' 2024-04-14T23:54:22.467 INFO:tasks.workunit.client.0.smithi060.stdout:removed './280/37' 2024-04-14T23:54:22.467 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './280' 2024-04-14T23:54:22.467 INFO:tasks.workunit.client.0.smithi060.stdout:removed './281/21' 2024-04-14T23:54:22.467 INFO:tasks.workunit.client.0.smithi060.stdout:removed './281/27' 2024-04-14T23:54:22.467 INFO:tasks.workunit.client.0.smithi060.stdout:removed './281/31' 2024-04-14T23:54:22.467 INFO:tasks.workunit.client.0.smithi060.stdout:removed './281/36' 2024-04-14T23:54:22.467 INFO:tasks.workunit.client.0.smithi060.stdout:removed './281/20' 2024-04-14T23:54:22.467 INFO:tasks.workunit.client.0.smithi060.stdout:removed './281/39' 2024-04-14T23:54:22.467 INFO:tasks.workunit.client.0.smithi060.stdout:removed './281/34' 2024-04-14T23:54:22.467 INFO:tasks.workunit.client.0.smithi060.stdout:removed './281/38' 2024-04-14T23:54:22.467 INFO:tasks.workunit.client.0.smithi060.stdout:removed './281/28' 2024-04-14T23:54:22.468 INFO:tasks.workunit.client.0.smithi060.stdout:removed './281/24' 2024-04-14T23:54:22.468 INFO:tasks.workunit.client.0.smithi060.stdout:removed './281/29' 2024-04-14T23:54:22.468 INFO:tasks.workunit.client.0.smithi060.stdout:removed './281/32' 2024-04-14T23:54:22.468 INFO:tasks.workunit.client.0.smithi060.stdout:removed './281/22' 2024-04-14T23:54:22.468 INFO:tasks.workunit.client.0.smithi060.stdout:removed './281/25' 2024-04-14T23:54:22.468 INFO:tasks.workunit.client.0.smithi060.stdout:removed './281/30' 2024-04-14T23:54:22.468 INFO:tasks.workunit.client.0.smithi060.stdout:removed './281/33' 2024-04-14T23:54:22.468 INFO:tasks.workunit.client.0.smithi060.stdout:removed './281/23' 2024-04-14T23:54:22.468 INFO:tasks.workunit.client.0.smithi060.stdout:removed './281/35' 2024-04-14T23:54:22.468 INFO:tasks.workunit.client.0.smithi060.stdout:removed './281/26' 2024-04-14T23:54:22.468 INFO:tasks.workunit.client.0.smithi060.stdout:removed './281/37' 2024-04-14T23:54:22.468 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './281' 2024-04-14T23:54:22.468 INFO:tasks.workunit.client.0.smithi060.stdout:removed './282/21' 2024-04-14T23:54:22.468 INFO:tasks.workunit.client.0.smithi060.stdout:removed './282/27' 2024-04-14T23:54:22.469 INFO:tasks.workunit.client.0.smithi060.stdout:removed './282/31' 2024-04-14T23:54:22.469 INFO:tasks.workunit.client.0.smithi060.stdout:removed './282/36' 2024-04-14T23:54:22.469 INFO:tasks.workunit.client.0.smithi060.stdout:removed './282/20' 2024-04-14T23:54:22.469 INFO:tasks.workunit.client.0.smithi060.stdout:removed './282/39' 2024-04-14T23:54:22.469 INFO:tasks.workunit.client.0.smithi060.stdout:removed './282/34' 2024-04-14T23:54:22.469 INFO:tasks.workunit.client.0.smithi060.stdout:removed './282/38' 2024-04-14T23:54:22.469 INFO:tasks.workunit.client.0.smithi060.stdout:removed './282/28' 2024-04-14T23:54:22.469 INFO:tasks.workunit.client.0.smithi060.stdout:removed './282/24' 2024-04-14T23:54:22.469 INFO:tasks.workunit.client.0.smithi060.stdout:removed './282/29' 2024-04-14T23:54:22.469 INFO:tasks.workunit.client.0.smithi060.stdout:removed './282/32' 2024-04-14T23:54:22.469 INFO:tasks.workunit.client.0.smithi060.stdout:removed './282/22' 2024-04-14T23:54:22.469 INFO:tasks.workunit.client.0.smithi060.stdout:removed './282/25' 2024-04-14T23:54:22.469 INFO:tasks.workunit.client.0.smithi060.stdout:removed './282/30' 2024-04-14T23:54:22.469 INFO:tasks.workunit.client.0.smithi060.stdout:removed './282/33' 2024-04-14T23:54:22.469 INFO:tasks.workunit.client.0.smithi060.stdout:removed './282/23' 2024-04-14T23:54:22.470 INFO:tasks.workunit.client.0.smithi060.stdout:removed './282/35' 2024-04-14T23:54:22.470 INFO:tasks.workunit.client.0.smithi060.stdout:removed './282/26' 2024-04-14T23:54:22.470 INFO:tasks.workunit.client.0.smithi060.stdout:removed './282/37' 2024-04-14T23:54:22.470 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './282' 2024-04-14T23:54:22.470 INFO:tasks.workunit.client.0.smithi060.stdout:removed './283/21' 2024-04-14T23:54:22.470 INFO:tasks.workunit.client.0.smithi060.stdout:removed './283/27' 2024-04-14T23:54:22.470 INFO:tasks.workunit.client.0.smithi060.stdout:removed './283/31' 2024-04-14T23:54:22.470 INFO:tasks.workunit.client.0.smithi060.stdout:removed './283/36' 2024-04-14T23:54:22.470 INFO:tasks.workunit.client.0.smithi060.stdout:removed './283/20' 2024-04-14T23:54:22.470 INFO:tasks.workunit.client.0.smithi060.stdout:removed './283/39' 2024-04-14T23:54:22.470 INFO:tasks.workunit.client.0.smithi060.stdout:removed './283/34' 2024-04-14T23:54:22.471 INFO:tasks.workunit.client.0.smithi060.stdout:removed './283/38' 2024-04-14T23:54:22.471 INFO:tasks.workunit.client.0.smithi060.stdout:removed './283/28' 2024-04-14T23:54:22.471 INFO:tasks.workunit.client.0.smithi060.stdout:removed './283/24' 2024-04-14T23:54:22.471 INFO:tasks.workunit.client.0.smithi060.stdout:removed './283/29' 2024-04-14T23:54:22.471 INFO:tasks.workunit.client.0.smithi060.stdout:removed './283/32' 2024-04-14T23:54:22.471 INFO:tasks.workunit.client.0.smithi060.stdout:removed './283/22' 2024-04-14T23:54:22.471 INFO:tasks.workunit.client.0.smithi060.stdout:removed './283/25' 2024-04-14T23:54:22.471 INFO:tasks.workunit.client.0.smithi060.stdout:removed './283/30' 2024-04-14T23:54:22.471 INFO:tasks.workunit.client.0.smithi060.stdout:removed './283/33' 2024-04-14T23:54:22.471 INFO:tasks.workunit.client.0.smithi060.stdout:removed './283/23' 2024-04-14T23:54:22.471 INFO:tasks.workunit.client.0.smithi060.stdout:removed './283/35' 2024-04-14T23:54:22.471 INFO:tasks.workunit.client.0.smithi060.stdout:removed './283/26' 2024-04-14T23:54:22.471 INFO:tasks.workunit.client.0.smithi060.stdout:removed './283/37' 2024-04-14T23:54:22.471 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './283' 2024-04-14T23:54:22.471 INFO:tasks.workunit.client.0.smithi060.stdout:removed './284/21' 2024-04-14T23:54:22.472 INFO:tasks.workunit.client.0.smithi060.stdout:removed './284/27' 2024-04-14T23:54:22.472 INFO:tasks.workunit.client.0.smithi060.stdout:removed './284/31' 2024-04-14T23:54:22.472 INFO:tasks.workunit.client.0.smithi060.stdout:removed './284/36' 2024-04-14T23:54:22.472 INFO:tasks.workunit.client.0.smithi060.stdout:removed './284/20' 2024-04-14T23:54:22.472 INFO:tasks.workunit.client.0.smithi060.stdout:removed './284/39' 2024-04-14T23:54:22.472 INFO:tasks.workunit.client.0.smithi060.stdout:removed './284/34' 2024-04-14T23:54:22.472 INFO:tasks.workunit.client.0.smithi060.stdout:removed './284/38' 2024-04-14T23:54:22.472 INFO:tasks.workunit.client.0.smithi060.stdout:removed './284/28' 2024-04-14T23:54:22.472 INFO:tasks.workunit.client.0.smithi060.stdout:removed './284/24' 2024-04-14T23:54:22.472 INFO:tasks.workunit.client.0.smithi060.stdout:removed './284/29' 2024-04-14T23:54:22.472 INFO:tasks.workunit.client.0.smithi060.stdout:removed './284/32' 2024-04-14T23:54:22.472 INFO:tasks.workunit.client.0.smithi060.stdout:removed './284/22' 2024-04-14T23:54:22.472 INFO:tasks.workunit.client.0.smithi060.stdout:removed './284/25' 2024-04-14T23:54:22.472 INFO:tasks.workunit.client.0.smithi060.stdout:removed './284/30' 2024-04-14T23:54:22.472 INFO:tasks.workunit.client.0.smithi060.stdout:removed './284/33' 2024-04-14T23:54:22.473 INFO:tasks.workunit.client.0.smithi060.stdout:removed './284/23' 2024-04-14T23:54:22.473 INFO:tasks.workunit.client.0.smithi060.stdout:removed './284/35' 2024-04-14T23:54:22.473 INFO:tasks.workunit.client.0.smithi060.stdout:removed './284/26' 2024-04-14T23:54:22.473 INFO:tasks.workunit.client.0.smithi060.stdout:removed './284/37' 2024-04-14T23:54:22.473 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './284' 2024-04-14T23:54:22.473 INFO:tasks.workunit.client.0.smithi060.stdout:removed './285/21' 2024-04-14T23:54:22.473 INFO:tasks.workunit.client.0.smithi060.stdout:removed './285/27' 2024-04-14T23:54:22.473 INFO:tasks.workunit.client.0.smithi060.stdout:removed './285/31' 2024-04-14T23:54:22.473 INFO:tasks.workunit.client.0.smithi060.stdout:removed './285/36' 2024-04-14T23:54:22.473 INFO:tasks.workunit.client.0.smithi060.stdout:removed './285/20' 2024-04-14T23:54:22.473 INFO:tasks.workunit.client.0.smithi060.stdout:removed './285/39' 2024-04-14T23:54:22.473 INFO:tasks.workunit.client.0.smithi060.stdout:removed './285/34' 2024-04-14T23:54:22.473 INFO:tasks.workunit.client.0.smithi060.stdout:removed './285/38' 2024-04-14T23:54:22.473 INFO:tasks.workunit.client.0.smithi060.stdout:removed './285/28' 2024-04-14T23:54:22.473 INFO:tasks.workunit.client.0.smithi060.stdout:removed './285/24' 2024-04-14T23:54:22.474 INFO:tasks.workunit.client.0.smithi060.stdout:removed './285/29' 2024-04-14T23:54:22.474 INFO:tasks.workunit.client.0.smithi060.stdout:removed './285/32' 2024-04-14T23:54:22.474 INFO:tasks.workunit.client.0.smithi060.stdout:removed './285/22' 2024-04-14T23:54:22.474 INFO:tasks.workunit.client.0.smithi060.stdout:removed './285/25' 2024-04-14T23:54:22.474 INFO:tasks.workunit.client.0.smithi060.stdout:removed './285/30' 2024-04-14T23:54:22.474 INFO:tasks.workunit.client.0.smithi060.stdout:removed './285/33' 2024-04-14T23:54:22.474 INFO:tasks.workunit.client.0.smithi060.stdout:removed './285/23' 2024-04-14T23:54:22.474 INFO:tasks.workunit.client.0.smithi060.stdout:removed './285/35' 2024-04-14T23:54:22.474 INFO:tasks.workunit.client.0.smithi060.stdout:removed './285/26' 2024-04-14T23:54:22.474 INFO:tasks.workunit.client.0.smithi060.stdout:removed './285/37' 2024-04-14T23:54:22.474 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './285' 2024-04-14T23:54:22.474 INFO:tasks.workunit.client.0.smithi060.stdout:removed './286/21' 2024-04-14T23:54:22.474 INFO:tasks.workunit.client.0.smithi060.stdout:removed './286/27' 2024-04-14T23:54:22.474 INFO:tasks.workunit.client.0.smithi060.stdout:removed './286/31' 2024-04-14T23:54:22.475 INFO:tasks.workunit.client.0.smithi060.stdout:removed './286/36' 2024-04-14T23:54:22.475 INFO:tasks.workunit.client.0.smithi060.stdout:removed './286/20' 2024-04-14T23:54:22.475 INFO:tasks.workunit.client.0.smithi060.stdout:removed './286/39' 2024-04-14T23:54:22.475 INFO:tasks.workunit.client.0.smithi060.stdout:removed './286/34' 2024-04-14T23:54:22.475 INFO:tasks.workunit.client.0.smithi060.stdout:removed './286/38' 2024-04-14T23:54:22.475 INFO:tasks.workunit.client.0.smithi060.stdout:removed './286/28' 2024-04-14T23:54:22.475 INFO:tasks.workunit.client.0.smithi060.stdout:removed './286/24' 2024-04-14T23:54:22.475 INFO:tasks.workunit.client.0.smithi060.stdout:removed './286/29' 2024-04-14T23:54:22.475 INFO:tasks.workunit.client.0.smithi060.stdout:removed './286/32' 2024-04-14T23:54:22.475 INFO:tasks.workunit.client.0.smithi060.stdout:removed './286/22' 2024-04-14T23:54:22.475 INFO:tasks.workunit.client.0.smithi060.stdout:removed './286/25' 2024-04-14T23:54:22.475 INFO:tasks.workunit.client.0.smithi060.stdout:removed './286/30' 2024-04-14T23:54:22.475 INFO:tasks.workunit.client.0.smithi060.stdout:removed './286/33' 2024-04-14T23:54:22.475 INFO:tasks.workunit.client.0.smithi060.stdout:removed './286/23' 2024-04-14T23:54:22.475 INFO:tasks.workunit.client.0.smithi060.stdout:removed './286/35' 2024-04-14T23:54:22.476 INFO:tasks.workunit.client.0.smithi060.stdout:removed './286/26' 2024-04-14T23:54:22.476 INFO:tasks.workunit.client.0.smithi060.stdout:removed './286/37' 2024-04-14T23:54:22.476 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './286' 2024-04-14T23:54:23.522 INFO:tasks.workunit.client.0.smithi060.stdout:removed './287/21' 2024-04-14T23:54:23.522 INFO:tasks.workunit.client.0.smithi060.stdout:removed './287/27' 2024-04-14T23:54:23.522 INFO:tasks.workunit.client.0.smithi060.stdout:removed './287/31' 2024-04-14T23:54:23.522 INFO:tasks.workunit.client.0.smithi060.stdout:removed './287/36' 2024-04-14T23:54:23.522 INFO:tasks.workunit.client.0.smithi060.stdout:removed './287/20' 2024-04-14T23:54:23.522 INFO:tasks.workunit.client.0.smithi060.stdout:removed './287/39' 2024-04-14T23:54:23.523 INFO:tasks.workunit.client.0.smithi060.stdout:removed './287/34' 2024-04-14T23:54:23.523 INFO:tasks.workunit.client.0.smithi060.stdout:removed './287/38' 2024-04-14T23:54:23.523 INFO:tasks.workunit.client.0.smithi060.stdout:removed './287/28' 2024-04-14T23:54:23.523 INFO:tasks.workunit.client.0.smithi060.stdout:removed './287/24' 2024-04-14T23:54:23.523 INFO:tasks.workunit.client.0.smithi060.stdout:removed './287/29' 2024-04-14T23:54:23.523 INFO:tasks.workunit.client.0.smithi060.stdout:removed './287/32' 2024-04-14T23:54:23.523 INFO:tasks.workunit.client.0.smithi060.stdout:removed './287/22' 2024-04-14T23:54:23.523 INFO:tasks.workunit.client.0.smithi060.stdout:removed './287/25' 2024-04-14T23:54:23.523 INFO:tasks.workunit.client.0.smithi060.stdout:removed './287/30' 2024-04-14T23:54:23.523 INFO:tasks.workunit.client.0.smithi060.stdout:removed './287/33' 2024-04-14T23:54:23.523 INFO:tasks.workunit.client.0.smithi060.stdout:removed './287/23' 2024-04-14T23:54:23.523 INFO:tasks.workunit.client.0.smithi060.stdout:removed './287/35' 2024-04-14T23:54:23.523 INFO:tasks.workunit.client.0.smithi060.stdout:removed './287/26' 2024-04-14T23:54:23.523 INFO:tasks.workunit.client.0.smithi060.stdout:removed './287/37' 2024-04-14T23:54:23.524 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './287' 2024-04-14T23:54:23.524 INFO:tasks.workunit.client.0.smithi060.stdout:removed './288/21' 2024-04-14T23:54:23.524 INFO:tasks.workunit.client.0.smithi060.stdout:removed './288/27' 2024-04-14T23:54:23.524 INFO:tasks.workunit.client.0.smithi060.stdout:removed './288/31' 2024-04-14T23:54:23.524 INFO:tasks.workunit.client.0.smithi060.stdout:removed './288/36' 2024-04-14T23:54:23.524 INFO:tasks.workunit.client.0.smithi060.stdout:removed './288/20' 2024-04-14T23:54:23.524 INFO:tasks.workunit.client.0.smithi060.stdout:removed './288/39' 2024-04-14T23:54:23.524 INFO:tasks.workunit.client.0.smithi060.stdout:removed './288/34' 2024-04-14T23:54:23.524 INFO:tasks.workunit.client.0.smithi060.stdout:removed './288/38' 2024-04-14T23:54:23.524 INFO:tasks.workunit.client.0.smithi060.stdout:removed './288/28' 2024-04-14T23:54:23.524 INFO:tasks.workunit.client.0.smithi060.stdout:removed './288/24' 2024-04-14T23:54:23.524 INFO:tasks.workunit.client.0.smithi060.stdout:removed './288/29' 2024-04-14T23:54:23.525 INFO:tasks.workunit.client.0.smithi060.stdout:removed './288/32' 2024-04-14T23:54:23.525 INFO:tasks.workunit.client.0.smithi060.stdout:removed './288/22' 2024-04-14T23:54:23.525 INFO:tasks.workunit.client.0.smithi060.stdout:removed './288/25' 2024-04-14T23:54:23.525 INFO:tasks.workunit.client.0.smithi060.stdout:removed './288/30' 2024-04-14T23:54:23.525 INFO:tasks.workunit.client.0.smithi060.stdout:removed './288/33' 2024-04-14T23:54:23.525 INFO:tasks.workunit.client.0.smithi060.stdout:removed './288/23' 2024-04-14T23:54:23.525 INFO:tasks.workunit.client.0.smithi060.stdout:removed './288/35' 2024-04-14T23:54:23.525 INFO:tasks.workunit.client.0.smithi060.stdout:removed './288/26' 2024-04-14T23:54:23.525 INFO:tasks.workunit.client.0.smithi060.stdout:removed './288/37' 2024-04-14T23:54:23.525 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './288' 2024-04-14T23:54:23.525 INFO:tasks.workunit.client.0.smithi060.stdout:removed './289/21' 2024-04-14T23:54:23.525 INFO:tasks.workunit.client.0.smithi060.stdout:removed './289/27' 2024-04-14T23:54:23.525 INFO:tasks.workunit.client.0.smithi060.stdout:removed './289/31' 2024-04-14T23:54:23.525 INFO:tasks.workunit.client.0.smithi060.stdout:removed './289/36' 2024-04-14T23:54:23.526 INFO:tasks.workunit.client.0.smithi060.stdout:removed './289/20' 2024-04-14T23:54:23.526 INFO:tasks.workunit.client.0.smithi060.stdout:removed './289/39' 2024-04-14T23:54:23.526 INFO:tasks.workunit.client.0.smithi060.stdout:removed './289/34' 2024-04-14T23:54:23.526 INFO:tasks.workunit.client.0.smithi060.stdout:removed './289/38' 2024-04-14T23:54:23.526 INFO:tasks.workunit.client.0.smithi060.stdout:removed './289/28' 2024-04-14T23:54:23.526 INFO:tasks.workunit.client.0.smithi060.stdout:removed './289/24' 2024-04-14T23:54:23.526 INFO:tasks.workunit.client.0.smithi060.stdout:removed './289/29' 2024-04-14T23:54:23.526 INFO:tasks.workunit.client.0.smithi060.stdout:removed './289/32' 2024-04-14T23:54:23.526 INFO:tasks.workunit.client.0.smithi060.stdout:removed './289/22' 2024-04-14T23:54:23.526 INFO:tasks.workunit.client.0.smithi060.stdout:removed './289/25' 2024-04-14T23:54:23.526 INFO:tasks.workunit.client.0.smithi060.stdout:removed './289/30' 2024-04-14T23:54:23.527 INFO:tasks.workunit.client.0.smithi060.stdout:removed './289/33' 2024-04-14T23:54:23.527 INFO:tasks.workunit.client.0.smithi060.stdout:removed './289/23' 2024-04-14T23:54:23.527 INFO:tasks.workunit.client.0.smithi060.stdout:removed './289/35' 2024-04-14T23:54:23.527 INFO:tasks.workunit.client.0.smithi060.stdout:removed './289/26' 2024-04-14T23:54:23.527 INFO:tasks.workunit.client.0.smithi060.stdout:removed './289/37' 2024-04-14T23:54:23.527 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './289' 2024-04-14T23:54:23.527 INFO:tasks.workunit.client.0.smithi060.stdout:removed './290/21' 2024-04-14T23:54:23.527 INFO:tasks.workunit.client.0.smithi060.stdout:removed './290/27' 2024-04-14T23:54:23.527 INFO:tasks.workunit.client.0.smithi060.stdout:removed './290/31' 2024-04-14T23:54:23.527 INFO:tasks.workunit.client.0.smithi060.stdout:removed './290/36' 2024-04-14T23:54:23.527 INFO:tasks.workunit.client.0.smithi060.stdout:removed './290/20' 2024-04-14T23:54:23.527 INFO:tasks.workunit.client.0.smithi060.stdout:removed './290/39' 2024-04-14T23:54:23.527 INFO:tasks.workunit.client.0.smithi060.stdout:removed './290/34' 2024-04-14T23:54:23.528 INFO:tasks.workunit.client.0.smithi060.stdout:removed './290/38' 2024-04-14T23:54:23.528 INFO:tasks.workunit.client.0.smithi060.stdout:removed './290/28' 2024-04-14T23:54:23.528 INFO:tasks.workunit.client.0.smithi060.stdout:removed './290/24' 2024-04-14T23:54:23.528 INFO:tasks.workunit.client.0.smithi060.stdout:removed './290/29' 2024-04-14T23:54:23.528 INFO:tasks.workunit.client.0.smithi060.stdout:removed './290/32' 2024-04-14T23:54:23.528 INFO:tasks.workunit.client.0.smithi060.stdout:removed './290/22' 2024-04-14T23:54:23.528 INFO:tasks.workunit.client.0.smithi060.stdout:removed './290/25' 2024-04-14T23:54:23.528 INFO:tasks.workunit.client.0.smithi060.stdout:removed './290/30' 2024-04-14T23:54:23.528 INFO:tasks.workunit.client.0.smithi060.stdout:removed './290/33' 2024-04-14T23:54:23.528 INFO:tasks.workunit.client.0.smithi060.stdout:removed './290/23' 2024-04-14T23:54:23.528 INFO:tasks.workunit.client.0.smithi060.stdout:removed './290/35' 2024-04-14T23:54:23.528 INFO:tasks.workunit.client.0.smithi060.stdout:removed './290/26' 2024-04-14T23:54:23.529 INFO:tasks.workunit.client.0.smithi060.stdout:removed './290/37' 2024-04-14T23:54:23.529 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './290' 2024-04-14T23:54:23.529 INFO:tasks.workunit.client.0.smithi060.stdout:removed './291/21' 2024-04-14T23:54:23.529 INFO:tasks.workunit.client.0.smithi060.stdout:removed './291/27' 2024-04-14T23:54:23.529 INFO:tasks.workunit.client.0.smithi060.stdout:removed './291/31' 2024-04-14T23:54:23.529 INFO:tasks.workunit.client.0.smithi060.stdout:removed './291/36' 2024-04-14T23:54:23.529 INFO:tasks.workunit.client.0.smithi060.stdout:removed './291/20' 2024-04-14T23:54:23.529 INFO:tasks.workunit.client.0.smithi060.stdout:removed './291/39' 2024-04-14T23:54:23.529 INFO:tasks.workunit.client.0.smithi060.stdout:removed './291/34' 2024-04-14T23:54:23.529 INFO:tasks.workunit.client.0.smithi060.stdout:removed './291/38' 2024-04-14T23:54:23.529 INFO:tasks.workunit.client.0.smithi060.stdout:removed './291/28' 2024-04-14T23:54:23.529 INFO:tasks.workunit.client.0.smithi060.stdout:removed './291/24' 2024-04-14T23:54:23.529 INFO:tasks.workunit.client.0.smithi060.stdout:removed './291/29' 2024-04-14T23:54:23.529 INFO:tasks.workunit.client.0.smithi060.stdout:removed './291/32' 2024-04-14T23:54:23.529 INFO:tasks.workunit.client.0.smithi060.stdout:removed './291/22' 2024-04-14T23:54:23.530 INFO:tasks.workunit.client.0.smithi060.stdout:removed './291/25' 2024-04-14T23:54:23.530 INFO:tasks.workunit.client.0.smithi060.stdout:removed './291/30' 2024-04-14T23:54:23.530 INFO:tasks.workunit.client.0.smithi060.stdout:removed './291/33' 2024-04-14T23:54:23.530 INFO:tasks.workunit.client.0.smithi060.stdout:removed './291/23' 2024-04-14T23:54:23.530 INFO:tasks.workunit.client.0.smithi060.stdout:removed './291/35' 2024-04-14T23:54:23.530 INFO:tasks.workunit.client.0.smithi060.stdout:removed './291/26' 2024-04-14T23:54:23.530 INFO:tasks.workunit.client.0.smithi060.stdout:removed './291/37' 2024-04-14T23:54:23.530 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './291' 2024-04-14T23:54:23.530 INFO:tasks.workunit.client.0.smithi060.stdout:removed './292/21' 2024-04-14T23:54:23.530 INFO:tasks.workunit.client.0.smithi060.stdout:removed './292/27' 2024-04-14T23:54:23.530 INFO:tasks.workunit.client.0.smithi060.stdout:removed './292/31' 2024-04-14T23:54:23.530 INFO:tasks.workunit.client.0.smithi060.stdout:removed './292/36' 2024-04-14T23:54:23.530 INFO:tasks.workunit.client.0.smithi060.stdout:removed './292/20' 2024-04-14T23:54:23.530 INFO:tasks.workunit.client.0.smithi060.stdout:removed './292/39' 2024-04-14T23:54:23.530 INFO:tasks.workunit.client.0.smithi060.stdout:removed './292/34' 2024-04-14T23:54:23.531 INFO:tasks.workunit.client.0.smithi060.stdout:removed './292/38' 2024-04-14T23:54:23.531 INFO:tasks.workunit.client.0.smithi060.stdout:removed './292/28' 2024-04-14T23:54:23.531 INFO:tasks.workunit.client.0.smithi060.stdout:removed './292/24' 2024-04-14T23:54:23.531 INFO:tasks.workunit.client.0.smithi060.stdout:removed './292/29' 2024-04-14T23:54:23.531 INFO:tasks.workunit.client.0.smithi060.stdout:removed './292/32' 2024-04-14T23:54:23.531 INFO:tasks.workunit.client.0.smithi060.stdout:removed './292/22' 2024-04-14T23:54:23.531 INFO:tasks.workunit.client.0.smithi060.stdout:removed './292/25' 2024-04-14T23:54:23.531 INFO:tasks.workunit.client.0.smithi060.stdout:removed './292/30' 2024-04-14T23:54:23.531 INFO:tasks.workunit.client.0.smithi060.stdout:removed './292/33' 2024-04-14T23:54:23.531 INFO:tasks.workunit.client.0.smithi060.stdout:removed './292/23' 2024-04-14T23:54:23.531 INFO:tasks.workunit.client.0.smithi060.stdout:removed './292/35' 2024-04-14T23:54:23.531 INFO:tasks.workunit.client.0.smithi060.stdout:removed './292/26' 2024-04-14T23:54:23.531 INFO:tasks.workunit.client.0.smithi060.stdout:removed './292/37' 2024-04-14T23:54:23.531 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './292' 2024-04-14T23:54:23.531 INFO:tasks.workunit.client.0.smithi060.stdout:removed './293/21' 2024-04-14T23:54:23.532 INFO:tasks.workunit.client.0.smithi060.stdout:removed './293/27' 2024-04-14T23:54:23.532 INFO:tasks.workunit.client.0.smithi060.stdout:removed './293/31' 2024-04-14T23:54:23.532 INFO:tasks.workunit.client.0.smithi060.stdout:removed './293/36' 2024-04-14T23:54:23.532 INFO:tasks.workunit.client.0.smithi060.stdout:removed './293/20' 2024-04-14T23:54:23.532 INFO:tasks.workunit.client.0.smithi060.stdout:removed './293/39' 2024-04-14T23:54:23.532 INFO:tasks.workunit.client.0.smithi060.stdout:removed './293/34' 2024-04-14T23:54:23.532 INFO:tasks.workunit.client.0.smithi060.stdout:removed './293/38' 2024-04-14T23:54:23.532 INFO:tasks.workunit.client.0.smithi060.stdout:removed './293/28' 2024-04-14T23:54:23.532 INFO:tasks.workunit.client.0.smithi060.stdout:removed './293/24' 2024-04-14T23:54:23.532 INFO:tasks.workunit.client.0.smithi060.stdout:removed './293/29' 2024-04-14T23:54:23.532 INFO:tasks.workunit.client.0.smithi060.stdout:removed './293/32' 2024-04-14T23:54:23.532 INFO:tasks.workunit.client.0.smithi060.stdout:removed './293/22' 2024-04-14T23:54:23.532 INFO:tasks.workunit.client.0.smithi060.stdout:removed './293/25' 2024-04-14T23:54:23.532 INFO:tasks.workunit.client.0.smithi060.stdout:removed './293/30' 2024-04-14T23:54:23.533 INFO:tasks.workunit.client.0.smithi060.stdout:removed './293/33' 2024-04-14T23:54:23.533 INFO:tasks.workunit.client.0.smithi060.stdout:removed './293/23' 2024-04-14T23:54:23.533 INFO:tasks.workunit.client.0.smithi060.stdout:removed './293/35' 2024-04-14T23:54:23.533 INFO:tasks.workunit.client.0.smithi060.stdout:removed './293/26' 2024-04-14T23:54:23.533 INFO:tasks.workunit.client.0.smithi060.stdout:removed './293/37' 2024-04-14T23:54:23.533 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './293' 2024-04-14T23:54:23.533 INFO:tasks.workunit.client.0.smithi060.stdout:removed './294/21' 2024-04-14T23:54:23.533 INFO:tasks.workunit.client.0.smithi060.stdout:removed './294/27' 2024-04-14T23:54:23.533 INFO:tasks.workunit.client.0.smithi060.stdout:removed './294/31' 2024-04-14T23:54:23.533 INFO:tasks.workunit.client.0.smithi060.stdout:removed './294/36' 2024-04-14T23:54:23.533 INFO:tasks.workunit.client.0.smithi060.stdout:removed './294/20' 2024-04-14T23:54:23.533 INFO:tasks.workunit.client.0.smithi060.stdout:removed './294/39' 2024-04-14T23:54:23.533 INFO:tasks.workunit.client.0.smithi060.stdout:removed './294/34' 2024-04-14T23:54:23.533 INFO:tasks.workunit.client.0.smithi060.stdout:removed './294/38' 2024-04-14T23:54:23.533 INFO:tasks.workunit.client.0.smithi060.stdout:removed './294/28' 2024-04-14T23:54:23.534 INFO:tasks.workunit.client.0.smithi060.stdout:removed './294/24' 2024-04-14T23:54:23.534 INFO:tasks.workunit.client.0.smithi060.stdout:removed './294/29' 2024-04-14T23:54:23.534 INFO:tasks.workunit.client.0.smithi060.stdout:removed './294/32' 2024-04-14T23:54:23.534 INFO:tasks.workunit.client.0.smithi060.stdout:removed './294/22' 2024-04-14T23:54:23.534 INFO:tasks.workunit.client.0.smithi060.stdout:removed './294/25' 2024-04-14T23:54:23.534 INFO:tasks.workunit.client.0.smithi060.stdout:removed './294/30' 2024-04-14T23:54:23.534 INFO:tasks.workunit.client.0.smithi060.stdout:removed './294/33' 2024-04-14T23:54:23.534 INFO:tasks.workunit.client.0.smithi060.stdout:removed './294/23' 2024-04-14T23:54:23.534 INFO:tasks.workunit.client.0.smithi060.stdout:removed './294/35' 2024-04-14T23:54:23.534 INFO:tasks.workunit.client.0.smithi060.stdout:removed './294/26' 2024-04-14T23:54:23.534 INFO:tasks.workunit.client.0.smithi060.stdout:removed './294/37' 2024-04-14T23:54:23.534 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './294' 2024-04-14T23:54:23.534 INFO:tasks.workunit.client.0.smithi060.stdout:removed './295/21' 2024-04-14T23:54:23.535 INFO:tasks.workunit.client.0.smithi060.stdout:removed './295/27' 2024-04-14T23:54:23.535 INFO:tasks.workunit.client.0.smithi060.stdout:removed './295/31' 2024-04-14T23:54:23.535 INFO:tasks.workunit.client.0.smithi060.stdout:removed './295/36' 2024-04-14T23:54:23.535 INFO:tasks.workunit.client.0.smithi060.stdout:removed './295/20' 2024-04-14T23:54:23.535 INFO:tasks.workunit.client.0.smithi060.stdout:removed './295/39' 2024-04-14T23:54:23.535 INFO:tasks.workunit.client.0.smithi060.stdout:removed './295/34' 2024-04-14T23:54:23.535 INFO:tasks.workunit.client.0.smithi060.stdout:removed './295/38' 2024-04-14T23:54:23.535 INFO:tasks.workunit.client.0.smithi060.stdout:removed './295/28' 2024-04-14T23:54:23.535 INFO:tasks.workunit.client.0.smithi060.stdout:removed './295/24' 2024-04-14T23:54:23.535 INFO:tasks.workunit.client.0.smithi060.stdout:removed './295/29' 2024-04-14T23:54:23.535 INFO:tasks.workunit.client.0.smithi060.stdout:removed './295/32' 2024-04-14T23:54:23.535 INFO:tasks.workunit.client.0.smithi060.stdout:removed './295/22' 2024-04-14T23:54:23.535 INFO:tasks.workunit.client.0.smithi060.stdout:removed './295/25' 2024-04-14T23:54:23.535 INFO:tasks.workunit.client.0.smithi060.stdout:removed './295/30' 2024-04-14T23:54:23.535 INFO:tasks.workunit.client.0.smithi060.stdout:removed './295/33' 2024-04-14T23:54:23.536 INFO:tasks.workunit.client.0.smithi060.stdout:removed './295/23' 2024-04-14T23:54:23.536 INFO:tasks.workunit.client.0.smithi060.stdout:removed './295/35' 2024-04-14T23:54:23.536 INFO:tasks.workunit.client.0.smithi060.stdout:removed './295/26' 2024-04-14T23:54:23.536 INFO:tasks.workunit.client.0.smithi060.stdout:removed './295/37' 2024-04-14T23:54:23.536 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './295' 2024-04-14T23:54:23.536 INFO:tasks.workunit.client.0.smithi060.stdout:removed './296/21' 2024-04-14T23:54:23.536 INFO:tasks.workunit.client.0.smithi060.stdout:removed './296/27' 2024-04-14T23:54:23.536 INFO:tasks.workunit.client.0.smithi060.stdout:removed './296/31' 2024-04-14T23:54:23.536 INFO:tasks.workunit.client.0.smithi060.stdout:removed './296/36' 2024-04-14T23:54:23.536 INFO:tasks.workunit.client.0.smithi060.stdout:removed './296/20' 2024-04-14T23:54:23.536 INFO:tasks.workunit.client.0.smithi060.stdout:removed './296/39' 2024-04-14T23:54:23.536 INFO:tasks.workunit.client.0.smithi060.stdout:removed './296/34' 2024-04-14T23:54:23.536 INFO:tasks.workunit.client.0.smithi060.stdout:removed './296/38' 2024-04-14T23:54:23.536 INFO:tasks.workunit.client.0.smithi060.stdout:removed './296/28' 2024-04-14T23:54:23.536 INFO:tasks.workunit.client.0.smithi060.stdout:removed './296/24' 2024-04-14T23:54:23.537 INFO:tasks.workunit.client.0.smithi060.stdout:removed './296/29' 2024-04-14T23:54:23.537 INFO:tasks.workunit.client.0.smithi060.stdout:removed './296/32' 2024-04-14T23:54:23.537 INFO:tasks.workunit.client.0.smithi060.stdout:removed './296/22' 2024-04-14T23:54:23.537 INFO:tasks.workunit.client.0.smithi060.stdout:removed './296/25' 2024-04-14T23:54:23.537 INFO:tasks.workunit.client.0.smithi060.stdout:removed './296/30' 2024-04-14T23:54:23.537 INFO:tasks.workunit.client.0.smithi060.stdout:removed './296/33' 2024-04-14T23:54:23.537 INFO:tasks.workunit.client.0.smithi060.stdout:removed './296/23' 2024-04-14T23:54:23.537 INFO:tasks.workunit.client.0.smithi060.stdout:removed './296/35' 2024-04-14T23:54:23.537 INFO:tasks.workunit.client.0.smithi060.stdout:removed './296/26' 2024-04-14T23:54:23.537 INFO:tasks.workunit.client.0.smithi060.stdout:removed './296/37' 2024-04-14T23:54:23.537 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './296' 2024-04-14T23:54:23.537 INFO:tasks.workunit.client.0.smithi060.stdout:removed './297/21' 2024-04-14T23:54:23.537 INFO:tasks.workunit.client.0.smithi060.stdout:removed './297/27' 2024-04-14T23:54:24.989 INFO:tasks.workunit.client.0.smithi060.stdout:removed './297/31' 2024-04-14T23:54:24.989 INFO:tasks.workunit.client.0.smithi060.stdout:removed './297/36' 2024-04-14T23:54:24.989 INFO:tasks.workunit.client.0.smithi060.stdout:removed './297/20' 2024-04-14T23:54:24.989 INFO:tasks.workunit.client.0.smithi060.stdout:removed './297/39' 2024-04-14T23:54:24.989 INFO:tasks.workunit.client.0.smithi060.stdout:removed './297/34' 2024-04-14T23:54:24.989 INFO:tasks.workunit.client.0.smithi060.stdout:removed './297/38' 2024-04-14T23:54:24.989 INFO:tasks.workunit.client.0.smithi060.stdout:removed './297/28' 2024-04-14T23:54:24.989 INFO:tasks.workunit.client.0.smithi060.stdout:removed './297/24' 2024-04-14T23:54:24.989 INFO:tasks.workunit.client.0.smithi060.stdout:removed './297/29' 2024-04-14T23:54:24.989 INFO:tasks.workunit.client.0.smithi060.stdout:removed './297/32' 2024-04-14T23:54:24.989 INFO:tasks.workunit.client.0.smithi060.stdout:removed './297/22' 2024-04-14T23:54:24.990 INFO:tasks.workunit.client.0.smithi060.stdout:removed './297/25' 2024-04-14T23:54:24.990 INFO:tasks.workunit.client.0.smithi060.stdout:removed './297/30' 2024-04-14T23:54:24.990 INFO:tasks.workunit.client.0.smithi060.stdout:removed './297/33' 2024-04-14T23:54:24.990 INFO:tasks.workunit.client.0.smithi060.stdout:removed './297/23' 2024-04-14T23:54:24.990 INFO:tasks.workunit.client.0.smithi060.stdout:removed './297/35' 2024-04-14T23:54:24.990 INFO:tasks.workunit.client.0.smithi060.stdout:removed './297/26' 2024-04-14T23:54:24.990 INFO:tasks.workunit.client.0.smithi060.stdout:removed './297/37' 2024-04-14T23:54:24.990 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './297' 2024-04-14T23:54:24.990 INFO:tasks.workunit.client.0.smithi060.stdout:removed './298/21' 2024-04-14T23:54:24.990 INFO:tasks.workunit.client.0.smithi060.stdout:removed './298/27' 2024-04-14T23:54:24.990 INFO:tasks.workunit.client.0.smithi060.stdout:removed './298/31' 2024-04-14T23:54:24.990 INFO:tasks.workunit.client.0.smithi060.stdout:removed './298/36' 2024-04-14T23:54:24.990 INFO:tasks.workunit.client.0.smithi060.stdout:removed './298/20' 2024-04-14T23:54:24.990 INFO:tasks.workunit.client.0.smithi060.stdout:removed './298/39' 2024-04-14T23:54:24.990 INFO:tasks.workunit.client.0.smithi060.stdout:removed './298/34' 2024-04-14T23:54:24.991 INFO:tasks.workunit.client.0.smithi060.stdout:removed './298/38' 2024-04-14T23:54:24.991 INFO:tasks.workunit.client.0.smithi060.stdout:removed './298/28' 2024-04-14T23:54:24.991 INFO:tasks.workunit.client.0.smithi060.stdout:removed './298/24' 2024-04-14T23:54:24.991 INFO:tasks.workunit.client.0.smithi060.stdout:removed './298/29' 2024-04-14T23:54:24.991 INFO:tasks.workunit.client.0.smithi060.stdout:removed './298/32' 2024-04-14T23:54:24.991 INFO:tasks.workunit.client.0.smithi060.stdout:removed './298/22' 2024-04-14T23:54:24.991 INFO:tasks.workunit.client.0.smithi060.stdout:removed './298/25' 2024-04-14T23:54:24.991 INFO:tasks.workunit.client.0.smithi060.stdout:removed './298/30' 2024-04-14T23:54:24.991 INFO:tasks.workunit.client.0.smithi060.stdout:removed './298/33' 2024-04-14T23:54:24.991 INFO:tasks.workunit.client.0.smithi060.stdout:removed './298/23' 2024-04-14T23:54:24.991 INFO:tasks.workunit.client.0.smithi060.stdout:removed './298/35' 2024-04-14T23:54:24.991 INFO:tasks.workunit.client.0.smithi060.stdout:removed './298/26' 2024-04-14T23:54:24.991 INFO:tasks.workunit.client.0.smithi060.stdout:removed './298/37' 2024-04-14T23:54:24.991 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './298' 2024-04-14T23:54:24.991 INFO:tasks.workunit.client.0.smithi060.stdout:removed './299/21' 2024-04-14T23:54:24.991 INFO:tasks.workunit.client.0.smithi060.stdout:removed './299/27' 2024-04-14T23:54:24.991 INFO:tasks.workunit.client.0.smithi060.stdout:removed './299/31' 2024-04-14T23:54:24.992 INFO:tasks.workunit.client.0.smithi060.stdout:removed './299/36' 2024-04-14T23:54:24.992 INFO:tasks.workunit.client.0.smithi060.stdout:removed './299/20' 2024-04-14T23:54:24.992 INFO:tasks.workunit.client.0.smithi060.stdout:removed './299/39' 2024-04-14T23:54:24.992 INFO:tasks.workunit.client.0.smithi060.stdout:removed './299/34' 2024-04-14T23:54:24.992 INFO:tasks.workunit.client.0.smithi060.stdout:removed './299/38' 2024-04-14T23:54:24.992 INFO:tasks.workunit.client.0.smithi060.stdout:removed './299/28' 2024-04-14T23:54:24.992 INFO:tasks.workunit.client.0.smithi060.stdout:removed './299/24' 2024-04-14T23:54:24.992 INFO:tasks.workunit.client.0.smithi060.stdout:removed './299/29' 2024-04-14T23:54:24.992 INFO:tasks.workunit.client.0.smithi060.stdout:removed './299/32' 2024-04-14T23:54:24.992 INFO:tasks.workunit.client.0.smithi060.stdout:removed './299/22' 2024-04-14T23:54:24.992 INFO:tasks.workunit.client.0.smithi060.stdout:removed './299/25' 2024-04-14T23:54:24.992 INFO:tasks.workunit.client.0.smithi060.stdout:removed './299/30' 2024-04-14T23:54:24.992 INFO:tasks.workunit.client.0.smithi060.stdout:removed './299/33' 2024-04-14T23:54:24.992 INFO:tasks.workunit.client.0.smithi060.stdout:removed './299/23' 2024-04-14T23:54:24.992 INFO:tasks.workunit.client.0.smithi060.stdout:removed './299/35' 2024-04-14T23:54:24.993 INFO:tasks.workunit.client.0.smithi060.stdout:removed './299/26' 2024-04-14T23:54:24.993 INFO:tasks.workunit.client.0.smithi060.stdout:removed './299/37' 2024-04-14T23:54:24.993 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './299' 2024-04-14T23:54:24.993 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/166/19' 2024-04-14T23:54:24.993 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/166/15' 2024-04-14T23:54:24.993 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/166/20' 2024-04-14T23:54:24.993 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/166/12' 2024-04-14T23:54:24.993 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/166/10' 2024-04-14T23:54:24.993 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/166/11' 2024-04-14T23:54:24.993 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/166/16' 2024-04-14T23:54:24.993 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/166/14' 2024-04-14T23:54:24.993 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/166/18' 2024-04-14T23:54:24.993 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/166/13' 2024-04-14T23:54:24.993 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/166/17' 2024-04-14T23:54:24.993 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/166' 2024-04-14T23:54:24.993 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/147/19' 2024-04-14T23:54:24.994 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/147/15' 2024-04-14T23:54:24.994 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/147/20' 2024-04-14T23:54:24.994 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/147/12' 2024-04-14T23:54:24.994 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/147/10' 2024-04-14T23:54:24.994 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/147/11' 2024-04-14T23:54:24.994 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/147/16' 2024-04-14T23:54:24.994 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/147/14' 2024-04-14T23:54:24.994 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/147/18' 2024-04-14T23:54:24.994 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/147/13' 2024-04-14T23:54:24.994 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/147/17' 2024-04-14T23:54:24.994 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/147' 2024-04-14T23:54:24.994 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/151/19' 2024-04-14T23:54:24.994 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/151/15' 2024-04-14T23:54:24.994 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/151/20' 2024-04-14T23:54:24.995 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/151/12' 2024-04-14T23:54:24.995 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/151/10' 2024-04-14T23:54:24.995 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/151/11' 2024-04-14T23:54:24.995 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/151/16' 2024-04-14T23:54:24.995 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/151/14' 2024-04-14T23:54:24.995 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/151/18' 2024-04-14T23:54:24.995 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/151/13' 2024-04-14T23:54:24.995 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/151/17' 2024-04-14T23:54:24.995 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/151' 2024-04-14T23:54:24.995 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/170/19' 2024-04-14T23:54:24.995 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/170/15' 2024-04-14T23:54:24.995 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/170/20' 2024-04-14T23:54:24.995 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/170/12' 2024-04-14T23:54:24.995 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/170/10' 2024-04-14T23:54:24.995 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/170/11' 2024-04-14T23:54:24.996 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/170/16' 2024-04-14T23:54:24.996 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/170/14' 2024-04-14T23:54:24.996 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/170/18' 2024-04-14T23:54:24.996 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/170/13' 2024-04-14T23:54:24.996 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/170/17' 2024-04-14T23:54:24.996 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/170' 2024-04-14T23:54:24.996 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/167/19' 2024-04-14T23:54:24.996 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/167/15' 2024-04-14T23:54:24.996 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/167/20' 2024-04-14T23:54:24.996 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/167/12' 2024-04-14T23:54:24.996 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/167/10' 2024-04-14T23:54:24.996 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/167/11' 2024-04-14T23:54:24.996 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/167/16' 2024-04-14T23:54:24.996 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/167/14' 2024-04-14T23:54:24.996 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/167/18' 2024-04-14T23:54:24.996 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/167/13' 2024-04-14T23:54:24.996 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/167/17' 2024-04-14T23:54:24.997 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/167' 2024-04-14T23:54:24.997 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/140/19' 2024-04-14T23:54:24.997 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/140/15' 2024-04-14T23:54:24.997 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/140/20' 2024-04-14T23:54:24.997 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/140/12' 2024-04-14T23:54:24.997 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/140/10' 2024-04-14T23:54:24.997 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/140/11' 2024-04-14T23:54:24.997 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/140/16' 2024-04-14T23:54:24.997 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/140/14' 2024-04-14T23:54:24.997 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/140/18' 2024-04-14T23:54:24.997 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/140/13' 2024-04-14T23:54:24.997 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/140/17' 2024-04-14T23:54:24.997 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/140' 2024-04-14T23:54:24.997 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/142/19' 2024-04-14T23:54:24.997 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/142/15' 2024-04-14T23:54:24.998 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/142/20' 2024-04-14T23:54:24.998 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/142/12' 2024-04-14T23:54:24.998 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/142/10' 2024-04-14T23:54:27.567 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/142/11' 2024-04-14T23:54:27.567 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/142/16' 2024-04-14T23:54:27.568 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/142/14' 2024-04-14T23:54:27.568 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/142/18' 2024-04-14T23:54:27.568 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/142/13' 2024-04-14T23:54:27.568 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/142/17' 2024-04-14T23:54:27.568 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/142' 2024-04-14T23:54:27.568 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/150/19' 2024-04-14T23:54:27.568 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/150/15' 2024-04-14T23:54:27.568 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/150/20' 2024-04-14T23:54:27.568 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/150/12' 2024-04-14T23:54:27.568 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/150/10' 2024-04-14T23:54:27.568 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/150/11' 2024-04-14T23:54:27.568 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/150/16' 2024-04-14T23:54:27.568 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/150/14' 2024-04-14T23:54:27.569 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/150/18' 2024-04-14T23:54:27.569 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/150/13' 2024-04-14T23:54:27.569 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/150/17' 2024-04-14T23:54:27.569 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/150' 2024-04-14T23:54:27.569 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/105/19' 2024-04-14T23:54:27.569 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/105/15' 2024-04-14T23:54:27.569 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/105/20' 2024-04-14T23:54:27.569 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/105/12' 2024-04-14T23:54:27.569 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/105/10' 2024-04-14T23:54:27.569 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/105/11' 2024-04-14T23:54:27.569 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/105/16' 2024-04-14T23:54:27.569 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/105/14' 2024-04-14T23:54:27.569 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/105/18' 2024-04-14T23:54:27.569 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/105/13' 2024-04-14T23:54:27.569 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/105/17' 2024-04-14T23:54:27.570 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/105' 2024-04-14T23:54:27.570 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/149/19' 2024-04-14T23:54:27.570 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/149/15' 2024-04-14T23:54:27.570 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/149/20' 2024-04-14T23:54:27.570 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/149/12' 2024-04-14T23:54:27.570 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/149/10' 2024-04-14T23:54:27.570 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/149/11' 2024-04-14T23:54:27.570 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/149/16' 2024-04-14T23:54:27.570 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/149/14' 2024-04-14T23:54:27.570 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/149/18' 2024-04-14T23:54:27.570 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/149/13' 2024-04-14T23:54:27.570 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/149/17' 2024-04-14T23:54:27.570 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/149' 2024-04-14T23:54:27.570 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/109/19' 2024-04-14T23:54:27.571 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/109/15' 2024-04-14T23:54:27.571 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/109/20' 2024-04-14T23:54:27.571 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/109/12' 2024-04-14T23:54:27.571 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/109/10' 2024-04-14T23:54:27.571 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/109/11' 2024-04-14T23:54:27.571 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/109/16' 2024-04-14T23:54:27.571 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/109/14' 2024-04-14T23:54:27.571 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/109/18' 2024-04-14T23:54:27.571 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/109/13' 2024-04-14T23:54:27.571 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/109/17' 2024-04-14T23:54:27.571 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/109' 2024-04-14T23:54:27.571 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/158/19' 2024-04-14T23:54:27.571 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/158/15' 2024-04-14T23:54:27.571 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/158/20' 2024-04-14T23:54:27.571 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/158/12' 2024-04-14T23:54:27.571 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/158/10' 2024-04-14T23:54:27.572 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/158/11' 2024-04-14T23:54:27.572 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/158/16' 2024-04-14T23:54:27.572 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/158/14' 2024-04-14T23:54:27.572 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/158/18' 2024-04-14T23:54:27.572 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/158/13' 2024-04-14T23:54:27.572 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/158/17' 2024-04-14T23:54:27.572 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/158' 2024-04-14T23:54:27.572 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/106/19' 2024-04-14T23:54:27.572 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/106/15' 2024-04-14T23:54:27.572 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/106/20' 2024-04-14T23:54:27.572 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/106/12' 2024-04-14T23:54:27.572 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/106/10' 2024-04-14T23:54:27.572 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/106/11' 2024-04-14T23:54:27.572 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/106/16' 2024-04-14T23:54:27.572 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/106/14' 2024-04-14T23:54:27.573 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/106/18' 2024-04-14T23:54:27.573 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/106/13' 2024-04-14T23:54:27.573 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/106/17' 2024-04-14T23:54:27.573 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/106' 2024-04-14T23:54:27.573 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/189/19' 2024-04-14T23:54:27.573 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/189/15' 2024-04-14T23:54:27.573 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/189/20' 2024-04-14T23:54:27.573 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/189/12' 2024-04-14T23:54:27.573 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/189/10' 2024-04-14T23:54:27.573 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/189/11' 2024-04-14T23:54:27.573 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/189/16' 2024-04-14T23:54:27.573 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/189/14' 2024-04-14T23:54:27.573 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/189/18' 2024-04-14T23:54:27.573 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/189/13' 2024-04-14T23:54:27.573 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/189/17' 2024-04-14T23:54:27.574 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/189' 2024-04-14T23:54:27.574 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/133/19' 2024-04-14T23:54:27.574 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/133/15' 2024-04-14T23:54:27.574 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/133/20' 2024-04-14T23:54:27.574 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/133/12' 2024-04-14T23:54:27.574 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/133/10' 2024-04-14T23:54:27.574 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/133/11' 2024-04-14T23:54:27.574 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/133/16' 2024-04-14T23:54:27.574 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/133/14' 2024-04-14T23:54:27.574 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/133/18' 2024-04-14T23:54:27.575 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/133/13' 2024-04-14T23:54:27.575 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/133/17' 2024-04-14T23:54:27.575 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/133' 2024-04-14T23:54:27.575 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/173/19' 2024-04-14T23:54:27.575 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/173/15' 2024-04-14T23:54:27.575 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/173/20' 2024-04-14T23:54:27.575 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/173/12' 2024-04-14T23:54:27.575 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/173/10' 2024-04-14T23:54:27.575 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/173/11' 2024-04-14T23:54:29.110 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/173/16' 2024-04-14T23:54:29.110 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/173/14' 2024-04-14T23:54:29.110 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/173/18' 2024-04-14T23:54:29.110 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/173/13' 2024-04-14T23:54:29.110 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/173/17' 2024-04-14T23:54:29.110 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/173' 2024-04-14T23:54:29.110 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/174/19' 2024-04-14T23:54:29.111 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/174/15' 2024-04-14T23:54:29.111 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/174/20' 2024-04-14T23:54:29.111 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/174/12' 2024-04-14T23:54:29.111 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/174/10' 2024-04-14T23:54:29.111 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/174/11' 2024-04-14T23:54:29.111 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/174/16' 2024-04-14T23:54:29.111 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/174/14' 2024-04-14T23:54:29.111 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/174/18' 2024-04-14T23:54:29.111 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/174/13' 2024-04-14T23:54:29.111 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/174/17' 2024-04-14T23:54:29.111 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/174' 2024-04-14T23:54:29.111 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/155/19' 2024-04-14T23:54:29.111 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/155/15' 2024-04-14T23:54:29.111 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/155/20' 2024-04-14T23:54:29.111 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/155/12' 2024-04-14T23:54:29.111 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/155/10' 2024-04-14T23:54:29.112 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/155/11' 2024-04-14T23:54:29.112 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/155/16' 2024-04-14T23:54:29.112 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/155/14' 2024-04-14T23:54:29.112 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/155/18' 2024-04-14T23:54:29.112 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/155/13' 2024-04-14T23:54:29.112 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/155/17' 2024-04-14T23:54:29.112 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/155' 2024-04-14T23:54:29.112 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/114/19' 2024-04-14T23:54:29.112 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/114/15' 2024-04-14T23:54:29.112 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/114/20' 2024-04-14T23:54:29.112 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/114/12' 2024-04-14T23:54:29.112 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/114/10' 2024-04-14T23:54:29.112 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/114/11' 2024-04-14T23:54:29.113 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/114/16' 2024-04-14T23:54:29.113 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/114/14' 2024-04-14T23:54:29.113 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/114/18' 2024-04-14T23:54:29.113 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/114/13' 2024-04-14T23:54:29.113 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/114/17' 2024-04-14T23:54:29.113 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/114' 2024-04-14T23:54:29.113 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/159/19' 2024-04-14T23:54:29.113 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/159/15' 2024-04-14T23:54:29.113 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/159/20' 2024-04-14T23:54:29.113 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/159/12' 2024-04-14T23:54:29.113 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/159/10' 2024-04-14T23:54:29.113 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/159/11' 2024-04-14T23:54:29.113 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/159/16' 2024-04-14T23:54:29.113 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/159/14' 2024-04-14T23:54:29.114 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/159/18' 2024-04-14T23:54:29.114 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/159/13' 2024-04-14T23:54:29.114 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/159/17' 2024-04-14T23:54:29.114 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/159' 2024-04-14T23:54:29.114 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/187/19' 2024-04-14T23:54:29.114 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/187/15' 2024-04-14T23:54:29.114 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/187/20' 2024-04-14T23:54:29.114 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/187/12' 2024-04-14T23:54:29.114 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/187/10' 2024-04-14T23:54:29.114 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/187/11' 2024-04-14T23:54:29.114 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/187/16' 2024-04-14T23:54:29.114 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/187/14' 2024-04-14T23:54:29.114 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/187/18' 2024-04-14T23:54:29.114 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/187/13' 2024-04-14T23:54:29.115 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/187/17' 2024-04-14T23:54:29.115 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/187' 2024-04-14T23:54:29.115 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/185/19' 2024-04-14T23:54:29.115 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/185/15' 2024-04-14T23:54:29.115 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/185/20' 2024-04-14T23:54:29.115 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/185/12' 2024-04-14T23:54:29.115 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/185/10' 2024-04-14T23:54:29.115 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/185/11' 2024-04-14T23:54:29.115 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/185/16' 2024-04-14T23:54:29.115 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/185/14' 2024-04-14T23:54:29.115 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/185/18' 2024-04-14T23:54:29.115 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/185/13' 2024-04-14T23:54:29.115 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/185/17' 2024-04-14T23:54:29.116 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/185' 2024-04-14T23:54:29.116 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/120/19' 2024-04-14T23:54:29.116 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/120/15' 2024-04-14T23:54:29.116 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/120/20' 2024-04-14T23:54:29.116 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/120/12' 2024-04-14T23:54:29.116 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/120/10' 2024-04-14T23:54:29.116 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/120/11' 2024-04-14T23:54:29.116 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/120/16' 2024-04-14T23:54:29.116 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/120/14' 2024-04-14T23:54:29.116 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/120/18' 2024-04-14T23:54:29.116 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/120/13' 2024-04-14T23:54:29.116 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/120/17' 2024-04-14T23:54:29.116 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/120' 2024-04-14T23:54:29.116 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/110/19' 2024-04-14T23:54:29.117 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/110/15' 2024-04-14T23:54:29.117 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/110/20' 2024-04-14T23:54:29.117 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/110/12' 2024-04-14T23:54:29.117 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/110/10' 2024-04-14T23:54:29.117 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/110/11' 2024-04-14T23:54:29.117 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/110/16' 2024-04-14T23:54:29.117 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/110/14' 2024-04-14T23:54:29.117 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/110/18' 2024-04-14T23:54:29.117 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/110/13' 2024-04-14T23:54:29.117 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/110/17' 2024-04-14T23:54:29.117 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/110' 2024-04-14T23:54:29.117 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/137/19' 2024-04-14T23:54:29.117 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/137/15' 2024-04-14T23:54:29.117 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/137/20' 2024-04-14T23:54:29.117 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/137/12' 2024-04-14T23:54:29.118 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/137/10' 2024-04-14T23:54:29.118 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/137/11' 2024-04-14T23:54:29.118 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/137/16' 2024-04-14T23:54:29.723 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:54:29.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:29.936+0000 7fe7b3e00640 1 Processor -- start 2024-04-14T23:54:29.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:29.937+0000 7fe7b3e00640 1 -- start start 2024-04-14T23:54:29.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:29.938+0000 7fe7b3e00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe7b4154d50 0x7fe7b415b640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:29.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:29.938+0000 7fe7b3e00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7b406c450 con 0x7fe7b4071290 2024-04-14T23:54:29.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:29.938+0000 7fe7b3e00640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7b406c730 con 0x7fe7b40c3b20 2024-04-14T23:54:29.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:29.938+0000 7fe7b3e00640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fe7b40c9110 con 0x7fe7b4154d50 2024-04-14T23:54:29.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:29.939+0000 7fe7b2a00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe7b4154d50 0x7fe7b415b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:29.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:29.939+0000 7fe7b2000640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fe7b4071290 0x7fe7b40ce380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:40010/0 (socket says 172.21.15.60:40010) 2024-04-14T23:54:29.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:29.939+0000 7fe7b2000640 1 -- 172.21.15.60:0/3747064310 learned_addr learned my addr 172.21.15.60:0/3747064310 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:54:29.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:29.939+0000 7fe7b2a00640 1 -- 172.21.15.60:0/3747064310 >> v1:172.21.15.186:6789/0 conn(0x7fe7b40c3b20 legacy=0x7fe7b40c3f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:29.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:29.939+0000 7fe7b2a00640 1 -- 172.21.15.60:0/3747064310 >> v1:172.21.15.60:6789/0 conn(0x7fe7b4071290 legacy=0x7fe7b40ce380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:29.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:29.939+0000 7fe7b2a00640 1 -- 172.21.15.60:0/3747064310 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7b406ca10 con 0x7fe7b4154d50 2024-04-14T23:54:29.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:29.939+0000 7fe7b1600640 1 -- 172.21.15.60:0/3747064310 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1794289607 0 0) 0x7fe7b406c450 con 0x7fe7b4071290 2024-04-14T23:54:29.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:29.940+0000 7fe7b2a00640 1 --2- 172.21.15.60:0/3747064310 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe7b4154d50 0x7fe7b415b640 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fe7a40097a0 tx=0x7fe7a402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ea38cf14c681c74 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:54:29.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:29.941+0000 7fe7b1600640 1 -- 172.21.15.60:0/3747064310 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7a4034070 con 0x7fe7b4154d50 2024-04-14T23:54:29.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:29.941+0000 7fe7b1600640 1 -- 172.21.15.60:0/3747064310 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7a4030610 con 0x7fe7b4154d50 2024-04-14T23:54:29.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:29.941+0000 7fe7b1600640 1 -- 172.21.15.60:0/3747064310 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7a40309e0 con 0x7fe7b4154d50 2024-04-14T23:54:29.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:29.941+0000 7fe7b3e00640 1 -- 172.21.15.60:0/3747064310 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe7b4154d50 msgr2=0x7fe7b415b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:29.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:29.941+0000 7fe7b3e00640 1 --2- 172.21.15.60:0/3747064310 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe7b4154d50 0x7fe7b415b640 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fe7a40097a0 tx=0x7fe7a402b960 comp rx=0 tx=0).stop 2024-04-14T23:54:29.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:29.942+0000 7fe7b3e00640 1 -- 172.21.15.60:0/3747064310 shutdown_connections 2024-04-14T23:54:29.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:29.942+0000 7fe7b3e00640 1 --2- 172.21.15.60:0/3747064310 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe7b4154d50 0x7fe7b415b640 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:29.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:29.942+0000 7fe7b3e00640 1 -- 172.21.15.60:0/3747064310 >> 172.21.15.60:0/3747064310 conn(0x7fe7b4066f10 msgr2=0x7fe7b4067320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:54:29.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:29.942+0000 7fe7b3e00640 1 -- 172.21.15.60:0/3747064310 shutdown_connections 2024-04-14T23:54:29.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:29.942+0000 7fe7b3e00640 1 -- 172.21.15.60:0/3747064310 wait complete. 2024-04-14T23:54:29.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:29.944+0000 7fe7b3e00640 1 Processor -- start 2024-04-14T23:54:29.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:29.944+0000 7fe7b3e00640 1 -- start start 2024-04-14T23:54:29.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:29.945+0000 7fe7b3e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe7b4071290 0x7fe7b41f1a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:29.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:29.945+0000 7fe7b3e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe7b40c3b20 0x7fe7b40c6b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:29.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:29.945+0000 7fe7b3e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe7b4154d50 0x7fe7b41f1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:29.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:29.945+0000 7fe7b3e00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fe7b406d2b0 con 0x7fe7b4154d50 2024-04-14T23:54:29.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:29.945+0000 7fe7b3e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe7b406cfd0 con 0x7fe7b4071290 2024-04-14T23:54:29.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:29.945+0000 7fe7b3e00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fe7b4155130 con 0x7fe7b40c3b20 2024-04-14T23:54:29.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:29.945+0000 7fe7b2a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe7b4071290 0x7fe7b41f1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:29.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:29.945+0000 7fe7b2a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe7b4071290 0x7fe7b41f1a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:33914/0 (socket says 172.21.15.60:33914) 2024-04-14T23:54:29.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:29.945+0000 7fe7b2a00640 1 -- 172.21.15.60:0/2048318463 learned_addr learned my addr 172.21.15.60:0/2048318463 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:54:29.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:29.945+0000 7fe7b3400640 1 --2- 172.21.15.60:0/2048318463 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe7b4154d50 0x7fe7b41f1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:29.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:29.946+0000 7fe7b2000640 1 --2- 172.21.15.60:0/2048318463 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe7b40c3b20 0x7fe7b40c6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:29.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:29.946+0000 7fe7b2a00640 1 -- 172.21.15.60:0/2048318463 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe7b40c3b20 msgr2=0x7fe7b40c6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:29.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:29.946+0000 7fe7b2a00640 1 --2- 172.21.15.60:0/2048318463 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe7b40c3b20 0x7fe7b40c6b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:29.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:29.946+0000 7fe7b2a00640 1 -- 172.21.15.60:0/2048318463 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe7b4154d50 msgr2=0x7fe7b41f1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:29.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:29.946+0000 7fe7b2a00640 1 --2- 172.21.15.60:0/2048318463 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe7b4154d50 0x7fe7b41f1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:29.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:29.946+0000 7fe7b2a00640 1 -- 172.21.15.60:0/2048318463 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7b40c5ba0 con 0x7fe7b4071290 2024-04-14T23:54:29.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:29.946+0000 7fe7b3400640 1 --2- 172.21.15.60:0/2048318463 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe7b4154d50 0x7fe7b41f1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:54:29.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:29.946+0000 7fe7b2a00640 1 --2- 172.21.15.60:0/2048318463 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe7b4071290 0x7fe7b41f1a90 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe7a4000c00 tx=0x7fe7a40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:54:29.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:29.947+0000 7fe7aaa00640 1 -- 172.21.15.60:0/2048318463 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7a4047070 con 0x7fe7b4071290 2024-04-14T23:54:29.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:29.947+0000 7fe7aaa00640 1 -- 172.21.15.60:0/2048318463 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7a4004030 con 0x7fe7b4071290 2024-04-14T23:54:29.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:29.947+0000 7fe7aaa00640 1 -- 172.21.15.60:0/2048318463 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7a4034040 con 0x7fe7b4071290 2024-04-14T23:54:29.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:29.947+0000 7fe7b3e00640 1 -- 172.21.15.60:0/2048318463 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7b40c5e80 con 0x7fe7b4071290 2024-04-14T23:54:29.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:29.947+0000 7fe7b3e00640 1 -- 172.21.15.60:0/2048318463 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7b41f0c10 con 0x7fe7b4071290 2024-04-14T23:54:29.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:29.948+0000 7fe7b3e00640 1 -- 172.21.15.60:0/2048318463 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7b4076970 con 0x7fe7b4071290 2024-04-14T23:54:29.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:29.949+0000 7fe7aaa00640 1 -- 172.21.15.60:0/2048318463 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7fe7a404a070 con 0x7fe7b4071290 2024-04-14T23:54:29.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:29.950+0000 7fe7aaa00640 1 --2- 172.21.15.60:0/2048318463 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fe79006b990 0x7fe79006de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:29.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:29.950+0000 7fe7aaa00640 1 -- 172.21.15.60:0/2048318463 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 6025+0+0 (secure 0 0 0) 0x7fe7a40ac500 con 0x7fe7b4071290 2024-04-14T23:54:29.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:29.950+0000 7fe7b2000640 1 --2- 172.21.15.60:0/2048318463 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fe79006b990 0x7fe79006de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:29.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:29.951+0000 7fe7b2000640 1 --2- 172.21.15.60:0/2048318463 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fe79006b990 0x7fe79006de50 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe79c001920 tx=0x7fe79c0018b0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:54:29.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:29.955+0000 7fe7aaa00640 1 -- 172.21.15.60:0/2048318463 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe7b4076970 con 0x7fe7b4071290 2024-04-14T23:54:30.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.196+0000 7fe7b3e00640 1 -- 172.21.15.60:0/2048318463 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe7b404f540 con 0x7fe7b4071290 2024-04-14T23:54:30.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.198+0000 7fe7aaa00640 1 -- 172.21.15.60:0/2048318463 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6839 (secure 0 0 0) 0x7fe7a407a850 con 0x7fe7b4071290 2024-04-14T23:54:30.203 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:54:30.203 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4986,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/1971580590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1971580590},{"type":"v1","addr":"172.21.15.60:6833","nonce":1971580590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:52:53.627240+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4839},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4839":{"gid":4839,"name":"c","rank":0,"incarnation":15,"state":"up:active","state_seq":24,"addr":"172.21.15.60:6835/1714133698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1714133698},{"type":"v1","addr":"172.21.15.60:6835","nonce":1714133698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4839,"qdb_cluster":[4839]},"id":1}]} 2024-04-14T23:54:30.204 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 20 2024-04-14T23:54:30.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.201+0000 7fe7b3e00640 1 -- 172.21.15.60:0/2048318463 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fe79006b990 msgr2=0x7fe79006de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:30.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.201+0000 7fe7b3e00640 1 --2- 172.21.15.60:0/2048318463 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fe79006b990 0x7fe79006de50 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe79c001920 tx=0x7fe79c0018b0 comp rx=0 tx=0).stop 2024-04-14T23:54:30.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.201+0000 7fe7b3e00640 1 -- 172.21.15.60:0/2048318463 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe7b4071290 msgr2=0x7fe7b41f1a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:30.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.201+0000 7fe7b3e00640 1 --2- 172.21.15.60:0/2048318463 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe7b4071290 0x7fe7b41f1a90 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe7a4000c00 tx=0x7fe7a40046c0 comp rx=0 tx=0).stop 2024-04-14T23:54:30.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.201+0000 7fe7b3e00640 1 -- 172.21.15.60:0/2048318463 shutdown_connections 2024-04-14T23:54:30.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.201+0000 7fe7b3e00640 1 --2- 172.21.15.60:0/2048318463 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fe79006b990 0x7fe79006de50 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:30.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.201+0000 7fe7b3e00640 1 --2- 172.21.15.60:0/2048318463 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe7b4154d50 0x7fe7b41f1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:30.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.201+0000 7fe7b3e00640 1 --2- 172.21.15.60:0/2048318463 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe7b40c3b20 0x7fe7b40c6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:30.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.201+0000 7fe7b3e00640 1 --2- 172.21.15.60:0/2048318463 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe7b4071290 0x7fe7b41f1a90 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:30.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.201+0000 7fe7b3e00640 1 -- 172.21.15.60:0/2048318463 >> 172.21.15.60:0/2048318463 conn(0x7fe7b4066f10 msgr2=0x7fe7b40c4f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:54:30.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.201+0000 7fe7b3e00640 1 -- 172.21.15.60:0/2048318463 shutdown_connections 2024-04-14T23:54:30.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.201+0000 7fe7b3e00640 1 -- 172.21.15.60:0/2048318463 wait complete. 2024-04-14T23:54:30.226 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=0) 2024-04-14T23:54:30.226 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-14T23:54:30.227 INFO:teuthology.orchestra.run:waiting for 300 2024-04-14T23:54:30.259 INFO:tasks.ceph.mds.c:Stopped 2024-04-14T23:54:30.259 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:54:30.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.523+0000 7f73e2e00640 1 Processor -- start 2024-04-14T23:54:30.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.523+0000 7f73e2e00640 1 -- start start 2024-04-14T23:54:30.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.525+0000 7f73e2e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73dc154b20 0x7f73dc0cbf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:30.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.525+0000 7f73e2e00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f73dc15e930 0x7f73dc15ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:30.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.525+0000 7f73e2e00640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73dc06cda0 con 0x7f73dc0cc720 2024-04-14T23:54:30.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.525+0000 7f73e2e00640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f73dc06d2f0 con 0x7f73dc15e930 2024-04-14T23:54:30.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.525+0000 7f73e2e00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f73dc0cbc20 con 0x7f73dc154b20 2024-04-14T23:54:30.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.526+0000 7f73e2400640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f73dc15e930 0x7f73dc15ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:30.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.525+0000 7f73e1a00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73dc154b20 0x7f73dc0cbf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:30.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.526+0000 7f73e2400640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f73dc15e930 0x7f73dc15ad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:33918/0 (socket says 172.21.15.60:33918) 2024-04-14T23:54:30.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.526+0000 7f73e2400640 1 -- 172.21.15.60:0/2844862128 learned_addr learned my addr 172.21.15.60:0/2844862128 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:54:30.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.526+0000 7f73e1000640 1 --1- 172.21.15.60:0/2844862128 >> v1:172.21.15.186:6789/0 conn(0x7f73dc0cc720 0x7f73dc0d0230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.60:44448/0 (socket says 172.21.15.60:44448) 2024-04-14T23:54:30.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.526+0000 7f73e2400640 1 -- 172.21.15.60:0/2844862128 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73dc154b20 msgr2=0x7f73dc0cbf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:54:30.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.526+0000 7f73e2400640 1 --2- 172.21.15.60:0/2844862128 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73dc154b20 0x7f73dc0cbf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:30.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.526+0000 7f73e2400640 1 -- 172.21.15.60:0/2844862128 >> v1:172.21.15.186:6789/0 conn(0x7f73dc0cc720 legacy=0x7f73dc0d0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:30.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.526+0000 7f73e2400640 1 -- 172.21.15.60:0/2844862128 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73dc06d010 con 0x7f73dc15e930 2024-04-14T23:54:30.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.527+0000 7f73e2400640 1 --2- 172.21.15.60:0/2844862128 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73dc15e930 0x7f73dc15ad20 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f73d4009870 tx=0x7f73d402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c4c1ae617c3ba6ff server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:54:30.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.527+0000 7f73dbe00640 1 -- 172.21.15.60:0/2844862128 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73d4034070 con 0x7f73dc15e930 2024-04-14T23:54:30.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.528+0000 7f73dbe00640 1 -- 172.21.15.60:0/2844862128 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73d4030840 con 0x7f73dc15e930 2024-04-14T23:54:30.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.528+0000 7f73dbe00640 1 -- 172.21.15.60:0/2844862128 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73d4030c10 con 0x7f73dc15e930 2024-04-14T23:54:30.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.528+0000 7f73e2e00640 1 -- 172.21.15.60:0/2844862128 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73dc15e930 msgr2=0x7f73dc15ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:30.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.528+0000 7f73e2e00640 1 --2- 172.21.15.60:0/2844862128 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73dc15e930 0x7f73dc15ad20 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f73d4009870 tx=0x7f73d402b980 comp rx=0 tx=0).stop 2024-04-14T23:54:30.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.528+0000 7f73e2e00640 1 -- 172.21.15.60:0/2844862128 shutdown_connections 2024-04-14T23:54:30.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.528+0000 7f73e2e00640 1 --2- 172.21.15.60:0/2844862128 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73dc15e930 0x7f73dc15ad20 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:30.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.528+0000 7f73e2e00640 1 --2- 172.21.15.60:0/2844862128 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73dc154b20 0x7f73dc0cbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:30.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.528+0000 7f73e2e00640 1 -- 172.21.15.60:0/2844862128 >> 172.21.15.60:0/2844862128 conn(0x7f73dc06ddb0 msgr2=0x7f73dc066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:54:30.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.528+0000 7f73e2e00640 1 -- 172.21.15.60:0/2844862128 shutdown_connections 2024-04-14T23:54:30.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.529+0000 7f73e2e00640 1 -- 172.21.15.60:0/2844862128 wait complete. 2024-04-14T23:54:30.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.530+0000 7f73e2e00640 1 Processor -- start 2024-04-14T23:54:30.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.530+0000 7f73e2e00640 1 -- start start 2024-04-14T23:54:30.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.531+0000 7f73e2e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73dc0cc720 0x7f73dc0c5e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:30.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.531+0000 7f73e2e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73dc154b20 0x7f73dc155d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:30.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.532+0000 7f73e2e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73dc15e930 0x7f73dc14abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:30.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.532+0000 7f73e2e00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f73dc155080 con 0x7f73dc0cc720 2024-04-14T23:54:30.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.532+0000 7f73e2e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f73dc154f00 con 0x7f73dc154b20 2024-04-14T23:54:30.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.532+0000 7f73e2e00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f73dc0d0940 con 0x7f73dc15e930 2024-04-14T23:54:30.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.532+0000 7f73e1a00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73dc0cc720 0x7f73dc0c5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:30.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.532+0000 7f73e1a00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73dc0cc720 0x7f73dc0c5e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37318/0 (socket says 172.21.15.60:37318) 2024-04-14T23:54:30.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.532+0000 7f73e1a00640 1 -- 172.21.15.60:0/3304227019 learned_addr learned my addr 172.21.15.60:0/3304227019 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:54:30.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.532+0000 7f73e1000640 1 --2- 172.21.15.60:0/3304227019 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73dc154b20 0x7f73dc155d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:30.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.532+0000 7f73e2400640 1 --2- 172.21.15.60:0/3304227019 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73dc15e930 0x7f73dc14abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:30.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.532+0000 7f73e1a00640 1 -- 172.21.15.60:0/3304227019 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73dc15e930 msgr2=0x7f73dc14abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:30.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.532+0000 7f73e1a00640 1 --2- 172.21.15.60:0/3304227019 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73dc15e930 0x7f73dc14abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:30.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.533+0000 7f73e1a00640 1 -- 172.21.15.60:0/3304227019 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73dc154b20 msgr2=0x7f73dc155d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:30.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.533+0000 7f73e1a00640 1 --2- 172.21.15.60:0/3304227019 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73dc154b20 0x7f73dc155d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:30.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.533+0000 7f73e1a00640 1 -- 172.21.15.60:0/3304227019 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73dc15a030 con 0x7f73dc0cc720 2024-04-14T23:54:30.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.533+0000 7f73e2400640 1 --2- 172.21.15.60:0/3304227019 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73dc15e930 0x7f73dc14abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:54:30.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.533+0000 7f73e1000640 1 --2- 172.21.15.60:0/3304227019 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73dc154b20 0x7f73dc155d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:54:30.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.533+0000 7f73e1a00640 1 --2- 172.21.15.60:0/3304227019 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73dc0cc720 0x7f73dc0c5e30 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f73d000e8a0 tx=0x7f73d000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:54:30.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.533+0000 7f73d9600640 1 -- 172.21.15.60:0/3304227019 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73d0017070 con 0x7f73dc0cc720 2024-04-14T23:54:30.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.533+0000 7f73d9600640 1 -- 172.21.15.60:0/3304227019 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73d000ee90 con 0x7f73dc0cc720 2024-04-14T23:54:30.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.533+0000 7f73e2e00640 1 -- 172.21.15.60:0/3304227019 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73dc156ae0 con 0x7f73dc0cc720 2024-04-14T23:54:30.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.533+0000 7f73d9600640 1 -- 172.21.15.60:0/3304227019 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73d000cb00 con 0x7f73dc0cc720 2024-04-14T23:54:30.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.533+0000 7f73e2e00640 1 -- 172.21.15.60:0/3304227019 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73dc156d80 con 0x7f73dc0cc720 2024-04-14T23:54:30.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.535+0000 7f73e2e00640 1 -- 172.21.15.60:0/3304227019 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73dc04f540 con 0x7f73dc0cc720 2024-04-14T23:54:30.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.535+0000 7f73d9600640 1 -- 172.21.15.60:0/3304227019 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f73d001a070 con 0x7f73dc0cc720 2024-04-14T23:54:30.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.537+0000 7f73d9600640 1 --2- 172.21.15.60:0/3304227019 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f73c006b990 0x7f73c006de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:30.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.537+0000 7f73d9600640 1 -- 172.21.15.60:0/3304227019 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 6025+0+0 (secure 0 0 0) 0x7f73d008aa70 con 0x7f73dc0cc720 2024-04-14T23:54:30.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.537+0000 7f73e1000640 1 --2- 172.21.15.60:0/3304227019 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f73c006b990 0x7f73c006de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:30.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.538+0000 7f73e1000640 1 --2- 172.21.15.60:0/3304227019 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f73c006b990 0x7f73c006de50 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f73dc0b35b0 tx=0x7f73cc001630 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:54:30.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.541+0000 7f73d9600640 1 -- 172.21.15.60:0/3304227019 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f73d0058ee0 con 0x7f73dc0cc720 2024-04-14T23:54:30.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.783+0000 7f73e2e00640 1 -- 172.21.15.60:0/3304227019 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f73dc1583a0 con 0x7f73dc0cc720 2024-04-14T23:54:30.790 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.784+0000 7f73d9600640 1 -- 172.21.15.60:0/3304227019 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6839 (secure 0 0 0) 0x7f73d005cf80 con 0x7f73dc0cc720 2024-04-14T23:54:30.790 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:54:30.790 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4986,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/1971580590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1971580590},{"type":"v1","addr":"172.21.15.60:6833","nonce":1971580590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:52:53.627240+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4839},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4839":{"gid":4839,"name":"c","rank":0,"incarnation":15,"state":"up:active","state_seq":24,"addr":"172.21.15.60:6835/1714133698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1714133698},{"type":"v1","addr":"172.21.15.60:6835","nonce":1714133698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4839,"qdb_cluster":[4839]},"id":1}]} 2024-04-14T23:54:30.790 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 20 2024-04-14T23:54:30.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.787+0000 7f73e2e00640 1 -- 172.21.15.60:0/3304227019 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f73c006b990 msgr2=0x7f73c006de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:30.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.787+0000 7f73e2e00640 1 --2- 172.21.15.60:0/3304227019 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f73c006b990 0x7f73c006de50 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f73dc0b35b0 tx=0x7f73cc001630 comp rx=0 tx=0).stop 2024-04-14T23:54:30.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.787+0000 7f73e2e00640 1 -- 172.21.15.60:0/3304227019 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73dc0cc720 msgr2=0x7f73dc0c5e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:30.793 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.787+0000 7f73e2e00640 1 --2- 172.21.15.60:0/3304227019 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73dc0cc720 0x7f73dc0c5e30 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f73d000e8a0 tx=0x7f73d000ed60 comp rx=0 tx=0).stop 2024-04-14T23:54:30.793 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.788+0000 7f73e2e00640 1 -- 172.21.15.60:0/3304227019 shutdown_connections 2024-04-14T23:54:30.793 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.788+0000 7f73e2e00640 1 --2- 172.21.15.60:0/3304227019 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f73c006b990 0x7f73c006de50 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:30.793 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.788+0000 7f73e2e00640 1 --2- 172.21.15.60:0/3304227019 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73dc15e930 0x7f73dc14abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:30.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.788+0000 7f73e2e00640 1 --2- 172.21.15.60:0/3304227019 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73dc154b20 0x7f73dc155d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:30.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.788+0000 7f73e2e00640 1 --2- 172.21.15.60:0/3304227019 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73dc0cc720 0x7f73dc0c5e30 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:30.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.788+0000 7f73e2e00640 1 -- 172.21.15.60:0/3304227019 >> 172.21.15.60:0/3304227019 conn(0x7f73dc06ddb0 msgr2=0x7f73dc1522f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:54:30.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.788+0000 7f73e2e00640 1 -- 172.21.15.60:0/3304227019 shutdown_connections 2024-04-14T23:54:30.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:30.788+0000 7f73e2e00640 1 -- 172.21.15.60:0/3304227019 wait complete. 2024-04-14T23:54:30.811 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-14T23:54:32.812 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:54:33.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.041+0000 7fbabb200640 1 Processor -- start 2024-04-14T23:54:33.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.041+0000 7fbabb200640 1 -- start start 2024-04-14T23:54:33.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.043+0000 7fbabb200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbabc0b1e00 0x7fbabc0c6950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:33.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.043+0000 7fbabb200640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fbabc073320 0x7fbabc0c7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:33.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.043+0000 7fbabb200640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbabc152fb0 con 0x7fbabc0c84e0 2024-04-14T23:54:33.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.043+0000 7fbabb200640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fbabc0c6540 con 0x7fbabc073320 2024-04-14T23:54:33.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.043+0000 7fbabb200640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fbabc0b21e0 con 0x7fbabc0b1e00 2024-04-14T23:54:33.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.043+0000 7fbaba800640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fbabc0c84e0 0x7fbabc0c3d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54938/0 (socket says 172.21.15.60:54938) 2024-04-14T23:54:33.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.043+0000 7fbaba800640 1 -- 172.21.15.60:0/3648165476 learned_addr learned my addr 172.21.15.60:0/3648165476 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:54:33.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.043+0000 7fbab9400640 1 --2- 172.21.15.60:0/3648165476 >> v2:172.21.15.186:3300/0 conn(0x7fbabc073320 0x7fbabc0c7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:33.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.043+0000 7fbab9e00640 1 --2- 172.21.15.60:0/3648165476 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbabc0b1e00 0x7fbabc0c6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:33.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.044+0000 7fbab8a00640 1 -- 172.21.15.60:0/3648165476 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2446458106 0 0) 0x7fbabc152fb0 con 0x7fbabc0c84e0 2024-04-14T23:54:33.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.044+0000 7fbab8a00640 1 -- 172.21.15.60:0/3648165476 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbaa0003520 con 0x7fbabc0c84e0 2024-04-14T23:54:33.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.044+0000 7fbab9e00640 1 -- 172.21.15.60:0/3648165476 >> v2:172.21.15.186:3300/0 conn(0x7fbabc073320 msgr2=0x7fbabc0c7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:54:33.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.044+0000 7fbab9e00640 1 --2- 172.21.15.60:0/3648165476 >> v2:172.21.15.186:3300/0 conn(0x7fbabc073320 0x7fbabc0c7fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:33.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.044+0000 7fbab9e00640 1 -- 172.21.15.60:0/3648165476 >> v1:172.21.15.60:6789/0 conn(0x7fbabc0c84e0 legacy=0x7fbabc0c3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:33.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.044+0000 7fbab9e00640 1 -- 172.21.15.60:0/3648165476 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbabc159310 con 0x7fbabc0b1e00 2024-04-14T23:54:33.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.044+0000 7fbab9400640 1 --2- 172.21.15.60:0/3648165476 >> v2:172.21.15.186:3300/0 conn(0x7fbabc073320 0x7fbabc0c7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:54:33.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.045+0000 7fbab9e00640 1 --2- 172.21.15.60:0/3648165476 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbabc0b1e00 0x7fbabc0c6950 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fbaac0097a0 tx=0x7fbaac02b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d2056747f96d1abb server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:54:33.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.045+0000 7fbab8a00640 1 -- 172.21.15.60:0/3648165476 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaac034070 con 0x7fbabc0b1e00 2024-04-14T23:54:33.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.045+0000 7fbab8a00640 1 -- 172.21.15.60:0/3648165476 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbaac030800 con 0x7fbabc0b1e00 2024-04-14T23:54:33.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.045+0000 7fbab8a00640 1 -- 172.21.15.60:0/3648165476 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaac030bd0 con 0x7fbabc0b1e00 2024-04-14T23:54:33.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.046+0000 7fbabb200640 1 -- 172.21.15.60:0/3648165476 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbabc0b1e00 msgr2=0x7fbabc0c6950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:33.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.046+0000 7fbabb200640 1 --2- 172.21.15.60:0/3648165476 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbabc0b1e00 0x7fbabc0c6950 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fbaac0097a0 tx=0x7fbaac02b940 comp rx=0 tx=0).stop 2024-04-14T23:54:33.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.046+0000 7fbabb200640 1 -- 172.21.15.60:0/3648165476 shutdown_connections 2024-04-14T23:54:33.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.046+0000 7fbabb200640 1 --2- 172.21.15.60:0/3648165476 >> v2:172.21.15.186:3300/0 conn(0x7fbabc073320 0x7fbabc0c7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:33.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.046+0000 7fbabb200640 1 --2- 172.21.15.60:0/3648165476 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbabc0b1e00 0x7fbabc0c6950 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:33.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.046+0000 7fbabb200640 1 -- 172.21.15.60:0/3648165476 >> 172.21.15.60:0/3648165476 conn(0x7fbabc0c8b50 msgr2=0x7fbabc0c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:54:33.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.046+0000 7fbabb200640 1 -- 172.21.15.60:0/3648165476 shutdown_connections 2024-04-14T23:54:33.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.046+0000 7fbabb200640 1 -- 172.21.15.60:0/3648165476 wait complete. 2024-04-14T23:54:33.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.048+0000 7fbabb200640 1 Processor -- start 2024-04-14T23:54:33.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.048+0000 7fbabb200640 1 -- start start 2024-04-14T23:54:33.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.049+0000 7fbabb200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbabc073320 0x7fbabc1554a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:33.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.049+0000 7fbabb200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbabc0b1e00 0x7fbabc153b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:33.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.049+0000 7fbabb200640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbabc0c84e0 0x7fbabc1f03e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:33.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.049+0000 7fbabb200640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fbabc0c38b0 con 0x7fbabc0c84e0 2024-04-14T23:54:33.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.049+0000 7fbabb200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbabc159890 con 0x7fbabc073320 2024-04-14T23:54:33.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.049+0000 7fbabb200640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fbabc0c5f80 con 0x7fbabc0b1e00 2024-04-14T23:54:33.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.050+0000 7fbaba800640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbabc0c84e0 0x7fbabc1f03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:33.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.050+0000 7fbab9400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbabc0b1e00 0x7fbabc153b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:33.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.050+0000 7fbaba800640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbabc0c84e0 0x7fbabc1f03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49798/0 (socket says 172.21.15.60:49798) 2024-04-14T23:54:33.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.050+0000 7fbaba800640 1 -- 172.21.15.60:0/3440522675 learned_addr learned my addr 172.21.15.60:0/3440522675 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:54:33.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.050+0000 7fbab9e00640 1 --2- 172.21.15.60:0/3440522675 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbabc073320 0x7fbabc1554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:33.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.050+0000 7fbaba800640 1 -- 172.21.15.60:0/3440522675 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbabc0b1e00 msgr2=0x7fbabc153b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:33.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.050+0000 7fbaba800640 1 --2- 172.21.15.60:0/3440522675 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbabc0b1e00 0x7fbabc153b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:33.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.050+0000 7fbaba800640 1 -- 172.21.15.60:0/3440522675 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbabc073320 msgr2=0x7fbabc1554a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:33.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.050+0000 7fbaba800640 1 --2- 172.21.15.60:0/3440522675 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbabc073320 0x7fbabc1554a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:33.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.050+0000 7fbaba800640 1 -- 172.21.15.60:0/3440522675 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbabc1e5cf0 con 0x7fbabc0c84e0 2024-04-14T23:54:33.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.051+0000 7fbaba800640 1 --2- 172.21.15.60:0/3440522675 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbabc0c84e0 0x7fbabc1f03e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fbab400cb70 tx=0x7fbab400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:54:33.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.051+0000 7fbab2000640 1 -- 172.21.15.60:0/3440522675 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbab4017070 con 0x7fbabc0c84e0 2024-04-14T23:54:33.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.051+0000 7fbab2000640 1 -- 172.21.15.60:0/3440522675 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbab400ad40 con 0x7fbabc0c84e0 2024-04-14T23:54:33.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.051+0000 7fbabb200640 1 -- 172.21.15.60:0/3440522675 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbabc155ae0 con 0x7fbabc0c84e0 2024-04-14T23:54:33.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.051+0000 7fbab2000640 1 -- 172.21.15.60:0/3440522675 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbab4012620 con 0x7fbabc0c84e0 2024-04-14T23:54:33.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.051+0000 7fbabb200640 1 -- 172.21.15.60:0/3440522675 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbabc156760 con 0x7fbabc0c84e0 2024-04-14T23:54:33.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.052+0000 7fbabb200640 1 -- 172.21.15.60:0/3440522675 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbabc04f540 con 0x7fbabc0c84e0 2024-04-14T23:54:33.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.053+0000 7fbab2000640 1 -- 172.21.15.60:0/3440522675 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7fbab401a070 con 0x7fbabc0c84e0 2024-04-14T23:54:33.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.055+0000 7fbab2000640 1 --2- 172.21.15.60:0/3440522675 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fba9c06b990 0x7fba9c06de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:33.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.055+0000 7fbab2000640 1 -- 172.21.15.60:0/3440522675 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 6025+0+0 (secure 0 0 0) 0x7fbab408a7f0 con 0x7fbabc0c84e0 2024-04-14T23:54:33.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.055+0000 7fbab9e00640 1 --2- 172.21.15.60:0/3440522675 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fba9c06b990 0x7fba9c06de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:33.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.056+0000 7fbab9e00640 1 --2- 172.21.15.60:0/3440522675 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fba9c06b990 0x7fba9c06de50 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fbaac036000 tx=0x7fbaac03b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:54:33.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.059+0000 7fbab2000640 1 -- 172.21.15.60:0/3440522675 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fbab4058bd0 con 0x7fbabc0c84e0 2024-04-14T23:54:33.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.301+0000 7fbabb200640 1 -- 172.21.15.60:0/3440522675 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbabc1560a0 con 0x7fbabc0c84e0 2024-04-14T23:54:33.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.303+0000 7fbab2000640 1 -- 172.21.15.60:0/3440522675 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6839 (secure 0 0 0) 0x7fbab405cc70 con 0x7fbabc0c84e0 2024-04-14T23:54:33.308 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:54:33.308 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4986,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/1971580590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1971580590},{"type":"v1","addr":"172.21.15.60:6833","nonce":1971580590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:52:53.627240+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4839},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4839":{"gid":4839,"name":"c","rank":0,"incarnation":15,"state":"up:active","state_seq":24,"addr":"172.21.15.60:6835/1714133698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1714133698},{"type":"v1","addr":"172.21.15.60:6835","nonce":1714133698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4839,"qdb_cluster":[4839]},"id":1}]} 2024-04-14T23:54:33.308 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 20 2024-04-14T23:54:33.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.306+0000 7fbabb200640 1 -- 172.21.15.60:0/3440522675 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fba9c06b990 msgr2=0x7fba9c06de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:33.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.306+0000 7fbabb200640 1 --2- 172.21.15.60:0/3440522675 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fba9c06b990 0x7fba9c06de50 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fbaac036000 tx=0x7fbaac03b040 comp rx=0 tx=0).stop 2024-04-14T23:54:33.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.306+0000 7fbabb200640 1 -- 172.21.15.60:0/3440522675 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbabc0c84e0 msgr2=0x7fbabc1f03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:33.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.306+0000 7fbabb200640 1 --2- 172.21.15.60:0/3440522675 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbabc0c84e0 0x7fbabc1f03e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fbab400cb70 tx=0x7fbab400a430 comp rx=0 tx=0).stop 2024-04-14T23:54:33.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.306+0000 7fbabb200640 1 -- 172.21.15.60:0/3440522675 shutdown_connections 2024-04-14T23:54:33.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.306+0000 7fbabb200640 1 --2- 172.21.15.60:0/3440522675 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fba9c06b990 0x7fba9c06de50 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:33.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.306+0000 7fbabb200640 1 --2- 172.21.15.60:0/3440522675 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbabc0c84e0 0x7fbabc1f03e0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:33.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.306+0000 7fbabb200640 1 --2- 172.21.15.60:0/3440522675 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbabc0b1e00 0x7fbabc153b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:33.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.306+0000 7fbabb200640 1 --2- 172.21.15.60:0/3440522675 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbabc073320 0x7fbabc1554a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:33.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.306+0000 7fbabb200640 1 -- 172.21.15.60:0/3440522675 >> 172.21.15.60:0/3440522675 conn(0x7fbabc0c8b50 msgr2=0x7fbabc0c2ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:54:33.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.306+0000 7fbabb200640 1 -- 172.21.15.60:0/3440522675 shutdown_connections 2024-04-14T23:54:33.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:33.306+0000 7fbabb200640 1 -- 172.21.15.60:0/3440522675 wait complete. 2024-04-14T23:54:33.331 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-14T23:54:35.332 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:54:35.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.561+0000 7f2c7d800640 1 Processor -- start 2024-04-14T23:54:35.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.562+0000 7f2c7d800640 1 -- start start 2024-04-14T23:54:35.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.563+0000 7f2c7d800640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2c780c6950 0x7f2c7806b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:35.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.563+0000 7f2c7d800640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c7806b850 con 0x7f2c780c7080 2024-04-14T23:54:35.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.563+0000 7f2c7d800640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c7806bb30 con 0x7f2c7815ba70 2024-04-14T23:54:35.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.563+0000 7f2c7d800640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f2c7806c740 con 0x7f2c780c6950 2024-04-14T23:54:35.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.563+0000 7f2c77e00640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f2c780c7080 0x7f2c7815b360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54948/0 (socket says 172.21.15.60:54948) 2024-04-14T23:54:35.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.563+0000 7f2c77e00640 1 -- 172.21.15.60:0/891053325 learned_addr learned my addr 172.21.15.60:0/891053325 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:54:35.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.564+0000 7f2c6fe00640 1 --2- 172.21.15.60:0/891053325 >> v2:172.21.15.60:3300/0 conn(0x7f2c780c6950 0x7f2c7806b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:35.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.564+0000 7f2c77400640 1 -- 172.21.15.60:0/891053325 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1171949785 0 0) 0x7f2c7806b850 con 0x7f2c780c7080 2024-04-14T23:54:35.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.564+0000 7f2c77400640 1 -- 172.21.15.60:0/891053325 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2c5c003520 con 0x7f2c780c7080 2024-04-14T23:54:35.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.564+0000 7f2c77400640 1 -- 172.21.15.60:0/891053325 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2200670354 0 0) 0x7f2c7806bb30 con 0x7f2c7815ba70 2024-04-14T23:54:35.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.564+0000 7f2c77400640 1 -- 172.21.15.60:0/891053325 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2c7806b850 con 0x7f2c7815ba70 2024-04-14T23:54:35.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.564+0000 7f2c6fe00640 1 -- 172.21.15.60:0/891053325 >> v1:172.21.15.186:6789/0 conn(0x7f2c7815ba70 legacy=0x7f2c78070d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:35.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.564+0000 7f2c6fe00640 1 -- 172.21.15.60:0/891053325 >> v1:172.21.15.60:6789/0 conn(0x7f2c780c7080 legacy=0x7f2c7815b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:35.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.564+0000 7f2c6fe00640 1 -- 172.21.15.60:0/891053325 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c7806be10 con 0x7f2c780c6950 2024-04-14T23:54:35.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.565+0000 7f2c6fe00640 1 --2- 172.21.15.60:0/891053325 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c780c6950 0x7f2c7806b1c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f2c600097a0 tx=0x7f2c6002b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cb0b5a5380a9e9c server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:54:35.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.566+0000 7f2c77400640 1 -- 172.21.15.60:0/891053325 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c6002f810 con 0x7f2c780c6950 2024-04-14T23:54:35.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.566+0000 7f2c77400640 1 -- 172.21.15.60:0/891053325 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c6002f9b0 con 0x7f2c780c6950 2024-04-14T23:54:35.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.566+0000 7f2c77400640 1 -- 172.21.15.60:0/891053325 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c6002fd60 con 0x7f2c780c6950 2024-04-14T23:54:35.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.566+0000 7f2c7d800640 1 -- 172.21.15.60:0/891053325 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c780c6950 msgr2=0x7f2c7806b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:35.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.566+0000 7f2c7d800640 1 --2- 172.21.15.60:0/891053325 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c780c6950 0x7f2c7806b1c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f2c600097a0 tx=0x7f2c6002b9f0 comp rx=0 tx=0).stop 2024-04-14T23:54:35.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.566+0000 7f2c7d800640 1 -- 172.21.15.60:0/891053325 shutdown_connections 2024-04-14T23:54:35.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.566+0000 7f2c7d800640 1 --2- 172.21.15.60:0/891053325 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c780c6950 0x7f2c7806b1c0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:35.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.566+0000 7f2c7d800640 1 -- 172.21.15.60:0/891053325 >> 172.21.15.60:0/891053325 conn(0x7f2c780c8b50 msgr2=0x7f2c780c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:54:35.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.567+0000 7f2c7d800640 1 -- 172.21.15.60:0/891053325 shutdown_connections 2024-04-14T23:54:35.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.567+0000 7f2c7d800640 1 -- 172.21.15.60:0/891053325 wait complete. 2024-04-14T23:54:35.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.568+0000 7f2c7d800640 1 Processor -- start 2024-04-14T23:54:35.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.569+0000 7f2c7d800640 1 -- start start 2024-04-14T23:54:35.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.569+0000 7f2c7d800640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2c780c6950 0x7f2c78153cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:35.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.569+0000 7f2c7d800640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c780c7080 0x7f2c780caea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:35.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.569+0000 7f2c7d800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c7815ba70 0x7f2c780cb3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:35.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.569+0000 7f2c7d800640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f2c780c7460 con 0x7f2c780c7080 2024-04-14T23:54:35.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.569+0000 7f2c7d800640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2c7806c3d0 con 0x7f2c7815ba70 2024-04-14T23:54:35.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.569+0000 7f2c7d800640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f2c780c75e0 con 0x7f2c780c6950 2024-04-14T23:54:35.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.570+0000 7f2c6fe00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c780c7080 0x7f2c780caea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:35.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.570+0000 7f2c6fe00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c780c7080 0x7f2c780caea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49816/0 (socket says 172.21.15.60:49816) 2024-04-14T23:54:35.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.570+0000 7f2c6fe00640 1 -- 172.21.15.60:0/2996815202 learned_addr learned my addr 172.21.15.60:0/2996815202 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:54:35.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.570+0000 7f2c77e00640 1 --2- 172.21.15.60:0/2996815202 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2c780c6950 0x7f2c78153cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:35.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.570+0000 7f2c7ce00640 1 --2- 172.21.15.60:0/2996815202 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c7815ba70 0x7f2c780cb3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:35.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.570+0000 7f2c6fe00640 1 -- 172.21.15.60:0/2996815202 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2c780c6950 msgr2=0x7f2c78153cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:35.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.570+0000 7f2c6fe00640 1 --2- 172.21.15.60:0/2996815202 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2c780c6950 0x7f2c78153cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:35.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.570+0000 7f2c6fe00640 1 -- 172.21.15.60:0/2996815202 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c7815ba70 msgr2=0x7f2c780cb3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:35.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.570+0000 7f2c6fe00640 1 --2- 172.21.15.60:0/2996815202 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c7815ba70 0x7f2c780cb3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:35.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.570+0000 7f2c6fe00640 1 -- 172.21.15.60:0/2996815202 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c78154210 con 0x7f2c780c7080 2024-04-14T23:54:35.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.571+0000 7f2c6fe00640 1 --2- 172.21.15.60:0/2996815202 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c780c7080 0x7f2c780caea0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f2c600098c0 tx=0x7f2c60004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:54:35.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.571+0000 7f2c74c00640 1 -- 172.21.15.60:0/2996815202 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c60047070 con 0x7f2c780c7080 2024-04-14T23:54:35.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.571+0000 7f2c74c00640 1 -- 172.21.15.60:0/2996815202 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c600378f0 con 0x7f2c780c7080 2024-04-14T23:54:35.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.571+0000 7f2c74c00640 1 -- 172.21.15.60:0/2996815202 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c60037ce0 con 0x7f2c780c7080 2024-04-14T23:54:35.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.571+0000 7f2c7d800640 1 -- 172.21.15.60:0/2996815202 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c781f0490 con 0x7f2c780c7080 2024-04-14T23:54:35.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.571+0000 7f2c7d800640 1 -- 172.21.15.60:0/2996815202 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c781f0a30 con 0x7f2c780c7080 2024-04-14T23:54:35.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.572+0000 7f2c7d800640 1 -- 172.21.15.60:0/2996815202 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c7804f540 con 0x7f2c780c7080 2024-04-14T23:54:35.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.573+0000 7f2c74c00640 1 -- 172.21.15.60:0/2996815202 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f2c6004a070 con 0x7f2c780c7080 2024-04-14T23:54:35.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.574+0000 7f2c74c00640 1 --2- 172.21.15.60:0/2996815202 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f2c5406b990 0x7f2c5406de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:35.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.574+0000 7f2c74c00640 1 -- 172.21.15.60:0/2996815202 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 6025+0+0 (secure 0 0 0) 0x7f2c600ac690 con 0x7f2c780c7080 2024-04-14T23:54:35.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.575+0000 7f2c77e00640 1 --2- 172.21.15.60:0/2996815202 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f2c5406b990 0x7f2c5406de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:35.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.575+0000 7f2c77e00640 1 --2- 172.21.15.60:0/2996815202 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f2c5406b990 0x7f2c5406de50 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f2c68000f60 tx=0x7f2c68007d70 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:54:35.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.579+0000 7f2c74c00640 1 -- 172.21.15.60:0/2996815202 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2c78154210 con 0x7f2c780c7080 2024-04-14T23:54:35.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.823+0000 7f2c7d800640 1 -- 172.21.15.60:0/2996815202 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2c78154eb0 con 0x7f2c780c7080 2024-04-14T23:54:35.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.825+0000 7f2c74c00640 1 -- 172.21.15.60:0/2996815202 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6839 (secure 0 0 0) 0x7f2c60033030 con 0x7f2c780c7080 2024-04-14T23:54:35.830 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:54:35.830 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4986,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/1971580590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1971580590},{"type":"v1","addr":"172.21.15.60:6833","nonce":1971580590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:52:53.627240+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4839},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4839":{"gid":4839,"name":"c","rank":0,"incarnation":15,"state":"up:active","state_seq":24,"addr":"172.21.15.60:6835/1714133698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1714133698},{"type":"v1","addr":"172.21.15.60:6835","nonce":1714133698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4839,"qdb_cluster":[4839]},"id":1}]} 2024-04-14T23:54:35.830 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 20 2024-04-14T23:54:35.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.827+0000 7f2c7d800640 1 -- 172.21.15.60:0/2996815202 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f2c5406b990 msgr2=0x7f2c5406de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:35.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.828+0000 7f2c7d800640 1 --2- 172.21.15.60:0/2996815202 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f2c5406b990 0x7f2c5406de50 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f2c68000f60 tx=0x7f2c68007d70 comp rx=0 tx=0).stop 2024-04-14T23:54:35.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.828+0000 7f2c7d800640 1 -- 172.21.15.60:0/2996815202 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c780c7080 msgr2=0x7f2c780caea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:35.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.828+0000 7f2c7d800640 1 --2- 172.21.15.60:0/2996815202 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c780c7080 0x7f2c780caea0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f2c600098c0 tx=0x7f2c60004730 comp rx=0 tx=0).stop 2024-04-14T23:54:35.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.828+0000 7f2c7d800640 1 -- 172.21.15.60:0/2996815202 shutdown_connections 2024-04-14T23:54:35.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.828+0000 7f2c7d800640 1 --2- 172.21.15.60:0/2996815202 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f2c5406b990 0x7f2c5406de50 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:35.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.828+0000 7f2c7d800640 1 --2- 172.21.15.60:0/2996815202 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c7815ba70 0x7f2c780cb3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:35.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.828+0000 7f2c7d800640 1 --2- 172.21.15.60:0/2996815202 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c780c7080 0x7f2c780caea0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:35.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.828+0000 7f2c7d800640 1 --2- 172.21.15.60:0/2996815202 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2c780c6950 0x7f2c78153cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:35.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.828+0000 7f2c7d800640 1 -- 172.21.15.60:0/2996815202 >> 172.21.15.60:0/2996815202 conn(0x7f2c780c8b50 msgr2=0x7f2c7815a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:54:35.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.828+0000 7f2c7d800640 1 -- 172.21.15.60:0/2996815202 shutdown_connections 2024-04-14T23:54:35.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:35.828+0000 7f2c7d800640 1 -- 172.21.15.60:0/2996815202 wait complete. 2024-04-14T23:54:35.854 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-14T23:54:37.854 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:54:38.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.073+0000 7f67a3a00640 1 Processor -- start 2024-04-14T23:54:38.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.073+0000 7f67a3a00640 1 -- start start 2024-04-14T23:54:38.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.074+0000 7f67a3a00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f679c0cd160 0x7f679c15ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:38.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.074+0000 7f67a3a00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f679c0cb750 0x7f679c154b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:38.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.074+0000 7f67a3a00640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f679c0b3f50 con 0x7f679c0cd540 2024-04-14T23:54:38.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.074+0000 7f67a3a00640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f679c15b370 con 0x7f679c0cb750 2024-04-14T23:54:38.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.074+0000 7f67a3a00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f679c0b39b0 con 0x7f679c0cd160 2024-04-14T23:54:38.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.075+0000 7f67a1000640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f679c0cd160 0x7f679c15ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:38.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.075+0000 7f67a1000640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f679c0cd160 0x7f679c15ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.60:43494/0 (socket says 172.21.15.60:43494) 2024-04-14T23:54:38.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.075+0000 7f67a1000640 1 -- 172.21.15.60:0/3830173297 learned_addr learned my addr 172.21.15.60:0/3830173297 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:54:38.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.075+0000 7f67a1a00640 1 --2- 172.21.15.60:0/3830173297 >> v2:172.21.15.60:3300/0 conn(0x7f679c0cb750 0x7f679c154b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:38.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.075+0000 7f67a1000640 1 -- 172.21.15.60:0/3830173297 >> v2:172.21.15.60:3300/0 conn(0x7f679c0cb750 msgr2=0x7f679c154b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:54:38.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.075+0000 7f67a1000640 1 --2- 172.21.15.60:0/3830173297 >> v2:172.21.15.60:3300/0 conn(0x7f679c0cb750 0x7f679c154b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:38.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.075+0000 7f67a1000640 1 -- 172.21.15.60:0/3830173297 >> v1:172.21.15.186:6789/0 conn(0x7f679c0cd540 legacy=0x7f679c0b3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:38.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.075+0000 7f67a1000640 1 -- 172.21.15.60:0/3830173297 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f679c0b3c70 con 0x7f679c0cd160 2024-04-14T23:54:38.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.075+0000 7f679b400640 1 -- 172.21.15.60:0/3830173297 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3866377377 0 0) 0x7f679c0b3f50 con 0x7f679c0cd540 2024-04-14T23:54:38.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.077+0000 7f67a1000640 1 --2- 172.21.15.60:0/3830173297 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f679c0cd160 0x7f679c15ae30 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f678c009e50 tx=0x7f678c02bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e32a42c04a74f87 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:54:38.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.077+0000 7f679b400640 1 -- 172.21.15.60:0/3830173297 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f678c02cb00 con 0x7f679c0cd160 2024-04-14T23:54:38.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.077+0000 7f679b400640 1 -- 172.21.15.60:0/3830173297 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f678c02cca0 con 0x7f679c0cd160 2024-04-14T23:54:38.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.077+0000 7f679b400640 1 -- 172.21.15.60:0/3830173297 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f678c034610 con 0x7f679c0cd160 2024-04-14T23:54:38.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.078+0000 7f67a3a00640 1 -- 172.21.15.60:0/3830173297 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f679c0cd160 msgr2=0x7f679c15ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:38.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.078+0000 7f67a3a00640 1 --2- 172.21.15.60:0/3830173297 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f679c0cd160 0x7f679c15ae30 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f678c009e50 tx=0x7f678c02bf60 comp rx=0 tx=0).stop 2024-04-14T23:54:38.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.078+0000 7f67a3a00640 1 -- 172.21.15.60:0/3830173297 shutdown_connections 2024-04-14T23:54:38.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.078+0000 7f67a3a00640 1 --2- 172.21.15.60:0/3830173297 >> v2:172.21.15.60:3300/0 conn(0x7f679c0cb750 0x7f679c154b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:38.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.078+0000 7f67a3a00640 1 --2- 172.21.15.60:0/3830173297 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f679c0cd160 0x7f679c15ae30 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:38.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.078+0000 7f67a3a00640 1 -- 172.21.15.60:0/3830173297 >> 172.21.15.60:0/3830173297 conn(0x7f679c066f10 msgr2=0x7f679c0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:54:38.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.078+0000 7f67a3a00640 1 -- 172.21.15.60:0/3830173297 shutdown_connections 2024-04-14T23:54:38.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.078+0000 7f67a3a00640 1 -- 172.21.15.60:0/3830173297 wait complete. 2024-04-14T23:54:38.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.080+0000 7f67a3a00640 1 Processor -- start 2024-04-14T23:54:38.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.080+0000 7f67a3a00640 1 -- start start 2024-04-14T23:54:38.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.081+0000 7f67a3a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f679c0cb750 0x7f679c1e5c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:38.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.081+0000 7f67a3a00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f679c0cd160 0x7f679c1e4a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:38.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.081+0000 7f67a3a00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f679c0cd540 0x7f679c1f0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:38.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.081+0000 7f67a3a00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f679c15ea70 con 0x7f679c0cd160 2024-04-14T23:54:38.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.081+0000 7f67a3a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f679c15e790 con 0x7f679c0cb750 2024-04-14T23:54:38.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.081+0000 7f67a3a00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f679c0ccb20 con 0x7f679c0cd540 2024-04-14T23:54:38.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.081+0000 7f679be00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f679c0cd160 0x7f679c1e4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:38.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.081+0000 7f679be00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f679c0cd160 0x7f679c1e4a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49840/0 (socket says 172.21.15.60:49840) 2024-04-14T23:54:38.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.081+0000 7f679be00640 1 -- 172.21.15.60:0/1880512197 learned_addr learned my addr 172.21.15.60:0/1880512197 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:54:38.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.081+0000 7f67a1000640 1 --2- 172.21.15.60:0/1880512197 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f679c0cb750 0x7f679c1e5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:38.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.081+0000 7f67a1a00640 1 --2- 172.21.15.60:0/1880512197 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f679c0cd540 0x7f679c1f0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:38.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.082+0000 7f67a1000640 1 -- 172.21.15.60:0/1880512197 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f679c0cd540 msgr2=0x7f679c1f0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:38.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.082+0000 7f67a1000640 1 --2- 172.21.15.60:0/1880512197 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f679c0cd540 0x7f679c1f0c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:38.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.082+0000 7f67a1000640 1 -- 172.21.15.60:0/1880512197 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f679c0cd160 msgr2=0x7f679c1e4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:38.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.082+0000 7f67a1000640 1 --2- 172.21.15.60:0/1880512197 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f679c0cd160 0x7f679c1e4a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:38.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.082+0000 7f67a1000640 1 -- 172.21.15.60:0/1880512197 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f679c0c5fc0 con 0x7f679c0cb750 2024-04-14T23:54:38.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.082+0000 7f67a1000640 1 --2- 172.21.15.60:0/1880512197 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f679c0cb750 0x7f679c1e5c20 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f678c032000 tx=0x7f678c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:54:38.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.082+0000 7f6798c00640 1 -- 172.21.15.60:0/1880512197 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f678c043020 con 0x7f679c0cb750 2024-04-14T23:54:38.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.082+0000 7f6798c00640 1 -- 172.21.15.60:0/1880512197 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f678c02ce50 con 0x7f679c0cb750 2024-04-14T23:54:38.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.082+0000 7f6798c00640 1 -- 172.21.15.60:0/1880512197 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f678c009510 con 0x7f679c0cb750 2024-04-14T23:54:38.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.082+0000 7f67a3a00640 1 -- 172.21.15.60:0/1880512197 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f679c0c6db0 con 0x7f679c0cb750 2024-04-14T23:54:38.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.083+0000 7f67a3a00640 1 -- 172.21.15.60:0/1880512197 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f679c0c6540 con 0x7f679c0cb750 2024-04-14T23:54:38.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.084+0000 7f67a3a00640 1 -- 172.21.15.60:0/1880512197 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f679c076970 con 0x7f679c0cb750 2024-04-14T23:54:38.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.085+0000 7f6798c00640 1 -- 172.21.15.60:0/1880512197 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f678c046020 con 0x7f679c0cb750 2024-04-14T23:54:38.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.086+0000 7f6798c00640 1 --2- 172.21.15.60:0/1880512197 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f677c06ba10 0x7f677c06ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:38.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.086+0000 7f6798c00640 1 -- 172.21.15.60:0/1880512197 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 6025+0+0 (secure 0 0 0) 0x7f678c0ace30 con 0x7f679c0cb750 2024-04-14T23:54:38.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.086+0000 7f679be00640 1 --2- 172.21.15.60:0/1880512197 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f677c06ba10 0x7f677c06ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:38.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.087+0000 7f679be00640 1 --2- 172.21.15.60:0/1880512197 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f677c06ba10 0x7f677c06ded0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f679c152a40 tx=0x7f6788009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:54:38.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.091+0000 7f6798c00640 1 -- 172.21.15.60:0/1880512197 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f678c04d030 con 0x7f679c0cb750 2024-04-14T23:54:38.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.333+0000 7f67a3a00640 1 -- 172.21.15.60:0/1880512197 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f679c0ca6c0 con 0x7f679c0cb750 2024-04-14T23:54:38.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.334+0000 7f6798c00640 1 -- 172.21.15.60:0/1880512197 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6839 (secure 0 0 0) 0x7f678c07b200 con 0x7f679c0cb750 2024-04-14T23:54:38.340 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:54:38.340 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4986,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/1971580590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1971580590},{"type":"v1","addr":"172.21.15.60:6833","nonce":1971580590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:52:53.627240+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4839},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4839":{"gid":4839,"name":"c","rank":0,"incarnation":15,"state":"up:active","state_seq":24,"addr":"172.21.15.60:6835/1714133698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1714133698},{"type":"v1","addr":"172.21.15.60:6835","nonce":1714133698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4839,"qdb_cluster":[4839]},"id":1}]} 2024-04-14T23:54:38.340 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 20 2024-04-14T23:54:38.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.337+0000 7f67a3a00640 1 -- 172.21.15.60:0/1880512197 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f677c06ba10 msgr2=0x7f677c06ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:38.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.337+0000 7f67a3a00640 1 --2- 172.21.15.60:0/1880512197 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f677c06ba10 0x7f677c06ded0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f679c152a40 tx=0x7f6788009040 comp rx=0 tx=0).stop 2024-04-14T23:54:38.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.337+0000 7f67a3a00640 1 -- 172.21.15.60:0/1880512197 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f679c0cb750 msgr2=0x7f679c1e5c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:38.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.337+0000 7f67a3a00640 1 --2- 172.21.15.60:0/1880512197 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f679c0cb750 0x7f679c1e5c20 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f678c032000 tx=0x7f678c004a40 comp rx=0 tx=0).stop 2024-04-14T23:54:38.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.337+0000 7f67a3a00640 1 -- 172.21.15.60:0/1880512197 shutdown_connections 2024-04-14T23:54:38.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.337+0000 7f67a3a00640 1 --2- 172.21.15.60:0/1880512197 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f677c06ba10 0x7f677c06ded0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:38.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.337+0000 7f67a3a00640 1 --2- 172.21.15.60:0/1880512197 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f679c0cd540 0x7f679c1f0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:38.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.337+0000 7f67a3a00640 1 --2- 172.21.15.60:0/1880512197 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f679c0cd160 0x7f679c1e4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:38.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.337+0000 7f67a3a00640 1 --2- 172.21.15.60:0/1880512197 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f679c0cb750 0x7f679c1e5c20 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:38.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.337+0000 7f67a3a00640 1 -- 172.21.15.60:0/1880512197 >> 172.21.15.60:0/1880512197 conn(0x7f679c066f10 msgr2=0x7f679c0b4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:54:38.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.337+0000 7f67a3a00640 1 -- 172.21.15.60:0/1880512197 shutdown_connections 2024-04-14T23:54:38.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:38.338+0000 7f67a3a00640 1 -- 172.21.15.60:0/1880512197 wait complete. 2024-04-14T23:54:38.362 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-14T23:54:40.363 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:54:40.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.591+0000 7f7d42600640 1 Processor -- start 2024-04-14T23:54:40.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.592+0000 7f7d42600640 1 -- start start 2024-04-14T23:54:40.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.593+0000 7f7d42600640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7d3c073320 0x7f7d3c0c6950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:40.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.593+0000 7f7d42600640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7d3c0c3d90 0x7f7d3c0c41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:40.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.593+0000 7f7d42600640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d3c152fb0 con 0x7f7d3c0b1e00 2024-04-14T23:54:40.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.593+0000 7f7d42600640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f7d3c0c6540 con 0x7f7d3c073320 2024-04-14T23:54:40.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.593+0000 7f7d42600640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f7d3c0b21e0 con 0x7f7d3c0c3d90 2024-04-14T23:54:40.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.593+0000 7f7d41200640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f7d3c0b1e00 0x7f7d3c0c7fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54950/0 (socket says 172.21.15.60:54950) 2024-04-14T23:54:40.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.593+0000 7f7d41200640 1 -- 172.21.15.60:0/2624835435 learned_addr learned my addr 172.21.15.60:0/2624835435 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:54:40.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.593+0000 7f7d3be00640 1 --2- 172.21.15.60:0/2624835435 >> v2:172.21.15.60:3300/0 conn(0x7f7d3c073320 0x7f7d3c0c6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:40.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.593+0000 7f7d41c00640 1 --2- 172.21.15.60:0/2624835435 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7d3c0c3d90 0x7f7d3c0c41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:40.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.594+0000 7f7d3b400640 1 -- 172.21.15.60:0/2624835435 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1429999742 0 0) 0x7f7d3c152fb0 con 0x7f7d3c0b1e00 2024-04-14T23:54:40.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.594+0000 7f7d3b400640 1 -- 172.21.15.60:0/2624835435 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7d24003520 con 0x7f7d3c0b1e00 2024-04-14T23:54:40.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.594+0000 7f7d3be00640 1 -- 172.21.15.60:0/2624835435 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7d3c0c3d90 msgr2=0x7f7d3c0c41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:54:40.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.594+0000 7f7d3be00640 1 --2- 172.21.15.60:0/2624835435 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7d3c0c3d90 0x7f7d3c0c41c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:40.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.594+0000 7f7d3be00640 1 -- 172.21.15.60:0/2624835435 >> v1:172.21.15.60:6789/0 conn(0x7f7d3c0b1e00 legacy=0x7f7d3c0c7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:40.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.594+0000 7f7d3be00640 1 -- 172.21.15.60:0/2624835435 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d3c159310 con 0x7f7d3c073320 2024-04-14T23:54:40.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.594+0000 7f7d41c00640 1 --2- 172.21.15.60:0/2624835435 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7d3c0c3d90 0x7f7d3c0c41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:54:40.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.595+0000 7f7d3be00640 1 --2- 172.21.15.60:0/2624835435 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d3c073320 0x7f7d3c0c6950 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c0097a0 tx=0x7f7d2c02b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6e51f421d3c97df server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:54:40.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.595+0000 7f7d3b400640 1 -- 172.21.15.60:0/2624835435 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c034070 con 0x7f7d3c073320 2024-04-14T23:54:40.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.595+0000 7f7d3b400640 1 -- 172.21.15.60:0/2624835435 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d2c030800 con 0x7f7d3c073320 2024-04-14T23:54:40.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.595+0000 7f7d3b400640 1 -- 172.21.15.60:0/2624835435 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c030bd0 con 0x7f7d3c073320 2024-04-14T23:54:40.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.596+0000 7f7d42600640 1 -- 172.21.15.60:0/2624835435 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d3c073320 msgr2=0x7f7d3c0c6950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:40.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.596+0000 7f7d42600640 1 --2- 172.21.15.60:0/2624835435 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d3c073320 0x7f7d3c0c6950 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c0097a0 tx=0x7f7d2c02b940 comp rx=0 tx=0).stop 2024-04-14T23:54:40.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.596+0000 7f7d42600640 1 -- 172.21.15.60:0/2624835435 shutdown_connections 2024-04-14T23:54:40.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.596+0000 7f7d42600640 1 --2- 172.21.15.60:0/2624835435 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d3c073320 0x7f7d3c0c6950 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:40.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.596+0000 7f7d42600640 1 --2- 172.21.15.60:0/2624835435 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7d3c0c3d90 0x7f7d3c0c41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:40.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.596+0000 7f7d42600640 1 -- 172.21.15.60:0/2624835435 >> 172.21.15.60:0/2624835435 conn(0x7f7d3c0c8b50 msgr2=0x7f7d3c0c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:54:40.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.596+0000 7f7d42600640 1 -- 172.21.15.60:0/2624835435 shutdown_connections 2024-04-14T23:54:40.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.596+0000 7f7d42600640 1 -- 172.21.15.60:0/2624835435 wait complete. 2024-04-14T23:54:40.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.598+0000 7f7d42600640 1 Processor -- start 2024-04-14T23:54:40.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.598+0000 7f7d42600640 1 -- start start 2024-04-14T23:54:40.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.599+0000 7f7d42600640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7d3c073320 0x7f7d3c153af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:40.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.599+0000 7f7d42600640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d3c0b1e00 0x7f7d3c0cb310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:40.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.599+0000 7f7d42600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d3c0c3d90 0x7f7d3c1f05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:40.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.599+0000 7f7d42600640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f7d3c0c38b0 con 0x7f7d3c0b1e00 2024-04-14T23:54:40.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.599+0000 7f7d42600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7d3c159890 con 0x7f7d3c0c3d90 2024-04-14T23:54:40.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.599+0000 7f7d42600640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f7d3c1595b0 con 0x7f7d3c073320 2024-04-14T23:54:40.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.599+0000 7f7d3be00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d3c0b1e00 0x7f7d3c0cb310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:40.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.599+0000 7f7d41c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d3c0c3d90 0x7f7d3c1f05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:40.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.599+0000 7f7d41200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7d3c073320 0x7f7d3c153af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:40.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.599+0000 7f7d41c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d3c0c3d90 0x7f7d3c1f05a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:38922/0 (socket says 172.21.15.60:38922) 2024-04-14T23:54:40.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.599+0000 7f7d41c00640 1 -- 172.21.15.60:0/2658660699 learned_addr learned my addr 172.21.15.60:0/2658660699 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:54:40.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.600+0000 7f7d3be00640 1 -- 172.21.15.60:0/2658660699 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7d3c073320 msgr2=0x7f7d3c153af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:40.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.600+0000 7f7d3be00640 1 --2- 172.21.15.60:0/2658660699 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7d3c073320 0x7f7d3c153af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:40.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.600+0000 7f7d3be00640 1 -- 172.21.15.60:0/2658660699 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d3c0c3d90 msgr2=0x7f7d3c1f05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:40.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.600+0000 7f7d3be00640 1 --2- 172.21.15.60:0/2658660699 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d3c0c3d90 0x7f7d3c1f05a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:40.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.600+0000 7f7d3be00640 1 -- 172.21.15.60:0/2658660699 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d3c156df0 con 0x7f7d3c0b1e00 2024-04-14T23:54:40.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.600+0000 7f7d3be00640 1 --2- 172.21.15.60:0/2658660699 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d3c0b1e00 0x7f7d3c0cb310 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c033000 tx=0x7f7d2c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:54:40.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.600+0000 7f7d38c00640 1 -- 172.21.15.60:0/2658660699 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c034040 con 0x7f7d3c0b1e00 2024-04-14T23:54:40.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.601+0000 7f7d38c00640 1 -- 172.21.15.60:0/2658660699 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d2c030e30 con 0x7f7d3c0b1e00 2024-04-14T23:54:40.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.601+0000 7f7d38c00640 1 -- 172.21.15.60:0/2658660699 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c0443f0 con 0x7f7d3c0b1e00 2024-04-14T23:54:40.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.601+0000 7f7d42600640 1 -- 172.21.15.60:0/2658660699 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d3c1555f0 con 0x7f7d3c0b1e00 2024-04-14T23:54:40.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.601+0000 7f7d42600640 1 -- 172.21.15.60:0/2658660699 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d3c156b10 con 0x7f7d3c0b1e00 2024-04-14T23:54:40.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.602+0000 7f7d42600640 1 -- 172.21.15.60:0/2658660699 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d3c0c4f80 con 0x7f7d3c0b1e00 2024-04-14T23:54:40.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.603+0000 7f7d38c00640 1 -- 172.21.15.60:0/2658660699 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f7d2c04b060 con 0x7f7d3c0b1e00 2024-04-14T23:54:40.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.604+0000 7f7d38c00640 1 --2- 172.21.15.60:0/2658660699 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f7d2006b9e0 0x7f7d2006dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:40.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.604+0000 7f7d38c00640 1 -- 172.21.15.60:0/2658660699 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 6025+0+0 (secure 0 0 0) 0x7f7d2c0ac6e0 con 0x7f7d3c0b1e00 2024-04-14T23:54:40.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.604+0000 7f7d41200640 1 --2- 172.21.15.60:0/2658660699 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f7d2006b9e0 0x7f7d2006dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:40.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.605+0000 7f7d41200640 1 --2- 172.21.15.60:0/2658660699 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f7d2006b9e0 0x7f7d2006dea0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f7d30005880 tx=0x7f7d30009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:54:40.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.609+0000 7f7d38c00640 1 -- 172.21.15.60:0/2658660699 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7d3c0c4f80 con 0x7f7d3c0b1e00 2024-04-14T23:54:40.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.848+0000 7f7d42600640 1 -- 172.21.15.60:0/2658660699 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7d3c0c3180 con 0x7f7d3c0b1e00 2024-04-14T23:54:40.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.849+0000 7f7d38c00640 1 -- 172.21.15.60:0/2658660699 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6839 (secure 0 0 0) 0x7f7d2c03f070 con 0x7f7d3c0b1e00 2024-04-14T23:54:40.855 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:54:40.855 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4986,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/1971580590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1971580590},{"type":"v1","addr":"172.21.15.60:6833","nonce":1971580590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:52:53.627240+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4839},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4839":{"gid":4839,"name":"c","rank":0,"incarnation":15,"state":"up:active","state_seq":24,"addr":"172.21.15.60:6835/1714133698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1714133698},{"type":"v1","addr":"172.21.15.60:6835","nonce":1714133698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4839,"qdb_cluster":[4839]},"id":1}]} 2024-04-14T23:54:40.856 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 20 2024-04-14T23:54:40.857 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.852+0000 7f7d42600640 1 -- 172.21.15.60:0/2658660699 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f7d2006b9e0 msgr2=0x7f7d2006dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:40.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.852+0000 7f7d42600640 1 --2- 172.21.15.60:0/2658660699 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f7d2006b9e0 0x7f7d2006dea0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f7d30005880 tx=0x7f7d30009040 comp rx=0 tx=0).stop 2024-04-14T23:54:40.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.852+0000 7f7d42600640 1 -- 172.21.15.60:0/2658660699 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d3c0b1e00 msgr2=0x7f7d3c0cb310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:40.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.852+0000 7f7d42600640 1 --2- 172.21.15.60:0/2658660699 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d3c0b1e00 0x7f7d3c0cb310 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c033000 tx=0x7f7d2c0046c0 comp rx=0 tx=0).stop 2024-04-14T23:54:40.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.852+0000 7f7d42600640 1 -- 172.21.15.60:0/2658660699 shutdown_connections 2024-04-14T23:54:40.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.852+0000 7f7d42600640 1 --2- 172.21.15.60:0/2658660699 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f7d2006b9e0 0x7f7d2006dea0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:40.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.852+0000 7f7d42600640 1 --2- 172.21.15.60:0/2658660699 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d3c0c3d90 0x7f7d3c1f05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:40.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.852+0000 7f7d42600640 1 --2- 172.21.15.60:0/2658660699 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d3c0b1e00 0x7f7d3c0cb310 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:40.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.852+0000 7f7d42600640 1 --2- 172.21.15.60:0/2658660699 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7d3c073320 0x7f7d3c153af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:40.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.852+0000 7f7d42600640 1 -- 172.21.15.60:0/2658660699 >> 172.21.15.60:0/2658660699 conn(0x7f7d3c0c8b50 msgr2=0x7f7d3c072b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:54:40.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.852+0000 7f7d42600640 1 -- 172.21.15.60:0/2658660699 shutdown_connections 2024-04-14T23:54:40.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:40.852+0000 7f7d42600640 1 -- 172.21.15.60:0/2658660699 wait complete. 2024-04-14T23:54:40.878 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-14T23:54:42.879 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:54:43.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.107+0000 7f1387e00640 1 Processor -- start 2024-04-14T23:54:43.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.107+0000 7f1387e00640 1 -- start start 2024-04-14T23:54:43.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.109+0000 7f1387e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f138815ba70 0x7f138806b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:43.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.109+0000 7f1387e00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f138806b850 con 0x7f13880c7080 2024-04-14T23:54:43.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.109+0000 7f1387e00640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f138806bb30 con 0x7f13880c6950 2024-04-14T23:54:43.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.109+0000 7f1387e00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f138806c740 con 0x7f138815ba70 2024-04-14T23:54:43.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.109+0000 7f1386a00640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f13880c7080 0x7f138815b360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:46490/0 (socket says 172.21.15.60:46490) 2024-04-14T23:54:43.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.109+0000 7f1386a00640 1 -- 172.21.15.60:0/3253950577 learned_addr learned my addr 172.21.15.60:0/3253950577 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:54:43.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.109+0000 7f1387400640 1 --2- 172.21.15.60:0/3253950577 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f138815ba70 0x7f138806b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:43.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.110+0000 7f1386000640 1 -- 172.21.15.60:0/3253950577 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2021753625 0 0) 0x7f138806bb30 con 0x7f13880c6950 2024-04-14T23:54:43.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.110+0000 7f1386000640 1 -- 172.21.15.60:0/3253950577 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f136c003520 con 0x7f13880c6950 2024-04-14T23:54:43.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.110+0000 7f1386000640 1 -- 172.21.15.60:0/3253950577 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4179507493 0 0) 0x7f138806b850 con 0x7f13880c7080 2024-04-14T23:54:43.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.110+0000 7f1387400640 1 -- 172.21.15.60:0/3253950577 >> v1:172.21.15.186:6789/0 conn(0x7f13880c6950 legacy=0x7f1388070d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:43.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.110+0000 7f1387400640 1 -- 172.21.15.60:0/3253950577 >> v1:172.21.15.60:6789/0 conn(0x7f13880c7080 legacy=0x7f138815b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:43.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.110+0000 7f1387400640 1 -- 172.21.15.60:0/3253950577 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f138806be10 con 0x7f138815ba70 2024-04-14T23:54:43.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.112+0000 7f1387400640 1 --2- 172.21.15.60:0/3253950577 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f138815ba70 0x7f138806b1c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f13800097d0 tx=0x7f138002b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=150d0ccf435439ba server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:54:43.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.112+0000 7f1386000640 1 -- 172.21.15.60:0/3253950577 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1380034070 con 0x7f138815ba70 2024-04-14T23:54:43.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.112+0000 7f1386000640 1 -- 172.21.15.60:0/3253950577 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1380030880 con 0x7f138815ba70 2024-04-14T23:54:43.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.112+0000 7f1386000640 1 -- 172.21.15.60:0/3253950577 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1380030c50 con 0x7f138815ba70 2024-04-14T23:54:43.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.113+0000 7f1387e00640 1 -- 172.21.15.60:0/3253950577 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f138815ba70 msgr2=0x7f138806b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:43.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.113+0000 7f1387e00640 1 --2- 172.21.15.60:0/3253950577 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f138815ba70 0x7f138806b1c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f13800097d0 tx=0x7f138002b9c0 comp rx=0 tx=0).stop 2024-04-14T23:54:43.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.113+0000 7f1387e00640 1 -- 172.21.15.60:0/3253950577 shutdown_connections 2024-04-14T23:54:43.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.113+0000 7f1387e00640 1 --2- 172.21.15.60:0/3253950577 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f138815ba70 0x7f138806b1c0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:43.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.113+0000 7f1387e00640 1 -- 172.21.15.60:0/3253950577 >> 172.21.15.60:0/3253950577 conn(0x7f13880c8b50 msgr2=0x7f13880c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:54:43.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.113+0000 7f1387e00640 1 -- 172.21.15.60:0/3253950577 shutdown_connections 2024-04-14T23:54:43.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.113+0000 7f1387e00640 1 -- 172.21.15.60:0/3253950577 wait complete. 2024-04-14T23:54:43.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.115+0000 7f1387e00640 1 Processor -- start 2024-04-14T23:54:43.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.115+0000 7f1387e00640 1 -- start start 2024-04-14T23:54:43.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.116+0000 7f1387e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13880c6950 0x7f13880c12e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:43.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.116+0000 7f1387e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f13880c7080 0x7f13880c1820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:43.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.116+0000 7f1387e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f138815ba70 0x7f13880b0260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:43.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.116+0000 7f1387e00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f13880c7460 con 0x7f13880c6950 2024-04-14T23:54:43.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.116+0000 7f1387e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f138806c3d0 con 0x7f13880c7080 2024-04-14T23:54:43.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.116+0000 7f1387e00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f13880c75e0 con 0x7f138815ba70 2024-04-14T23:54:43.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.116+0000 7f1386a00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13880c6950 0x7f13880c12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:43.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.116+0000 7f1386a00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13880c6950 0x7f13880c12e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57508/0 (socket says 172.21.15.60:57508) 2024-04-14T23:54:43.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.116+0000 7f1386a00640 1 -- 172.21.15.60:0/157916091 learned_addr learned my addr 172.21.15.60:0/157916091 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:54:43.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.116+0000 7f137e000640 1 --2- 172.21.15.60:0/157916091 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f13880c7080 0x7f13880c1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:43.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.116+0000 7f1387400640 1 --2- 172.21.15.60:0/157916091 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f138815ba70 0x7f13880b0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:43.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.117+0000 7f1386a00640 1 -- 172.21.15.60:0/157916091 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f138815ba70 msgr2=0x7f13880b0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:43.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.117+0000 7f1386a00640 1 --2- 172.21.15.60:0/157916091 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f138815ba70 0x7f13880b0260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:43.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.117+0000 7f1386a00640 1 -- 172.21.15.60:0/157916091 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f13880c7080 msgr2=0x7f13880c1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:43.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.117+0000 7f1386a00640 1 --2- 172.21.15.60:0/157916091 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f13880c7080 0x7f13880c1820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:43.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.117+0000 7f1386a00640 1 -- 172.21.15.60:0/157916091 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13880afa50 con 0x7f13880c6950 2024-04-14T23:54:43.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.117+0000 7f1386a00640 1 --2- 172.21.15.60:0/157916091 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13880c6950 0x7f13880c12e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f137800c8f0 tx=0x7f137800cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:54:43.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.117+0000 7f137f400640 1 -- 172.21.15.60:0/157916091 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1378018070 con 0x7f13880c6950 2024-04-14T23:54:43.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.117+0000 7f137f400640 1 -- 172.21.15.60:0/157916091 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f137800a640 con 0x7f13880c6950 2024-04-14T23:54:43.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.117+0000 7f137f400640 1 -- 172.21.15.60:0/157916091 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f137800aa30 con 0x7f13880c6950 2024-04-14T23:54:43.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.117+0000 7f1387e00640 1 -- 172.21.15.60:0/157916091 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13880c0c70 con 0x7f13880c6950 2024-04-14T23:54:43.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.118+0000 7f1387e00640 1 -- 172.21.15.60:0/157916091 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13880b1460 con 0x7f13880c6950 2024-04-14T23:54:43.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.119+0000 7f1387e00640 1 -- 172.21.15.60:0/157916091 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f138804f540 con 0x7f13880c6950 2024-04-14T23:54:43.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.120+0000 7f137f400640 1 -- 172.21.15.60:0/157916091 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f137801b070 con 0x7f13880c6950 2024-04-14T23:54:43.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.121+0000 7f137f400640 1 --2- 172.21.15.60:0/157916091 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f136406b990 0x7f136406de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:43.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.121+0000 7f137f400640 1 -- 172.21.15.60:0/157916091 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 6025+0+0 (secure 0 0 0) 0x7f1378089f50 con 0x7f13880c6950 2024-04-14T23:54:43.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.121+0000 7f137e000640 1 --2- 172.21.15.60:0/157916091 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f136406b990 0x7f136406de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:43.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.122+0000 7f137e000640 1 --2- 172.21.15.60:0/157916091 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f136406b990 0x7f136406de50 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f1388156e60 tx=0x7f1370009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:54:43.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.126+0000 7f137f400640 1 -- 172.21.15.60:0/157916091 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f13780583c0 con 0x7f13880c6950 2024-04-14T23:54:43.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.370+0000 7f1387e00640 1 -- 172.21.15.60:0/157916091 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f13880c7970 con 0x7f13880c6950 2024-04-14T23:54:43.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.372+0000 7f137f400640 1 -- 172.21.15.60:0/157916091 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6839 (secure 0 0 0) 0x7f137805c460 con 0x7f13880c6950 2024-04-14T23:54:43.377 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:54:43.377 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4986,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/1971580590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1971580590},{"type":"v1","addr":"172.21.15.60:6833","nonce":1971580590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:52:53.627240+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4839},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4839":{"gid":4839,"name":"c","rank":0,"incarnation":15,"state":"up:active","state_seq":24,"addr":"172.21.15.60:6835/1714133698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1714133698},{"type":"v1","addr":"172.21.15.60:6835","nonce":1714133698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4839,"qdb_cluster":[4839]},"id":1}]} 2024-04-14T23:54:43.377 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 20 2024-04-14T23:54:43.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.374+0000 7f1387e00640 1 -- 172.21.15.60:0/157916091 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f136406b990 msgr2=0x7f136406de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:43.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.374+0000 7f1387e00640 1 --2- 172.21.15.60:0/157916091 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f136406b990 0x7f136406de50 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f1388156e60 tx=0x7f1370009040 comp rx=0 tx=0).stop 2024-04-14T23:54:43.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.374+0000 7f1387e00640 1 -- 172.21.15.60:0/157916091 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13880c6950 msgr2=0x7f13880c12e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:43.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.374+0000 7f1387e00640 1 --2- 172.21.15.60:0/157916091 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13880c6950 0x7f13880c12e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f137800c8f0 tx=0x7f137800cdb0 comp rx=0 tx=0).stop 2024-04-14T23:54:43.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.375+0000 7f1387e00640 1 -- 172.21.15.60:0/157916091 shutdown_connections 2024-04-14T23:54:43.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.375+0000 7f1387e00640 1 --2- 172.21.15.60:0/157916091 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f136406b990 0x7f136406de50 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:43.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.375+0000 7f1387e00640 1 --2- 172.21.15.60:0/157916091 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f138815ba70 0x7f13880b0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:43.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.375+0000 7f1387e00640 1 --2- 172.21.15.60:0/157916091 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f13880c7080 0x7f13880c1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:43.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.375+0000 7f1387e00640 1 --2- 172.21.15.60:0/157916091 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13880c6950 0x7f13880c12e0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:43.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.375+0000 7f1387e00640 1 -- 172.21.15.60:0/157916091 >> 172.21.15.60:0/157916091 conn(0x7f13880c8b50 msgr2=0x7f13880c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:54:43.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.375+0000 7f1387e00640 1 -- 172.21.15.60:0/157916091 shutdown_connections 2024-04-14T23:54:43.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:43.375+0000 7f1387e00640 1 -- 172.21.15.60:0/157916091 wait complete. 2024-04-14T23:54:43.397 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-14T23:54:45.398 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:54:45.492 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-14T23:54:45.501 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-14T23:54:45.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.622+0000 7ff389000640 1 Processor -- start 2024-04-14T23:54:45.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.623+0000 7ff389000640 1 -- start start 2024-04-14T23:54:45.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.624+0000 7ff389000640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff384152cf0 0x7ff38415c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:45.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.624+0000 7ff389000640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3841530d0 con 0x7ff3840c63c0 2024-04-14T23:54:45.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.624+0000 7ff389000640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3840c4da0 con 0x7ff384159450 2024-04-14T23:54:45.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.624+0000 7ff389000640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7ff3840751f0 con 0x7ff384152cf0 2024-04-14T23:54:45.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.625+0000 7ff381e00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff384152cf0 0x7ff38415c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:45.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.625+0000 7ff381400640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7ff3840c63c0 0x7ff3840c3f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:46502/0 (socket says 172.21.15.60:46502) 2024-04-14T23:54:45.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.625+0000 7ff381400640 1 -- 172.21.15.60:0/2161765954 learned_addr learned my addr 172.21.15.60:0/2161765954 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:54:45.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.625+0000 7ff381e00640 1 -- 172.21.15.60:0/2161765954 >> v1:172.21.15.186:6789/0 conn(0x7ff384159450 legacy=0x7ff3840c4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:45.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.625+0000 7ff380a00640 1 -- 172.21.15.60:0/2161765954 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4142181094 0 0) 0x7ff3841530d0 con 0x7ff3840c63c0 2024-04-14T23:54:45.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.626+0000 7ff381e00640 1 -- 172.21.15.60:0/2161765954 >> v1:172.21.15.60:6789/0 conn(0x7ff3840c63c0 legacy=0x7ff3840c3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:45.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.626+0000 7ff381e00640 1 -- 172.21.15.60:0/2161765954 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3840c8370 con 0x7ff384152cf0 2024-04-14T23:54:45.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.627+0000 7ff381e00640 1 --2- 172.21.15.60:0/2161765954 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff384152cf0 0x7ff38415c2a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ff374009720 tx=0x7ff37402b7c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70f7b8766b01b1e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:54:45.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.627+0000 7ff380a00640 1 -- 172.21.15.60:0/2161765954 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff374038070 con 0x7ff384152cf0 2024-04-14T23:54:45.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.627+0000 7ff380a00640 1 -- 172.21.15.60:0/2161765954 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff374031520 con 0x7ff384152cf0 2024-04-14T23:54:45.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.627+0000 7ff380a00640 1 -- 172.21.15.60:0/2161765954 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3740318f0 con 0x7ff384152cf0 2024-04-14T23:54:45.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.628+0000 7ff389000640 1 -- 172.21.15.60:0/2161765954 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff384152cf0 msgr2=0x7ff38415c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:45.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.628+0000 7ff389000640 1 --2- 172.21.15.60:0/2161765954 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff384152cf0 0x7ff38415c2a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ff374009720 tx=0x7ff37402b7c0 comp rx=0 tx=0).stop 2024-04-14T23:54:45.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.628+0000 7ff389000640 1 -- 172.21.15.60:0/2161765954 shutdown_connections 2024-04-14T23:54:45.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.628+0000 7ff389000640 1 --2- 172.21.15.60:0/2161765954 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff384152cf0 0x7ff38415c2a0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:45.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.628+0000 7ff389000640 1 -- 172.21.15.60:0/2161765954 >> 172.21.15.60:0/2161765954 conn(0x7ff3840cc170 msgr2=0x7ff3840cc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:54:45.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.628+0000 7ff389000640 1 -- 172.21.15.60:0/2161765954 shutdown_connections 2024-04-14T23:54:45.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.628+0000 7ff389000640 1 -- 172.21.15.60:0/2161765954 wait complete. 2024-04-14T23:54:45.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.630+0000 7ff389000640 1 Processor -- start 2024-04-14T23:54:45.635 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.630+0000 7ff389000640 1 -- start start 2024-04-14T23:54:45.636 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.631+0000 7ff389000640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff3840c63c0 0x7ff3841f0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:45.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.631+0000 7ff389000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff384152cf0 0x7ff384071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:45.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.631+0000 7ff389000640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff384159450 0x7ff38415a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:45.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.631+0000 7ff389000640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff3840c8610 con 0x7ff384159450 2024-04-14T23:54:45.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.631+0000 7ff389000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff3840c3af0 con 0x7ff384152cf0 2024-04-14T23:54:45.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.631+0000 7ff389000640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7ff3840c88f0 con 0x7ff3840c63c0 2024-04-14T23:54:45.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.631+0000 7ff382800640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff384159450 0x7ff38415a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:45.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.631+0000 7ff381400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff384152cf0 0x7ff384071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:45.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.631+0000 7ff381e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff3840c63c0 0x7ff3841f0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:45.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.631+0000 7ff382800640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff384159450 0x7ff38415a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57530/0 (socket says 172.21.15.60:57530) 2024-04-14T23:54:45.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.631+0000 7ff382800640 1 -- 172.21.15.60:0/1821803898 learned_addr learned my addr 172.21.15.60:0/1821803898 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:54:45.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.631+0000 7ff381400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff384152cf0 0x7ff384071910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:48300/0 (socket says 172.21.15.60:48300) 2024-04-14T23:54:45.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.632+0000 7ff382800640 1 -- 172.21.15.60:0/1821803898 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff3840c63c0 msgr2=0x7ff3841f0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:45.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.632+0000 7ff382800640 1 --2- 172.21.15.60:0/1821803898 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff3840c63c0 0x7ff3841f0d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:45.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.632+0000 7ff382800640 1 -- 172.21.15.60:0/1821803898 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff384152cf0 msgr2=0x7ff384071910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:45.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.632+0000 7ff382800640 1 --2- 172.21.15.60:0/1821803898 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff384152cf0 0x7ff384071910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:45.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.632+0000 7ff382800640 1 -- 172.21.15.60:0/1821803898 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff384159fd0 con 0x7ff384159450 2024-04-14T23:54:45.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.632+0000 7ff382800640 1 --2- 172.21.15.60:0/1821803898 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff384159450 0x7ff38415a790 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff37c00cb20 tx=0x7ff37c0020b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:54:45.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.632+0000 7ff37a000640 1 -- 172.21.15.60:0/1821803898 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff37c017070 con 0x7ff384159450 2024-04-14T23:54:45.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.632+0000 7ff37a000640 1 -- 172.21.15.60:0/1821803898 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff37c003da0 con 0x7ff384159450 2024-04-14T23:54:45.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.632+0000 7ff37a000640 1 -- 172.21.15.60:0/1821803898 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff37c00a920 con 0x7ff384159450 2024-04-14T23:54:45.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.632+0000 7ff389000640 1 -- 172.21.15.60:0/1821803898 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3841f1360 con 0x7ff384159450 2024-04-14T23:54:45.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.632+0000 7ff389000640 1 -- 172.21.15.60:0/1821803898 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3841f1920 con 0x7ff384159450 2024-04-14T23:54:45.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.633+0000 7ff389000640 1 -- 172.21.15.60:0/1821803898 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff384076970 con 0x7ff384159450 2024-04-14T23:54:45.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.635+0000 7ff37a000640 1 -- 172.21.15.60:0/1821803898 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7ff37c01a070 con 0x7ff384159450 2024-04-14T23:54:45.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.636+0000 7ff37a000640 1 --2- 172.21.15.60:0/1821803898 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff36406b9c0 0x7ff36406de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:45.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.636+0000 7ff37a000640 1 -- 172.21.15.60:0/1821803898 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 6025+0+0 (secure 0 0 0) 0x7ff37c08a1e0 con 0x7ff384159450 2024-04-14T23:54:45.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.636+0000 7ff381e00640 1 --2- 172.21.15.60:0/1821803898 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff36406b9c0 0x7ff36406de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:45.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.637+0000 7ff381e00640 1 --2- 172.21.15.60:0/1821803898 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff36406b9c0 0x7ff36406de80 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff374035000 tx=0x7ff374030000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:54:45.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.641+0000 7ff37a000640 1 -- 172.21.15.60:0/1821803898 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff37c058530 con 0x7ff384159450 2024-04-14T23:54:45.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.887+0000 7ff389000640 1 -- 172.21.15.60:0/1821803898 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff3841562f0 con 0x7ff384159450 2024-04-14T23:54:45.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.889+0000 7ff37a000640 1 -- 172.21.15.60:0/1821803898 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6839 (secure 0 0 0) 0x7ff37c05c5d0 con 0x7ff384159450 2024-04-14T23:54:45.894 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:54:45.894 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4986,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/1971580590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1971580590},{"type":"v1","addr":"172.21.15.60:6833","nonce":1971580590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:52:53.627240+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4839},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4839":{"gid":4839,"name":"c","rank":0,"incarnation":15,"state":"up:active","state_seq":24,"addr":"172.21.15.60:6835/1714133698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1714133698},{"type":"v1","addr":"172.21.15.60:6835","nonce":1714133698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4839,"qdb_cluster":[4839]},"id":1}]} 2024-04-14T23:54:45.895 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 20 2024-04-14T23:54:45.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.891+0000 7ff389000640 1 -- 172.21.15.60:0/1821803898 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff36406b9c0 msgr2=0x7ff36406de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:45.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.891+0000 7ff389000640 1 --2- 172.21.15.60:0/1821803898 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff36406b9c0 0x7ff36406de80 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff374035000 tx=0x7ff374030000 comp rx=0 tx=0).stop 2024-04-14T23:54:45.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.891+0000 7ff389000640 1 -- 172.21.15.60:0/1821803898 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff384159450 msgr2=0x7ff38415a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:45.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.891+0000 7ff389000640 1 --2- 172.21.15.60:0/1821803898 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff384159450 0x7ff38415a790 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff37c00cb20 tx=0x7ff37c0020b0 comp rx=0 tx=0).stop 2024-04-14T23:54:45.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.892+0000 7ff389000640 1 -- 172.21.15.60:0/1821803898 shutdown_connections 2024-04-14T23:54:45.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.892+0000 7ff389000640 1 --2- 172.21.15.60:0/1821803898 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff36406b9c0 0x7ff36406de80 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:45.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.892+0000 7ff389000640 1 --2- 172.21.15.60:0/1821803898 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff384159450 0x7ff38415a790 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:45.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.892+0000 7ff389000640 1 --2- 172.21.15.60:0/1821803898 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff384152cf0 0x7ff384071910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:45.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.892+0000 7ff389000640 1 --2- 172.21.15.60:0/1821803898 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff3840c63c0 0x7ff3841f0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:45.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.892+0000 7ff389000640 1 -- 172.21.15.60:0/1821803898 >> 172.21.15.60:0/1821803898 conn(0x7ff3840cc170 msgr2=0x7ff38406e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:54:45.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.892+0000 7ff389000640 1 -- 172.21.15.60:0/1821803898 shutdown_connections 2024-04-14T23:54:45.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:45.892+0000 7ff389000640 1 -- 172.21.15.60:0/1821803898 wait complete. 2024-04-14T23:54:45.916 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-14T23:54:47.917 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:54:48.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.146+0000 7f8fd5800640 1 Processor -- start 2024-04-14T23:54:48.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.146+0000 7f8fd5800640 1 -- start start 2024-04-14T23:54:48.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.148+0000 7f8fd5800640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8fd0154b20 0x7f8fd00cbf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:48.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.149+0000 7f8fd5800640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8fd015e930 0x7f8fd015ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:48.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.149+0000 7f8fd5800640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fd006cda0 con 0x7f8fd00cc720 2024-04-14T23:54:48.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.149+0000 7f8fd5800640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f8fd006d2f0 con 0x7f8fd015e930 2024-04-14T23:54:48.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.149+0000 7f8fd5800640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f8fd00cbc20 con 0x7f8fd0154b20 2024-04-14T23:54:48.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.149+0000 7f8fcfe00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8fd0154b20 0x7f8fd00cbf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:48.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.149+0000 7f8fcf400640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f8fd00cc720 0x7f8fd00d0230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.60:57786/0 (socket says 172.21.15.60:57786) 2024-04-14T23:54:48.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.149+0000 7f8fcf400640 1 -- 172.21.15.60:0/2584089943 learned_addr learned my addr 172.21.15.60:0/2584089943 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:54:48.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.149+0000 7f8fd4e00640 1 --2- 172.21.15.60:0/2584089943 >> v2:172.21.15.60:3300/0 conn(0x7f8fd015e930 0x7f8fd015ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:48.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.149+0000 7f8fcfe00640 1 -- 172.21.15.60:0/2584089943 >> v2:172.21.15.60:3300/0 conn(0x7f8fd015e930 msgr2=0x7f8fd015ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:54:48.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.149+0000 7f8fcfe00640 1 --2- 172.21.15.60:0/2584089943 >> v2:172.21.15.60:3300/0 conn(0x7f8fd015e930 0x7f8fd015ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:48.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.149+0000 7f8fcfe00640 1 -- 172.21.15.60:0/2584089943 >> v1:172.21.15.186:6789/0 conn(0x7f8fd00cc720 legacy=0x7f8fd00d0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:48.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.150+0000 7f8fcfe00640 1 -- 172.21.15.60:0/2584089943 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fd006d010 con 0x7f8fd0154b20 2024-04-14T23:54:48.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.150+0000 7f8fcea00640 1 -- 172.21.15.60:0/2584089943 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 660577997 0 0) 0x7f8fd006cda0 con 0x7f8fd00cc720 2024-04-14T23:54:48.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.151+0000 7f8fcfe00640 1 --2- 172.21.15.60:0/2584089943 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8fd0154b20 0x7f8fd00cbf00 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8fc00097a0 tx=0x7f8fc002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c693456cc31bf30a server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:54:48.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.151+0000 7f8fcea00640 1 -- 172.21.15.60:0/2584089943 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc0035020 con 0x7f8fd0154b20 2024-04-14T23:54:48.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.151+0000 7f8fcea00640 1 -- 172.21.15.60:0/2584089943 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fc0031830 con 0x7f8fd0154b20 2024-04-14T23:54:48.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.151+0000 7f8fcea00640 1 -- 172.21.15.60:0/2584089943 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc0031c00 con 0x7f8fd0154b20 2024-04-14T23:54:48.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.152+0000 7f8fd5800640 1 -- 172.21.15.60:0/2584089943 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8fd0154b20 msgr2=0x7f8fd00cbf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:48.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.152+0000 7f8fd5800640 1 --2- 172.21.15.60:0/2584089943 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8fd0154b20 0x7f8fd00cbf00 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8fc00097a0 tx=0x7f8fc002b8d0 comp rx=0 tx=0).stop 2024-04-14T23:54:48.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.152+0000 7f8fd5800640 1 -- 172.21.15.60:0/2584089943 shutdown_connections 2024-04-14T23:54:48.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.152+0000 7f8fd5800640 1 --2- 172.21.15.60:0/2584089943 >> v2:172.21.15.60:3300/0 conn(0x7f8fd015e930 0x7f8fd015ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:48.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.152+0000 7f8fd5800640 1 --2- 172.21.15.60:0/2584089943 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8fd0154b20 0x7f8fd00cbf00 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:48.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.152+0000 7f8fd5800640 1 -- 172.21.15.60:0/2584089943 >> 172.21.15.60:0/2584089943 conn(0x7f8fd006ddb0 msgr2=0x7f8fd0066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:54:48.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.152+0000 7f8fd5800640 1 -- 172.21.15.60:0/2584089943 shutdown_connections 2024-04-14T23:54:48.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.152+0000 7f8fd5800640 1 -- 172.21.15.60:0/2584089943 wait complete. 2024-04-14T23:54:48.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.154+0000 7f8fd5800640 1 Processor -- start 2024-04-14T23:54:48.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.154+0000 7f8fd5800640 1 -- start start 2024-04-14T23:54:48.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.156+0000 7f8fd5800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8fd00cc720 0x7f8fd00b49d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:48.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.156+0000 7f8fd5800640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8fd0154b20 0x7f8fd00b4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:48.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.156+0000 7f8fd5800640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8fd015e930 0x7f8fd0159790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:48.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.156+0000 7f8fd5800640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f8fd0155080 con 0x7f8fd0154b20 2024-04-14T23:54:48.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.156+0000 7f8fd5800640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8fd0154f00 con 0x7f8fd00cc720 2024-04-14T23:54:48.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.156+0000 7f8fd5800640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f8fd00d0940 con 0x7f8fd015e930 2024-04-14T23:54:48.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.156+0000 7f8fcf400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8fd0154b20 0x7f8fd00b4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:48.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.156+0000 7f8fcfe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8fd00cc720 0x7f8fd00b49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:48.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.156+0000 7f8fd4e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8fd015e930 0x7f8fd0159790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:48.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.156+0000 7f8fcfe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8fd00cc720 0x7f8fd00b49d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:48316/0 (socket says 172.21.15.60:48316) 2024-04-14T23:54:48.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.156+0000 7f8fcf400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8fd0154b20 0x7f8fd00b4f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57554/0 (socket says 172.21.15.60:57554) 2024-04-14T23:54:48.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.156+0000 7f8fcfe00640 1 -- 172.21.15.60:0/3927763603 learned_addr learned my addr 172.21.15.60:0/3927763603 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:54:48.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.156+0000 7f8fd4e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8fd015e930 0x7f8fd0159790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.60:33424/0 (socket says 172.21.15.60:33424) 2024-04-14T23:54:48.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.157+0000 7f8fd4e00640 1 -- 172.21.15.60:0/3927763603 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8fd00cc720 msgr2=0x7f8fd00b49d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:48.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.157+0000 7f8fd4e00640 1 --2- 172.21.15.60:0/3927763603 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8fd00cc720 0x7f8fd00b49d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:48.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.157+0000 7f8fd4e00640 1 -- 172.21.15.60:0/3927763603 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8fd0154b20 msgr2=0x7f8fd00b4f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:48.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.157+0000 7f8fd4e00640 1 --2- 172.21.15.60:0/3927763603 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8fd0154b20 0x7f8fd00b4f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:48.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.157+0000 7f8fd4e00640 1 -- 172.21.15.60:0/3927763603 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fd01f5690 con 0x7f8fd015e930 2024-04-14T23:54:48.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.157+0000 7f8fcfe00640 1 --2- 172.21.15.60:0/3927763603 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8fd00cc720 0x7f8fd00b49d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:54:48.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.157+0000 7f8fcf400640 1 --2- 172.21.15.60:0/3927763603 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8fd0154b20 0x7f8fd00b4f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:54:48.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.157+0000 7f8fd4e00640 1 --2- 172.21.15.60:0/3927763603 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8fd015e930 0x7f8fd0159790 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8fc800ebd0 tx=0x7f8fc800c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:54:48.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.157+0000 7f8fc7e00640 1 -- 172.21.15.60:0/3927763603 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc8018020 con 0x7f8fd015e930 2024-04-14T23:54:48.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.157+0000 7f8fc7e00640 1 -- 172.21.15.60:0/3927763603 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fc800cd20 con 0x7f8fd015e930 2024-04-14T23:54:48.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.157+0000 7f8fc7e00640 1 -- 172.21.15.60:0/3927763603 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc80146c0 con 0x7f8fd015e930 2024-04-14T23:54:48.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.157+0000 7f8fd5800640 1 -- 172.21.15.60:0/3927763603 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fd00c6390 con 0x7f8fd015e930 2024-04-14T23:54:48.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.158+0000 7f8fd5800640 1 -- 172.21.15.60:0/3927763603 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fd0076970 con 0x7f8fd015e930 2024-04-14T23:54:48.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.159+0000 7f8fd5800640 1 -- 172.21.15.60:0/3927763603 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fd00c68b0 con 0x7f8fd015e930 2024-04-14T23:54:48.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.160+0000 7f8fc7e00640 1 -- 172.21.15.60:0/3927763603 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f8fc8005d10 con 0x7f8fd015e930 2024-04-14T23:54:48.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.161+0000 7f8fc7e00640 1 --2- 172.21.15.60:0/3927763603 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f8fb006b9c0 0x7f8fb006de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:48.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.161+0000 7f8fc7e00640 1 -- 172.21.15.60:0/3927763603 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(34..34 src has 1..34) ==== 6025+0+0 (secure 0 0 0) 0x7f8fc808b510 con 0x7f8fd015e930 2024-04-14T23:54:48.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.161+0000 7f8fcfe00640 1 --2- 172.21.15.60:0/3927763603 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f8fb006b9c0 0x7f8fb006de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:48.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.162+0000 7f8fcfe00640 1 --2- 172.21.15.60:0/3927763603 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f8fb006b9c0 0x7f8fb006de80 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f8fc002b910 tx=0x7f8fc003c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:54:48.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.165+0000 7f8fc7e00640 1 -- 172.21.15.60:0/3927763603 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8fd00c68b0 con 0x7f8fd015e930 2024-04-14T23:54:48.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.409+0000 7f8fd5800640 1 -- 172.21.15.60:0/3927763603 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8fd004f540 con 0x7f8fd015e930 2024-04-14T23:54:48.415 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.410+0000 7f8fc7e00640 1 -- 172.21.15.60:0/3927763603 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6839 (secure 0 0 0) 0x7f8fc8022030 con 0x7f8fd015e930 2024-04-14T23:54:48.415 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:54:48.415 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4986,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/1971580590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1971580590},{"type":"v1","addr":"172.21.15.60:6833","nonce":1971580590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:52:53.627240+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4839},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4839":{"gid":4839,"name":"c","rank":0,"incarnation":15,"state":"up:active","state_seq":24,"addr":"172.21.15.60:6835/1714133698","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1714133698},{"type":"v1","addr":"172.21.15.60:6835","nonce":1714133698}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4839,"qdb_cluster":[4839]},"id":1}]} 2024-04-14T23:54:48.415 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 20 2024-04-14T23:54:48.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.412+0000 7f8fd5800640 1 -- 172.21.15.60:0/3927763603 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f8fb006b9c0 msgr2=0x7f8fb006de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:48.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.412+0000 7f8fd5800640 1 --2- 172.21.15.60:0/3927763603 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f8fb006b9c0 0x7f8fb006de80 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f8fc002b910 tx=0x7f8fc003c040 comp rx=0 tx=0).stop 2024-04-14T23:54:48.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.412+0000 7f8fd5800640 1 -- 172.21.15.60:0/3927763603 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8fd015e930 msgr2=0x7f8fd0159790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:48.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.412+0000 7f8fd5800640 1 --2- 172.21.15.60:0/3927763603 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8fd015e930 0x7f8fd0159790 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8fc800ebd0 tx=0x7f8fc800c490 comp rx=0 tx=0).stop 2024-04-14T23:54:48.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.413+0000 7f8fd5800640 1 -- 172.21.15.60:0/3927763603 shutdown_connections 2024-04-14T23:54:48.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.413+0000 7f8fd5800640 1 --2- 172.21.15.60:0/3927763603 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f8fb006b9c0 0x7f8fb006de80 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:48.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.413+0000 7f8fd5800640 1 --2- 172.21.15.60:0/3927763603 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8fd015e930 0x7f8fd0159790 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:48.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.413+0000 7f8fd5800640 1 --2- 172.21.15.60:0/3927763603 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8fd0154b20 0x7f8fd00b4f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:48.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.413+0000 7f8fd5800640 1 --2- 172.21.15.60:0/3927763603 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8fd00cc720 0x7f8fd00b49d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:48.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.413+0000 7f8fd5800640 1 -- 172.21.15.60:0/3927763603 >> 172.21.15.60:0/3927763603 conn(0x7f8fd006ddb0 msgr2=0x7f8fd014fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:54:48.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.413+0000 7f8fd5800640 1 -- 172.21.15.60:0/3927763603 shutdown_connections 2024-04-14T23:54:48.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:48.413+0000 7f8fd5800640 1 -- 172.21.15.60:0/3927763603 wait complete. 2024-04-14T23:54:48.439 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-14T23:54:50.440 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:54:50.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.666+0000 7f67bd600640 1 Processor -- start 2024-04-14T23:54:50.672 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.667+0000 7f67bd600640 1 -- start start 2024-04-14T23:54:50.674 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.668+0000 7f67bd600640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f67b8152820 0x7f67b8152c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:50.674 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.668+0000 7f67bd600640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f67b8071220 0x7f67b80ce380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:50.674 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.668+0000 7f67bd600640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67b80c9100 con 0x7f67b80c3410 2024-04-14T23:54:50.674 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.669+0000 7f67bd600640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f67b80c93a0 con 0x7f67b8071220 2024-04-14T23:54:50.674 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.669+0000 7f67bd600640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f67b80c9680 con 0x7f67b8152820 2024-04-14T23:54:50.674 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.669+0000 7f67b7400640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f67b80c3410 0x7f67b8159280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:46514/0 (socket says 172.21.15.60:46514) 2024-04-14T23:54:50.674 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.669+0000 7f67b7400640 1 -- 172.21.15.60:0/58988232 learned_addr learned my addr 172.21.15.60:0/58988232 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:54:50.674 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.669+0000 7f67bcc00640 1 --2- 172.21.15.60:0/58988232 >> v2:172.21.15.186:3300/0 conn(0x7f67b8071220 0x7f67b80ce380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:50.674 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.669+0000 7f67b7e00640 1 --2- 172.21.15.60:0/58988232 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f67b8152820 0x7f67b8152c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:50.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.669+0000 7f67b6a00640 1 -- 172.21.15.60:0/58988232 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 447151477 0 0) 0x7f67b80c9100 con 0x7f67b80c3410 2024-04-14T23:54:50.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.669+0000 7f67b7e00640 1 -- 172.21.15.60:0/58988232 >> v2:172.21.15.186:3300/0 conn(0x7f67b8071220 msgr2=0x7f67b80ce380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:54:50.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.669+0000 7f67b7e00640 1 --2- 172.21.15.60:0/58988232 >> v2:172.21.15.186:3300/0 conn(0x7f67b8071220 0x7f67b80ce380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:50.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.669+0000 7f67b7e00640 1 -- 172.21.15.60:0/58988232 >> v1:172.21.15.60:6789/0 conn(0x7f67b80c3410 legacy=0x7f67b8159280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:50.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.669+0000 7f67b7e00640 1 -- 172.21.15.60:0/58988232 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67b80c9960 con 0x7f67b8152820 2024-04-14T23:54:50.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.670+0000 7f67bcc00640 1 --2- 172.21.15.60:0/58988232 >> v2:172.21.15.186:3300/0 conn(0x7f67b8071220 0x7f67b80ce380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:54:50.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.671+0000 7f67b7e00640 1 --2- 172.21.15.60:0/58988232 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f67b8152820 0x7f67b8152c00 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f67a00097a0 tx=0x7f67a002b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76926ba03367ffc4 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:54:50.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.671+0000 7f67b6a00640 1 -- 172.21.15.60:0/58988232 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67a0034070 con 0x7f67b8152820 2024-04-14T23:54:50.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.671+0000 7f67b6a00640 1 -- 172.21.15.60:0/58988232 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67a0030800 con 0x7f67b8152820 2024-04-14T23:54:50.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.671+0000 7f67b6a00640 1 -- 172.21.15.60:0/58988232 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67a0030bd0 con 0x7f67b8152820 2024-04-14T23:54:50.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.672+0000 7f67bd600640 1 -- 172.21.15.60:0/58988232 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f67b8152820 msgr2=0x7f67b8152c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:50.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.672+0000 7f67bd600640 1 --2- 172.21.15.60:0/58988232 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f67b8152820 0x7f67b8152c00 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f67a00097a0 tx=0x7f67a002b940 comp rx=0 tx=0).stop 2024-04-14T23:54:50.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.672+0000 7f67bd600640 1 -- 172.21.15.60:0/58988232 shutdown_connections 2024-04-14T23:54:50.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.672+0000 7f67bd600640 1 --2- 172.21.15.60:0/58988232 >> v2:172.21.15.186:3300/0 conn(0x7f67b8071220 0x7f67b80ce380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:50.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.672+0000 7f67bd600640 1 --2- 172.21.15.60:0/58988232 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f67b8152820 0x7f67b8152c00 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:50.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.672+0000 7f67bd600640 1 -- 172.21.15.60:0/58988232 >> 172.21.15.60:0/58988232 conn(0x7f67b8066f10 msgr2=0x7f67b8067320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:54:50.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.672+0000 7f67bd600640 1 -- 172.21.15.60:0/58988232 shutdown_connections 2024-04-14T23:54:50.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.672+0000 7f67bd600640 1 -- 172.21.15.60:0/58988232 wait complete. 2024-04-14T23:54:50.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.674+0000 7f67bd600640 1 Processor -- start 2024-04-14T23:54:50.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.674+0000 7f67bd600640 1 -- start start 2024-04-14T23:54:50.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.675+0000 7f67bd600640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f67b8071220 0x7f67b80b2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:50.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.675+0000 7f67bd600640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67b80c3410 0x7f67b806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:50.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.675+0000 7f67bd600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67b8152820 0x7f67b80cc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:50.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.675+0000 7f67bd600640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f67b80c8df0 con 0x7f67b80c3410 2024-04-14T23:54:50.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.675+0000 7f67bd600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f67b8158d60 con 0x7f67b8152820 2024-04-14T23:54:50.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.675+0000 7f67bd600640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f67b80b0fa0 con 0x7f67b8071220 2024-04-14T23:54:50.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.675+0000 7f67b7400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67b80c3410 0x7f67b806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:50.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.675+0000 7f67b7400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67b80c3410 0x7f67b806d810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57558/0 (socket says 172.21.15.60:57558) 2024-04-14T23:54:50.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.675+0000 7f67b7400640 1 -- 172.21.15.60:0/1004165203 learned_addr learned my addr 172.21.15.60:0/1004165203 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:54:50.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.675+0000 7f67b7e00640 1 --2- 172.21.15.60:0/1004165203 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f67b8071220 0x7f67b80b2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:50.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.675+0000 7f67bcc00640 1 --2- 172.21.15.60:0/1004165203 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67b8152820 0x7f67b80cc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:50.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.676+0000 7f67b7400640 1 -- 172.21.15.60:0/1004165203 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f67b8071220 msgr2=0x7f67b80b2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:50.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.676+0000 7f67b7400640 1 --2- 172.21.15.60:0/1004165203 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f67b8071220 0x7f67b80b2ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:50.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.676+0000 7f67b7400640 1 -- 172.21.15.60:0/1004165203 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67b8152820 msgr2=0x7f67b80cc1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:50.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.676+0000 7f67b7400640 1 --2- 172.21.15.60:0/1004165203 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67b8152820 0x7f67b80cc1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:50.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.676+0000 7f67b7400640 1 -- 172.21.15.60:0/1004165203 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67b80b1e50 con 0x7f67b80c3410 2024-04-14T23:54:50.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.676+0000 7f67b7400640 1 --2- 172.21.15.60:0/1004165203 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67b80c3410 0x7f67b806d810 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f67a800cc10 tx=0x7f67a800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:54:50.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.676+0000 7f67afe00640 1 -- 172.21.15.60:0/1004165203 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67a8017070 con 0x7f67b80c3410 2024-04-14T23:54:50.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.676+0000 7f67afe00640 1 -- 172.21.15.60:0/1004165203 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67a800ad00 con 0x7f67b80c3410 2024-04-14T23:54:50.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.676+0000 7f67afe00640 1 -- 172.21.15.60:0/1004165203 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67a8012640 con 0x7f67b80c3410 2024-04-14T23:54:50.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.676+0000 7f67bd600640 1 -- 172.21.15.60:0/1004165203 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67b81f06b0 con 0x7f67b80c3410 2024-04-14T23:54:50.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.676+0000 7f67bd600640 1 -- 172.21.15.60:0/1004165203 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67b81f0c70 con 0x7f67b80c3410 2024-04-14T23:54:50.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.678+0000 7f67bd600640 1 -- 172.21.15.60:0/1004165203 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67b80b3ad0 con 0x7f67b80c3410 2024-04-14T23:54:50.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.679+0000 7f67afe00640 1 -- 172.21.15.60:0/1004165203 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f67a801a070 con 0x7f67b80c3410 2024-04-14T23:54:50.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.679+0000 7f67afe00640 1 --2- 172.21.15.60:0/1004165203 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f679406b990 0x7f679406de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:50.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.680+0000 7f67afe00640 1 -- 172.21.15.60:0/1004165203 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6083+0+0 (secure 0 0 0) 0x7f67a808a6d0 con 0x7f67b80c3410 2024-04-14T23:54:50.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.680+0000 7f67b7e00640 1 --2- 172.21.15.60:0/1004165203 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f679406b990 0x7f679406de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:50.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.681+0000 7f67b7e00640 1 --2- 172.21.15.60:0/1004165203 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f679406b990 0x7f679406de50 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f67a0036000 tx=0x7f67a0008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:54:50.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.684+0000 7f67afe00640 1 -- 172.21.15.60:0/1004165203 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f67b80b3ad0 con 0x7f67b80c3410 2024-04-14T23:54:50.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.923+0000 7f67bd600640 1 -- 172.21.15.60:0/1004165203 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f67b80b06a0 con 0x7f67b80c3410 2024-04-14T23:54:50.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.925+0000 7f67afe00640 1 -- 172.21.15.60:0/1004165203 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+6006 (secure 0 0 0) 0x7f67b80b06a0 con 0x7f67b80c3410 2024-04-14T23:54:50.930 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:54:50.930 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:54:48.943349+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4986},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4986":{"gid":4986,"name":"a","rank":0,"incarnation":21,"state":"up:replay","state_seq":1,"addr":"172.21.15.60:6833/1971580590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1971580590},{"type":"v1","addr":"172.21.15.60:6833","nonce":1971580590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-14T23:54:50.930 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 21 2024-04-14T23:54:50.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.927+0000 7f67bd600640 1 -- 172.21.15.60:0/1004165203 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f679406b990 msgr2=0x7f679406de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:50.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.927+0000 7f67bd600640 1 --2- 172.21.15.60:0/1004165203 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f679406b990 0x7f679406de50 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f67a0036000 tx=0x7f67a0008030 comp rx=0 tx=0).stop 2024-04-14T23:54:50.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.928+0000 7f67bd600640 1 -- 172.21.15.60:0/1004165203 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67b80c3410 msgr2=0x7f67b806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:50.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.928+0000 7f67bd600640 1 --2- 172.21.15.60:0/1004165203 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67b80c3410 0x7f67b806d810 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f67a800cc10 tx=0x7f67a800a430 comp rx=0 tx=0).stop 2024-04-14T23:54:50.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.928+0000 7f67bd600640 1 -- 172.21.15.60:0/1004165203 shutdown_connections 2024-04-14T23:54:50.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.928+0000 7f67bd600640 1 --2- 172.21.15.60:0/1004165203 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f679406b990 0x7f679406de50 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:50.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.928+0000 7f67bd600640 1 --2- 172.21.15.60:0/1004165203 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67b8152820 0x7f67b80cc1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:50.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.928+0000 7f67bd600640 1 --2- 172.21.15.60:0/1004165203 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67b80c3410 0x7f67b806d810 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:50.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.928+0000 7f67bd600640 1 --2- 172.21.15.60:0/1004165203 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f67b8071220 0x7f67b80b2ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:50.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.928+0000 7f67bd600640 1 -- 172.21.15.60:0/1004165203 >> 172.21.15.60:0/1004165203 conn(0x7f67b8066f10 msgr2=0x7f67b8151d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:54:50.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.928+0000 7f67bd600640 1 -- 172.21.15.60:0/1004165203 shutdown_connections 2024-04-14T23:54:50.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:50.928+0000 7f67bd600640 1 -- 172.21.15.60:0/1004165203 wait complete. 2024-04-14T23:54:50.954 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c down, removed from mdsmap 2024-04-14T23:54:50.954 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-04-14T23:54:50.954 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:54:51.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.180+0000 7fc331c00640 1 Processor -- start 2024-04-14T23:54:51.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.180+0000 7fc331c00640 1 -- start start 2024-04-14T23:54:51.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.182+0000 7fc331c00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fc32c15c8a0 0x7fc32c072390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:51.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.182+0000 7fc331c00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc32c0c6950 0x7fc32c1698f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:51.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.182+0000 7fc331c00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc32c071bf0 con 0x7fc32c0c86d0 2024-04-14T23:54:51.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.182+0000 7fc331c00640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fc32c15cc80 con 0x7fc32c0c6950 2024-04-14T23:54:51.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.182+0000 7fc331c00640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fc32c1632a0 con 0x7fc32c15c8a0 2024-04-14T23:54:51.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.183+0000 7fc331200640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc32c0c6950 0x7fc32c1698f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:51.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.183+0000 7fc32b400640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fc32c0c86d0 0x7fc32c1625c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:46518/0 (socket says 172.21.15.60:46518) 2024-04-14T23:54:51.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.183+0000 7fc32b400640 1 -- 172.21.15.60:0/1490109593 learned_addr learned my addr 172.21.15.60:0/1490109593 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:54:51.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.183+0000 7fc32be00640 1 --2- 172.21.15.60:0/1490109593 >> v2:172.21.15.186:3300/0 conn(0x7fc32c15c8a0 0x7fc32c072390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:51.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.183+0000 7fc331200640 1 -- 172.21.15.60:0/1490109593 >> v2:172.21.15.186:3300/0 conn(0x7fc32c15c8a0 msgr2=0x7fc32c072390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:54:51.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.183+0000 7fc331200640 1 --2- 172.21.15.60:0/1490109593 >> v2:172.21.15.186:3300/0 conn(0x7fc32c15c8a0 0x7fc32c072390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:51.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.183+0000 7fc331200640 1 -- 172.21.15.60:0/1490109593 >> v1:172.21.15.60:6789/0 conn(0x7fc32c0c86d0 legacy=0x7fc32c1625c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:51.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.183+0000 7fc331200640 1 -- 172.21.15.60:0/1490109593 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc32c0c6fb0 con 0x7fc32c0c6950 2024-04-14T23:54:51.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.183+0000 7fc32aa00640 1 -- 172.21.15.60:0/1490109593 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1625189283 0 0) 0x7fc314001110 con 0x7fc32c0c86d0 2024-04-14T23:54:51.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.185+0000 7fc331200640 1 --2- 172.21.15.60:0/1490109593 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc32c0c6950 0x7fc32c1698f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc32c0c7630 tx=0x7fc32402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4374184ae7fe70d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:54:51.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.185+0000 7fc32aa00640 1 -- 172.21.15.60:0/1490109593 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc324039070 con 0x7fc32c0c6950 2024-04-14T23:54:51.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.185+0000 7fc32aa00640 1 -- 172.21.15.60:0/1490109593 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc32402cc80 con 0x7fc32c0c6950 2024-04-14T23:54:51.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.185+0000 7fc32aa00640 1 -- 172.21.15.60:0/1490109593 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc324034660 con 0x7fc32c0c6950 2024-04-14T23:54:51.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.186+0000 7fc331c00640 1 -- 172.21.15.60:0/1490109593 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc32c0c6950 msgr2=0x7fc32c1698f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:51.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.186+0000 7fc331c00640 1 --2- 172.21.15.60:0/1490109593 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc32c0c6950 0x7fc32c1698f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc32c0c7630 tx=0x7fc32402bed0 comp rx=0 tx=0).stop 2024-04-14T23:54:51.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.186+0000 7fc331c00640 1 -- 172.21.15.60:0/1490109593 shutdown_connections 2024-04-14T23:54:51.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.186+0000 7fc331c00640 1 --2- 172.21.15.60:0/1490109593 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc32c0c6950 0x7fc32c1698f0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:51.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.186+0000 7fc331c00640 1 --2- 172.21.15.60:0/1490109593 >> v2:172.21.15.186:3300/0 conn(0x7fc32c15c8a0 0x7fc32c072390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:51.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.186+0000 7fc331c00640 1 -- 172.21.15.60:0/1490109593 >> 172.21.15.60:0/1490109593 conn(0x7fc32c0c8b50 msgr2=0x7fc32c0c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:54:51.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.186+0000 7fc331c00640 1 -- 172.21.15.60:0/1490109593 shutdown_connections 2024-04-14T23:54:51.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.186+0000 7fc331c00640 1 -- 172.21.15.60:0/1490109593 wait complete. 2024-04-14T23:54:51.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.187+0000 7fc331c00640 1 Processor -- start 2024-04-14T23:54:51.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.188+0000 7fc331c00640 1 -- start start 2024-04-14T23:54:51.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.188+0000 7fc331c00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc32c0c6950 0x7fc32c1550f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:51.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.188+0000 7fc331c00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc32c0c86d0 0x7fc32c1618e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:51.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.189+0000 7fc331c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc32c15c8a0 0x7fc32c1f7c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:51.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.189+0000 7fc331c00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fc32c0728d0 con 0x7fc32c0c6950 2024-04-14T23:54:51.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.189+0000 7fc331c00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc32c071e70 con 0x7fc32c15c8a0 2024-04-14T23:54:51.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.189+0000 7fc331c00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fc32c06b1c0 con 0x7fc32c0c86d0 2024-04-14T23:54:51.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.189+0000 7fc32be00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc32c0c6950 0x7fc32c1550f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:51.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.189+0000 7fc32be00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc32c0c6950 0x7fc32c1550f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57572/0 (socket says 172.21.15.60:57572) 2024-04-14T23:54:51.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.189+0000 7fc32be00640 1 -- 172.21.15.60:0/2096608646 learned_addr learned my addr 172.21.15.60:0/2096608646 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:54:51.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.189+0000 7fc32b400640 1 --2- 172.21.15.60:0/2096608646 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc32c0c86d0 0x7fc32c1618e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:51.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.189+0000 7fc331200640 1 --2- 172.21.15.60:0/2096608646 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc32c15c8a0 0x7fc32c1f7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:51.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.189+0000 7fc32be00640 1 -- 172.21.15.60:0/2096608646 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc32c0c86d0 msgr2=0x7fc32c1618e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:51.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.189+0000 7fc32be00640 1 --2- 172.21.15.60:0/2096608646 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc32c0c86d0 0x7fc32c1618e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:51.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.189+0000 7fc32be00640 1 -- 172.21.15.60:0/2096608646 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc32c15c8a0 msgr2=0x7fc32c1f7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:51.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.189+0000 7fc32be00640 1 --2- 172.21.15.60:0/2096608646 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc32c15c8a0 0x7fc32c1f7c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:51.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.189+0000 7fc32be00640 1 -- 172.21.15.60:0/2096608646 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc32c15d620 con 0x7fc32c0c6950 2024-04-14T23:54:51.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.190+0000 7fc32be00640 1 --2- 172.21.15.60:0/2096608646 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc32c0c6950 0x7fc32c1550f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc31c00e9e0 tx=0x7fc31c00eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:54:51.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.190+0000 7fc323e00640 1 -- 172.21.15.60:0/2096608646 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc31c017070 con 0x7fc32c0c6950 2024-04-14T23:54:51.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.190+0000 7fc323e00640 1 -- 172.21.15.60:0/2096608646 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc31c00c9d0 con 0x7fc32c0c6950 2024-04-14T23:54:51.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.190+0000 7fc323e00640 1 -- 172.21.15.60:0/2096608646 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc31c00cdc0 con 0x7fc32c0c6950 2024-04-14T23:54:51.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.190+0000 7fc331c00640 1 -- 172.21.15.60:0/2096608646 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc32c15d060 con 0x7fc32c0c6950 2024-04-14T23:54:51.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.190+0000 7fc331c00640 1 -- 172.21.15.60:0/2096608646 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc32c15f250 con 0x7fc32c0c6950 2024-04-14T23:54:51.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.191+0000 7fc331c00640 1 -- 172.21.15.60:0/2096608646 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc32c04f540 con 0x7fc32c0c6950 2024-04-14T23:54:51.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.192+0000 7fc323e00640 1 -- 172.21.15.60:0/2096608646 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7fc31c01a070 con 0x7fc32c0c6950 2024-04-14T23:54:51.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.194+0000 7fc323e00640 1 --2- 172.21.15.60:0/2096608646 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fc30806b990 0x7fc30806de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:51.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.194+0000 7fc323e00640 1 -- 172.21.15.60:0/2096608646 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6083+0+0 (secure 0 0 0) 0x7fc31c08ad30 con 0x7fc32c0c6950 2024-04-14T23:54:51.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.194+0000 7fc32b400640 1 --2- 172.21.15.60:0/2096608646 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fc30806b990 0x7fc30806de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:51.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.195+0000 7fc32b400640 1 --2- 172.21.15.60:0/2096608646 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fc30806b990 0x7fc30806de50 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc32c15ef50 tx=0x7fc314007b50 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:54:51.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.198+0000 7fc323e00640 1 -- 172.21.15.60:0/2096608646 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc31c0590d0 con 0x7fc32c0c6950 2024-04-14T23:54:51.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.442+0000 7fc331c00640 1 -- 172.21.15.60:0/2096608646 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc32c0c7970 con 0x7fc32c0c6950 2024-04-14T23:54:51.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.443+0000 7fc323e00640 1 -- 172.21.15.60:0/2096608646 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+6006 (secure 0 0 0) 0x7fc31c05d170 con 0x7fc32c0c6950 2024-04-14T23:54:51.448 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 21 2024-04-14T23:54:51.448 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:54:51.448 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:54:48.943349+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4986},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4986":{"gid":4986,"name":"a","rank":0,"incarnation":21,"state":"up:replay","state_seq":1,"addr":"172.21.15.60:6833/1971580590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1971580590},{"type":"v1","addr":"172.21.15.60:6833","nonce":1971580590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-14T23:54:51.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.445+0000 7fc331c00640 1 -- 172.21.15.60:0/2096608646 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fc30806b990 msgr2=0x7fc30806de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:51.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.445+0000 7fc331c00640 1 --2- 172.21.15.60:0/2096608646 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fc30806b990 0x7fc30806de50 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc32c15ef50 tx=0x7fc314007b50 comp rx=0 tx=0).stop 2024-04-14T23:54:51.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.445+0000 7fc331c00640 1 -- 172.21.15.60:0/2096608646 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc32c0c6950 msgr2=0x7fc32c1550f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:51.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.445+0000 7fc331c00640 1 --2- 172.21.15.60:0/2096608646 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc32c0c6950 0x7fc32c1550f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc31c00e9e0 tx=0x7fc31c00eea0 comp rx=0 tx=0).stop 2024-04-14T23:54:51.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.445+0000 7fc331c00640 1 -- 172.21.15.60:0/2096608646 shutdown_connections 2024-04-14T23:54:51.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.445+0000 7fc331c00640 1 --2- 172.21.15.60:0/2096608646 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fc30806b990 0x7fc30806de50 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:51.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.445+0000 7fc331c00640 1 --2- 172.21.15.60:0/2096608646 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc32c15c8a0 0x7fc32c1f7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:51.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.445+0000 7fc331c00640 1 --2- 172.21.15.60:0/2096608646 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc32c0c86d0 0x7fc32c1618e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:51.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.445+0000 7fc331c00640 1 --2- 172.21.15.60:0/2096608646 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc32c0c6950 0x7fc32c1550f0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:51.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.445+0000 7fc331c00640 1 -- 172.21.15.60:0/2096608646 >> 172.21.15.60:0/2096608646 conn(0x7fc32c0c8b50 msgr2=0x7fc32c15a1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:54:51.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.445+0000 7fc331c00640 1 -- 172.21.15.60:0/2096608646 shutdown_connections 2024-04-14T23:54:51.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:51.445+0000 7fc331c00640 1 -- 172.21.15.60:0/2096608646 wait complete. 2024-04-14T23:54:51.469 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 21, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-14T23:48:42.700188+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 21, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4986": { "addr": "172.21.15.60:6833/1971580590", "addrs": { "addrvec": [ { "addr": "172.21.15.60:6832", "nonce": 1971580590, "type": "v2" }, { "addr": "172.21.15.60:6833", "nonce": 1971580590, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4986, "incarnation": 21, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 35, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-14T23:54:48.943349+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4986 } } } ], "standbys": [ { "addr": "172.21.15.60:6837/1527815634", "addrs": { "addrvec": [ { "addr": "172.21.15.60:6836", "nonce": 1527815634, "type": "v2" }, { "addr": "172.21.15.60:6837", "nonce": 1527815634, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4197, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6838/3958319486", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6835", "nonce": 3958319486, "type": "v2" }, { "addr": "172.21.15.186:6838", "nonce": 3958319486, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4220, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6837/3910951185", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6834", "nonce": 3910951185, "type": "v2" }, { "addr": "172.21.15.186:6837", "nonce": 3910951185, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4264, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6839/2155020404", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6836", "nonce": 2155020404, "type": "v2" }, { "addr": "172.21.15.186:6839", "nonce": 2155020404, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4267, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-14T23:54:53.470 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:54:53.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.696+0000 7f4e07e00640 1 Processor -- start 2024-04-14T23:54:53.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.697+0000 7f4e07e00640 1 -- start start 2024-04-14T23:54:53.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.698+0000 7f4e07e00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4e0815b640 0x7f4e080ce380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:53.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.698+0000 7f4e07e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e080ce8c0 0x7f4e080c3b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:53.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.698+0000 7f4e07e00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e0806c450 con 0x7f4e08154d50 2024-04-14T23:54:53.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.698+0000 7f4e07e00640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f4e080c9110 con 0x7f4e0815b640 2024-04-14T23:54:53.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.698+0000 7f4e07e00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f4e0806c730 con 0x7f4e080ce8c0 2024-04-14T23:54:53.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.699+0000 7f4e06000640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4e0815b640 0x7f4e080ce380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:53.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.699+0000 7f4e06a00640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f4e08154d50 0x7f4e08071290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33224/0 (socket says 172.21.15.60:33224) 2024-04-14T23:54:53.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.699+0000 7f4e06a00640 1 -- 172.21.15.60:0/2542321205 learned_addr learned my addr 172.21.15.60:0/2542321205 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:54:53.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.699+0000 7f4e07400640 1 --2- 172.21.15.60:0/2542321205 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e080ce8c0 0x7f4e080c3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:53.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.699+0000 7f4e06000640 1 -- 172.21.15.60:0/2542321205 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e080ce8c0 msgr2=0x7f4e080c3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:54:53.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.699+0000 7f4e06000640 1 --2- 172.21.15.60:0/2542321205 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e080ce8c0 0x7f4e080c3b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:53.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.699+0000 7f4e06000640 1 -- 172.21.15.60:0/2542321205 >> v1:172.21.15.60:6789/0 conn(0x7f4e08154d50 legacy=0x7f4e08071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:53.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.700+0000 7f4e06000640 1 -- 172.21.15.60:0/2542321205 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e0806ca10 con 0x7f4e0815b640 2024-04-14T23:54:53.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.700+0000 7f4e05600640 1 -- 172.21.15.60:0/2542321205 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 96815111 0 0) 0x7f4e0806c450 con 0x7f4e08154d50 2024-04-14T23:54:53.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.701+0000 7f4e06000640 1 --2- 172.21.15.60:0/2542321205 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e0815b640 0x7f4e080ce380 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f4df40097a0 tx=0x7f4df402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8a80c885f00438f server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:54:53.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.701+0000 7f4e05600640 1 -- 172.21.15.60:0/2542321205 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4df40315a0 con 0x7f4e0815b640 2024-04-14T23:54:53.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.701+0000 7f4e05600640 1 -- 172.21.15.60:0/2542321205 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4df4037070 con 0x7f4e0815b640 2024-04-14T23:54:53.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.701+0000 7f4e05600640 1 -- 172.21.15.60:0/2542321205 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4df4031960 con 0x7f4e0815b640 2024-04-14T23:54:53.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.702+0000 7f4e07e00640 1 -- 172.21.15.60:0/2542321205 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e0815b640 msgr2=0x7f4e080ce380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:53.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.702+0000 7f4e07e00640 1 --2- 172.21.15.60:0/2542321205 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e0815b640 0x7f4e080ce380 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f4df40097a0 tx=0x7f4df402b840 comp rx=0 tx=0).stop 2024-04-14T23:54:53.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.702+0000 7f4e07e00640 1 -- 172.21.15.60:0/2542321205 shutdown_connections 2024-04-14T23:54:53.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.702+0000 7f4e07e00640 1 --2- 172.21.15.60:0/2542321205 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e0815b640 0x7f4e080ce380 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:53.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.702+0000 7f4e07e00640 1 --2- 172.21.15.60:0/2542321205 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e080ce8c0 0x7f4e080c3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:53.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.702+0000 7f4e07e00640 1 -- 172.21.15.60:0/2542321205 >> 172.21.15.60:0/2542321205 conn(0x7f4e08066f10 msgr2=0x7f4e08067320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:54:53.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.702+0000 7f4e07e00640 1 -- 172.21.15.60:0/2542321205 shutdown_connections 2024-04-14T23:54:53.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.702+0000 7f4e07e00640 1 -- 172.21.15.60:0/2542321205 wait complete. 2024-04-14T23:54:53.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.704+0000 7f4e07e00640 1 Processor -- start 2024-04-14T23:54:53.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.705+0000 7f4e07e00640 1 -- start start 2024-04-14T23:54:53.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.706+0000 7f4e07e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e080ce8c0 0x7f4e080b3160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:53.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.706+0000 7f4e07e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e08154d50 0x7f4e081f38a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:53.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.706+0000 7f4e07e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e0815b640 0x7f4e0814ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:53.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.706+0000 7f4e07e00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f4e0815ba90 con 0x7f4e08154d50 2024-04-14T23:54:53.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.706+0000 7f4e07e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4e0806cfd0 con 0x7f4e080ce8c0 2024-04-14T23:54:53.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.706+0000 7f4e07e00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f4e0806d2b0 con 0x7f4e0815b640 2024-04-14T23:54:53.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.706+0000 7f4e06a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e080ce8c0 0x7f4e080b3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:53.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.706+0000 7f4e07400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e0815b640 0x7f4e0814ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:53.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.706+0000 7f4e06a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e080ce8c0 0x7f4e080b3160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:44850/0 (socket says 172.21.15.60:44850) 2024-04-14T23:54:53.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.706+0000 7f4e06a00640 1 -- 172.21.15.60:0/4210086195 learned_addr learned my addr 172.21.15.60:0/4210086195 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:54:53.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.706+0000 7f4e06000640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e08154d50 0x7f4e081f38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:53.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.707+0000 7f4e07400640 1 -- 172.21.15.60:0/4210086195 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e080ce8c0 msgr2=0x7f4e080b3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:53.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.707+0000 7f4e07400640 1 --2- 172.21.15.60:0/4210086195 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e080ce8c0 0x7f4e080b3160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:53.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.707+0000 7f4e07400640 1 -- 172.21.15.60:0/4210086195 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e08154d50 msgr2=0x7f4e081f38a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:53.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.707+0000 7f4e07400640 1 --2- 172.21.15.60:0/4210086195 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e08154d50 0x7f4e081f38a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:53.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.707+0000 7f4e07400640 1 -- 172.21.15.60:0/4210086195 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e080c57d0 con 0x7f4e0815b640 2024-04-14T23:54:53.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.707+0000 7f4e07400640 1 --2- 172.21.15.60:0/4210086195 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e0815b640 0x7f4e0814ac80 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4e00009d90 tx=0x7f4e0000c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:54:53.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.707+0000 7f4dfea00640 1 -- 172.21.15.60:0/4210086195 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e00018020 con 0x7f4e0815b640 2024-04-14T23:54:53.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.707+0000 7f4dfea00640 1 -- 172.21.15.60:0/4210086195 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e00014480 con 0x7f4e0815b640 2024-04-14T23:54:53.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.707+0000 7f4dfea00640 1 -- 172.21.15.60:0/4210086195 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e00014870 con 0x7f4e0815b640 2024-04-14T23:54:53.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.707+0000 7f4e07e00640 1 -- 172.21.15.60:0/4210086195 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e080b1290 con 0x7f4e0815b640 2024-04-14T23:54:53.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.708+0000 7f4e07e00640 1 -- 172.21.15.60:0/4210086195 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e080b3ef0 con 0x7f4e0815b640 2024-04-14T23:54:53.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.709+0000 7f4e07e00640 1 -- 172.21.15.60:0/4210086195 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e0804f540 con 0x7f4e0815b640 2024-04-14T23:54:53.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.710+0000 7f4dfea00640 1 -- 172.21.15.60:0/4210086195 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f4e0001b020 con 0x7f4e0815b640 2024-04-14T23:54:53.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.710+0000 7f4dfea00640 1 --2- 172.21.15.60:0/4210086195 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f4de806b9c0 0x7f4de806de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:53.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.711+0000 7f4dfea00640 1 -- 172.21.15.60:0/4210086195 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(35..35 src has 1..35) ==== 6083+0+0 (secure 0 0 0) 0x7f4e0008b750 con 0x7f4e0815b640 2024-04-14T23:54:53.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.711+0000 7f4e06a00640 1 --2- 172.21.15.60:0/4210086195 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f4de806b9c0 0x7f4de806de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:53.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.711+0000 7f4e06a00640 1 --2- 172.21.15.60:0/4210086195 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f4de806b9c0 0x7f4de806de80 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4df8000f60 tx=0x7f4df8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:54:53.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.715+0000 7f4dfea00640 1 -- 172.21.15.60:0/4210086195 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4e00022020 con 0x7f4e0815b640 2024-04-14T23:54:53.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.964+0000 7f4e07e00640 1 -- 172.21.15.60:0/4210086195 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4e080c3410 con 0x7f4e0815b640 2024-04-14T23:54:53.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.966+0000 7f4dfea00640 1 -- 172.21.15.60:0/4210086195 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+6006 (secure 0 0 0) 0x7f4e00010040 con 0x7f4e0815b640 2024-04-14T23:54:53.972 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 21 2024-04-14T23:54:53.972 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:54:53.972 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:54:48.943349+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4986},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4986":{"gid":4986,"name":"a","rank":0,"incarnation":21,"state":"up:replay","state_seq":1,"addr":"172.21.15.60:6833/1971580590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1971580590},{"type":"v1","addr":"172.21.15.60:6833","nonce":1971580590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-14T23:54:53.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.969+0000 7f4e07e00640 1 -- 172.21.15.60:0/4210086195 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f4de806b9c0 msgr2=0x7f4de806de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:53.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.969+0000 7f4e07e00640 1 --2- 172.21.15.60:0/4210086195 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f4de806b9c0 0x7f4de806de80 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4df8000f60 tx=0x7f4df8009040 comp rx=0 tx=0).stop 2024-04-14T23:54:53.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.969+0000 7f4e07e00640 1 -- 172.21.15.60:0/4210086195 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e0815b640 msgr2=0x7f4e0814ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:53.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.969+0000 7f4e07e00640 1 --2- 172.21.15.60:0/4210086195 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e0815b640 0x7f4e0814ac80 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4e00009d90 tx=0x7f4e0000c490 comp rx=0 tx=0).stop 2024-04-14T23:54:53.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.969+0000 7f4e07e00640 1 -- 172.21.15.60:0/4210086195 shutdown_connections 2024-04-14T23:54:53.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.969+0000 7f4e07e00640 1 --2- 172.21.15.60:0/4210086195 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f4de806b9c0 0x7f4de806de80 secure :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4df8000f60 tx=0x7f4df8009040 comp rx=0 tx=0).stop 2024-04-14T23:54:53.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.969+0000 7f4e07e00640 1 --2- 172.21.15.60:0/4210086195 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e0815b640 0x7f4e0814ac80 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:53.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.969+0000 7f4e07e00640 1 --2- 172.21.15.60:0/4210086195 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e08154d50 0x7f4e081f38a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:53.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.969+0000 7f4e07e00640 1 --2- 172.21.15.60:0/4210086195 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e080ce8c0 0x7f4e080b3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:53.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.969+0000 7f4e07e00640 1 -- 172.21.15.60:0/4210086195 >> 172.21.15.60:0/4210086195 conn(0x7f4e08066f10 msgr2=0x7f4e080cb470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:54:53.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.969+0000 7f4e07e00640 1 -- 172.21.15.60:0/4210086195 shutdown_connections 2024-04-14T23:54:53.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:53.969+0000 7f4e07e00640 1 -- 172.21.15.60:0/4210086195 wait complete. 2024-04-14T23:54:53.998 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-14T23:54:55.999 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:54:56.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.185+0000 7f369b200640 1 Processor -- start 2024-04-14T23:54:56.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.185+0000 7f369b200640 1 -- start start 2024-04-14T23:54:56.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.187+0000 7f369b200640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f36940b1e00 0x7f36940c6950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:56.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.187+0000 7f369b200640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3694152fb0 con 0x7f3694073320 2024-04-14T23:54:56.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.187+0000 7f369b200640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3694159310 con 0x7f36940c3d90 2024-04-14T23:54:56.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.187+0000 7f369b200640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f36940c6540 con 0x7f36940b1e00 2024-04-14T23:54:56.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.187+0000 7f3699400640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f3694073320 0x7f36940c7fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33230/0 (socket says 172.21.15.60:33230) 2024-04-14T23:54:56.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.187+0000 7f3699400640 1 -- 172.21.15.60:0/6137107 learned_addr learned my addr 172.21.15.60:0/6137107 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:54:56.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.187+0000 7f3699e00640 1 --2- 172.21.15.60:0/6137107 >> v2:172.21.15.186:3300/0 conn(0x7f36940b1e00 0x7f36940c6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:56.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.187+0000 7f3698a00640 1 -- 172.21.15.60:0/6137107 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2167322785 0 0) 0x7f3694152fb0 con 0x7f3694073320 2024-04-14T23:54:56.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.187+0000 7f3698a00640 1 -- 172.21.15.60:0/6137107 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f367c003520 con 0x7f3694073320 2024-04-14T23:54:56.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.188+0000 7f3699e00640 1 -- 172.21.15.60:0/6137107 >> v1:172.21.15.186:6789/0 conn(0x7f36940c3d90 legacy=0x7f36940c41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:56.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.188+0000 7f3698a00640 1 -- 172.21.15.60:0/6137107 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4043369276 0 0) 0x7f3694159310 con 0x7f36940c3d90 2024-04-14T23:54:56.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.188+0000 7f3699e00640 1 -- 172.21.15.60:0/6137107 >> v1:172.21.15.60:6789/0 conn(0x7f3694073320 legacy=0x7f36940c7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:56.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.188+0000 7f3699e00640 1 -- 172.21.15.60:0/6137107 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36940c38b0 con 0x7f36940b1e00 2024-04-14T23:54:56.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.189+0000 7f3699e00640 1 --2- 172.21.15.60:0/6137107 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36940b1e00 0x7f36940c6950 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f36941594e0 tx=0x7f368802b9d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=89321633c9d91da8 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:54:56.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.189+0000 7f3698a00640 1 -- 172.21.15.60:0/6137107 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3688034070 con 0x7f36940b1e00 2024-04-14T23:54:56.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.189+0000 7f3698a00640 1 -- 172.21.15.60:0/6137107 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3688030890 con 0x7f36940b1e00 2024-04-14T23:54:56.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.189+0000 7f3698a00640 1 -- 172.21.15.60:0/6137107 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3688030c40 con 0x7f36940b1e00 2024-04-14T23:54:56.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.190+0000 7f369b200640 1 -- 172.21.15.60:0/6137107 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36940b1e00 msgr2=0x7f36940c6950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:56.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.190+0000 7f369b200640 1 --2- 172.21.15.60:0/6137107 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36940b1e00 0x7f36940c6950 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f36941594e0 tx=0x7f368802b9d0 comp rx=0 tx=0).stop 2024-04-14T23:54:56.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.190+0000 7f369b200640 1 -- 172.21.15.60:0/6137107 shutdown_connections 2024-04-14T23:54:56.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.190+0000 7f369b200640 1 --2- 172.21.15.60:0/6137107 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36940b1e00 0x7f36940c6950 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:56.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.190+0000 7f369b200640 1 -- 172.21.15.60:0/6137107 >> 172.21.15.60:0/6137107 conn(0x7f36940c8b50 msgr2=0x7f36940c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:54:56.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.190+0000 7f369b200640 1 -- 172.21.15.60:0/6137107 shutdown_connections 2024-04-14T23:54:56.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.190+0000 7f369b200640 1 -- 172.21.15.60:0/6137107 wait complete. 2024-04-14T23:54:56.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.192+0000 7f369b200640 1 Processor -- start 2024-04-14T23:54:56.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.192+0000 7f369b200640 1 -- start start 2024-04-14T23:54:56.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.194+0000 7f369b200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3694073320 0x7f36941e3c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:56.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.194+0000 7f369b200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f36940b1e00 0x7f36941e0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:56.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.194+0000 7f369b200640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f36940c3d90 0x7f3694157c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:56.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.194+0000 7f369b200640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f3694159890 con 0x7f36940c3d90 2024-04-14T23:54:56.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.194+0000 7f369b200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f36940c5f80 con 0x7f3694073320 2024-04-14T23:54:56.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.194+0000 7f369b200640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f36940c6260 con 0x7f36940b1e00 2024-04-14T23:54:56.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.194+0000 7f369a800640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f36940c3d90 0x7f3694157c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:56.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.194+0000 7f369a800640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f36940c3d90 0x7f3694157c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51046/0 (socket says 172.21.15.60:51046) 2024-04-14T23:54:56.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.194+0000 7f369a800640 1 -- 172.21.15.60:0/4105199524 learned_addr learned my addr 172.21.15.60:0/4105199524 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:54:56.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.194+0000 7f3699e00640 1 --2- 172.21.15.60:0/4105199524 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3694073320 0x7f36941e3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:56.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.194+0000 7f3699400640 1 --2- 172.21.15.60:0/4105199524 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f36940b1e00 0x7f36941e0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:56.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.195+0000 7f3699e00640 1 -- 172.21.15.60:0/4105199524 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f36940b1e00 msgr2=0x7f36941e0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:56.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.195+0000 7f3699e00640 1 --2- 172.21.15.60:0/4105199524 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f36940b1e00 0x7f36941e0bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:56.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.195+0000 7f3699e00640 1 -- 172.21.15.60:0/4105199524 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f36940c3d90 msgr2=0x7f3694157c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:56.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.195+0000 7f3699e00640 1 --2- 172.21.15.60:0/4105199524 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f36940c3d90 0x7f3694157c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:56.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.195+0000 7f3699e00640 1 -- 172.21.15.60:0/4105199524 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36941e4260 con 0x7f3694073320 2024-04-14T23:54:56.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.195+0000 7f369a800640 1 --2- 172.21.15.60:0/4105199524 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f36940c3d90 0x7f3694157c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:54:56.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.195+0000 7f3699e00640 1 --2- 172.21.15.60:0/4105199524 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3694073320 0x7f36941e3c20 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f3688033000 tx=0x7f36880046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:54:56.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.195+0000 7f368e000640 1 -- 172.21.15.60:0/4105199524 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3688048070 con 0x7f3694073320 2024-04-14T23:54:56.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.195+0000 7f368e000640 1 -- 172.21.15.60:0/4105199524 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3688030690 con 0x7f3694073320 2024-04-14T23:54:56.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.195+0000 7f368e000640 1 -- 172.21.15.60:0/4105199524 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3688034040 con 0x7f3694073320 2024-04-14T23:54:56.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.195+0000 7f369b200640 1 -- 172.21.15.60:0/4105199524 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36941e4540 con 0x7f3694073320 2024-04-14T23:54:56.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.196+0000 7f369b200640 1 -- 172.21.15.60:0/4105199524 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36941f1fb0 con 0x7f3694073320 2024-04-14T23:54:56.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.197+0000 7f369b200640 1 -- 172.21.15.60:0/4105199524 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f369404f540 con 0x7f3694073320 2024-04-14T23:54:56.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.198+0000 7f368e000640 1 -- 172.21.15.60:0/4105199524 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f3688037070 con 0x7f3694073320 2024-04-14T23:54:56.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.198+0000 7f368e000640 1 --2- 172.21.15.60:0/4105199524 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f367806b990 0x7f367806de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:56.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.198+0000 7f368e000640 1 -- 172.21.15.60:0/4105199524 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6083+0+0 (secure 0 0 0) 0x7f36880ac470 con 0x7f3694073320 2024-04-14T23:54:56.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.199+0000 7f3699400640 1 --2- 172.21.15.60:0/4105199524 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f367806b990 0x7f367806de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:56.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.199+0000 7f3699400640 1 --2- 172.21.15.60:0/4105199524 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f367806b990 0x7f367806de50 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f3694158f70 tx=0x7f3684009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:54:56.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.203+0000 7f368e000640 1 -- 172.21.15.60:0/4105199524 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f368807a780 con 0x7f3694073320 2024-04-14T23:54:56.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.432+0000 7f369b200640 1 -- 172.21.15.60:0/4105199524 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f36940b21e0 con 0x7f3694073320 2024-04-14T23:54:56.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.434+0000 7f368e000640 1 -- 172.21.15.60:0/4105199524 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+6006 (secure 0 0 0) 0x7f368807e820 con 0x7f3694073320 2024-04-14T23:54:56.439 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:54:56.439 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:54:48.943349+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4986},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4986":{"gid":4986,"name":"a","rank":0,"incarnation":21,"state":"up:replay","state_seq":1,"addr":"172.21.15.60:6833/1971580590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1971580590},{"type":"v1","addr":"172.21.15.60:6833","nonce":1971580590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-14T23:54:56.439 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 21 2024-04-14T23:54:56.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.436+0000 7f369b200640 1 -- 172.21.15.60:0/4105199524 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f367806b990 msgr2=0x7f367806de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:56.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.436+0000 7f369b200640 1 --2- 172.21.15.60:0/4105199524 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f367806b990 0x7f367806de50 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f3694158f70 tx=0x7f3684009040 comp rx=0 tx=0).stop 2024-04-14T23:54:56.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.436+0000 7f369b200640 1 -- 172.21.15.60:0/4105199524 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3694073320 msgr2=0x7f36941e3c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:56.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.436+0000 7f369b200640 1 --2- 172.21.15.60:0/4105199524 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3694073320 0x7f36941e3c20 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f3688033000 tx=0x7f36880046c0 comp rx=0 tx=0).stop 2024-04-14T23:54:56.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.436+0000 7f369b200640 1 -- 172.21.15.60:0/4105199524 shutdown_connections 2024-04-14T23:54:56.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.436+0000 7f369b200640 1 --2- 172.21.15.60:0/4105199524 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f367806b990 0x7f367806de50 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:56.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.436+0000 7f369b200640 1 --2- 172.21.15.60:0/4105199524 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f36940c3d90 0x7f3694157c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:56.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.436+0000 7f369b200640 1 --2- 172.21.15.60:0/4105199524 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f36940b1e00 0x7f36941e0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:56.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.436+0000 7f369b200640 1 --2- 172.21.15.60:0/4105199524 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3694073320 0x7f36941e3c20 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:56.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.436+0000 7f369b200640 1 -- 172.21.15.60:0/4105199524 >> 172.21.15.60:0/4105199524 conn(0x7f36940c8b50 msgr2=0x7f3694152700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:54:56.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.437+0000 7f369b200640 1 -- 172.21.15.60:0/4105199524 shutdown_connections 2024-04-14T23:54:56.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:56.437+0000 7f369b200640 1 -- 172.21.15.60:0/4105199524 wait complete. 2024-04-14T23:54:56.458 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-14T23:54:58.459 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:54:58.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.685+0000 7f7665200640 1 Processor -- start 2024-04-14T23:54:58.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.685+0000 7f7665200640 1 -- start start 2024-04-14T23:54:58.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.687+0000 7f7665200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f76600b1e00 0x7f76600c6950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:58.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.687+0000 7f7665200640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7660073320 0x7f76600c7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:58.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.687+0000 7f7665200640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f76600c84e0 0x7f76600c3d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:58.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.687+0000 7f7665200640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f76600c6540 con 0x7f7660073320 2024-04-14T23:54:58.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.687+0000 7f7665200640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f76600b21e0 con 0x7f76600c84e0 2024-04-14T23:54:58.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.687+0000 7f7665200640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f7660159890 con 0x7f76600b1e00 2024-04-14T23:54:58.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.687+0000 7f765ea00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7660073320 0x7f76600c7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:58.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.687+0000 7f765f400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f76600b1e00 0x7f76600c6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:58.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.687+0000 7f765f400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f76600b1e00 0x7f76600c6950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.60:53214/0 (socket says 172.21.15.60:53214) 2024-04-14T23:54:58.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.687+0000 7f765fe00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f76600c84e0 0x7f76600c3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:58.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.687+0000 7f765ea00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7660073320 0x7f76600c7fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51062/0 (socket says 172.21.15.60:51062) 2024-04-14T23:54:58.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.687+0000 7f765f400640 1 -- 172.21.15.60:0/1425119797 learned_addr learned my addr 172.21.15.60:0/1425119797 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:54:58.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.688+0000 7f765f400640 1 -- 172.21.15.60:0/1425119797 >> v2:172.21.15.186:3300/0 conn(0x7f76600c84e0 msgr2=0x7f76600c3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:54:58.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.688+0000 7f765f400640 1 --2- 172.21.15.60:0/1425119797 >> v2:172.21.15.186:3300/0 conn(0x7f76600c84e0 0x7f76600c3d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:58.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.688+0000 7f765f400640 1 -- 172.21.15.60:0/1425119797 >> v2:172.21.15.60:3300/0 conn(0x7f7660073320 msgr2=0x7f76600c7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:54:58.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.688+0000 7f765f400640 1 --2- 172.21.15.60:0/1425119797 >> v2:172.21.15.60:3300/0 conn(0x7f7660073320 0x7f76600c7fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:58.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.688+0000 7f765f400640 1 -- 172.21.15.60:0/1425119797 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7660152fb0 con 0x7f76600b1e00 2024-04-14T23:54:58.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.688+0000 7f765ea00640 1 --2- 172.21.15.60:0/1425119797 >> v2:172.21.15.60:3300/0 conn(0x7f7660073320 0x7f76600c7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:54:58.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.689+0000 7f765f400640 1 --2- 172.21.15.60:0/1425119797 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f76600b1e00 0x7f76600c6950 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f7660156460 tx=0x7f765002b990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=caaff63921dfb7df server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:54:58.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.689+0000 7f765e000640 1 -- 172.21.15.60:0/1425119797 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7650034070 con 0x7f76600b1e00 2024-04-14T23:54:58.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.689+0000 7f765e000640 1 -- 172.21.15.60:0/1425119797 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7650030850 con 0x7f76600b1e00 2024-04-14T23:54:58.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.689+0000 7f765e000640 1 -- 172.21.15.60:0/1425119797 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7650030c20 con 0x7f76600b1e00 2024-04-14T23:54:58.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.690+0000 7f7665200640 1 -- 172.21.15.60:0/1425119797 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f76600b1e00 msgr2=0x7f76600c6950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:58.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.690+0000 7f7665200640 1 --2- 172.21.15.60:0/1425119797 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f76600b1e00 0x7f76600c6950 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f7660156460 tx=0x7f765002b990 comp rx=0 tx=0).stop 2024-04-14T23:54:58.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.690+0000 7f7665200640 1 -- 172.21.15.60:0/1425119797 shutdown_connections 2024-04-14T23:54:58.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.690+0000 7f7665200640 1 --2- 172.21.15.60:0/1425119797 >> v2:172.21.15.186:3300/0 conn(0x7f76600c84e0 0x7f76600c3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:58.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.690+0000 7f7665200640 1 --2- 172.21.15.60:0/1425119797 >> v2:172.21.15.60:3300/0 conn(0x7f7660073320 0x7f76600c7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:58.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.690+0000 7f7665200640 1 --2- 172.21.15.60:0/1425119797 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f76600b1e00 0x7f76600c6950 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:58.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.690+0000 7f7665200640 1 -- 172.21.15.60:0/1425119797 >> 172.21.15.60:0/1425119797 conn(0x7f76600c8b50 msgr2=0x7f76600c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:54:58.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.690+0000 7f7665200640 1 -- 172.21.15.60:0/1425119797 shutdown_connections 2024-04-14T23:54:58.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.690+0000 7f7665200640 1 -- 172.21.15.60:0/1425119797 wait complete. 2024-04-14T23:54:58.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.692+0000 7f7665200640 1 Processor -- start 2024-04-14T23:54:58.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.692+0000 7f7665200640 1 -- start start 2024-04-14T23:54:58.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.693+0000 7f7665200640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7660073320 0x7f7660153970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:58.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.693+0000 7f7665200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76600b1e00 0x7f7660155bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:58.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.693+0000 7f7665200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f76600c84e0 0x7f76601e5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:58.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.693+0000 7f7665200640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f76600c38b0 con 0x7f7660073320 2024-04-14T23:54:58.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.693+0000 7f7665200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7660159310 con 0x7f76600b1e00 2024-04-14T23:54:58.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.693+0000 7f7665200640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f76600c5f80 con 0x7f76600c84e0 2024-04-14T23:54:58.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.693+0000 7f765f400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7660073320 0x7f7660153970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:58.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.693+0000 7f765f400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7660073320 0x7f7660153970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51066/0 (socket says 172.21.15.60:51066) 2024-04-14T23:54:58.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.693+0000 7f765f400640 1 -- 172.21.15.60:0/3389282063 learned_addr learned my addr 172.21.15.60:0/3389282063 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:54:58.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.694+0000 7f765ea00640 1 --2- 172.21.15.60:0/3389282063 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76600b1e00 0x7f7660155bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:58.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.694+0000 7f765fe00640 1 --2- 172.21.15.60:0/3389282063 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f76600c84e0 0x7f76601e5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:58.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.694+0000 7f765f400640 1 -- 172.21.15.60:0/3389282063 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f76600c84e0 msgr2=0x7f76601e5280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:58.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.694+0000 7f765f400640 1 --2- 172.21.15.60:0/3389282063 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f76600c84e0 0x7f76601e5280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:58.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.694+0000 7f765f400640 1 -- 172.21.15.60:0/3389282063 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76600b1e00 msgr2=0x7f7660155bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:58.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.694+0000 7f765f400640 1 --2- 172.21.15.60:0/3389282063 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76600b1e00 0x7f7660155bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:58.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.694+0000 7f765f400640 1 -- 172.21.15.60:0/3389282063 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7660154020 con 0x7f7660073320 2024-04-14T23:54:58.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.694+0000 7f765f400640 1 --2- 172.21.15.60:0/3389282063 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7660073320 0x7f7660153970 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7650000c00 tx=0x7f7650004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:54:58.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.695+0000 7f7657400640 1 -- 172.21.15.60:0/3389282063 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7650048070 con 0x7f7660073320 2024-04-14T23:54:58.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.695+0000 7f7657400640 1 -- 172.21.15.60:0/3389282063 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7650039a10 con 0x7f7660073320 2024-04-14T23:54:58.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.695+0000 7f7657400640 1 -- 172.21.15.60:0/3389282063 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7650034040 con 0x7f7660073320 2024-04-14T23:54:58.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.695+0000 7f7665200640 1 -- 172.21.15.60:0/3389282063 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76601e6280 con 0x7f7660073320 2024-04-14T23:54:58.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.695+0000 7f7665200640 1 -- 172.21.15.60:0/3389282063 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76601e6840 con 0x7f7660073320 2024-04-14T23:54:58.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.696+0000 7f7665200640 1 -- 172.21.15.60:0/3389282063 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f766004f540 con 0x7f7660073320 2024-04-14T23:54:58.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.697+0000 7f7657400640 1 -- 172.21.15.60:0/3389282063 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f765004b070 con 0x7f7660073320 2024-04-14T23:54:58.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.698+0000 7f7657400640 1 --2- 172.21.15.60:0/3389282063 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f764006b9c0 0x7f764006de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:54:58.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.698+0000 7f7657400640 1 -- 172.21.15.60:0/3389282063 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6083+0+0 (secure 0 0 0) 0x7f76500ac960 con 0x7f7660073320 2024-04-14T23:54:58.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.699+0000 7f765ea00640 1 --2- 172.21.15.60:0/3389282063 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f764006b9c0 0x7f764006de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:54:58.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.699+0000 7f765ea00640 1 --2- 172.21.15.60:0/3389282063 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f764006b9c0 0x7f764006de80 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7660156430 tx=0x7f764c009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:54:58.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.703+0000 7f7657400640 1 -- 172.21.15.60:0/3389282063 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f765007ad90 con 0x7f7660073320 2024-04-14T23:54:58.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.942+0000 7f7665200640 1 -- 172.21.15.60:0/3389282063 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f76600c9e50 con 0x7f7660073320 2024-04-14T23:54:58.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.943+0000 7f7657400640 1 -- 172.21.15.60:0/3389282063 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6010 (secure 0 0 0) 0x7f765007ee30 con 0x7f7660073320 2024-04-14T23:54:58.948 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:54:58.949 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:54:56.124386+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4986},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4986":{"gid":4986,"name":"a","rank":0,"incarnation":21,"state":"up:reconnect","state_seq":30,"addr":"172.21.15.60:6833/1971580590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1971580590},{"type":"v1","addr":"172.21.15.60:6833","nonce":1971580590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-14T23:54:58.949 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 22 2024-04-14T23:54:58.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.946+0000 7f7665200640 1 -- 172.21.15.60:0/3389282063 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f764006b9c0 msgr2=0x7f764006de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:58.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.946+0000 7f7665200640 1 --2- 172.21.15.60:0/3389282063 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f764006b9c0 0x7f764006de80 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7660156430 tx=0x7f764c009190 comp rx=0 tx=0).stop 2024-04-14T23:54:58.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.946+0000 7f7665200640 1 -- 172.21.15.60:0/3389282063 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7660073320 msgr2=0x7f7660153970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:54:58.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.946+0000 7f7665200640 1 --2- 172.21.15.60:0/3389282063 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7660073320 0x7f7660153970 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7650000c00 tx=0x7f7650004730 comp rx=0 tx=0).stop 2024-04-14T23:54:58.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.946+0000 7f7665200640 1 -- 172.21.15.60:0/3389282063 shutdown_connections 2024-04-14T23:54:58.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.946+0000 7f7665200640 1 --2- 172.21.15.60:0/3389282063 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f764006b9c0 0x7f764006de80 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:58.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.946+0000 7f7665200640 1 --2- 172.21.15.60:0/3389282063 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f76600c84e0 0x7f76601e5280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:58.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.946+0000 7f7665200640 1 --2- 172.21.15.60:0/3389282063 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76600b1e00 0x7f7660155bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:58.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.946+0000 7f7665200640 1 --2- 172.21.15.60:0/3389282063 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7660073320 0x7f7660153970 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:54:58.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.946+0000 7f7665200640 1 -- 172.21.15.60:0/3389282063 >> 172.21.15.60:0/3389282063 conn(0x7f76600c8b50 msgr2=0x7f76600720d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:54:58.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.946+0000 7f7665200640 1 -- 172.21.15.60:0/3389282063 shutdown_connections 2024-04-14T23:54:58.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:54:58.947+0000 7f7665200640 1 -- 172.21.15.60:0/3389282063 wait complete. 2024-04-14T23:54:58.969 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-14T23:55:00.971 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:55:01.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.191+0000 7fe5fb000640 1 Processor -- start 2024-04-14T23:55:01.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.191+0000 7fe5fb000640 1 -- start start 2024-04-14T23:55:01.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.193+0000 7fe5fb000640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe5f40c8df0 0x7fe5f40c93d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:55:01.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.193+0000 7fe5fb000640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe5f40c9910 0x7fe5f40c9d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:55:01.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.193+0000 7fe5fb000640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5f41530a0 con 0x7fe5f4152cc0 2024-04-14T23:55:01.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.193+0000 7fe5fb000640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fe5f40c4f80 con 0x7fe5f40c9910 2024-04-14T23:55:01.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.193+0000 7fe5fb000640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fe5f40717b0 con 0x7fe5f40c8df0 2024-04-14T23:55:01.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.193+0000 7fe5f9c00640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fe5f4152cc0 0x7fe5f4070270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33242/0 (socket says 172.21.15.60:33242) 2024-04-14T23:55:01.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.193+0000 7fe5f9c00640 1 -- 172.21.15.60:0/1183888787 learned_addr learned my addr 172.21.15.60:0/1183888787 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:55:01.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.193+0000 7fe5f9200640 1 --2- 172.21.15.60:0/1183888787 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe5f40c8df0 0x7fe5f40c93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:55:01.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.193+0000 7fe5fa600640 1 --2- 172.21.15.60:0/1183888787 >> v2:172.21.15.186:3300/0 conn(0x7fe5f40c9910 0x7fe5f40c9d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:55:01.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.194+0000 7fe5f3e00640 1 -- 172.21.15.60:0/1183888787 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2120538484 0 0) 0x7fe5f41530a0 con 0x7fe5f4152cc0 2024-04-14T23:55:01.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.194+0000 7fe5f3e00640 1 -- 172.21.15.60:0/1183888787 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe5dc003520 con 0x7fe5f4152cc0 2024-04-14T23:55:01.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.194+0000 7fe5fa600640 1 -- 172.21.15.60:0/1183888787 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe5f40c8df0 msgr2=0x7fe5f40c93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:55:01.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.194+0000 7fe5fa600640 1 --2- 172.21.15.60:0/1183888787 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe5f40c8df0 0x7fe5f40c93d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:01.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.194+0000 7fe5fa600640 1 -- 172.21.15.60:0/1183888787 >> v1:172.21.15.60:6789/0 conn(0x7fe5f4152cc0 legacy=0x7fe5f4070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:55:01.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.194+0000 7fe5fa600640 1 -- 172.21.15.60:0/1183888787 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5f41595f0 con 0x7fe5f40c9910 2024-04-14T23:55:01.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.194+0000 7fe5f9200640 1 --2- 172.21.15.60:0/1183888787 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe5f40c8df0 0x7fe5f40c93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:55:01.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.195+0000 7fe5fa600640 1 --2- 172.21.15.60:0/1183888787 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5f40c9910 0x7fe5f40c9d40 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec009d90 tx=0x7fe5ec02bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cdd5029cffec7c01 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:55:01.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.195+0000 7fe5f3e00640 1 -- 172.21.15.60:0/1183888787 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5ec039070 con 0x7fe5f40c9910 2024-04-14T23:55:01.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.195+0000 7fe5f3e00640 1 -- 172.21.15.60:0/1183888787 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5ec02cce0 con 0x7fe5f40c9910 2024-04-14T23:55:01.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.195+0000 7fe5f3e00640 1 -- 172.21.15.60:0/1183888787 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5ec034660 con 0x7fe5f40c9910 2024-04-14T23:55:01.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.196+0000 7fe5fb000640 1 -- 172.21.15.60:0/1183888787 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5f40c9910 msgr2=0x7fe5f40c9d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:55:01.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.196+0000 7fe5fb000640 1 --2- 172.21.15.60:0/1183888787 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5f40c9910 0x7fe5f40c9d40 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec009d90 tx=0x7fe5ec02bf30 comp rx=0 tx=0).stop 2024-04-14T23:55:01.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.196+0000 7fe5fb000640 1 -- 172.21.15.60:0/1183888787 shutdown_connections 2024-04-14T23:55:01.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.196+0000 7fe5fb000640 1 --2- 172.21.15.60:0/1183888787 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5f40c9910 0x7fe5f40c9d40 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:01.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.196+0000 7fe5fb000640 1 --2- 172.21.15.60:0/1183888787 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe5f40c8df0 0x7fe5f40c93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:01.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.196+0000 7fe5fb000640 1 -- 172.21.15.60:0/1183888787 >> 172.21.15.60:0/1183888787 conn(0x7fe5f4066f10 msgr2=0x7fe5f4067320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:55:01.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.196+0000 7fe5fb000640 1 -- 172.21.15.60:0/1183888787 shutdown_connections 2024-04-14T23:55:01.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.196+0000 7fe5fb000640 1 -- 172.21.15.60:0/1183888787 wait complete. 2024-04-14T23:55:01.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.198+0000 7fe5fb000640 1 Processor -- start 2024-04-14T23:55:01.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.198+0000 7fe5fb000640 1 -- start start 2024-04-14T23:55:01.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.199+0000 7fe5fb000640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5f40c8df0 0x7fe5f41567d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:55:01.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.199+0000 7fe5fb000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5f40c9910 0x7fe5f41f42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:55:01.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.199+0000 7fe5fb000640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe5f4152cc0 0x7fe5f40b2330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:55:01.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.199+0000 7fe5fb000640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fe5f406f900 con 0x7fe5f40c8df0 2024-04-14T23:55:01.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.199+0000 7fe5fb000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe5f41598d0 con 0x7fe5f40c9910 2024-04-14T23:55:01.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.199+0000 7fe5fb000640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fe5f40b0e70 con 0x7fe5f4152cc0 2024-04-14T23:55:01.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.199+0000 7fe5f9c00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5f40c8df0 0x7fe5f41567d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:55:01.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.199+0000 7fe5f9c00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5f40c8df0 0x7fe5f41567d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51072/0 (socket says 172.21.15.60:51072) 2024-04-14T23:55:01.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.199+0000 7fe5f9c00640 1 -- 172.21.15.60:0/3791914013 learned_addr learned my addr 172.21.15.60:0/3791914013 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:55:01.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.199+0000 7fe5fa600640 1 --2- 172.21.15.60:0/3791914013 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe5f4152cc0 0x7fe5f40b2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:55:01.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.199+0000 7fe5f9200640 1 --2- 172.21.15.60:0/3791914013 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5f40c9910 0x7fe5f41f42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:55:01.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.200+0000 7fe5f9c00640 1 -- 172.21.15.60:0/3791914013 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe5f4152cc0 msgr2=0x7fe5f40b2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:55:01.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.200+0000 7fe5f9c00640 1 --2- 172.21.15.60:0/3791914013 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe5f4152cc0 0x7fe5f40b2330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:01.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.200+0000 7fe5f9c00640 1 -- 172.21.15.60:0/3791914013 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5f40c9910 msgr2=0x7fe5f41f42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:55:01.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.200+0000 7fe5f9c00640 1 --2- 172.21.15.60:0/3791914013 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5f40c9910 0x7fe5f41f42e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:01.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.200+0000 7fe5f9c00640 1 -- 172.21.15.60:0/3791914013 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5f4158820 con 0x7fe5f40c8df0 2024-04-14T23:55:01.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.200+0000 7fe5f9c00640 1 --2- 172.21.15.60:0/3791914013 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5f40c8df0 0x7fe5f41567d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fe5e800c3c0 tx=0x7fe5e800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:55:01.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.200+0000 7fe5f1600640 1 -- 172.21.15.60:0/3791914013 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5e8017070 con 0x7fe5f40c8df0 2024-04-14T23:55:01.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.200+0000 7fe5f1600640 1 -- 172.21.15.60:0/3791914013 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5e8005ce0 con 0x7fe5f40c8df0 2024-04-14T23:55:01.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.200+0000 7fe5f1600640 1 -- 172.21.15.60:0/3791914013 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5e80072f0 con 0x7fe5f40c8df0 2024-04-14T23:55:01.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.200+0000 7fe5fb000640 1 -- 172.21.15.60:0/3791914013 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5f40c8850 con 0x7fe5f40c8df0 2024-04-14T23:55:01.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.201+0000 7fe5fb000640 1 -- 172.21.15.60:0/3791914013 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5f4158540 con 0x7fe5f40c8df0 2024-04-14T23:55:01.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.202+0000 7fe5fb000640 1 -- 172.21.15.60:0/3791914013 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5f404f540 con 0x7fe5f40c8df0 2024-04-14T23:55:01.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.203+0000 7fe5f1600640 1 -- 172.21.15.60:0/3791914013 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7fe5e801a070 con 0x7fe5f40c8df0 2024-04-14T23:55:01.209 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.204+0000 7fe5f1600640 1 --2- 172.21.15.60:0/3791914013 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fe5d406b990 0x7fe5d406de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:55:01.209 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.204+0000 7fe5f1600640 1 -- 172.21.15.60:0/3791914013 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6083+0+0 (secure 0 0 0) 0x7fe5e8089d50 con 0x7fe5f40c8df0 2024-04-14T23:55:01.209 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.204+0000 7fe5f9200640 1 --2- 172.21.15.60:0/3791914013 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fe5d406b990 0x7fe5d406de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:55:01.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.205+0000 7fe5f9200640 1 --2- 172.21.15.60:0/3791914013 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fe5d406b990 0x7fe5d406de50 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe5e0009560 tx=0x7fe5e0009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:55:01.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.209+0000 7fe5f1600640 1 -- 172.21.15.60:0/3791914013 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe5e80580f0 con 0x7fe5f40c8df0 2024-04-14T23:55:01.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.447+0000 7fe5fb000640 1 -- 172.21.15.60:0/3791914013 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe5f4070050 con 0x7fe5f40c8df0 2024-04-14T23:55:01.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.449+0000 7fe5f1600640 1 -- 172.21.15.60:0/3791914013 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6010 (secure 0 0 0) 0x7fe5e805c190 con 0x7fe5f40c8df0 2024-04-14T23:55:01.454 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:55:01.454 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:54:56.124386+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4986},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4986":{"gid":4986,"name":"a","rank":0,"incarnation":21,"state":"up:reconnect","state_seq":30,"addr":"172.21.15.60:6833/1971580590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1971580590},{"type":"v1","addr":"172.21.15.60:6833","nonce":1971580590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-14T23:55:01.454 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 22 2024-04-14T23:55:01.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.452+0000 7fe5fb000640 1 -- 172.21.15.60:0/3791914013 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fe5d406b990 msgr2=0x7fe5d406de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:55:01.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.452+0000 7fe5fb000640 1 --2- 172.21.15.60:0/3791914013 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fe5d406b990 0x7fe5d406de50 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe5e0009560 tx=0x7fe5e0009190 comp rx=0 tx=0).stop 2024-04-14T23:55:01.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.452+0000 7fe5fb000640 1 -- 172.21.15.60:0/3791914013 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5f40c8df0 msgr2=0x7fe5f41567d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:55:01.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.452+0000 7fe5fb000640 1 --2- 172.21.15.60:0/3791914013 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5f40c8df0 0x7fe5f41567d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fe5e800c3c0 tx=0x7fe5e800c880 comp rx=0 tx=0).stop 2024-04-14T23:55:01.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.452+0000 7fe5fb000640 1 -- 172.21.15.60:0/3791914013 shutdown_connections 2024-04-14T23:55:01.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.452+0000 7fe5fb000640 1 --2- 172.21.15.60:0/3791914013 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fe5d406b990 0x7fe5d406de50 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:01.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.452+0000 7fe5fb000640 1 --2- 172.21.15.60:0/3791914013 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe5f4152cc0 0x7fe5f40b2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:01.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.452+0000 7fe5fb000640 1 --2- 172.21.15.60:0/3791914013 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5f40c9910 0x7fe5f41f42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:01.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.452+0000 7fe5fb000640 1 --2- 172.21.15.60:0/3791914013 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5f40c8df0 0x7fe5f41567d0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:01.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.452+0000 7fe5fb000640 1 -- 172.21.15.60:0/3791914013 >> 172.21.15.60:0/3791914013 conn(0x7fe5f4066f10 msgr2=0x7fe5f4074810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:55:01.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.452+0000 7fe5fb000640 1 -- 172.21.15.60:0/3791914013 shutdown_connections 2024-04-14T23:55:01.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:01.452+0000 7fe5fb000640 1 -- 172.21.15.60:0/3791914013 wait complete. 2024-04-14T23:55:01.477 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-14T23:55:03.478 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:55:03.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.709+0000 7fd7e2800640 1 Processor -- start 2024-04-14T23:55:03.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.709+0000 7fd7e2800640 1 -- start start 2024-04-14T23:55:03.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.710+0000 7fd7e2800640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd7dc0cd160 0x7fd7dc15ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:55:03.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.710+0000 7fd7e2800640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd7dc0cb750 0x7fd7dc154b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:55:03.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.710+0000 7fd7e2800640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7dc0b3f50 con 0x7fd7dc0cd540 2024-04-14T23:55:03.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.711+0000 7fd7e2800640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fd7dc15b370 con 0x7fd7dc0cb750 2024-04-14T23:55:03.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.711+0000 7fd7e2800640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fd7dc0b39b0 con 0x7fd7dc0cd160 2024-04-14T23:55:03.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.711+0000 7fd7db400640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd7dc0cd160 0x7fd7dc15ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:55:03.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.711+0000 7fd7daa00640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fd7dc0cd540 0x7fd7dc0b3190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:50020/0 (socket says 172.21.15.60:50020) 2024-04-14T23:55:03.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.711+0000 7fd7daa00640 1 -- 172.21.15.60:0/2045239909 learned_addr learned my addr 172.21.15.60:0/2045239909 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:55:03.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.711+0000 7fd7dbe00640 1 --2- 172.21.15.60:0/2045239909 >> v2:172.21.15.60:3300/0 conn(0x7fd7dc0cb750 0x7fd7dc154b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:55:03.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.712+0000 7fd7db400640 1 -- 172.21.15.60:0/2045239909 >> v2:172.21.15.60:3300/0 conn(0x7fd7dc0cb750 msgr2=0x7fd7dc154b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:55:03.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.712+0000 7fd7db400640 1 --2- 172.21.15.60:0/2045239909 >> v2:172.21.15.60:3300/0 conn(0x7fd7dc0cb750 0x7fd7dc154b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:03.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.712+0000 7fd7db400640 1 -- 172.21.15.60:0/2045239909 >> v1:172.21.15.60:6789/0 conn(0x7fd7dc0cd540 legacy=0x7fd7dc0b3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:55:03.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.712+0000 7fd7db400640 1 -- 172.21.15.60:0/2045239909 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7dc0b3c70 con 0x7fd7dc0cd160 2024-04-14T23:55:03.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.713+0000 7fd7db400640 1 --2- 172.21.15.60:0/2045239909 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7dc0cd160 0x7fd7dc15ae30 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc067ef0 tx=0x7fd7cc02bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aab0a509bb8795d server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:55:03.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.713+0000 7fd7da000640 1 -- 172.21.15.60:0/2045239909 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7cc039070 con 0x7fd7dc0cd160 2024-04-14T23:55:03.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.713+0000 7fd7da000640 1 -- 172.21.15.60:0/2045239909 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7cc02cd00 con 0x7fd7dc0cd160 2024-04-14T23:55:03.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.713+0000 7fd7da000640 1 -- 172.21.15.60:0/2045239909 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7cc034660 con 0x7fd7dc0cd160 2024-04-14T23:55:03.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.714+0000 7fd7e2800640 1 -- 172.21.15.60:0/2045239909 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7dc0cd160 msgr2=0x7fd7dc15ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:55:03.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.714+0000 7fd7e2800640 1 --2- 172.21.15.60:0/2045239909 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7dc0cd160 0x7fd7dc15ae30 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc067ef0 tx=0x7fd7cc02bf50 comp rx=0 tx=0).stop 2024-04-14T23:55:03.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.714+0000 7fd7e2800640 1 -- 172.21.15.60:0/2045239909 shutdown_connections 2024-04-14T23:55:03.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.714+0000 7fd7e2800640 1 --2- 172.21.15.60:0/2045239909 >> v2:172.21.15.60:3300/0 conn(0x7fd7dc0cb750 0x7fd7dc154b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:03.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.714+0000 7fd7e2800640 1 --2- 172.21.15.60:0/2045239909 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7dc0cd160 0x7fd7dc15ae30 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:03.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.714+0000 7fd7e2800640 1 -- 172.21.15.60:0/2045239909 >> 172.21.15.60:0/2045239909 conn(0x7fd7dc066f10 msgr2=0x7fd7dc0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:55:03.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.714+0000 7fd7e2800640 1 -- 172.21.15.60:0/2045239909 shutdown_connections 2024-04-14T23:55:03.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.714+0000 7fd7e2800640 1 -- 172.21.15.60:0/2045239909 wait complete. 2024-04-14T23:55:03.721 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.716+0000 7fd7e2800640 1 Processor -- start 2024-04-14T23:55:03.721 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.716+0000 7fd7e2800640 1 -- start start 2024-04-14T23:55:03.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.717+0000 7fd7e2800640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd7dc0cb750 0x7fd7dc15aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:55:03.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.717+0000 7fd7e2800640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd7dc0cd160 0x7fd7dc1f43f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:55:03.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.717+0000 7fd7e2800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7dc0cd540 0x7fd7dc1f4930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:55:03.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.717+0000 7fd7e2800640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fd7dc15ea70 con 0x7fd7dc0cb750 2024-04-14T23:55:03.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.717+0000 7fd7e2800640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd7dc15e790 con 0x7fd7dc0cd540 2024-04-14T23:55:03.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.717+0000 7fd7e2800640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fd7dc0ccb20 con 0x7fd7dc0cd160 2024-04-14T23:55:03.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.717+0000 7fd7db400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd7dc0cb750 0x7fd7dc15aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:55:03.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.717+0000 7fd7daa00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd7dc0cd160 0x7fd7dc1f43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:55:03.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.717+0000 7fd7db400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd7dc0cb750 0x7fd7dc15aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35502/0 (socket says 172.21.15.60:35502) 2024-04-14T23:55:03.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.717+0000 7fd7db400640 1 -- 172.21.15.60:0/1512483577 learned_addr learned my addr 172.21.15.60:0/1512483577 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:55:03.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.717+0000 7fd7dbe00640 1 --2- 172.21.15.60:0/1512483577 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7dc0cd540 0x7fd7dc1f4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:55:03.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.718+0000 7fd7db400640 1 -- 172.21.15.60:0/1512483577 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd7dc0cd160 msgr2=0x7fd7dc1f43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:55:03.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.718+0000 7fd7db400640 1 --2- 172.21.15.60:0/1512483577 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd7dc0cd160 0x7fd7dc1f43f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:03.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.718+0000 7fd7db400640 1 -- 172.21.15.60:0/1512483577 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7dc0cd540 msgr2=0x7fd7dc1f4930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:55:03.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.718+0000 7fd7db400640 1 --2- 172.21.15.60:0/1512483577 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7dc0cd540 0x7fd7dc1f4930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:03.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.718+0000 7fd7db400640 1 -- 172.21.15.60:0/1512483577 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7dc15d6f0 con 0x7fd7dc0cb750 2024-04-14T23:55:03.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.718+0000 7fd7daa00640 1 --2- 172.21.15.60:0/1512483577 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd7dc0cd160 0x7fd7dc1f43f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:55:03.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.718+0000 7fd7dbe00640 1 --2- 172.21.15.60:0/1512483577 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7dc0cd540 0x7fd7dc1f4930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:55:03.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.718+0000 7fd7db400640 1 --2- 172.21.15.60:0/1512483577 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd7dc0cb750 0x7fd7dc15aae0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd7cc037040 tx=0x7fd7cc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:55:03.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.718+0000 7fd7d3400640 1 -- 172.21.15.60:0/1512483577 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7cc039050 con 0x7fd7dc0cb750 2024-04-14T23:55:03.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.718+0000 7fd7d3400640 1 -- 172.21.15.60:0/1512483577 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7cc045070 con 0x7fd7dc0cb750 2024-04-14T23:55:03.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.719+0000 7fd7e2800640 1 -- 172.21.15.60:0/1512483577 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7dc1f19a0 con 0x7fd7dc0cb750 2024-04-14T23:55:03.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.719+0000 7fd7d3400640 1 -- 172.21.15.60:0/1512483577 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7cc03e850 con 0x7fd7dc0cb750 2024-04-14T23:55:03.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.719+0000 7fd7e2800640 1 -- 172.21.15.60:0/1512483577 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7dc15a2e0 con 0x7fd7dc0cb750 2024-04-14T23:55:03.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.720+0000 7fd7e2800640 1 -- 172.21.15.60:0/1512483577 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7dc0cff90 con 0x7fd7dc0cb750 2024-04-14T23:55:03.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.721+0000 7fd7d3400640 1 -- 172.21.15.60:0/1512483577 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7fd7cc03e9f0 con 0x7fd7dc0cb750 2024-04-14T23:55:03.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.722+0000 7fd7d3400640 1 --2- 172.21.15.60:0/1512483577 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fd7b806b990 0x7fd7b806de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:55:03.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.722+0000 7fd7d3400640 1 -- 172.21.15.60:0/1512483577 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6083+0+0 (secure 0 0 0) 0x7fd7cc0ad5f0 con 0x7fd7dc0cb750 2024-04-14T23:55:03.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.722+0000 7fd7daa00640 1 --2- 172.21.15.60:0/1512483577 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fd7b806b990 0x7fd7b806de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:55:03.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.723+0000 7fd7daa00640 1 --2- 172.21.15.60:0/1512483577 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fd7b806b990 0x7fd7b806de50 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc15c7e0 tx=0x7fd7c4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:55:03.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.726+0000 7fd7d3400640 1 -- 172.21.15.60:0/1512483577 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd7dc0cff90 con 0x7fd7dc0cb750 2024-04-14T23:55:03.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.970+0000 7fd7e2800640 1 -- 172.21.15.60:0/1512483577 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd7dc076970 con 0x7fd7dc0cb750 2024-04-14T23:55:03.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.971+0000 7fd7d3400640 1 -- 172.21.15.60:0/1512483577 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6007 (secure 0 0 0) 0x7fd7cc07bb40 con 0x7fd7dc0cb750 2024-04-14T23:55:03.976 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:55:03.976 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:55:01.480605+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4986},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4986":{"gid":4986,"name":"a","rank":0,"incarnation":21,"state":"up:rejoin","state_seq":32,"addr":"172.21.15.60:6833/1971580590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1971580590},{"type":"v1","addr":"172.21.15.60:6833","nonce":1971580590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-14T23:55:03.976 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 23 2024-04-14T23:55:03.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.974+0000 7fd7e2800640 1 -- 172.21.15.60:0/1512483577 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fd7b806b990 msgr2=0x7fd7b806de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:55:03.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.974+0000 7fd7e2800640 1 --2- 172.21.15.60:0/1512483577 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fd7b806b990 0x7fd7b806de50 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc15c7e0 tx=0x7fd7c4009040 comp rx=0 tx=0).stop 2024-04-14T23:55:03.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.974+0000 7fd7e2800640 1 -- 172.21.15.60:0/1512483577 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd7dc0cb750 msgr2=0x7fd7dc15aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:55:03.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.974+0000 7fd7e2800640 1 --2- 172.21.15.60:0/1512483577 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd7dc0cb750 0x7fd7dc15aae0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd7cc037040 tx=0x7fd7cc004a40 comp rx=0 tx=0).stop 2024-04-14T23:55:03.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.974+0000 7fd7e2800640 1 -- 172.21.15.60:0/1512483577 shutdown_connections 2024-04-14T23:55:03.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.974+0000 7fd7e2800640 1 --2- 172.21.15.60:0/1512483577 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fd7b806b990 0x7fd7b806de50 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:03.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.974+0000 7fd7e2800640 1 --2- 172.21.15.60:0/1512483577 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7dc0cd540 0x7fd7dc1f4930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:03.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.974+0000 7fd7e2800640 1 --2- 172.21.15.60:0/1512483577 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd7dc0cd160 0x7fd7dc1f43f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:03.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.974+0000 7fd7e2800640 1 --2- 172.21.15.60:0/1512483577 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd7dc0cb750 0x7fd7dc15aae0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:03.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.974+0000 7fd7e2800640 1 -- 172.21.15.60:0/1512483577 >> 172.21.15.60:0/1512483577 conn(0x7fd7dc066f10 msgr2=0x7fd7dc0cef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:55:03.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.974+0000 7fd7e2800640 1 -- 172.21.15.60:0/1512483577 shutdown_connections 2024-04-14T23:55:03.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:03.974+0000 7fd7e2800640 1 -- 172.21.15.60:0/1512483577 wait complete. 2024-04-14T23:55:03.998 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 23, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-14T23:48:42.700188+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 23, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4986": { "addr": "172.21.15.60:6833/1971580590", "addrs": { "addrvec": [ { "addr": "172.21.15.60:6832", "nonce": 1971580590, "type": "v2" }, { "addr": "172.21.15.60:6833", "nonce": 1971580590, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4986, "incarnation": 21, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:rejoin", "state_seq": 32 } }, "last_failure": 0, "last_failure_osd_epoch": 35, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-14T23:55:01.480605+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4986 } } } ], "standbys": [ { "addr": "172.21.15.60:6837/1527815634", "addrs": { "addrvec": [ { "addr": "172.21.15.60:6836", "nonce": 1527815634, "type": "v2" }, { "addr": "172.21.15.60:6837", "nonce": 1527815634, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4197, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6838/3958319486", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6835", "nonce": 3958319486, "type": "v2" }, { "addr": "172.21.15.186:6838", "nonce": 3958319486, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4220, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6837/3910951185", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6834", "nonce": 3910951185, "type": "v2" }, { "addr": "172.21.15.186:6837", "nonce": 3910951185, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4264, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6839/2155020404", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6836", "nonce": 2155020404, "type": "v2" }, { "addr": "172.21.15.186:6839", "nonce": 2155020404, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4267, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-14T23:55:06.000 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:55:06.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.216+0000 7f9defe00640 1 Processor -- start 2024-04-14T23:55:06.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.216+0000 7f9defe00640 1 -- start start 2024-04-14T23:55:06.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.217+0000 7f9defe00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9df0073320 0x7f9df00c6950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:55:06.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.217+0000 7f9defe00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9df00c3d90 0x7f9df00c41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:55:06.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.217+0000 7f9defe00640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9df0152fb0 con 0x7f9df00b1e00 2024-04-14T23:55:06.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.217+0000 7f9defe00640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f9df00c6540 con 0x7f9df0073320 2024-04-14T23:55:06.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.217+0000 7f9defe00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f9df00b21e0 con 0x7f9df00c3d90 2024-04-14T23:55:06.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.217+0000 7f9def400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9df00c3d90 0x7f9df00c41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:55:06.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.217+0000 7f9deea00640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f9df00b1e00 0x7f9df00c7fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.60:58822/0 (socket says 172.21.15.60:58822) 2024-04-14T23:55:06.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.217+0000 7f9dee000640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9df0073320 0x7f9df00c6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:55:06.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.217+0000 7f9deea00640 1 -- 172.21.15.60:0/1771721878 learned_addr learned my addr 172.21.15.60:0/1771721878 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:55:06.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.218+0000 7f9ded600640 1 -- 172.21.15.60:0/1771721878 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1196239080 0 0) 0x7f9df0152fb0 con 0x7f9df00b1e00 2024-04-14T23:55:06.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.218+0000 7f9def400640 1 -- 172.21.15.60:0/1771721878 >> v2:172.21.15.60:3300/0 conn(0x7f9df0073320 msgr2=0x7f9df00c6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:55:06.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.218+0000 7f9def400640 1 --2- 172.21.15.60:0/1771721878 >> v2:172.21.15.60:3300/0 conn(0x7f9df0073320 0x7f9df00c6950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:06.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.218+0000 7f9def400640 1 -- 172.21.15.60:0/1771721878 >> v1:172.21.15.186:6789/0 conn(0x7f9df00b1e00 legacy=0x7f9df00c7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:55:06.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.218+0000 7f9def400640 1 -- 172.21.15.60:0/1771721878 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9df0159310 con 0x7f9df00c3d90 2024-04-14T23:55:06.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.219+0000 7f9def400640 1 --2- 172.21.15.60:0/1771721878 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9df00c3d90 0x7f9df00c41c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9de8009d90 tx=0x7f9de802bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74d56869fa202cba server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:55:06.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.219+0000 7f9ded600640 1 -- 172.21.15.60:0/1771721878 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de8039070 con 0x7f9df00c3d90 2024-04-14T23:55:06.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.219+0000 7f9ded600640 1 -- 172.21.15.60:0/1771721878 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9de802cce0 con 0x7f9df00c3d90 2024-04-14T23:55:06.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.219+0000 7f9ded600640 1 -- 172.21.15.60:0/1771721878 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de8034660 con 0x7f9df00c3d90 2024-04-14T23:55:06.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.220+0000 7f9defe00640 1 -- 172.21.15.60:0/1771721878 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9df00c3d90 msgr2=0x7f9df00c41c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:55:06.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.220+0000 7f9defe00640 1 --2- 172.21.15.60:0/1771721878 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9df00c3d90 0x7f9df00c41c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9de8009d90 tx=0x7f9de802bf30 comp rx=0 tx=0).stop 2024-04-14T23:55:06.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.220+0000 7f9defe00640 1 -- 172.21.15.60:0/1771721878 shutdown_connections 2024-04-14T23:55:06.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.220+0000 7f9defe00640 1 --2- 172.21.15.60:0/1771721878 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9df00c3d90 0x7f9df00c41c0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:06.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.220+0000 7f9defe00640 1 --2- 172.21.15.60:0/1771721878 >> v2:172.21.15.60:3300/0 conn(0x7f9df0073320 0x7f9df00c6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:06.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.220+0000 7f9defe00640 1 -- 172.21.15.60:0/1771721878 >> 172.21.15.60:0/1771721878 conn(0x7f9df00c8b50 msgr2=0x7f9df00c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:55:06.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.220+0000 7f9defe00640 1 -- 172.21.15.60:0/1771721878 shutdown_connections 2024-04-14T23:55:06.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.220+0000 7f9defe00640 1 -- 172.21.15.60:0/1771721878 wait complete. 2024-04-14T23:55:06.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.222+0000 7f9defe00640 1 Processor -- start 2024-04-14T23:55:06.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.222+0000 7f9defe00640 1 -- start start 2024-04-14T23:55:06.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.223+0000 7f9defe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9df0073320 0x7f9df00b11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:55:06.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.223+0000 7f9defe00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9df00b1e00 0x7f9df0151540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:55:06.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.223+0000 7f9defe00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9df00c3d90 0x7f9df014f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:55:06.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.223+0000 7f9defe00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f9df00c38b0 con 0x7f9df00b1e00 2024-04-14T23:55:06.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.223+0000 7f9defe00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9df0159890 con 0x7f9df0073320 2024-04-14T23:55:06.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.223+0000 7f9defe00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f9df01595b0 con 0x7f9df00c3d90 2024-04-14T23:55:06.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.223+0000 7f9def400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9df00c3d90 0x7f9df014f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:55:06.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.223+0000 7f9deea00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9df0073320 0x7f9df00b11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:55:06.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.223+0000 7f9dee000640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9df00b1e00 0x7f9df0151540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:55:06.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.223+0000 7f9def400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9df00c3d90 0x7f9df014f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.60:55848/0 (socket says 172.21.15.60:55848) 2024-04-14T23:55:06.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.223+0000 7f9dee000640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9df00b1e00 0x7f9df0151540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35518/0 (socket says 172.21.15.60:35518) 2024-04-14T23:55:06.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.223+0000 7f9def400640 1 -- 172.21.15.60:0/4198263332 learned_addr learned my addr 172.21.15.60:0/4198263332 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:55:06.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.224+0000 7f9deea00640 1 -- 172.21.15.60:0/4198263332 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9df00c3d90 msgr2=0x7f9df014f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:55:06.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.224+0000 7f9deea00640 1 --2- 172.21.15.60:0/4198263332 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9df00c3d90 0x7f9df014f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:06.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.224+0000 7f9deea00640 1 -- 172.21.15.60:0/4198263332 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9df00b1e00 msgr2=0x7f9df0151540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:55:06.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.224+0000 7f9deea00640 1 --2- 172.21.15.60:0/4198263332 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9df00b1e00 0x7f9df0151540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:06.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.224+0000 7f9deea00640 1 -- 172.21.15.60:0/4198263332 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9df01f0730 con 0x7f9df0073320 2024-04-14T23:55:06.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.224+0000 7f9dee000640 1 --2- 172.21.15.60:0/4198263332 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9df00b1e00 0x7f9df0151540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:55:06.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.224+0000 7f9def400640 1 --2- 172.21.15.60:0/4198263332 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9df00c3d90 0x7f9df014f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:55:06.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.224+0000 7f9deea00640 1 --2- 172.21.15.60:0/4198263332 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9df0073320 0x7f9df00b11e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9de000c450 tx=0x7f9de000c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:55:06.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.225+0000 7f9de6a00640 1 -- 172.21.15.60:0/4198263332 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de00074e0 con 0x7f9df0073320 2024-04-14T23:55:06.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.225+0000 7f9de6a00640 1 -- 172.21.15.60:0/4198263332 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9de00047e0 con 0x7f9df0073320 2024-04-14T23:55:06.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.225+0000 7f9de6a00640 1 -- 172.21.15.60:0/4198263332 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de0004bd0 con 0x7f9df0073320 2024-04-14T23:55:06.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.225+0000 7f9defe00640 1 -- 172.21.15.60:0/4198263332 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9df01e5130 con 0x7f9df0073320 2024-04-14T23:55:06.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.225+0000 7f9defe00640 1 -- 172.21.15.60:0/4198263332 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9df01e56f0 con 0x7f9df0073320 2024-04-14T23:55:06.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.226+0000 7f9defe00640 1 -- 172.21.15.60:0/4198263332 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9df004f540 con 0x7f9df0073320 2024-04-14T23:55:06.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.227+0000 7f9de6a00640 1 -- 172.21.15.60:0/4198263332 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f9de0011070 con 0x7f9df0073320 2024-04-14T23:55:06.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.228+0000 7f9de6a00640 1 --2- 172.21.15.60:0/4198263332 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f9dd006b990 0x7f9dd006de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:55:06.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.228+0000 7f9de6a00640 1 -- 172.21.15.60:0/4198263332 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6083+0+0 (secure 0 0 0) 0x7f9de008a5d0 con 0x7f9df0073320 2024-04-14T23:55:06.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.228+0000 7f9dee000640 1 --2- 172.21.15.60:0/4198263332 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f9dd006b990 0x7f9dd006de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:55:06.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.229+0000 7f9dee000640 1 --2- 172.21.15.60:0/4198263332 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f9dd006b990 0x7f9dd006de50 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f9df006c080 tx=0x7f9ddc009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:55:06.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.233+0000 7f9de6a00640 1 -- 172.21.15.60:0/4198263332 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f9de00588e0 con 0x7f9df0073320 2024-04-14T23:55:06.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.473+0000 7f9defe00640 1 -- 172.21.15.60:0/4198263332 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9df00c3030 con 0x7f9df0073320 2024-04-14T23:55:06.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.474+0000 7f9de6a00640 1 -- 172.21.15.60:0/4198263332 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6007 (secure 0 0 0) 0x7f9de005c980 con 0x7f9df0073320 2024-04-14T23:55:06.479 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:55:06.479 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:55:01.480605+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4986},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4986":{"gid":4986,"name":"a","rank":0,"incarnation":21,"state":"up:rejoin","state_seq":32,"addr":"172.21.15.60:6833/1971580590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1971580590},{"type":"v1","addr":"172.21.15.60:6833","nonce":1971580590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-14T23:55:06.480 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 23 2024-04-14T23:55:06.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.477+0000 7f9defe00640 1 -- 172.21.15.60:0/4198263332 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f9dd006b990 msgr2=0x7f9dd006de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:55:06.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.477+0000 7f9defe00640 1 --2- 172.21.15.60:0/4198263332 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f9dd006b990 0x7f9dd006de50 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f9df006c080 tx=0x7f9ddc009190 comp rx=0 tx=0).stop 2024-04-14T23:55:06.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.477+0000 7f9defe00640 1 -- 172.21.15.60:0/4198263332 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9df0073320 msgr2=0x7f9df00b11e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:55:06.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.477+0000 7f9defe00640 1 --2- 172.21.15.60:0/4198263332 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9df0073320 0x7f9df00b11e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9de000c450 tx=0x7f9de000c910 comp rx=0 tx=0).stop 2024-04-14T23:55:06.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.477+0000 7f9defe00640 1 -- 172.21.15.60:0/4198263332 shutdown_connections 2024-04-14T23:55:06.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.477+0000 7f9defe00640 1 --2- 172.21.15.60:0/4198263332 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f9dd006b990 0x7f9dd006de50 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:06.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.477+0000 7f9defe00640 1 --2- 172.21.15.60:0/4198263332 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9df00c3d90 0x7f9df014f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:06.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.477+0000 7f9defe00640 1 --2- 172.21.15.60:0/4198263332 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9df00b1e00 0x7f9df0151540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:06.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.477+0000 7f9defe00640 1 --2- 172.21.15.60:0/4198263332 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9df0073320 0x7f9df00b11e0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:06.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.477+0000 7f9defe00640 1 -- 172.21.15.60:0/4198263332 >> 172.21.15.60:0/4198263332 conn(0x7f9df00c8b50 msgr2=0x7f9df0158680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:55:06.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.477+0000 7f9defe00640 1 -- 172.21.15.60:0/4198263332 shutdown_connections 2024-04-14T23:55:06.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:06.477+0000 7f9defe00640 1 -- 172.21.15.60:0/4198263332 wait complete. 2024-04-14T23:55:06.500 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-14T23:55:08.500 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:55:08.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.723+0000 7fd5abe00640 1 Processor -- start 2024-04-14T23:55:08.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.723+0000 7fd5abe00640 1 -- start start 2024-04-14T23:55:08.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.725+0000 7fd5abe00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd5a40c8360 0x7fd5a415ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:55:08.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.725+0000 7fd5abe00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd5a40c8740 0x7fd5a406f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:55:08.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.725+0000 7fd5abe00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5a415f7c0 con 0x7fd5a40c75c0 2024-04-14T23:55:08.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.725+0000 7fd5abe00640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fd5a40c6b70 con 0x7fd5a40c8740 2024-04-14T23:55:08.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.725+0000 7fd5abe00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fd5a415faa0 con 0x7fd5a40c8360 2024-04-14T23:55:08.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.725+0000 7fd5a9e00640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fd5a40c75c0 0x7fd5a4158e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:50036/0 (socket says 172.21.15.60:50036) 2024-04-14T23:55:08.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.725+0000 7fd5a9e00640 1 -- 172.21.15.60:0/3202751106 learned_addr learned my addr 172.21.15.60:0/3202751106 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:55:08.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.725+0000 7fd5a8a00640 1 --2- 172.21.15.60:0/3202751106 >> v2:172.21.15.186:3300/0 conn(0x7fd5a40c8740 0x7fd5a406f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:55:08.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.725+0000 7fd5a9400640 1 --2- 172.21.15.60:0/3202751106 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd5a40c8360 0x7fd5a415ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:55:08.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.726+0000 7fd5a3e00640 1 -- 172.21.15.60:0/3202751106 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2840886735 0 0) 0x7fd5a415f7c0 con 0x7fd5a40c75c0 2024-04-14T23:55:08.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.726+0000 7fd5a3e00640 1 -- 172.21.15.60:0/3202751106 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd58c003520 con 0x7fd5a40c75c0 2024-04-14T23:55:08.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.726+0000 7fd5a8a00640 1 -- 172.21.15.60:0/3202751106 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd5a40c8360 msgr2=0x7fd5a415ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:55:08.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.726+0000 7fd5a8a00640 1 --2- 172.21.15.60:0/3202751106 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd5a40c8360 0x7fd5a415ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:08.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.726+0000 7fd5a8a00640 1 -- 172.21.15.60:0/3202751106 >> v1:172.21.15.60:6789/0 conn(0x7fd5a40c75c0 legacy=0x7fd5a4158e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:55:08.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.726+0000 7fd5a8a00640 1 -- 172.21.15.60:0/3202751106 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5a415fd80 con 0x7fd5a40c8740 2024-04-14T23:55:08.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.726+0000 7fd5a9400640 1 --2- 172.21.15.60:0/3202751106 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd5a40c8360 0x7fd5a415ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:55:08.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.727+0000 7fd5a8a00640 1 --2- 172.21.15.60:0/3202751106 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5a40c8740 0x7fd5a406f310 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd5900097a0 tx=0x7fd59002b940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b601b273fad1b96a server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:55:08.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.728+0000 7fd5a3e00640 1 -- 172.21.15.60:0/3202751106 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd590034070 con 0x7fd5a40c8740 2024-04-14T23:55:08.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.728+0000 7fd5a3e00640 1 -- 172.21.15.60:0/3202751106 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd590030800 con 0x7fd5a40c8740 2024-04-14T23:55:08.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.728+0000 7fd5a3e00640 1 -- 172.21.15.60:0/3202751106 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd590030bd0 con 0x7fd5a40c8740 2024-04-14T23:55:08.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.728+0000 7fd5abe00640 1 -- 172.21.15.60:0/3202751106 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5a40c8740 msgr2=0x7fd5a406f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:55:08.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.728+0000 7fd5abe00640 1 --2- 172.21.15.60:0/3202751106 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5a40c8740 0x7fd5a406f310 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd5900097a0 tx=0x7fd59002b940 comp rx=0 tx=0).stop 2024-04-14T23:55:08.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.728+0000 7fd5abe00640 1 -- 172.21.15.60:0/3202751106 shutdown_connections 2024-04-14T23:55:08.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.728+0000 7fd5abe00640 1 --2- 172.21.15.60:0/3202751106 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5a40c8740 0x7fd5a406f310 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:08.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.728+0000 7fd5abe00640 1 --2- 172.21.15.60:0/3202751106 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd5a40c8360 0x7fd5a415ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:08.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.728+0000 7fd5abe00640 1 -- 172.21.15.60:0/3202751106 >> 172.21.15.60:0/3202751106 conn(0x7fd5a40cc170 msgr2=0x7fd5a40cc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:55:08.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.728+0000 7fd5abe00640 1 -- 172.21.15.60:0/3202751106 shutdown_connections 2024-04-14T23:55:08.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.729+0000 7fd5abe00640 1 -- 172.21.15.60:0/3202751106 wait complete. 2024-04-14T23:55:08.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.731+0000 7fd5abe00640 1 Processor -- start 2024-04-14T23:55:08.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.731+0000 7fd5abe00640 1 -- start start 2024-04-14T23:55:08.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.732+0000 7fd5abe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5a40c75c0 0x7fd5a415e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:55:08.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.732+0000 7fd5abe00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd5a40c8360 0x7fd5a40cb230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:55:08.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.732+0000 7fd5abe00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd5a40c8740 0x7fd5a4161830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:55:08.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.732+0000 7fd5abe00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a415f4c0 con 0x7fd5a40c8360 2024-04-14T23:55:08.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.732+0000 7fd5abe00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a40c70a0 con 0x7fd5a40c75c0 2024-04-14T23:55:08.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.732+0000 7fd5abe00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fd5a4071030 con 0x7fd5a40c8740 2024-04-14T23:55:08.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.732+0000 7fd5a8a00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd5a40c8360 0x7fd5a40cb230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:55:08.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.732+0000 7fd5a9400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5a40c75c0 0x7fd5a415e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:55:08.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.732+0000 7fd5a9e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd5a40c8740 0x7fd5a4161830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:55:08.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.732+0000 7fd5a9400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5a40c75c0 0x7fd5a415e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:35158/0 (socket says 172.21.15.60:35158) 2024-04-14T23:55:08.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.732+0000 7fd5a9400640 1 -- 172.21.15.60:0/1429764830 learned_addr learned my addr 172.21.15.60:0/1429764830 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:55:08.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.733+0000 7fd5a9400640 1 -- 172.21.15.60:0/1429764830 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd5a40c8740 msgr2=0x7fd5a4161830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:55:08.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.733+0000 7fd5a9400640 1 --2- 172.21.15.60:0/1429764830 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd5a40c8740 0x7fd5a4161830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:08.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.733+0000 7fd5a9400640 1 -- 172.21.15.60:0/1429764830 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd5a40c8360 msgr2=0x7fd5a40cb230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:55:08.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.733+0000 7fd5a9400640 1 --2- 172.21.15.60:0/1429764830 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd5a40c8360 0x7fd5a40cb230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:08.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.733+0000 7fd5a9400640 1 -- 172.21.15.60:0/1429764830 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5a415ce90 con 0x7fd5a40c75c0 2024-04-14T23:55:08.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.733+0000 7fd5a9400640 1 --2- 172.21.15.60:0/1429764830 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5a40c75c0 0x7fd5a415e990 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fd59800ebe0 tx=0x7fd59800c520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:55:08.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.733+0000 7fd5a1600640 1 -- 172.21.15.60:0/1429764830 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd598017070 con 0x7fd5a40c75c0 2024-04-14T23:55:08.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.734+0000 7fd5a1600640 1 -- 172.21.15.60:0/1429764830 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd59800cc30 con 0x7fd5a40c75c0 2024-04-14T23:55:08.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.734+0000 7fd5a1600640 1 -- 172.21.15.60:0/1429764830 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd598013470 con 0x7fd5a40c75c0 2024-04-14T23:55:08.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.734+0000 7fd5abe00640 1 -- 172.21.15.60:0/1429764830 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5a41f5480 con 0x7fd5a40c75c0 2024-04-14T23:55:08.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.734+0000 7fd5abe00640 1 -- 172.21.15.60:0/1429764830 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5a415d430 con 0x7fd5a40c75c0 2024-04-14T23:55:08.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.735+0000 7fd5abe00640 1 -- 172.21.15.60:0/1429764830 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5a4152f10 con 0x7fd5a40c75c0 2024-04-14T23:55:08.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.736+0000 7fd5a1600640 1 -- 172.21.15.60:0/1429764830 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7fd59801a070 con 0x7fd5a40c75c0 2024-04-14T23:55:08.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.737+0000 7fd5a1600640 1 --2- 172.21.15.60:0/1429764830 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fd58406b9c0 0x7fd58406de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:55:08.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.737+0000 7fd5a1600640 1 -- 172.21.15.60:0/1429764830 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6083+0+0 (secure 0 0 0) 0x7fd59808ac90 con 0x7fd5a40c75c0 2024-04-14T23:55:08.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.737+0000 7fd5a8a00640 1 --2- 172.21.15.60:0/1429764830 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fd58406b9c0 0x7fd58406de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:55:08.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.738+0000 7fd5a8a00640 1 --2- 172.21.15.60:0/1429764830 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fd58406b9c0 0x7fd58406de80 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fd590036000 tx=0x7fd59003b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:55:08.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.741+0000 7fd5a1600640 1 -- 172.21.15.60:0/1429764830 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd5a4152f10 con 0x7fd5a40c75c0 2024-04-14T23:55:08.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.981+0000 7fd5abe00640 1 -- 172.21.15.60:0/1429764830 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd5a4076970 con 0x7fd5a40c75c0 2024-04-14T23:55:08.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.983+0000 7fd5a1600640 1 -- 172.21.15.60:0/1429764830 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6013 (secure 0 0 0) 0x7fd598059030 con 0x7fd5a40c75c0 2024-04-14T23:55:08.989 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:55:08.989 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:55:06.601086+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4986},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4986":{"gid":4986,"name":"a","rank":0,"incarnation":21,"state":"up:clientreplay","state_seq":34,"addr":"172.21.15.60:6833/1971580590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1971580590},{"type":"v1","addr":"172.21.15.60:6833","nonce":1971580590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-14T23:55:08.989 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 24 2024-04-14T23:55:08.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.985+0000 7fd5abe00640 1 -- 172.21.15.60:0/1429764830 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fd58406b9c0 msgr2=0x7fd58406de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:55:08.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.985+0000 7fd5abe00640 1 --2- 172.21.15.60:0/1429764830 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fd58406b9c0 0x7fd58406de80 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fd590036000 tx=0x7fd59003b040 comp rx=0 tx=0).stop 2024-04-14T23:55:08.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.985+0000 7fd5abe00640 1 -- 172.21.15.60:0/1429764830 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5a40c75c0 msgr2=0x7fd5a415e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:55:08.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.985+0000 7fd5abe00640 1 --2- 172.21.15.60:0/1429764830 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5a40c75c0 0x7fd5a415e990 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fd59800ebe0 tx=0x7fd59800c520 comp rx=0 tx=0).stop 2024-04-14T23:55:08.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.986+0000 7fd5abe00640 1 -- 172.21.15.60:0/1429764830 shutdown_connections 2024-04-14T23:55:08.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.986+0000 7fd5abe00640 1 --2- 172.21.15.60:0/1429764830 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fd58406b9c0 0x7fd58406de80 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:08.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.986+0000 7fd5abe00640 1 --2- 172.21.15.60:0/1429764830 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd5a40c8740 0x7fd5a4161830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:08.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.986+0000 7fd5abe00640 1 --2- 172.21.15.60:0/1429764830 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd5a40c8360 0x7fd5a40cb230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:08.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.986+0000 7fd5abe00640 1 --2- 172.21.15.60:0/1429764830 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5a40c75c0 0x7fd5a415e990 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:08.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.986+0000 7fd5abe00640 1 -- 172.21.15.60:0/1429764830 >> 172.21.15.60:0/1429764830 conn(0x7fd5a40cc170 msgr2=0x7fd5a4072080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:55:08.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.986+0000 7fd5abe00640 1 -- 172.21.15.60:0/1429764830 shutdown_connections 2024-04-14T23:55:08.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:08.986+0000 7fd5abe00640 1 -- 172.21.15.60:0/1429764830 wait complete. 2024-04-14T23:55:09.012 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-14T23:55:11.014 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:55:11.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.240+0000 7f3cb1a00640 1 Processor -- start 2024-04-14T23:55:11.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.241+0000 7f3cb1a00640 1 -- start start 2024-04-14T23:55:11.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.242+0000 7f3cb1a00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f3cac0b1e00 0x7f3cac0c6950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:55:11.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.242+0000 7f3cb1a00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3cac073320 0x7f3cac0c7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:55:11.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.242+0000 7f3cb1a00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cac152fb0 con 0x7f3cac0c84e0 2024-04-14T23:55:11.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.242+0000 7f3cb1a00640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f3cac0c6540 con 0x7f3cac073320 2024-04-14T23:55:11.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.242+0000 7f3cb1a00640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f3cac0b21e0 con 0x7f3cac0b1e00 2024-04-14T23:55:11.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.243+0000 7f3cab400640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3cac073320 0x7f3cac0c7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:55:11.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.243+0000 7f3cab400640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3cac073320 0x7f3cac0c7fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35544/0 (socket says 172.21.15.60:35544) 2024-04-14T23:55:11.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.243+0000 7f3cab400640 1 -- 172.21.15.60:0/3926838460 learned_addr learned my addr 172.21.15.60:0/3926838460 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:55:11.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.243+0000 7f3cabe00640 1 --2- 172.21.15.60:0/3926838460 >> v2:172.21.15.186:3300/0 conn(0x7f3cac0b1e00 0x7f3cac0c6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:55:11.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.243+0000 7f3cab400640 1 -- 172.21.15.60:0/3926838460 >> v2:172.21.15.186:3300/0 conn(0x7f3cac0b1e00 msgr2=0x7f3cac0c6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:55:11.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.243+0000 7f3cab400640 1 --2- 172.21.15.60:0/3926838460 >> v2:172.21.15.186:3300/0 conn(0x7f3cac0b1e00 0x7f3cac0c6950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:11.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.243+0000 7f3cab400640 1 -- 172.21.15.60:0/3926838460 >> v1:172.21.15.60:6789/0 conn(0x7f3cac0c84e0 legacy=0x7f3cac0c3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:55:11.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.243+0000 7f3cab400640 1 -- 172.21.15.60:0/3926838460 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cac159310 con 0x7f3cac073320 2024-04-14T23:55:11.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.244+0000 7f3cab400640 1 --2- 172.21.15.60:0/3926838460 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3cac073320 0x7f3cac0c7fa0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f3cac0c36e0 tx=0x7f3c9802ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa9c89df6bd2443 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:55:11.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.245+0000 7f3caaa00640 1 -- 172.21.15.60:0/3926838460 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c9802f850 con 0x7f3cac073320 2024-04-14T23:55:11.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.245+0000 7f3caaa00640 1 -- 172.21.15.60:0/3926838460 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c9802f9f0 con 0x7f3cac073320 2024-04-14T23:55:11.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.245+0000 7f3caaa00640 1 -- 172.21.15.60:0/3926838460 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c9802fdc0 con 0x7f3cac073320 2024-04-14T23:55:11.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.245+0000 7f3cb1a00640 1 -- 172.21.15.60:0/3926838460 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3cac073320 msgr2=0x7f3cac0c7fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:55:11.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.245+0000 7f3cb1a00640 1 --2- 172.21.15.60:0/3926838460 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3cac073320 0x7f3cac0c7fa0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f3cac0c36e0 tx=0x7f3c9802ba30 comp rx=0 tx=0).stop 2024-04-14T23:55:11.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.245+0000 7f3cb1a00640 1 -- 172.21.15.60:0/3926838460 shutdown_connections 2024-04-14T23:55:11.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.245+0000 7f3cb1a00640 1 --2- 172.21.15.60:0/3926838460 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3cac073320 0x7f3cac0c7fa0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:11.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.245+0000 7f3cb1a00640 1 --2- 172.21.15.60:0/3926838460 >> v2:172.21.15.186:3300/0 conn(0x7f3cac0b1e00 0x7f3cac0c6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:11.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.245+0000 7f3cb1a00640 1 -- 172.21.15.60:0/3926838460 >> 172.21.15.60:0/3926838460 conn(0x7f3cac0c8b50 msgr2=0x7f3cac0c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:55:11.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.245+0000 7f3cb1a00640 1 -- 172.21.15.60:0/3926838460 shutdown_connections 2024-04-14T23:55:11.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.246+0000 7f3cb1a00640 1 -- 172.21.15.60:0/3926838460 wait complete. 2024-04-14T23:55:11.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.247+0000 7f3cb1a00640 1 Processor -- start 2024-04-14T23:55:11.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.247+0000 7f3cb1a00640 1 -- start start 2024-04-14T23:55:11.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.248+0000 7f3cb1a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3cac073320 0x7f3cac15c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:55:11.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.248+0000 7f3cb1a00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3cac0b1e00 0x7f3cac0bff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:55:11.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.248+0000 7f3cb1a00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3cac0c84e0 0x7f3cac1513f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:55:11.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.248+0000 7f3cb1a00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f3cac0c38b0 con 0x7f3cac0c84e0 2024-04-14T23:55:11.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.248+0000 7f3cb1a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3cac159890 con 0x7f3cac073320 2024-04-14T23:55:11.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.248+0000 7f3cb1a00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f3cac0c5f80 con 0x7f3cac0b1e00 2024-04-14T23:55:11.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.249+0000 7f3cab400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3cac0b1e00 0x7f3cac0bff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:55:11.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.249+0000 7f3cab400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3cac0b1e00 0x7f3cac0bff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.60:55876/0 (socket says 172.21.15.60:55876) 2024-04-14T23:55:11.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.249+0000 7f3cab400640 1 -- 172.21.15.60:0/3717950303 learned_addr learned my addr 172.21.15.60:0/3717950303 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:55:11.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.249+0000 7f3cb1000640 1 --2- 172.21.15.60:0/3717950303 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3cac0c84e0 0x7f3cac1513f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:55:11.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.249+0000 7f3cabe00640 1 --2- 172.21.15.60:0/3717950303 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3cac073320 0x7f3cac15c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:55:11.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.249+0000 7f3cab400640 1 -- 172.21.15.60:0/3717950303 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3cac073320 msgr2=0x7f3cac15c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:55:11.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.249+0000 7f3cab400640 1 --2- 172.21.15.60:0/3717950303 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3cac073320 0x7f3cac15c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:11.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.249+0000 7f3cab400640 1 -- 172.21.15.60:0/3717950303 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3cac0c84e0 msgr2=0x7f3cac1513f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:55:11.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.249+0000 7f3cab400640 1 --2- 172.21.15.60:0/3717950303 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3cac0c84e0 0x7f3cac1513f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:11.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.249+0000 7f3cab400640 1 -- 172.21.15.60:0/3717950303 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cac0c0870 con 0x7f3cac0b1e00 2024-04-14T23:55:11.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.250+0000 7f3cab400640 1 --2- 172.21.15.60:0/3717950303 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3cac0b1e00 0x7f3cac0bff50 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f3c98000c00 tx=0x7f3c98004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:55:11.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.250+0000 7f3ca3e00640 1 -- 172.21.15.60:0/3717950303 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c98047070 con 0x7f3cac0b1e00 2024-04-14T23:55:11.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.250+0000 7f3ca3e00640 1 -- 172.21.15.60:0/3717950303 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c98037cb0 con 0x7f3cac0b1e00 2024-04-14T23:55:11.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.250+0000 7f3ca3e00640 1 -- 172.21.15.60:0/3717950303 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c98037e60 con 0x7f3cac0b1e00 2024-04-14T23:55:11.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.250+0000 7f3cb1a00640 1 -- 172.21.15.60:0/3717950303 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cac153950 con 0x7f3cac0b1e00 2024-04-14T23:55:11.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.250+0000 7f3cb1a00640 1 -- 172.21.15.60:0/3717950303 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cac153f10 con 0x7f3cac0b1e00 2024-04-14T23:55:11.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.251+0000 7f3cb1a00640 1 -- 172.21.15.60:0/3717950303 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cac04f540 con 0x7f3cac0b1e00 2024-04-14T23:55:11.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.252+0000 7f3ca3e00640 1 -- 172.21.15.60:0/3717950303 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f3c9804a070 con 0x7f3cac0b1e00 2024-04-14T23:55:11.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.253+0000 7f3ca3e00640 1 --2- 172.21.15.60:0/3717950303 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f3c8c06b990 0x7f3c8c06de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:55:11.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.253+0000 7f3ca3e00640 1 -- 172.21.15.60:0/3717950303 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(35..35 src has 1..35) ==== 6083+0+0 (secure 0 0 0) 0x7f3c980acd30 con 0x7f3cac0b1e00 2024-04-14T23:55:11.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.254+0000 7f3cabe00640 1 --2- 172.21.15.60:0/3717950303 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f3c8c06b990 0x7f3c8c06de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:55:11.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.254+0000 7f3cabe00640 1 --2- 172.21.15.60:0/3717950303 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f3c8c06b990 0x7f3c8c06de50 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f3c9c006fd0 tx=0x7f3c9c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:55:11.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.258+0000 7f3ca3e00640 1 -- 172.21.15.60:0/3717950303 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3c98033030 con 0x7f3cac0b1e00 2024-04-14T23:55:11.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.498+0000 7f3cb1a00640 1 -- 172.21.15.60:0/3717950303 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3cac0c3330 con 0x7f3cac0b1e00 2024-04-14T23:55:11.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.500+0000 7f3ca3e00640 1 -- 172.21.15.60:0/3717950303 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6013 (secure 0 0 0) 0x7f3c9807b0d0 con 0x7f3cac0b1e00 2024-04-14T23:55:11.506 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:55:11.506 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:55:06.601086+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4986},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4986":{"gid":4986,"name":"a","rank":0,"incarnation":21,"state":"up:clientreplay","state_seq":34,"addr":"172.21.15.60:6833/1971580590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1971580590},{"type":"v1","addr":"172.21.15.60:6833","nonce":1971580590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-14T23:55:11.506 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 24 2024-04-14T23:55:11.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.502+0000 7f3cb1a00640 1 -- 172.21.15.60:0/3717950303 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f3c8c06b990 msgr2=0x7f3c8c06de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:55:11.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.502+0000 7f3cb1a00640 1 --2- 172.21.15.60:0/3717950303 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f3c8c06b990 0x7f3c8c06de50 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f3c9c006fd0 tx=0x7f3c9c008040 comp rx=0 tx=0).stop 2024-04-14T23:55:11.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.502+0000 7f3cb1a00640 1 -- 172.21.15.60:0/3717950303 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3cac0b1e00 msgr2=0x7f3cac0bff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:55:11.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.502+0000 7f3cb1a00640 1 --2- 172.21.15.60:0/3717950303 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3cac0b1e00 0x7f3cac0bff50 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f3c98000c00 tx=0x7f3c98004730 comp rx=0 tx=0).stop 2024-04-14T23:55:11.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.502+0000 7f3cb1a00640 1 -- 172.21.15.60:0/3717950303 shutdown_connections 2024-04-14T23:55:11.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.502+0000 7f3cb1a00640 1 --2- 172.21.15.60:0/3717950303 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f3c8c06b990 0x7f3c8c06de50 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:11.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.502+0000 7f3cb1a00640 1 --2- 172.21.15.60:0/3717950303 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3cac0c84e0 0x7f3cac1513f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:11.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.502+0000 7f3cb1a00640 1 --2- 172.21.15.60:0/3717950303 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3cac0b1e00 0x7f3cac0bff50 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:11.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.502+0000 7f3cb1a00640 1 --2- 172.21.15.60:0/3717950303 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3cac073320 0x7f3cac15c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:11.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.503+0000 7f3cb1a00640 1 -- 172.21.15.60:0/3717950303 >> 172.21.15.60:0/3717950303 conn(0x7f3cac0c8b50 msgr2=0x7f3cac157be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:55:11.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.503+0000 7f3cb1a00640 1 -- 172.21.15.60:0/3717950303 shutdown_connections 2024-04-14T23:55:11.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:11.503+0000 7f3cb1a00640 1 -- 172.21.15.60:0/3717950303 wait complete. 2024-04-14T23:55:11.530 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-14T23:55:13.531 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:55:13.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:13.750+0000 7f28b5600640 1 Processor -- start 2024-04-14T23:55:13.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:13.750+0000 7f28b5600640 1 -- start start 2024-04-14T23:55:13.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:13.752+0000 7f28b5600640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f28b00b1e00 0x7f28b00c6950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:55:13.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:13.752+0000 7f28b5600640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f28b0073320 0x7f28b00c7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:55:13.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:13.752+0000 7f28b5600640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28b0152fb0 con 0x7f28b00c84e0 2024-04-14T23:55:13.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:13.752+0000 7f28b5600640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f28b00c6540 con 0x7f28b00b1e00 2024-04-14T23:55:13.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:13.752+0000 7f28b5600640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f28b00b21e0 con 0x7f28b0073320 2024-04-14T23:55:13.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:13.752+0000 7f28b4c00640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f28b00c84e0 0x7f28b00c3d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:41534/0 (socket says 172.21.15.60:41534) 2024-04-14T23:55:13.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:13.752+0000 7f28afe00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f28b00b1e00 0x7f28b00c6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:55:13.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:13.752+0000 7f28b4c00640 1 -- 172.21.15.60:0/1362106734 learned_addr learned my addr 172.21.15.60:0/1362106734 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:55:13.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:13.752+0000 7f28af400640 1 --2- 172.21.15.60:0/1362106734 >> v2:172.21.15.186:3300/0 conn(0x7f28b0073320 0x7f28b00c7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:55:13.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:13.753+0000 7f28aea00640 1 -- 172.21.15.60:0/1362106734 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 943994093 0 0) 0x7f28b0152fb0 con 0x7f28b00c84e0 2024-04-14T23:55:13.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:13.753+0000 7f28afe00640 1 -- 172.21.15.60:0/1362106734 >> v2:172.21.15.186:3300/0 conn(0x7f28b0073320 msgr2=0x7f28b00c7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:55:13.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:13.753+0000 7f28afe00640 1 --2- 172.21.15.60:0/1362106734 >> v2:172.21.15.186:3300/0 conn(0x7f28b0073320 0x7f28b00c7fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:13.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:13.753+0000 7f28afe00640 1 -- 172.21.15.60:0/1362106734 >> v1:172.21.15.60:6789/0 conn(0x7f28b00c84e0 legacy=0x7f28b00c3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:55:13.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:13.753+0000 7f28afe00640 1 -- 172.21.15.60:0/1362106734 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28b0159310 con 0x7f28b00b1e00 2024-04-14T23:55:13.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:13.754+0000 7f28afe00640 1 --2- 172.21.15.60:0/1362106734 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28b00b1e00 0x7f28b00c6950 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f28b006e450 tx=0x7f28a002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84f6071554c01eff server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:55:13.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:13.754+0000 7f28aea00640 1 -- 172.21.15.60:0/1362106734 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28a0034070 con 0x7f28b00b1e00 2024-04-14T23:55:13.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:13.754+0000 7f28aea00640 1 -- 172.21.15.60:0/1362106734 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28a0030610 con 0x7f28b00b1e00 2024-04-14T23:55:13.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:13.754+0000 7f28aea00640 1 -- 172.21.15.60:0/1362106734 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28a00309e0 con 0x7f28b00b1e00 2024-04-14T23:55:13.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:13.755+0000 7f28b5600640 1 -- 172.21.15.60:0/1362106734 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28b00b1e00 msgr2=0x7f28b00c6950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:55:13.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:13.755+0000 7f28b5600640 1 --2- 172.21.15.60:0/1362106734 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28b00b1e00 0x7f28b00c6950 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f28b006e450 tx=0x7f28a002b960 comp rx=0 tx=0).stop 2024-04-14T23:55:13.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:13.755+0000 7f28b5600640 1 -- 172.21.15.60:0/1362106734 shutdown_connections 2024-04-14T23:55:13.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:13.755+0000 7f28b5600640 1 --2- 172.21.15.60:0/1362106734 >> v2:172.21.15.186:3300/0 conn(0x7f28b0073320 0x7f28b00c7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:13.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:13.755+0000 7f28b5600640 1 --2- 172.21.15.60:0/1362106734 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28b00b1e00 0x7f28b00c6950 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:13.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:13.755+0000 7f28b5600640 1 -- 172.21.15.60:0/1362106734 >> 172.21.15.60:0/1362106734 conn(0x7f28b00c8b50 msgr2=0x7f28b00c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:55:13.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:13.755+0000 7f28b5600640 1 -- 172.21.15.60:0/1362106734 shutdown_connections 2024-04-14T23:55:13.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:13.755+0000 7f28b5600640 1 -- 172.21.15.60:0/1362106734 wait complete. 2024-04-14T23:55:13.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:13.757+0000 7f28b5600640 1 Processor -- start 2024-04-14T23:55:13.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:13.757+0000 7f28b5600640 1 -- start start 2024-04-14T23:55:13.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:13.758+0000 7f28b5600640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f28b0073320 0x7f28b00727c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:55:13.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:13.758+0000 7f28b5600640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28b00b1e00 0x7f28b0157a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:55:13.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:13.758+0000 7f28b5600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28b00c84e0 0x7f28b01f4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:55:13.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:13.758+0000 7f28b5600640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f28b00c38b0 con 0x7f28b00b1e00 2024-04-14T23:55:13.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:13.758+0000 7f28b5600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f28b0159890 con 0x7f28b00c84e0 2024-04-14T23:55:13.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:13.758+0000 7f28b5600640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f28b00c5f80 con 0x7f28b0073320 2024-04-14T23:55:13.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:13.758+0000 7f28afe00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f28b0073320 0x7f28b00727c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:55:13.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:13.758+0000 7f28afe00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f28b0073320 0x7f28b00727c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.60:41218/0 (socket says 172.21.15.60:41218) 2024-04-14T23:55:13.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:13.758+0000 7f28afe00640 1 -- 172.21.15.60:0/2211091928 learned_addr learned my addr 172.21.15.60:0/2211091928 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:55:13.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:13.758+0000 7f28b4c00640 1 --2- 172.21.15.60:0/2211091928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28b00c84e0 0x7f28b01f4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:55:13.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:13.758+0000 7f28af400640 1 --2- 172.21.15.60:0/2211091928 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28b00b1e00 0x7f28b0157a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:55:13.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:13.759+0000 7f28afe00640 1 -- 172.21.15.60:0/2211091928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28b00c84e0 msgr2=0x7f28b01f4600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:55:13.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:13.759+0000 7f28afe00640 1 --2- 172.21.15.60:0/2211091928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28b00c84e0 0x7f28b01f4600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:13.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:13.759+0000 7f28afe00640 1 -- 172.21.15.60:0/2211091928 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28b00b1e00 msgr2=0x7f28b0157a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:55:13.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:13.759+0000 7f28afe00640 1 --2- 172.21.15.60:0/2211091928 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28b00b1e00 0x7f28b0157a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:13.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:13.759+0000 7f28afe00640 1 -- 172.21.15.60:0/2211091928 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28b01e5520 con 0x7f28b0073320 2024-04-14T23:55:13.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:13.759+0000 7f28afe00640 1 --2- 172.21.15.60:0/2211091928 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f28b0073320 0x7f28b00727c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f28a0000c00 tx=0x7f28a00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:55:13.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:13.759+0000 7f28a7e00640 1 -- 172.21.15.60:0/2211091928 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28a0034040 con 0x7f28b0073320 2024-04-14T23:55:13.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:13.759+0000 7f28a7e00640 1 -- 172.21.15.60:0/2211091928 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28a0030410 con 0x7f28b0073320 2024-04-14T23:55:13.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:13.759+0000 7f28a7e00640 1 -- 172.21.15.60:0/2211091928 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28a0039c50 con 0x7f28b0073320 2024-04-14T23:55:13.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:13.759+0000 7f28b5600640 1 -- 172.21.15.60:0/2211091928 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28b0072470 con 0x7f28b0073320 2024-04-14T23:55:13.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:13.760+0000 7f28b5600640 1 -- 172.21.15.60:0/2211091928 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f28b01e5aa0 con 0x7f28b0073320 2024-04-14T23:55:13.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:13.761+0000 7f28b5600640 1 -- 172.21.15.60:0/2211091928 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28b004f540 con 0x7f28b0073320 2024-04-14T23:55:13.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:13.762+0000 7f28a7e00640 1 -- 172.21.15.60:0/2211091928 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f28a0037070 con 0x7f28b0073320 2024-04-14T23:55:13.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:13.763+0000 7f28a7e00640 1 --2- 172.21.15.60:0/2211091928 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f289006b9c0 0x7f289006de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:55:13.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:13.763+0000 7f28a7e00640 1 -- 172.21.15.60:0/2211091928 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(35..35 src has 1..35) ==== 6083+0+0 (secure 0 0 0) 0x7f28a00ac920 con 0x7f28b0073320 2024-04-14T23:55:13.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:13.763+0000 7f28af400640 1 --2- 172.21.15.60:0/2211091928 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f289006b9c0 0x7f289006de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:55:13.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:13.764+0000 7f28af400640 1 --2- 172.21.15.60:0/2211091928 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f289006b9c0 0x7f289006de80 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f28b0158320 tx=0x7f289c009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:55:13.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:13.767+0000 7f28a7e00640 1 -- 172.21.15.60:0/2211091928 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f28a007acc0 con 0x7f28b0073320 2024-04-14T23:55:14.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:14.006+0000 7f28b5600640 1 -- 172.21.15.60:0/2211091928 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f28b00cb250 con 0x7f28b0073320 2024-04-14T23:55:14.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:14.007+0000 7f28a7e00640 1 -- 172.21.15.60:0/2211091928 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6014 (secure 0 0 0) 0x7f28a007ed60 con 0x7f28b0073320 2024-04-14T23:55:14.014 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:55:14.014 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:55:13.553698+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4986},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4986":{"gid":4986,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":36,"addr":"172.21.15.60:6833/1971580590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1971580590},{"type":"v1","addr":"172.21.15.60:6833","nonce":1971580590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4986,"qdb_cluster":[4986]},"id":1}]} 2024-04-14T23:55:14.014 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 25 2024-04-14T23:55:14.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:14.010+0000 7f28b5600640 1 -- 172.21.15.60:0/2211091928 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f289006b9c0 msgr2=0x7f289006de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:55:14.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:14.010+0000 7f28b5600640 1 --2- 172.21.15.60:0/2211091928 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f289006b9c0 0x7f289006de80 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f28b0158320 tx=0x7f289c009190 comp rx=0 tx=0).stop 2024-04-14T23:55:14.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:14.010+0000 7f28b5600640 1 -- 172.21.15.60:0/2211091928 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f28b0073320 msgr2=0x7f28b00727c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:55:14.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:14.010+0000 7f28b5600640 1 --2- 172.21.15.60:0/2211091928 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f28b0073320 0x7f28b00727c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f28a0000c00 tx=0x7f28a00046c0 comp rx=0 tx=0).stop 2024-04-14T23:55:14.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:14.010+0000 7f28b5600640 1 -- 172.21.15.60:0/2211091928 shutdown_connections 2024-04-14T23:55:14.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:14.010+0000 7f28b5600640 1 --2- 172.21.15.60:0/2211091928 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f289006b9c0 0x7f289006de80 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:14.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:14.010+0000 7f28b5600640 1 --2- 172.21.15.60:0/2211091928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28b00c84e0 0x7f28b01f4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:14.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:14.010+0000 7f28b5600640 1 --2- 172.21.15.60:0/2211091928 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28b00b1e00 0x7f28b0157a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:14.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:14.010+0000 7f28b5600640 1 --2- 172.21.15.60:0/2211091928 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f28b0073320 0x7f28b00727c0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:14.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:14.010+0000 7f28b5600640 1 -- 172.21.15.60:0/2211091928 >> 172.21.15.60:0/2211091928 conn(0x7f28b00c8b50 msgr2=0x7f28b00b0ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:55:14.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:14.010+0000 7f28b5600640 1 -- 172.21.15.60:0/2211091928 shutdown_connections 2024-04-14T23:55:14.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:14.010+0000 7f28b5600640 1 -- 172.21.15.60:0/2211091928 wait complete. 2024-04-14T23:55:14.036 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=4839 2024-04-14T23:55:14.036 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 8 secs before reviving mds.c 2024-04-14T23:55:15.627 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-14T23:55:15.635 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-14T23:55:18.648 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/137/14' 2024-04-14T23:55:18.648 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/137/18' 2024-04-14T23:55:18.648 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/137/13' 2024-04-14T23:55:18.649 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/137/17' 2024-04-14T23:55:18.649 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/137' 2024-04-14T23:55:18.649 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/198/19' 2024-04-14T23:55:18.649 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/198/15' 2024-04-14T23:55:18.649 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/198/20' 2024-04-14T23:55:18.649 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/198/12' 2024-04-14T23:55:18.649 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/198/10' 2024-04-14T23:55:18.649 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/198/11' 2024-04-14T23:55:18.649 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/198/16' 2024-04-14T23:55:18.649 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/198/14' 2024-04-14T23:55:18.649 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/198/18' 2024-04-14T23:55:18.649 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/198/13' 2024-04-14T23:55:18.649 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/198/17' 2024-04-14T23:55:18.649 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/198' 2024-04-14T23:55:18.650 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/136/19' 2024-04-14T23:55:18.650 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/136/15' 2024-04-14T23:55:18.650 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/136/20' 2024-04-14T23:55:18.650 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/136/12' 2024-04-14T23:55:18.650 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/136/10' 2024-04-14T23:55:18.650 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/136/11' 2024-04-14T23:55:18.650 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/136/16' 2024-04-14T23:55:18.650 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/136/14' 2024-04-14T23:55:18.650 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/136/18' 2024-04-14T23:55:18.650 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/136/13' 2024-04-14T23:55:18.650 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/136/17' 2024-04-14T23:55:18.650 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/136' 2024-04-14T23:55:18.650 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/125/19' 2024-04-14T23:55:18.650 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/125/15' 2024-04-14T23:55:18.651 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/125/20' 2024-04-14T23:55:18.651 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/125/12' 2024-04-14T23:55:18.651 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/125/10' 2024-04-14T23:55:18.651 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/125/11' 2024-04-14T23:55:18.651 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/125/16' 2024-04-14T23:55:18.651 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/125/14' 2024-04-14T23:55:18.651 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/125/18' 2024-04-14T23:55:18.651 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/125/13' 2024-04-14T23:55:18.651 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/125/17' 2024-04-14T23:55:18.651 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/125' 2024-04-14T23:55:18.651 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/194/19' 2024-04-14T23:55:18.651 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/194/15' 2024-04-14T23:55:18.651 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/194/20' 2024-04-14T23:55:18.651 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/194/12' 2024-04-14T23:55:18.652 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/194/10' 2024-04-14T23:55:18.652 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/194/11' 2024-04-14T23:55:18.652 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/194/16' 2024-04-14T23:55:18.652 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/194/14' 2024-04-14T23:55:18.652 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/194/18' 2024-04-14T23:55:18.652 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/194/13' 2024-04-14T23:55:18.652 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/194/17' 2024-04-14T23:55:18.652 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/194' 2024-04-14T23:55:18.653 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/119/19' 2024-04-14T23:55:18.653 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/119/15' 2024-04-14T23:55:18.653 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/119/20' 2024-04-14T23:55:18.653 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/119/12' 2024-04-14T23:55:18.653 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/119/10' 2024-04-14T23:55:18.653 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/119/11' 2024-04-14T23:55:18.653 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/119/16' 2024-04-14T23:55:18.653 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/119/14' 2024-04-14T23:55:18.653 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/119/18' 2024-04-14T23:55:18.653 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/119/13' 2024-04-14T23:55:18.653 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/119/17' 2024-04-14T23:55:18.654 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/119' 2024-04-14T23:55:18.654 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/157/19' 2024-04-14T23:55:18.654 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/157/15' 2024-04-14T23:55:18.654 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/157/20' 2024-04-14T23:55:18.654 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/157/12' 2024-04-14T23:55:18.654 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/157/10' 2024-04-14T23:55:18.654 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/157/11' 2024-04-14T23:55:18.654 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/157/16' 2024-04-14T23:55:18.654 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/157/14' 2024-04-14T23:55:18.654 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/157/18' 2024-04-14T23:55:18.655 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/157/13' 2024-04-14T23:55:18.655 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/157/17' 2024-04-14T23:55:18.655 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/157' 2024-04-14T23:55:18.655 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/172/19' 2024-04-14T23:55:18.655 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/172/15' 2024-04-14T23:55:18.655 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/172/20' 2024-04-14T23:55:18.655 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/172/12' 2024-04-14T23:55:18.656 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/172/10' 2024-04-14T23:55:18.656 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/172/11' 2024-04-14T23:55:18.656 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/172/16' 2024-04-14T23:55:18.656 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/172/14' 2024-04-14T23:55:18.656 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/172/18' 2024-04-14T23:55:18.657 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/172/13' 2024-04-14T23:55:18.657 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/172/17' 2024-04-14T23:55:18.657 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/172' 2024-04-14T23:55:18.657 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/186/19' 2024-04-14T23:55:18.657 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/186/15' 2024-04-14T23:55:18.657 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/186/20' 2024-04-14T23:55:18.658 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/186/12' 2024-04-14T23:55:18.658 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/186/10' 2024-04-14T23:55:18.658 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/186/11' 2024-04-14T23:55:18.658 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/186/16' 2024-04-14T23:55:18.658 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/186/14' 2024-04-14T23:55:18.658 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/186/18' 2024-04-14T23:55:18.659 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/186/13' 2024-04-14T23:55:18.659 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/186/17' 2024-04-14T23:55:18.659 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/186' 2024-04-14T23:55:18.659 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/190/19' 2024-04-14T23:55:18.659 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/190/15' 2024-04-14T23:55:18.659 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/190/20' 2024-04-14T23:55:18.659 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/190/12' 2024-04-14T23:55:18.659 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/190/10' 2024-04-14T23:55:18.659 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/190/11' 2024-04-14T23:55:18.659 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/190/16' 2024-04-14T23:55:18.660 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/190/14' 2024-04-14T23:55:19.811 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/190/18' 2024-04-14T23:55:19.811 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/190/13' 2024-04-14T23:55:19.811 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/190/17' 2024-04-14T23:55:19.811 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/190' 2024-04-14T23:55:19.811 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/104/19' 2024-04-14T23:55:19.811 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/104/15' 2024-04-14T23:55:19.811 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/104/20' 2024-04-14T23:55:19.811 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/104/12' 2024-04-14T23:55:19.812 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/104/10' 2024-04-14T23:55:19.812 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/104/11' 2024-04-14T23:55:19.812 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/104/16' 2024-04-14T23:55:19.812 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/104/14' 2024-04-14T23:55:19.812 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/104/18' 2024-04-14T23:55:19.812 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/104/13' 2024-04-14T23:55:19.812 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/104/17' 2024-04-14T23:55:19.812 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/104' 2024-04-14T23:55:19.812 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/164/19' 2024-04-14T23:55:19.812 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/164/15' 2024-04-14T23:55:19.812 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/164/20' 2024-04-14T23:55:19.812 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/164/12' 2024-04-14T23:55:19.812 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/164/10' 2024-04-14T23:55:19.812 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/164/11' 2024-04-14T23:55:19.812 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/164/16' 2024-04-14T23:55:19.812 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/164/14' 2024-04-14T23:55:19.813 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/164/18' 2024-04-14T23:55:19.813 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/164/13' 2024-04-14T23:55:19.813 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/164/17' 2024-04-14T23:55:19.813 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/164' 2024-04-14T23:55:19.813 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/112/19' 2024-04-14T23:55:19.813 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/112/15' 2024-04-14T23:55:19.813 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/112/20' 2024-04-14T23:55:19.813 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/112/12' 2024-04-14T23:55:19.813 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/112/10' 2024-04-14T23:55:19.813 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/112/11' 2024-04-14T23:55:19.813 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/112/16' 2024-04-14T23:55:19.813 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/112/14' 2024-04-14T23:55:19.813 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/112/18' 2024-04-14T23:55:19.813 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/112/13' 2024-04-14T23:55:19.813 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/112/17' 2024-04-14T23:55:19.814 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/112' 2024-04-14T23:55:19.814 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/100/19' 2024-04-14T23:55:19.814 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/100/15' 2024-04-14T23:55:19.814 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/100/20' 2024-04-14T23:55:19.814 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/100/12' 2024-04-14T23:55:19.814 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/100/10' 2024-04-14T23:55:19.814 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/100/11' 2024-04-14T23:55:19.814 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/100/16' 2024-04-14T23:55:19.814 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/100/14' 2024-04-14T23:55:19.814 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/100/18' 2024-04-14T23:55:19.814 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/100/13' 2024-04-14T23:55:19.814 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/100/17' 2024-04-14T23:55:19.814 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/100' 2024-04-14T23:55:19.815 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/134/19' 2024-04-14T23:55:19.815 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/134/15' 2024-04-14T23:55:19.815 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/134/20' 2024-04-14T23:55:19.815 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/134/12' 2024-04-14T23:55:19.815 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/134/10' 2024-04-14T23:55:19.815 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/134/11' 2024-04-14T23:55:19.815 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/134/16' 2024-04-14T23:55:19.815 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/134/14' 2024-04-14T23:55:19.815 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/134/18' 2024-04-14T23:55:19.815 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/134/13' 2024-04-14T23:55:19.815 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/134/17' 2024-04-14T23:55:19.815 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/134' 2024-04-14T23:55:19.815 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/115/19' 2024-04-14T23:55:19.815 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/115/15' 2024-04-14T23:55:19.815 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/115/20' 2024-04-14T23:55:19.815 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/115/12' 2024-04-14T23:55:19.816 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/115/10' 2024-04-14T23:55:19.816 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/115/11' 2024-04-14T23:55:19.816 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/115/16' 2024-04-14T23:55:19.816 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/115/14' 2024-04-14T23:55:19.816 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/115/18' 2024-04-14T23:55:19.816 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/115/13' 2024-04-14T23:55:19.816 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/115/17' 2024-04-14T23:55:19.816 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/115' 2024-04-14T23:55:19.816 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/116/19' 2024-04-14T23:55:19.816 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/116/15' 2024-04-14T23:55:19.816 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/116/20' 2024-04-14T23:55:19.816 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/116/12' 2024-04-14T23:55:19.816 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/116/10' 2024-04-14T23:55:19.816 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/116/11' 2024-04-14T23:55:19.816 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/116/16' 2024-04-14T23:55:19.816 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/116/14' 2024-04-14T23:55:19.816 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/116/18' 2024-04-14T23:55:19.817 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/116/13' 2024-04-14T23:55:19.817 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/116/17' 2024-04-14T23:55:19.817 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/116' 2024-04-14T23:55:19.817 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/195/19' 2024-04-14T23:55:19.817 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/195/15' 2024-04-14T23:55:19.817 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/195/20' 2024-04-14T23:55:19.817 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/195/12' 2024-04-14T23:55:19.817 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/195/10' 2024-04-14T23:55:19.817 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/195/11' 2024-04-14T23:55:19.817 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/195/16' 2024-04-14T23:55:19.817 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/195/14' 2024-04-14T23:55:19.817 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/195/18' 2024-04-14T23:55:19.817 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/195/13' 2024-04-14T23:55:19.817 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/195/17' 2024-04-14T23:55:19.817 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/195' 2024-04-14T23:55:19.817 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/138/19' 2024-04-14T23:55:19.818 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/138/15' 2024-04-14T23:55:19.818 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/138/20' 2024-04-14T23:55:19.818 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/138/12' 2024-04-14T23:55:19.818 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/138/10' 2024-04-14T23:55:19.818 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/138/11' 2024-04-14T23:55:19.818 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/138/16' 2024-04-14T23:55:19.818 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/138/14' 2024-04-14T23:55:19.818 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/138/18' 2024-04-14T23:55:22.037 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.c 2024-04-14T23:55:22.037 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-14T23:55:22.038 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-14T23:55:22.046 INFO:tasks.ceph.mds.c:Started 2024-04-14T23:55:22.046 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:55:22.211 INFO:tasks.ceph.mds.c.smithi060.stdout:starting mds.c at 2024-04-14T23:55:22.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.231+0000 7ffae8400640 1 Processor -- start 2024-04-14T23:55:22.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.232+0000 7ffae8400640 1 -- start start 2024-04-14T23:55:22.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.233+0000 7ffae8400640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ffae00c63c0 0x7ffae00c6b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:55:22.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.233+0000 7ffae8400640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ffae015c9b0 0x7ffae00c70b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:55:22.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.233+0000 7ffae8400640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffae01530d0 con 0x7ffae0152cf0 2024-04-14T23:55:22.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.233+0000 7ffae8400640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7ffae00751f0 con 0x7ffae015c9b0 2024-04-14T23:55:22.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.233+0000 7ffae8400640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7ffae00c8370 con 0x7ffae00c63c0 2024-04-14T23:55:22.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.233+0000 7ffae5000640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ffae00c63c0 0x7ffae00c6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:55:22.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.233+0000 7ffae5000640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ffae00c63c0 0x7ffae00c6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:58274/0 (socket says 172.21.15.60:58274) 2024-04-14T23:55:22.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.233+0000 7ffae5000640 1 -- 172.21.15.60:0/2929159306 learned_addr learned my addr 172.21.15.60:0/2929159306 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:55:22.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.233+0000 7ffae6400640 1 --2- 172.21.15.60:0/2929159306 >> v2:172.21.15.60:3300/0 conn(0x7ffae015c9b0 0x7ffae00c70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:55:22.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.234+0000 7ffae5000640 1 -- 172.21.15.60:0/2929159306 >> v2:172.21.15.60:3300/0 conn(0x7ffae015c9b0 msgr2=0x7ffae00c70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:55:22.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.234+0000 7ffae5000640 1 --2- 172.21.15.60:0/2929159306 >> v2:172.21.15.60:3300/0 conn(0x7ffae015c9b0 0x7ffae00c70b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:22.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.234+0000 7ffae5000640 1 -- 172.21.15.60:0/2929159306 >> v1:172.21.15.60:6789/0 conn(0x7ffae0152cf0 legacy=0x7ffae015c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:55:22.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.234+0000 7ffae5000640 1 -- 172.21.15.60:0/2929159306 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffae00c3810 con 0x7ffae00c63c0 2024-04-14T23:55:22.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.235+0000 7ffae5000640 1 --2- 172.21.15.60:0/2929159306 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffae00c63c0 0x7ffae00c6b70 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ffae0070e80 tx=0x7ffacc02ba30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20ff507eaffea536 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:55:22.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.235+0000 7ffadfe00640 1 -- 172.21.15.60:0/2929159306 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffacc02f850 con 0x7ffae00c63c0 2024-04-14T23:55:22.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.235+0000 7ffadfe00640 1 -- 172.21.15.60:0/2929159306 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffacc02f9f0 con 0x7ffae00c63c0 2024-04-14T23:55:22.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.235+0000 7ffadfe00640 1 -- 172.21.15.60:0/2929159306 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffacc02fdc0 con 0x7ffae00c63c0 2024-04-14T23:55:22.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.236+0000 7ffae8400640 1 -- 172.21.15.60:0/2929159306 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffae00c63c0 msgr2=0x7ffae00c6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:55:22.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.236+0000 7ffae8400640 1 --2- 172.21.15.60:0/2929159306 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffae00c63c0 0x7ffae00c6b70 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ffae0070e80 tx=0x7ffacc02ba30 comp rx=0 tx=0).stop 2024-04-14T23:55:22.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.236+0000 7ffae8400640 1 -- 172.21.15.60:0/2929159306 shutdown_connections 2024-04-14T23:55:22.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.236+0000 7ffae8400640 1 --2- 172.21.15.60:0/2929159306 >> v2:172.21.15.60:3300/0 conn(0x7ffae015c9b0 0x7ffae00c70b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:22.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.236+0000 7ffae8400640 1 --2- 172.21.15.60:0/2929159306 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffae00c63c0 0x7ffae00c6b70 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:22.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.236+0000 7ffae8400640 1 -- 172.21.15.60:0/2929159306 >> 172.21.15.60:0/2929159306 conn(0x7ffae00cc170 msgr2=0x7ffae00cc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:55:22.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.236+0000 7ffae8400640 1 -- 172.21.15.60:0/2929159306 shutdown_connections 2024-04-14T23:55:22.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.236+0000 7ffae8400640 1 -- 172.21.15.60:0/2929159306 wait complete. 2024-04-14T23:55:22.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.238+0000 7ffae8400640 1 Processor -- start 2024-04-14T23:55:22.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.238+0000 7ffae8400640 1 -- start start 2024-04-14T23:55:22.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.239+0000 7ffae8400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffae00c63c0 0x7ffae01512a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:55:22.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.239+0000 7ffae8400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffae0152cf0 0x7ffae00cb230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:55:22.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.239+0000 7ffae8400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffae015c9b0 0x7ffae015b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:55:22.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.239+0000 7ffae8400640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ffae00c88f0 con 0x7ffae0152cf0 2024-04-14T23:55:22.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.239+0000 7ffae8400640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ffae00c8610 con 0x7ffae015c9b0 2024-04-14T23:55:22.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.239+0000 7ffae8400640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7ffae0159450 con 0x7ffae00c63c0 2024-04-14T23:55:22.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.239+0000 7ffae5000640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffae0152cf0 0x7ffae00cb230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:55:22.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.240+0000 7ffae5000640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffae0152cf0 0x7ffae00cb230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54724/0 (socket says 172.21.15.60:54724) 2024-04-14T23:55:22.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.240+0000 7ffae5000640 1 -- 172.21.15.60:0/4040431800 learned_addr learned my addr 172.21.15.60:0/4040431800 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:55:22.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.240+0000 7ffae5a00640 1 --2- 172.21.15.60:0/4040431800 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffae00c63c0 0x7ffae01512a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:55:22.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.240+0000 7ffae6400640 1 --2- 172.21.15.60:0/4040431800 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffae015c9b0 0x7ffae015b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:55:22.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.240+0000 7ffae5000640 1 -- 172.21.15.60:0/4040431800 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffae00c63c0 msgr2=0x7ffae01512a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:55:22.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.240+0000 7ffae5000640 1 --2- 172.21.15.60:0/4040431800 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffae00c63c0 0x7ffae01512a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:22.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.240+0000 7ffae5000640 1 -- 172.21.15.60:0/4040431800 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffae015c9b0 msgr2=0x7ffae015b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:55:22.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.240+0000 7ffae5000640 1 --2- 172.21.15.60:0/4040431800 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffae015c9b0 0x7ffae015b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:22.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.240+0000 7ffae5000640 1 -- 172.21.15.60:0/4040431800 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffae01f5870 con 0x7ffae0152cf0 2024-04-14T23:55:22.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.240+0000 7ffae5a00640 1 --2- 172.21.15.60:0/4040431800 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffae00c63c0 0x7ffae01512a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:55:22.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.240+0000 7ffae5000640 1 --2- 172.21.15.60:0/4040431800 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffae0152cf0 0x7ffae00cb230 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ffacc02ba70 tx=0x7ffacc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:55:22.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.241+0000 7ffadd600640 1 -- 172.21.15.60:0/4040431800 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffacc037d10 con 0x7ffae0152cf0 2024-04-14T23:55:22.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.241+0000 7ffadd600640 1 -- 172.21.15.60:0/4040431800 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffacc046070 con 0x7ffae0152cf0 2024-04-14T23:55:22.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.241+0000 7ffadd600640 1 -- 172.21.15.60:0/4040431800 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffacc041640 con 0x7ffae0152cf0 2024-04-14T23:55:22.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.241+0000 7ffae8400640 1 -- 172.21.15.60:0/4040431800 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffae015ba60 con 0x7ffae0152cf0 2024-04-14T23:55:22.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.241+0000 7ffae8400640 1 -- 172.21.15.60:0/4040431800 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffae01f4d90 con 0x7ffae0152cf0 2024-04-14T23:55:22.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.242+0000 7ffae8400640 1 -- 172.21.15.60:0/4040431800 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffae00c5be0 con 0x7ffae0152cf0 2024-04-14T23:55:22.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.243+0000 7ffadd600640 1 -- 172.21.15.60:0/4040431800 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7ffacc040070 con 0x7ffae0152cf0 2024-04-14T23:55:22.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.244+0000 7ffadd600640 1 --2- 172.21.15.60:0/4040431800 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ffac006b9c0 0x7ffac006de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:55:22.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.244+0000 7ffadd600640 1 -- 172.21.15.60:0/4040431800 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6083+0+0 (secure 0 0 0) 0x7ffacc0acf00 con 0x7ffae0152cf0 2024-04-14T23:55:22.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.245+0000 7ffae5a00640 1 --2- 172.21.15.60:0/4040431800 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ffac006b9c0 0x7ffac006de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:55:22.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.245+0000 7ffae5a00640 1 --2- 172.21.15.60:0/4040431800 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ffac006b9c0 0x7ffac006de80 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ffae01e7260 tx=0x7ffad4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:55:22.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.249+0000 7ffadd600640 1 -- 172.21.15.60:0/4040431800 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ffae00c5be0 con 0x7ffae0152cf0 2024-04-14T23:55:22.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.493+0000 7ffae8400640 1 -- 172.21.15.60:0/4040431800 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffae0076970 con 0x7ffae0152cf0 2024-04-14T23:55:22.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.495+0000 7ffadd600640 1 -- 172.21.15.60:0/4040431800 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6014 (secure 0 0 0) 0x7ffacc033030 con 0x7ffae0152cf0 2024-04-14T23:55:22.501 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:55:22.501 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:55:13.553698+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4986},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4986":{"gid":4986,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":36,"addr":"172.21.15.60:6833/1971580590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1971580590},{"type":"v1","addr":"172.21.15.60:6833","nonce":1971580590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4986,"qdb_cluster":[4986]},"id":1}]} 2024-04-14T23:55:22.501 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 25 2024-04-14T23:55:22.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.497+0000 7ffae8400640 1 -- 172.21.15.60:0/4040431800 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ffac006b9c0 msgr2=0x7ffac006de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:55:22.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.497+0000 7ffae8400640 1 --2- 172.21.15.60:0/4040431800 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ffac006b9c0 0x7ffac006de80 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ffae01e7260 tx=0x7ffad4009040 comp rx=0 tx=0).stop 2024-04-14T23:55:22.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.497+0000 7ffae8400640 1 -- 172.21.15.60:0/4040431800 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffae0152cf0 msgr2=0x7ffae00cb230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:55:22.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.497+0000 7ffae8400640 1 --2- 172.21.15.60:0/4040431800 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffae0152cf0 0x7ffae00cb230 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ffacc02ba70 tx=0x7ffacc004730 comp rx=0 tx=0).stop 2024-04-14T23:55:22.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.498+0000 7ffae8400640 1 -- 172.21.15.60:0/4040431800 shutdown_connections 2024-04-14T23:55:22.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.498+0000 7ffae8400640 1 --2- 172.21.15.60:0/4040431800 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ffac006b9c0 0x7ffac006de80 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:22.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.498+0000 7ffae8400640 1 --2- 172.21.15.60:0/4040431800 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffae015c9b0 0x7ffae015b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:22.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.498+0000 7ffae8400640 1 --2- 172.21.15.60:0/4040431800 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffae0152cf0 0x7ffae00cb230 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:22.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.498+0000 7ffae8400640 1 --2- 172.21.15.60:0/4040431800 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffae00c63c0 0x7ffae01512a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:22.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.498+0000 7ffae8400640 1 -- 172.21.15.60:0/4040431800 >> 172.21.15.60:0/4040431800 conn(0x7ffae00cc170 msgr2=0x7ffae00cc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:55:22.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.498+0000 7ffae8400640 1 -- 172.21.15.60:0/4040431800 shutdown_connections 2024-04-14T23:55:22.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:22.498+0000 7ffae8400640 1 -- 172.21.15.60:0/4040431800 wait complete. 2024-04-14T23:55:22.521 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2024-04-14T23:55:22.691 INFO:tasks.ceph.mds.a.smithi060.stderr:2024-04-14T23:55:22.685+0000 7f6cb7000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-14T23:55:23.061 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/138/13' 2024-04-14T23:55:23.061 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/138/17' 2024-04-14T23:55:23.061 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/138' 2024-04-14T23:55:23.061 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/135/19' 2024-04-14T23:55:23.061 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/135/15' 2024-04-14T23:55:23.061 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/135/20' 2024-04-14T23:55:23.061 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/135/12' 2024-04-14T23:55:23.061 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/135/10' 2024-04-14T23:55:23.062 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/135/11' 2024-04-14T23:55:23.062 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/135/16' 2024-04-14T23:55:23.062 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/135/14' 2024-04-14T23:55:23.062 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/135/18' 2024-04-14T23:55:23.062 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/135/13' 2024-04-14T23:55:23.062 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/135/17' 2024-04-14T23:55:23.062 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/135' 2024-04-14T23:55:23.062 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/117/19' 2024-04-14T23:55:23.062 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/117/15' 2024-04-14T23:55:23.062 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/117/20' 2024-04-14T23:55:23.062 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/117/12' 2024-04-14T23:55:23.062 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/117/10' 2024-04-14T23:55:23.063 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/117/11' 2024-04-14T23:55:23.063 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/117/16' 2024-04-14T23:55:23.063 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/117/14' 2024-04-14T23:55:23.063 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/117/18' 2024-04-14T23:55:23.063 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/117/13' 2024-04-14T23:55:23.063 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/117/17' 2024-04-14T23:55:23.063 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/117' 2024-04-14T23:55:23.063 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/143/19' 2024-04-14T23:55:23.063 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/143/15' 2024-04-14T23:55:23.063 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/143/20' 2024-04-14T23:55:23.063 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/143/12' 2024-04-14T23:55:23.063 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/143/10' 2024-04-14T23:55:23.064 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/143/11' 2024-04-14T23:55:23.064 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/143/16' 2024-04-14T23:55:23.064 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/143/14' 2024-04-14T23:55:23.064 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/143/18' 2024-04-14T23:55:23.064 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/143/13' 2024-04-14T23:55:23.064 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/143/17' 2024-04-14T23:55:23.064 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/143' 2024-04-14T23:55:23.064 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/180/19' 2024-04-14T23:55:23.064 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/180/15' 2024-04-14T23:55:23.064 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/180/20' 2024-04-14T23:55:23.064 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/180/12' 2024-04-14T23:55:23.064 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/180/10' 2024-04-14T23:55:23.064 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/180/11' 2024-04-14T23:55:23.065 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/180/16' 2024-04-14T23:55:23.065 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/180/14' 2024-04-14T23:55:23.065 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/180/18' 2024-04-14T23:55:23.065 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/180/13' 2024-04-14T23:55:23.065 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/180/17' 2024-04-14T23:55:23.065 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/180' 2024-04-14T23:55:23.065 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/126/19' 2024-04-14T23:55:23.065 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/126/15' 2024-04-14T23:55:23.065 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/126/20' 2024-04-14T23:55:23.065 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/126/12' 2024-04-14T23:55:23.065 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/126/10' 2024-04-14T23:55:23.065 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/126/11' 2024-04-14T23:55:23.066 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/126/16' 2024-04-14T23:55:23.066 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/126/14' 2024-04-14T23:55:23.066 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/126/18' 2024-04-14T23:55:23.066 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/126/13' 2024-04-14T23:55:23.066 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/126/17' 2024-04-14T23:55:23.066 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/126' 2024-04-14T23:55:23.066 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/161/19' 2024-04-14T23:55:23.066 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/161/15' 2024-04-14T23:55:23.066 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/161/20' 2024-04-14T23:55:23.066 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/161/12' 2024-04-14T23:55:23.066 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/161/10' 2024-04-14T23:55:23.066 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/161/11' 2024-04-14T23:55:23.067 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/161/16' 2024-04-14T23:55:23.067 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/161/14' 2024-04-14T23:55:23.067 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/161/18' 2024-04-14T23:55:23.067 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/161/13' 2024-04-14T23:55:23.067 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/161/17' 2024-04-14T23:55:23.067 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/161' 2024-04-14T23:55:23.067 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/176/19' 2024-04-14T23:55:23.067 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/176/15' 2024-04-14T23:55:23.067 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/176/20' 2024-04-14T23:55:23.067 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/176/12' 2024-04-14T23:55:23.067 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/176/10' 2024-04-14T23:55:23.068 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/176/11' 2024-04-14T23:55:23.068 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/176/16' 2024-04-14T23:55:23.068 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/176/14' 2024-04-14T23:55:23.068 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/176/18' 2024-04-14T23:55:23.068 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/176/13' 2024-04-14T23:55:23.068 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/176/17' 2024-04-14T23:55:23.068 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/176' 2024-04-14T23:55:23.068 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/111/19' 2024-04-14T23:55:23.068 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/111/15' 2024-04-14T23:55:23.068 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/111/20' 2024-04-14T23:55:23.068 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/111/12' 2024-04-14T23:55:23.068 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/111/10' 2024-04-14T23:55:23.068 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/111/11' 2024-04-14T23:55:23.069 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/111/16' 2024-04-14T23:55:23.069 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/111/14' 2024-04-14T23:55:23.069 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/111/18' 2024-04-14T23:55:23.069 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/111/13' 2024-04-14T23:55:23.069 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/111/17' 2024-04-14T23:55:23.069 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/111' 2024-04-14T23:55:23.069 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/191/19' 2024-04-14T23:55:23.069 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/191/15' 2024-04-14T23:55:23.069 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/191/20' 2024-04-14T23:55:23.069 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/191/12' 2024-04-14T23:55:23.069 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/191/10' 2024-04-14T23:55:23.069 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/191/11' 2024-04-14T23:55:23.070 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/191/16' 2024-04-14T23:55:23.070 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/191/14' 2024-04-14T23:55:23.070 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/191/18' 2024-04-14T23:55:23.070 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/191/13' 2024-04-14T23:55:24.522 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:55:24.562 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/191/17' 2024-04-14T23:55:24.562 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/191' 2024-04-14T23:55:24.562 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/184/19' 2024-04-14T23:55:24.562 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/184/15' 2024-04-14T23:55:24.562 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/184/20' 2024-04-14T23:55:24.562 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/184/12' 2024-04-14T23:55:24.562 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/184/10' 2024-04-14T23:55:24.562 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/184/11' 2024-04-14T23:55:24.562 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/184/16' 2024-04-14T23:55:24.562 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/184/14' 2024-04-14T23:55:24.563 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/184/18' 2024-04-14T23:55:24.563 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/184/13' 2024-04-14T23:55:24.563 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/184/17' 2024-04-14T23:55:24.563 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/184' 2024-04-14T23:55:24.563 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/192/19' 2024-04-14T23:55:24.563 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/192/15' 2024-04-14T23:55:24.563 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/192/20' 2024-04-14T23:55:24.563 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/192/12' 2024-04-14T23:55:24.563 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/192/10' 2024-04-14T23:55:24.563 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/192/11' 2024-04-14T23:55:24.563 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/192/16' 2024-04-14T23:55:24.563 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/192/14' 2024-04-14T23:55:24.563 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/192/18' 2024-04-14T23:55:24.563 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/192/13' 2024-04-14T23:55:24.563 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/192/17' 2024-04-14T23:55:24.564 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/192' 2024-04-14T23:55:24.564 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/103/19' 2024-04-14T23:55:24.564 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/103/15' 2024-04-14T23:55:24.564 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/103/20' 2024-04-14T23:55:24.564 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/103/12' 2024-04-14T23:55:24.564 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/103/10' 2024-04-14T23:55:24.564 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/103/11' 2024-04-14T23:55:24.564 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/103/16' 2024-04-14T23:55:24.564 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/103/14' 2024-04-14T23:55:24.564 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/103/18' 2024-04-14T23:55:24.564 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/103/13' 2024-04-14T23:55:24.564 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/103/17' 2024-04-14T23:55:24.564 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/103' 2024-04-14T23:55:24.564 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/128/19' 2024-04-14T23:55:24.565 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/128/15' 2024-04-14T23:55:24.565 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/128/20' 2024-04-14T23:55:24.565 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/128/12' 2024-04-14T23:55:24.565 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/128/10' 2024-04-14T23:55:24.565 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/128/11' 2024-04-14T23:55:24.565 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/128/16' 2024-04-14T23:55:24.565 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/128/14' 2024-04-14T23:55:24.565 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/128/18' 2024-04-14T23:55:24.565 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/128/13' 2024-04-14T23:55:24.565 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/128/17' 2024-04-14T23:55:24.565 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/128' 2024-04-14T23:55:24.565 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/144/19' 2024-04-14T23:55:24.565 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/144/15' 2024-04-14T23:55:24.565 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/144/20' 2024-04-14T23:55:24.565 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/144/12' 2024-04-14T23:55:24.565 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/144/10' 2024-04-14T23:55:24.566 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/144/11' 2024-04-14T23:55:24.566 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/144/16' 2024-04-14T23:55:24.566 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/144/14' 2024-04-14T23:55:24.566 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/144/18' 2024-04-14T23:55:24.566 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/144/13' 2024-04-14T23:55:24.566 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/144/17' 2024-04-14T23:55:24.566 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/144' 2024-04-14T23:55:24.566 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/146/19' 2024-04-14T23:55:24.566 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/146/15' 2024-04-14T23:55:24.566 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/146/20' 2024-04-14T23:55:24.566 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/146/12' 2024-04-14T23:55:24.566 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/146/10' 2024-04-14T23:55:24.566 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/146/11' 2024-04-14T23:55:24.566 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/146/16' 2024-04-14T23:55:24.566 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/146/14' 2024-04-14T23:55:24.567 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/146/18' 2024-04-14T23:55:24.567 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/146/13' 2024-04-14T23:55:24.567 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/146/17' 2024-04-14T23:55:24.567 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/146' 2024-04-14T23:55:24.567 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/141/19' 2024-04-14T23:55:24.567 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/141/15' 2024-04-14T23:55:24.567 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/141/20' 2024-04-14T23:55:24.567 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/141/12' 2024-04-14T23:55:24.567 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/141/10' 2024-04-14T23:55:24.568 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/141/11' 2024-04-14T23:55:24.568 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/141/16' 2024-04-14T23:55:24.568 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/141/14' 2024-04-14T23:55:24.568 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/141/18' 2024-04-14T23:55:24.568 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/141/13' 2024-04-14T23:55:24.568 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/141/17' 2024-04-14T23:55:24.569 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/141' 2024-04-14T23:55:24.569 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/182/19' 2024-04-14T23:55:24.569 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/182/15' 2024-04-14T23:55:24.569 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/182/20' 2024-04-14T23:55:24.569 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/182/12' 2024-04-14T23:55:24.569 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/182/10' 2024-04-14T23:55:24.569 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/182/11' 2024-04-14T23:55:24.569 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/182/16' 2024-04-14T23:55:24.569 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/182/14' 2024-04-14T23:55:24.569 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/182/18' 2024-04-14T23:55:24.569 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/182/13' 2024-04-14T23:55:24.569 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/182/17' 2024-04-14T23:55:24.569 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/182' 2024-04-14T23:55:24.569 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/107/19' 2024-04-14T23:55:24.569 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/107/15' 2024-04-14T23:55:24.570 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/107/20' 2024-04-14T23:55:24.570 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/107/12' 2024-04-14T23:55:24.570 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/107/10' 2024-04-14T23:55:24.570 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/107/11' 2024-04-14T23:55:24.570 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/107/16' 2024-04-14T23:55:24.570 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/107/14' 2024-04-14T23:55:24.570 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/107/18' 2024-04-14T23:55:24.570 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/107/13' 2024-04-14T23:55:24.570 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/107/17' 2024-04-14T23:55:24.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:24.730+0000 7f0eef200640 1 Processor -- start 2024-04-14T23:55:24.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:24.731+0000 7f0eef200640 1 -- start start 2024-04-14T23:55:24.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:24.732+0000 7f0eef200640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f0ef00cbd30 0x7f0ef00cc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:55:24.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:24.732+0000 7f0eef200640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ef00c5520 con 0x7f0ef0154cf0 2024-04-14T23:55:24.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:24.732+0000 7f0eef200640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ef015b040 con 0x7f0ef015a940 2024-04-14T23:55:24.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:24.732+0000 7f0eef200640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f0ef01550d0 con 0x7f0ef00cbd30 2024-04-14T23:55:24.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:24.733+0000 7f0eede00640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f0ef0154cf0 0x7f0ef006c310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:46854/0 (socket says 172.21.15.60:46854) 2024-04-14T23:55:24.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:24.733+0000 7f0eede00640 1 -- 172.21.15.60:0/3733059800 learned_addr learned my addr 172.21.15.60:0/3733059800 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:55:24.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:24.733+0000 7f0eee800640 1 --2- 172.21.15.60:0/3733059800 >> v2:172.21.15.186:3300/0 conn(0x7f0ef00cbd30 0x7f0ef00cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:55:24.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:24.733+0000 7f0eeca00640 1 -- 172.21.15.60:0/3733059800 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1973056569 0 0) 0x7f0ef00c5520 con 0x7f0ef0154cf0 2024-04-14T23:55:24.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:24.733+0000 7f0eeca00640 1 -- 172.21.15.60:0/3733059800 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ed4003520 con 0x7f0ef0154cf0 2024-04-14T23:55:24.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:24.733+0000 7f0eee800640 1 -- 172.21.15.60:0/3733059800 >> v1:172.21.15.186:6789/0 conn(0x7f0ef015a940 legacy=0x7f0ef00cb120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:55:24.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:24.733+0000 7f0eee800640 1 -- 172.21.15.60:0/3733059800 >> v1:172.21.15.60:6789/0 conn(0x7f0ef0154cf0 legacy=0x7f0ef006c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:55:24.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:24.733+0000 7f0eee800640 1 -- 172.21.15.60:0/3733059800 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ef015b300 con 0x7f0ef00cbd30 2024-04-14T23:55:24.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:24.734+0000 7f0eee800640 1 --2- 172.21.15.60:0/3733059800 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0ef00cbd30 0x7f0ef00cc160 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8009d90 tx=0x7f0ee802bfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ce58f84f2ce4364 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:55:24.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:24.735+0000 7f0eeca00640 1 -- 172.21.15.60:0/3733059800 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee8039070 con 0x7f0ef00cbd30 2024-04-14T23:55:24.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:24.735+0000 7f0eeca00640 1 -- 172.21.15.60:0/3733059800 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ee802cd80 con 0x7f0ef00cbd30 2024-04-14T23:55:24.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:24.735+0000 7f0eeca00640 1 -- 172.21.15.60:0/3733059800 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee8034640 con 0x7f0ef00cbd30 2024-04-14T23:55:24.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:24.735+0000 7f0eef200640 1 -- 172.21.15.60:0/3733059800 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0ef00cbd30 msgr2=0x7f0ef00cc160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:55:24.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:24.735+0000 7f0eef200640 1 --2- 172.21.15.60:0/3733059800 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0ef00cbd30 0x7f0ef00cc160 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8009d90 tx=0x7f0ee802bfd0 comp rx=0 tx=0).stop 2024-04-14T23:55:24.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:24.735+0000 7f0eef200640 1 -- 172.21.15.60:0/3733059800 shutdown_connections 2024-04-14T23:55:24.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:24.735+0000 7f0eef200640 1 --2- 172.21.15.60:0/3733059800 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0ef00cbd30 0x7f0ef00cc160 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:24.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:24.735+0000 7f0eef200640 1 -- 172.21.15.60:0/3733059800 >> 172.21.15.60:0/3733059800 conn(0x7f0ef00670f0 msgr2=0x7f0ef0067500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:55:24.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:24.736+0000 7f0eef200640 1 -- 172.21.15.60:0/3733059800 shutdown_connections 2024-04-14T23:55:24.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:24.736+0000 7f0eef200640 1 -- 172.21.15.60:0/3733059800 wait complete. 2024-04-14T23:55:24.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:24.738+0000 7f0eef200640 1 Processor -- start 2024-04-14T23:55:24.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:24.738+0000 7f0eef200640 1 -- start start 2024-04-14T23:55:24.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:24.739+0000 7f0eef200640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ef00cbd30 0x7f0ef00cf200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:55:24.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:24.739+0000 7f0eef200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0ef0154cf0 0x7f0ef01491a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:55:24.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:24.739+0000 7f0eef200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0ef015a940 0x7f0ef01e6840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:55:24.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:24.739+0000 7f0eef200640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef006bb10 con 0x7f0ef00cbd30 2024-04-14T23:55:24.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:24.739+0000 7f0eef200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef015b8c0 con 0x7f0ef0154cf0 2024-04-14T23:55:24.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:24.739+0000 7f0eef200640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f0ef006bdf0 con 0x7f0ef015a940 2024-04-14T23:55:24.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:24.739+0000 7f0eede00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ef00cbd30 0x7f0ef00cf200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:55:24.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:24.739+0000 7f0eede00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ef00cbd30 0x7f0ef00cf200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54734/0 (socket says 172.21.15.60:54734) 2024-04-14T23:55:24.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:24.739+0000 7f0eede00640 1 -- 172.21.15.60:0/1996955820 learned_addr learned my addr 172.21.15.60:0/1996955820 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:55:24.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:24.739+0000 7f0eed400640 1 --2- 172.21.15.60:0/1996955820 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0ef0154cf0 0x7f0ef01491a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:55:24.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:24.739+0000 7f0eee800640 1 --2- 172.21.15.60:0/1996955820 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0ef015a940 0x7f0ef01e6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:55:24.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:24.740+0000 7f0eed400640 1 -- 172.21.15.60:0/1996955820 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0ef015a940 msgr2=0x7f0ef01e6840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:55:24.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:24.740+0000 7f0eed400640 1 --2- 172.21.15.60:0/1996955820 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0ef015a940 0x7f0ef01e6840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:24.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:24.740+0000 7f0eed400640 1 -- 172.21.15.60:0/1996955820 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ef00cbd30 msgr2=0x7f0ef00cf200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:55:24.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:24.740+0000 7f0eed400640 1 --2- 172.21.15.60:0/1996955820 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ef00cbd30 0x7f0ef00cf200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:24.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:24.740+0000 7f0eed400640 1 -- 172.21.15.60:0/1996955820 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ef00ce340 con 0x7f0ef0154cf0 2024-04-14T23:55:24.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:24.740+0000 7f0eede00640 1 --2- 172.21.15.60:0/1996955820 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ef00cbd30 0x7f0ef00cf200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:55:24.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:24.740+0000 7f0eed400640 1 --2- 172.21.15.60:0/1996955820 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0ef0154cf0 0x7f0ef01491a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f0edc00c420 tx=0x7f0edc00c8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:55:24.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:24.740+0000 7f0ee6000640 1 -- 172.21.15.60:0/1996955820 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0edc0052d0 con 0x7f0ef0154cf0 2024-04-14T23:55:24.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:24.740+0000 7f0ee6000640 1 -- 172.21.15.60:0/1996955820 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0edc005490 con 0x7f0ef0154cf0 2024-04-14T23:55:24.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:24.740+0000 7f0ee6000640 1 -- 172.21.15.60:0/1996955820 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0edc007260 con 0x7f0ef0154cf0 2024-04-14T23:55:24.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:24.741+0000 7f0eef200640 1 -- 172.21.15.60:0/1996955820 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ef00ce600 con 0x7f0ef0154cf0 2024-04-14T23:55:24.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:24.741+0000 7f0eef200640 1 -- 172.21.15.60:0/1996955820 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ef0147c30 con 0x7f0ef0154cf0 2024-04-14T23:55:24.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:24.742+0000 7f0eef200640 1 -- 172.21.15.60:0/1996955820 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ef004f540 con 0x7f0ef0154cf0 2024-04-14T23:55:24.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:24.743+0000 7f0ee6000640 1 -- 172.21.15.60:0/1996955820 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f0edc007400 con 0x7f0ef0154cf0 2024-04-14T23:55:24.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:24.744+0000 7f0ee6000640 1 --2- 172.21.15.60:0/1996955820 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f0ed006b990 0x7f0ed006de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:55:24.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:24.744+0000 7f0ee6000640 1 -- 172.21.15.60:0/1996955820 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6083+0+0 (secure 0 0 0) 0x7f0edc089cd0 con 0x7f0ef0154cf0 2024-04-14T23:55:24.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:24.745+0000 7f0eede00640 1 --2- 172.21.15.60:0/1996955820 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f0ed006b990 0x7f0ed006de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:55:24.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:24.745+0000 7f0eede00640 1 --2- 172.21.15.60:0/1996955820 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f0ed006b990 0x7f0ed006de50 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0ee00011e0 tx=0x7f0ee0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:55:24.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:24.748+0000 7f0ee6000640 1 -- 172.21.15.60:0/1996955820 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0edc058070 con 0x7f0ef0154cf0 2024-04-14T23:55:25.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:24.998+0000 7f0eef200640 1 -- 172.21.15.60:0/1996955820 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0ef0154510 con 0x7f0ef0154cf0 2024-04-14T23:55:25.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:24.999+0000 7f0ee6000640 1 -- 172.21.15.60:0/1996955820 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6839 (secure 0 0 0) 0x7f0edc05c110 con 0x7f0ef0154cf0 2024-04-14T23:55:25.005 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:55:25.005 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5191,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/2356847578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2356847578},{"type":"v1","addr":"172.21.15.60:6835","nonce":2356847578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:55:13.553698+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4986},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4986":{"gid":4986,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":36,"addr":"172.21.15.60:6833/1971580590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1971580590},{"type":"v1","addr":"172.21.15.60:6833","nonce":1971580590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4986,"qdb_cluster":[4986]},"id":1}]} 2024-04-14T23:55:25.005 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 26 2024-04-14T23:55:25.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:25.001+0000 7f0eef200640 1 -- 172.21.15.60:0/1996955820 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f0ed006b990 msgr2=0x7f0ed006de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:55:25.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:25.001+0000 7f0eef200640 1 --2- 172.21.15.60:0/1996955820 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f0ed006b990 0x7f0ed006de50 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0ee00011e0 tx=0x7f0ee0009040 comp rx=0 tx=0).stop 2024-04-14T23:55:25.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:25.001+0000 7f0eef200640 1 -- 172.21.15.60:0/1996955820 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0ef0154cf0 msgr2=0x7f0ef01491a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:55:25.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:25.001+0000 7f0eef200640 1 --2- 172.21.15.60:0/1996955820 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0ef0154cf0 0x7f0ef01491a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f0edc00c420 tx=0x7f0edc00c8e0 comp rx=0 tx=0).stop 2024-04-14T23:55:25.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:25.002+0000 7f0eef200640 1 -- 172.21.15.60:0/1996955820 shutdown_connections 2024-04-14T23:55:25.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:25.002+0000 7f0eef200640 1 --2- 172.21.15.60:0/1996955820 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f0ed006b990 0x7f0ed006de50 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:25.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:25.002+0000 7f0eef200640 1 --2- 172.21.15.60:0/1996955820 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0ef015a940 0x7f0ef01e6840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:25.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:25.002+0000 7f0eef200640 1 --2- 172.21.15.60:0/1996955820 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0ef0154cf0 0x7f0ef01491a0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:25.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:25.002+0000 7f0eef200640 1 --2- 172.21.15.60:0/1996955820 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ef00cbd30 0x7f0ef00cf200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:55:25.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:25.002+0000 7f0eef200640 1 -- 172.21.15.60:0/1996955820 >> 172.21.15.60:0/1996955820 conn(0x7f0ef00670f0 msgr2=0x7f0ef0159c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:55:25.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:25.002+0000 7f0eef200640 1 -- 172.21.15.60:0/1996955820 shutdown_connections 2024-04-14T23:55:25.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:55:25.002+0000 7f0eef200640 1 -- 172.21.15.60:0/1996955820 wait complete. 2024-04-14T23:55:25.027 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported in up:standby state 2024-04-14T23:55:25.027 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 48 secs before thrashing 2024-04-14T23:55:25.503 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/107' 2024-04-14T23:55:25.503 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/178/19' 2024-04-14T23:55:25.503 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/178/15' 2024-04-14T23:55:25.503 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/178/20' 2024-04-14T23:55:25.504 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/178/12' 2024-04-14T23:55:25.504 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/178/10' 2024-04-14T23:55:25.504 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/178/11' 2024-04-14T23:55:25.504 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/178/16' 2024-04-14T23:55:25.504 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/178/14' 2024-04-14T23:55:25.504 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/178/18' 2024-04-14T23:55:25.504 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/178/13' 2024-04-14T23:55:25.504 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/178/17' 2024-04-14T23:55:25.504 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/178' 2024-04-14T23:55:25.504 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/156/19' 2024-04-14T23:55:25.504 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/156/15' 2024-04-14T23:55:25.504 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/156/20' 2024-04-14T23:55:25.504 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/156/12' 2024-04-14T23:55:25.504 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/156/10' 2024-04-14T23:55:25.505 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/156/11' 2024-04-14T23:55:25.505 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/156/16' 2024-04-14T23:55:25.505 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/156/14' 2024-04-14T23:55:25.505 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/156/18' 2024-04-14T23:55:25.505 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/156/13' 2024-04-14T23:55:25.505 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/156/17' 2024-04-14T23:55:25.505 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/156' 2024-04-14T23:55:25.505 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/131/19' 2024-04-14T23:55:25.505 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/131/15' 2024-04-14T23:55:25.505 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/131/20' 2024-04-14T23:55:25.505 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/131/12' 2024-04-14T23:55:25.505 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/131/10' 2024-04-14T23:55:25.505 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/131/11' 2024-04-14T23:55:25.505 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/131/16' 2024-04-14T23:55:25.505 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/131/14' 2024-04-14T23:55:25.506 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/131/18' 2024-04-14T23:55:25.506 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/131/13' 2024-04-14T23:55:25.506 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/131/17' 2024-04-14T23:55:25.506 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/131' 2024-04-14T23:55:25.506 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/123/19' 2024-04-14T23:55:25.506 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/123/15' 2024-04-14T23:55:25.506 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/123/20' 2024-04-14T23:55:25.506 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/123/12' 2024-04-14T23:55:25.506 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/123/10' 2024-04-14T23:55:25.506 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/123/11' 2024-04-14T23:55:25.506 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/123/16' 2024-04-14T23:55:25.506 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/123/14' 2024-04-14T23:55:25.506 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/123/18' 2024-04-14T23:55:25.506 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/123/13' 2024-04-14T23:55:25.506 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/123/17' 2024-04-14T23:55:25.507 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/123' 2024-04-14T23:55:25.507 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/160/19' 2024-04-14T23:55:25.507 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/160/15' 2024-04-14T23:55:25.507 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/160/20' 2024-04-14T23:55:25.507 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/160/12' 2024-04-14T23:55:25.507 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/160/10' 2024-04-14T23:55:25.507 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/160/11' 2024-04-14T23:55:25.507 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/160/16' 2024-04-14T23:55:25.507 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/160/14' 2024-04-14T23:55:25.507 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/160/18' 2024-04-14T23:55:25.507 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/160/13' 2024-04-14T23:55:25.507 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/160/17' 2024-04-14T23:55:25.507 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/160' 2024-04-14T23:55:25.507 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/165/19' 2024-04-14T23:55:25.508 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/165/15' 2024-04-14T23:55:25.508 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/165/20' 2024-04-14T23:55:25.508 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/165/12' 2024-04-14T23:55:25.508 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/165/10' 2024-04-14T23:55:25.508 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/165/11' 2024-04-14T23:55:25.508 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/165/16' 2024-04-14T23:55:25.508 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/165/14' 2024-04-14T23:55:25.508 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/165/18' 2024-04-14T23:55:25.508 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/165/13' 2024-04-14T23:55:25.508 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/165/17' 2024-04-14T23:55:25.508 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/165' 2024-04-14T23:55:25.508 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/101/19' 2024-04-14T23:55:25.508 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/101/15' 2024-04-14T23:55:25.508 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/101/20' 2024-04-14T23:55:25.508 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/101/12' 2024-04-14T23:55:25.509 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/101/10' 2024-04-14T23:55:25.509 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/101/11' 2024-04-14T23:55:25.509 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/101/16' 2024-04-14T23:55:25.509 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/101/14' 2024-04-14T23:55:25.509 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/101/18' 2024-04-14T23:55:25.509 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/101/13' 2024-04-14T23:55:25.509 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/101/17' 2024-04-14T23:55:25.509 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/101' 2024-04-14T23:55:25.509 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/148/19' 2024-04-14T23:55:25.509 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/148/15' 2024-04-14T23:55:25.509 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/148/20' 2024-04-14T23:55:25.509 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/148/12' 2024-04-14T23:55:25.509 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/148/10' 2024-04-14T23:55:25.509 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/148/11' 2024-04-14T23:55:25.509 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/148/16' 2024-04-14T23:55:25.510 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/148/14' 2024-04-14T23:55:25.510 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/148/18' 2024-04-14T23:55:25.510 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/148/13' 2024-04-14T23:55:25.510 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/148/17' 2024-04-14T23:55:25.510 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/148' 2024-04-14T23:55:25.510 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/168/19' 2024-04-14T23:55:25.510 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/168/15' 2024-04-14T23:55:25.510 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/168/20' 2024-04-14T23:55:25.510 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/168/12' 2024-04-14T23:55:25.510 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/168/10' 2024-04-14T23:55:25.510 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/168/11' 2024-04-14T23:55:25.510 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/168/16' 2024-04-14T23:55:25.510 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/168/14' 2024-04-14T23:55:25.511 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/168/18' 2024-04-14T23:55:25.511 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/168/13' 2024-04-14T23:55:25.511 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/168/17' 2024-04-14T23:55:25.511 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/168' 2024-04-14T23:55:27.493 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/179/19' 2024-04-14T23:55:27.494 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/179/15' 2024-04-14T23:55:27.494 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/179/20' 2024-04-14T23:55:27.494 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/179/12' 2024-04-14T23:55:27.494 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/179/10' 2024-04-14T23:55:27.494 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/179/11' 2024-04-14T23:55:27.494 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/179/16' 2024-04-14T23:55:27.494 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/179/14' 2024-04-14T23:55:27.494 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/179/18' 2024-04-14T23:55:27.494 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/179/13' 2024-04-14T23:55:27.494 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/179/17' 2024-04-14T23:55:27.494 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/179' 2024-04-14T23:55:27.494 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/132/19' 2024-04-14T23:55:27.495 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/132/15' 2024-04-14T23:55:27.495 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/132/20' 2024-04-14T23:55:27.495 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/132/12' 2024-04-14T23:55:27.495 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/132/10' 2024-04-14T23:55:27.495 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/132/11' 2024-04-14T23:55:27.495 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/132/16' 2024-04-14T23:55:27.495 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/132/14' 2024-04-14T23:55:27.495 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/132/18' 2024-04-14T23:55:27.495 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/132/13' 2024-04-14T23:55:27.495 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/132/17' 2024-04-14T23:55:27.495 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/132' 2024-04-14T23:55:27.495 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/199/19' 2024-04-14T23:55:27.495 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/199/15' 2024-04-14T23:55:27.495 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/199/20' 2024-04-14T23:55:27.496 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/199/12' 2024-04-14T23:55:27.496 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/199/10' 2024-04-14T23:55:27.496 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/199/11' 2024-04-14T23:55:27.496 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/199/16' 2024-04-14T23:55:27.496 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/199/14' 2024-04-14T23:55:27.496 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/199/18' 2024-04-14T23:55:27.496 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/199/13' 2024-04-14T23:55:27.496 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/199/17' 2024-04-14T23:55:27.496 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/199' 2024-04-14T23:55:27.496 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/163/19' 2024-04-14T23:55:27.496 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/163/15' 2024-04-14T23:55:27.496 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/163/20' 2024-04-14T23:55:27.496 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/163/12' 2024-04-14T23:55:27.496 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/163/10' 2024-04-14T23:55:27.497 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/163/11' 2024-04-14T23:55:27.497 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/163/16' 2024-04-14T23:55:27.497 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/163/14' 2024-04-14T23:55:27.497 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/163/18' 2024-04-14T23:55:27.497 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/163/13' 2024-04-14T23:55:27.497 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/163/17' 2024-04-14T23:55:27.497 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/163' 2024-04-14T23:55:27.497 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/153/19' 2024-04-14T23:55:27.497 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/153/15' 2024-04-14T23:55:27.497 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/153/20' 2024-04-14T23:55:27.497 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/153/12' 2024-04-14T23:55:27.497 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/153/10' 2024-04-14T23:55:27.497 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/153/11' 2024-04-14T23:55:27.497 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/153/16' 2024-04-14T23:55:27.497 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/153/14' 2024-04-14T23:55:27.498 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/153/18' 2024-04-14T23:55:27.498 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/153/13' 2024-04-14T23:55:27.498 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/153/17' 2024-04-14T23:55:27.498 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/153' 2024-04-14T23:55:27.498 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/122/19' 2024-04-14T23:55:27.498 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/122/15' 2024-04-14T23:55:27.498 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/122/20' 2024-04-14T23:55:27.498 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/122/12' 2024-04-14T23:55:27.498 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/122/10' 2024-04-14T23:55:27.498 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/122/11' 2024-04-14T23:55:27.498 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/122/16' 2024-04-14T23:55:27.498 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/122/14' 2024-04-14T23:55:27.499 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/122/18' 2024-04-14T23:55:27.499 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/122/13' 2024-04-14T23:55:27.499 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/122/17' 2024-04-14T23:55:27.499 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/122' 2024-04-14T23:55:27.499 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/162/19' 2024-04-14T23:55:27.499 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/162/15' 2024-04-14T23:55:27.499 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/162/20' 2024-04-14T23:55:27.499 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/162/12' 2024-04-14T23:55:27.499 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/162/10' 2024-04-14T23:55:27.500 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/162/11' 2024-04-14T23:55:27.500 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/162/16' 2024-04-14T23:55:27.500 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/162/14' 2024-04-14T23:55:27.500 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/162/18' 2024-04-14T23:55:27.500 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/162/13' 2024-04-14T23:55:27.500 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/162/17' 2024-04-14T23:55:27.500 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/162' 2024-04-14T23:55:27.500 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/197/19' 2024-04-14T23:55:27.501 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/197/15' 2024-04-14T23:55:27.501 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/197/20' 2024-04-14T23:55:27.501 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/197/12' 2024-04-14T23:55:27.501 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/197/10' 2024-04-14T23:55:27.501 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/197/11' 2024-04-14T23:55:27.501 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/197/16' 2024-04-14T23:55:27.501 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/197/14' 2024-04-14T23:55:27.501 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/197/18' 2024-04-14T23:55:27.501 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/197/13' 2024-04-14T23:55:27.501 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/197/17' 2024-04-14T23:55:27.502 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/197' 2024-04-14T23:55:27.502 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/113/19' 2024-04-14T23:55:27.502 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/113/15' 2024-04-14T23:55:27.502 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/113/20' 2024-04-14T23:55:27.502 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/113/12' 2024-04-14T23:55:27.502 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/113/10' 2024-04-14T23:55:27.502 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/113/11' 2024-04-14T23:55:27.502 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/113/16' 2024-04-14T23:55:27.502 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/113/14' 2024-04-14T23:55:27.502 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/113/18' 2024-04-14T23:55:27.502 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/113/13' 2024-04-14T23:55:27.502 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/113/17' 2024-04-14T23:55:27.503 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/113' 2024-04-14T23:55:27.503 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/152/19' 2024-04-14T23:55:28.764 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/152/15' 2024-04-14T23:55:28.764 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/152/20' 2024-04-14T23:55:28.764 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/152/12' 2024-04-14T23:55:28.764 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/152/10' 2024-04-14T23:55:28.764 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/152/11' 2024-04-14T23:55:28.764 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/152/16' 2024-04-14T23:55:28.764 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/152/14' 2024-04-14T23:55:28.764 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/152/18' 2024-04-14T23:55:28.764 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/152/13' 2024-04-14T23:55:28.764 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/152/17' 2024-04-14T23:55:28.764 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/152' 2024-04-14T23:55:28.765 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/121/19' 2024-04-14T23:55:28.765 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/121/15' 2024-04-14T23:55:28.765 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/121/20' 2024-04-14T23:55:28.765 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/121/12' 2024-04-14T23:55:28.765 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/121/10' 2024-04-14T23:55:28.765 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/121/11' 2024-04-14T23:55:28.765 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/121/16' 2024-04-14T23:55:28.765 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/121/14' 2024-04-14T23:55:28.765 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/121/18' 2024-04-14T23:55:28.765 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/121/13' 2024-04-14T23:55:28.765 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/121/17' 2024-04-14T23:55:28.765 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/121' 2024-04-14T23:55:28.765 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/118/19' 2024-04-14T23:55:28.765 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/118/15' 2024-04-14T23:55:28.765 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/118/20' 2024-04-14T23:55:28.766 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/118/12' 2024-04-14T23:55:28.766 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/118/10' 2024-04-14T23:55:28.766 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/118/11' 2024-04-14T23:55:28.766 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/118/16' 2024-04-14T23:55:28.766 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/118/14' 2024-04-14T23:55:28.766 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/118/18' 2024-04-14T23:55:28.766 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/118/13' 2024-04-14T23:55:28.766 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/118/17' 2024-04-14T23:55:28.766 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/118' 2024-04-14T23:55:28.766 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/196/19' 2024-04-14T23:55:28.766 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/196/15' 2024-04-14T23:55:28.766 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/196/20' 2024-04-14T23:55:28.766 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/196/12' 2024-04-14T23:55:28.766 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/196/10' 2024-04-14T23:55:28.767 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/196/11' 2024-04-14T23:55:28.767 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/196/16' 2024-04-14T23:55:28.767 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/196/14' 2024-04-14T23:55:28.767 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/196/18' 2024-04-14T23:55:28.767 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/196/13' 2024-04-14T23:55:28.767 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/196/17' 2024-04-14T23:55:28.767 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/196' 2024-04-14T23:55:28.767 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/124/19' 2024-04-14T23:55:28.767 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/124/15' 2024-04-14T23:55:28.767 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/124/20' 2024-04-14T23:55:28.767 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/124/12' 2024-04-14T23:55:28.767 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/124/10' 2024-04-14T23:55:28.767 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/124/11' 2024-04-14T23:55:28.767 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/124/16' 2024-04-14T23:55:28.767 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/124/14' 2024-04-14T23:55:28.768 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/124/18' 2024-04-14T23:55:28.768 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/124/13' 2024-04-14T23:55:28.768 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/124/17' 2024-04-14T23:55:28.768 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/124' 2024-04-14T23:55:28.768 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/188/19' 2024-04-14T23:55:28.768 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/188/15' 2024-04-14T23:55:28.768 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/188/20' 2024-04-14T23:55:28.768 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/188/12' 2024-04-14T23:55:28.768 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/188/10' 2024-04-14T23:55:28.768 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/188/11' 2024-04-14T23:55:28.768 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/188/16' 2024-04-14T23:55:28.768 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/188/14' 2024-04-14T23:55:28.768 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/188/18' 2024-04-14T23:55:28.768 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/188/13' 2024-04-14T23:55:28.768 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/188/17' 2024-04-14T23:55:28.769 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/188' 2024-04-14T23:55:28.769 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/145/19' 2024-04-14T23:55:28.769 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/145/15' 2024-04-14T23:55:28.769 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/145/20' 2024-04-14T23:55:28.769 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/145/12' 2024-04-14T23:55:28.769 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/145/10' 2024-04-14T23:55:28.769 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/145/11' 2024-04-14T23:55:28.769 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/145/16' 2024-04-14T23:55:28.769 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/145/14' 2024-04-14T23:55:28.769 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/145/18' 2024-04-14T23:55:28.769 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/145/13' 2024-04-14T23:55:28.769 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/145/17' 2024-04-14T23:55:28.769 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/145' 2024-04-14T23:55:28.769 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/130/19' 2024-04-14T23:55:28.769 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/130/15' 2024-04-14T23:55:28.770 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/130/20' 2024-04-14T23:55:28.770 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/130/12' 2024-04-14T23:55:28.770 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/130/10' 2024-04-14T23:55:28.770 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/130/11' 2024-04-14T23:55:28.770 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/130/16' 2024-04-14T23:55:28.770 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/130/14' 2024-04-14T23:55:28.770 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/130/18' 2024-04-14T23:55:28.770 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/130/13' 2024-04-14T23:55:28.770 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/130/17' 2024-04-14T23:55:28.770 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/130' 2024-04-14T23:55:28.770 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/193/19' 2024-04-14T23:55:28.770 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/193/15' 2024-04-14T23:55:28.770 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/193/20' 2024-04-14T23:55:28.770 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/193/12' 2024-04-14T23:55:28.771 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/193/10' 2024-04-14T23:55:28.771 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/193/11' 2024-04-14T23:55:28.771 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/193/16' 2024-04-14T23:55:28.771 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/193/14' 2024-04-14T23:55:28.771 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/193/18' 2024-04-14T23:55:28.771 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/193/13' 2024-04-14T23:55:28.771 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/193/17' 2024-04-14T23:55:28.771 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/193' 2024-04-14T23:55:28.771 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/175/19' 2024-04-14T23:55:28.771 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/175/15' 2024-04-14T23:55:31.088 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/175/20' 2024-04-14T23:55:31.088 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/175/12' 2024-04-14T23:55:31.088 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/175/10' 2024-04-14T23:55:31.088 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/175/11' 2024-04-14T23:55:31.088 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/175/16' 2024-04-14T23:55:31.088 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/175/14' 2024-04-14T23:55:31.089 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/175/18' 2024-04-14T23:55:31.089 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/175/13' 2024-04-14T23:55:31.089 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/175/17' 2024-04-14T23:55:31.089 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/175' 2024-04-14T23:55:31.089 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/181/19' 2024-04-14T23:55:31.089 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/181/15' 2024-04-14T23:55:31.089 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/181/20' 2024-04-14T23:55:31.089 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/181/12' 2024-04-14T23:55:31.089 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/181/10' 2024-04-14T23:55:31.089 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/181/11' 2024-04-14T23:55:31.089 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/181/16' 2024-04-14T23:55:31.089 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/181/14' 2024-04-14T23:55:31.089 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/181/18' 2024-04-14T23:55:31.089 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/181/13' 2024-04-14T23:55:31.089 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/181/17' 2024-04-14T23:55:31.089 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/181' 2024-04-14T23:55:31.090 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/127/19' 2024-04-14T23:55:31.090 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/127/15' 2024-04-14T23:55:31.090 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/127/20' 2024-04-14T23:55:31.090 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/127/12' 2024-04-14T23:55:31.090 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/127/10' 2024-04-14T23:55:31.090 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/127/11' 2024-04-14T23:55:31.090 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/127/16' 2024-04-14T23:55:31.090 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/127/14' 2024-04-14T23:55:31.090 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/127/18' 2024-04-14T23:55:31.090 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/127/13' 2024-04-14T23:55:31.090 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/127/17' 2024-04-14T23:55:31.090 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/127' 2024-04-14T23:55:31.090 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/108/19' 2024-04-14T23:55:31.090 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/108/15' 2024-04-14T23:55:31.091 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/108/20' 2024-04-14T23:55:31.091 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/108/12' 2024-04-14T23:55:31.091 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/108/10' 2024-04-14T23:55:31.091 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/108/11' 2024-04-14T23:55:31.091 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/108/16' 2024-04-14T23:55:31.091 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/108/14' 2024-04-14T23:55:31.091 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/108/18' 2024-04-14T23:55:31.091 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/108/13' 2024-04-14T23:55:31.091 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/108/17' 2024-04-14T23:55:31.091 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/108' 2024-04-14T23:55:31.091 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/183/19' 2024-04-14T23:55:31.091 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/183/15' 2024-04-14T23:55:31.091 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/183/20' 2024-04-14T23:55:31.091 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/183/12' 2024-04-14T23:55:31.091 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/183/10' 2024-04-14T23:55:31.091 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/183/11' 2024-04-14T23:55:31.092 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/183/16' 2024-04-14T23:55:31.092 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/183/14' 2024-04-14T23:55:31.092 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/183/18' 2024-04-14T23:55:31.092 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/183/13' 2024-04-14T23:55:31.092 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/183/17' 2024-04-14T23:55:31.092 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/183' 2024-04-14T23:55:31.092 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/177/19' 2024-04-14T23:55:31.092 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/177/15' 2024-04-14T23:55:31.092 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/177/20' 2024-04-14T23:55:31.092 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/177/12' 2024-04-14T23:55:31.092 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/177/10' 2024-04-14T23:55:31.092 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/177/11' 2024-04-14T23:55:31.092 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/177/16' 2024-04-14T23:55:31.092 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/177/14' 2024-04-14T23:55:31.092 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/177/18' 2024-04-14T23:55:31.093 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/177/13' 2024-04-14T23:55:31.093 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/177/17' 2024-04-14T23:55:31.093 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/177' 2024-04-14T23:55:31.093 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/169/19' 2024-04-14T23:55:31.093 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/169/15' 2024-04-14T23:55:31.093 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/169/20' 2024-04-14T23:55:31.093 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/169/12' 2024-04-14T23:55:31.093 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/169/10' 2024-04-14T23:55:31.093 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/169/11' 2024-04-14T23:55:31.093 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/169/16' 2024-04-14T23:55:31.093 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/169/14' 2024-04-14T23:55:31.093 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/169/18' 2024-04-14T23:55:31.093 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/169/13' 2024-04-14T23:55:31.093 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/169/17' 2024-04-14T23:55:31.093 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/169' 2024-04-14T23:55:31.093 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/154/19' 2024-04-14T23:55:31.094 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/154/15' 2024-04-14T23:55:31.094 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/154/20' 2024-04-14T23:55:31.094 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/154/12' 2024-04-14T23:55:31.094 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/154/10' 2024-04-14T23:55:31.094 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/154/11' 2024-04-14T23:55:31.094 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/154/16' 2024-04-14T23:55:31.094 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/154/14' 2024-04-14T23:55:31.094 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/154/18' 2024-04-14T23:55:31.094 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/154/13' 2024-04-14T23:55:31.094 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/154/17' 2024-04-14T23:55:31.094 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/154' 2024-04-14T23:55:31.094 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/171/19' 2024-04-14T23:55:31.094 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/171/15' 2024-04-14T23:55:31.094 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/171/20' 2024-04-14T23:55:31.094 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/171/12' 2024-04-14T23:55:31.095 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/171/10' 2024-04-14T23:55:31.095 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/171/11' 2024-04-14T23:55:31.095 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/171/16' 2024-04-14T23:55:31.095 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/171/14' 2024-04-14T23:55:31.095 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/171/18' 2024-04-14T23:55:31.095 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/171/13' 2024-04-14T23:55:31.095 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/171/17' 2024-04-14T23:55:31.095 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/171' 2024-04-14T23:55:31.095 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/139/19' 2024-04-14T23:55:31.095 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/139/15' 2024-04-14T23:55:31.095 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/139/20' 2024-04-14T23:55:31.357 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/139/12' 2024-04-14T23:55:31.357 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/139/10' 2024-04-14T23:55:31.357 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/139/11' 2024-04-14T23:55:31.357 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/139/16' 2024-04-14T23:55:31.358 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/139/14' 2024-04-14T23:55:31.358 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/139/18' 2024-04-14T23:55:31.358 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/139/13' 2024-04-14T23:55:31.358 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/139/17' 2024-04-14T23:55:31.358 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/139' 2024-04-14T23:55:31.358 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/102/19' 2024-04-14T23:55:31.358 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/102/15' 2024-04-14T23:55:31.358 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/102/20' 2024-04-14T23:55:31.358 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/102/12' 2024-04-14T23:55:31.358 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/102/10' 2024-04-14T23:55:31.358 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/102/11' 2024-04-14T23:55:31.358 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/102/16' 2024-04-14T23:55:31.358 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/102/14' 2024-04-14T23:55:31.358 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/102/18' 2024-04-14T23:55:31.359 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/102/13' 2024-04-14T23:55:31.359 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/102/17' 2024-04-14T23:55:31.359 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/102' 2024-04-14T23:55:31.359 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/129/19' 2024-04-14T23:55:31.359 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/129/15' 2024-04-14T23:55:31.359 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/129/20' 2024-04-14T23:55:31.359 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/129/12' 2024-04-14T23:55:31.359 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/129/10' 2024-04-14T23:55:31.359 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/129/11' 2024-04-14T23:55:31.359 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/129/16' 2024-04-14T23:55:31.359 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/129/14' 2024-04-14T23:55:31.359 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/129/18' 2024-04-14T23:55:31.359 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/129/13' 2024-04-14T23:55:31.359 INFO:tasks.workunit.client.0.smithi060.stdout:removed './copyofsnap1/test1/129/17' 2024-04-14T23:55:31.360 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1/129' 2024-04-14T23:55:31.360 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1/test1' 2024-04-14T23:55:31.360 INFO:tasks.workunit.client.0.smithi060.stdout:removed directory './copyofsnap1' 2024-04-14T23:55:31.361 INFO:tasks.workunit.client.0.smithi060.stdout:OK 2024-04-14T23:55:31.361 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-14T23:55:31.361 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-14T23:55:31.460 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-authwb.sh... 2024-04-14T23:55:31.460 DEBUG:teuthology.orchestra.run.smithi060:workunit test fs/snaps/snaptest-authwb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a9a752df26c63acad72e1b3569fd79a515ca0765 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-authwb.sh 2024-04-14T23:55:31.526 INFO:tasks.workunit.client.0.smithi060.stderr:+ set -e 2024-04-14T23:55:31.527 INFO:tasks.workunit.client.0.smithi060.stderr:+ touch foo 2024-04-14T23:55:31.531 INFO:tasks.workunit.client.0.smithi060.stderr:+ chmod +x foo 2024-04-14T23:55:31.534 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir .snap/s 2024-04-14T23:55:31.574 INFO:tasks.workunit.client.0.smithi060.stderr:+ find .snap/s/foo -executable 2024-04-14T23:55:31.575 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep foo 2024-04-14T23:55:31.581 INFO:tasks.workunit.client.0.smithi060.stdout:.snap/s/foo 2024-04-14T23:55:31.581 INFO:tasks.workunit.client.0.smithi060.stderr:+ rmdir .snap/s 2024-04-14T23:55:31.592 INFO:tasks.workunit.client.0.smithi060.stderr:+ rm foo 2024-04-14T23:55:31.597 INFO:tasks.workunit.client.0.smithi060.stderr:+ echo OK 2024-04-14T23:55:31.598 INFO:tasks.workunit.client.0.smithi060.stdout:OK 2024-04-14T23:55:31.599 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-14T23:55:31.599 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-14T23:55:31.720 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-capwb.sh... 2024-04-14T23:55:31.721 DEBUG:teuthology.orchestra.run.smithi060:workunit test fs/snaps/snaptest-capwb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a9a752df26c63acad72e1b3569fd79a515ca0765 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-capwb.sh 2024-04-14T23:55:31.772 INFO:tasks.workunit.client.0.smithi060.stderr:+ set -e 2024-04-14T23:55:31.772 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir foo 2024-04-14T23:55:31.775 INFO:tasks.workunit.client.0.smithi060.stderr:+ echo x 2024-04-14T23:55:31.776 INFO:tasks.workunit.client.0.smithi060.stderr:+ sync 2024-04-14T23:55:32.931 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir foo/.snap/ss 2024-04-14T23:55:33.383 INFO:tasks.workunit.client.0.smithi060.stderr:+ ln foo/x foo/xx 2024-04-14T23:55:33.391 INFO:tasks.workunit.client.0.smithi060.stderr:+ cat foo/.snap/ss/x 2024-04-14T23:55:33.406 INFO:tasks.workunit.client.0.smithi060.stdout:x 2024-04-14T23:55:33.406 INFO:tasks.workunit.client.0.smithi060.stderr:+ rmdir foo/.snap/ss 2024-04-14T23:55:33.417 INFO:tasks.workunit.client.0.smithi060.stderr:+ echo a 2024-04-14T23:55:33.420 INFO:tasks.workunit.client.0.smithi060.stderr:+ echo b 2024-04-14T23:55:33.422 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir foo/.snap/s 2024-04-14T23:55:33.602 INFO:tasks.workunit.client.0.smithi060.stderr:++ cat foo/.snap/s/a 2024-04-14T23:55:33.618 INFO:tasks.workunit.client.0.smithi060.stderr:+ r=a 2024-04-14T23:55:33.618 INFO:tasks.workunit.client.0.smithi060.stderr:+ '[' -z a ']' 2024-04-14T23:55:33.618 INFO:tasks.workunit.client.0.smithi060.stderr:+ ln foo/b foo/b2 2024-04-14T23:55:33.623 INFO:tasks.workunit.client.0.smithi060.stderr:+ cat foo/.snap/s/b 2024-04-14T23:55:33.656 INFO:tasks.workunit.client.0.smithi060.stdout:b 2024-04-14T23:55:33.656 INFO:tasks.workunit.client.0.smithi060.stdout:this used to hang: 2024-04-14T23:55:33.657 INFO:tasks.workunit.client.0.smithi060.stderr:+ echo 'this used to hang:' 2024-04-14T23:55:33.657 INFO:tasks.workunit.client.0.smithi060.stderr:+ echo more 2024-04-14T23:55:33.657 INFO:tasks.workunit.client.0.smithi060.stdout:oh, it didn't hang! good job. 2024-04-14T23:55:33.657 INFO:tasks.workunit.client.0.smithi060.stderr:+ echo 'oh, it didn'\''t hang! good job.' 2024-04-14T23:55:33.657 INFO:tasks.workunit.client.0.smithi060.stderr:+ cat foo/b 2024-04-14T23:55:33.660 INFO:tasks.workunit.client.0.smithi060.stdout:b 2024-04-14T23:55:33.660 INFO:tasks.workunit.client.0.smithi060.stdout:more 2024-04-14T23:55:33.661 INFO:tasks.workunit.client.0.smithi060.stderr:+ rmdir foo/.snap/s 2024-04-14T23:55:33.668 INFO:tasks.workunit.client.0.smithi060.stderr:+ rm -r foo 2024-04-14T23:55:33.700 INFO:tasks.workunit.client.0.smithi060.stdout:OK 2024-04-14T23:55:33.701 INFO:tasks.workunit.client.0.smithi060.stderr:+ echo OK 2024-04-14T23:55:33.702 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-14T23:55:33.702 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-14T23:55:33.782 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-dir-rename.sh... 2024-04-14T23:55:33.783 DEBUG:teuthology.orchestra.run.smithi060:workunit test fs/snaps/snaptest-dir-rename.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a9a752df26c63acad72e1b3569fd79a515ca0765 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-dir-rename.sh 2024-04-14T23:55:33.837 INFO:tasks.workunit.client.0.smithi060.stderr:+ set -e 2024-04-14T23:55:33.837 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir a 2024-04-14T23:55:33.841 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir .snap/bar 2024-04-14T23:55:33.872 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir a/.snap/foo 2024-04-14T23:55:33.901 INFO:tasks.workunit.client.0.smithi060.stderr:+ rmdir a/.snap/foo 2024-04-14T23:55:33.912 INFO:tasks.workunit.client.0.smithi060.stderr:+ rmdir a 2024-04-14T23:55:33.918 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/bar/a 2024-04-14T23:55:33.923 INFO:tasks.workunit.client.0.smithi060.stdout: File: .snap/bar/a 2024-04-14T23:55:33.923 INFO:tasks.workunit.client.0.smithi060.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-04-14T23:55:33.924 INFO:tasks.workunit.client.0.smithi060.stdout:Device: fdh/253d Inode: 1099511633797 Links: 2 2024-04-14T23:55:33.924 INFO:tasks.workunit.client.0.smithi060.stdout:Access: (0755/drwxr-xr-x) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-04-14T23:55:33.924 INFO:tasks.workunit.client.0.smithi060.stdout:Context: unconfined_u:object_r:unlabeled_t:s0 2024-04-14T23:55:33.924 INFO:tasks.workunit.client.0.smithi060.stdout:Access: 2024-04-14 23:55:33.833735791 +0000 2024-04-14T23:55:33.924 INFO:tasks.workunit.client.0.smithi060.stdout:Modify: 2024-04-14 23:55:33.833735791 +0000 2024-04-14T23:55:33.924 INFO:tasks.workunit.client.0.smithi060.stdout:Change: 2024-04-14 23:55:33.833735791 +0000 2024-04-14T23:55:33.924 INFO:tasks.workunit.client.0.smithi060.stdout: Birth: 2024-04-14 23:55:33.833735791 +0000 2024-04-14T23:55:33.924 INFO:tasks.workunit.client.0.smithi060.stderr:+ rmdir .snap/bar 2024-04-14T23:55:33.936 INFO:tasks.workunit.client.0.smithi060.stderr:+ echo OK 2024-04-14T23:55:33.936 INFO:tasks.workunit.client.0.smithi060.stdout:OK 2024-04-14T23:55:33.937 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-14T23:55:33.938 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-14T23:55:34.034 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-double-null.sh... 2024-04-14T23:55:34.035 DEBUG:teuthology.orchestra.run.smithi060:workunit test fs/snaps/snaptest-double-null.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a9a752df26c63acad72e1b3569fd79a515ca0765 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-double-null.sh 2024-04-14T23:55:34.104 INFO:tasks.workunit.client.0.smithi060.stderr:+ set -e 2024-04-14T23:55:34.105 INFO:tasks.workunit.client.0.smithi060.stderr:++ seq 1 20 2024-04-14T23:55:34.107 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 20` 2024-04-14T23:55:34.107 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir a 2024-04-14T23:55:34.113 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir a/.snap/one 2024-04-14T23:55:34.113 INFO:tasks.workunit.client.0.smithi060.stderr:+ cat 2024-04-14T23:55:34.151 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir a/.snap/two 2024-04-14T23:55:34.183 INFO:tasks.workunit.client.0.smithi060.stderr:+ chmod 777 a/foo 2024-04-14T23:55:34.187 INFO:tasks.workunit.client.0.smithi060.stderr:+ sync 2024-04-14T23:55:34.383 INFO:tasks.workunit.client.0.smithi060.stderr:+ ps 2024-04-14T23:55:34.416 INFO:tasks.workunit.client.0.smithi060.stdout: PID TTY TIME CMD 2024-04-14T23:55:34.416 INFO:tasks.workunit.client.0.smithi060.stdout: 4282 ? 00:00:00 systemd 2024-04-14T23:55:34.416 INFO:tasks.workunit.client.0.smithi060.stdout: 4284 ? 00:00:00 (sd-pam) 2024-04-14T23:55:34.416 INFO:tasks.workunit.client.0.smithi060.stdout: 22914 ? 00:00:02 sshd 2024-04-14T23:55:34.416 INFO:tasks.workunit.client.0.smithi060.stdout: 49915 ? 00:00:00 timeout 2024-04-14T23:55:34.416 INFO:tasks.workunit.client.0.smithi060.stdout: 49933 ? 00:00:00 snaptest-double 2024-04-14T23:55:34.416 INFO:tasks.workunit.client.0.smithi060.stdout: 49941 ? 00:00:00 ps 2024-04-14T23:55:34.417 INFO:tasks.workunit.client.0.smithi060.stderr:+ rmdir a/.snap/one a/.snap/two 2024-04-14T23:55:34.440 INFO:tasks.workunit.client.0.smithi060.stderr:+ rm a/foo 2024-04-14T23:55:34.448 INFO:tasks.workunit.client.0.smithi060.stderr:+ rmdir a 2024-04-14T23:55:34.462 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 20` 2024-04-14T23:55:34.463 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir a 2024-04-14T23:55:34.470 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir a/.snap/one 2024-04-14T23:55:34.470 INFO:tasks.workunit.client.0.smithi060.stderr:+ cat 2024-04-14T23:55:34.507 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir a/.snap/two 2024-04-14T23:55:34.540 INFO:tasks.workunit.client.0.smithi060.stderr:+ chmod 777 a/foo 2024-04-14T23:55:34.544 INFO:tasks.workunit.client.0.smithi060.stderr:+ sync 2024-04-14T23:55:34.630 INFO:tasks.workunit.client.0.smithi060.stderr:+ ps 2024-04-14T23:55:34.655 INFO:tasks.workunit.client.0.smithi060.stdout: PID TTY TIME CMD 2024-04-14T23:55:34.655 INFO:tasks.workunit.client.0.smithi060.stdout: 4282 ? 00:00:00 systemd 2024-04-14T23:55:34.655 INFO:tasks.workunit.client.0.smithi060.stdout: 4284 ? 00:00:00 (sd-pam) 2024-04-14T23:55:34.655 INFO:tasks.workunit.client.0.smithi060.stdout: 22914 ? 00:00:02 sshd 2024-04-14T23:55:34.656 INFO:tasks.workunit.client.0.smithi060.stdout: 49915 ? 00:00:00 timeout 2024-04-14T23:55:34.656 INFO:tasks.workunit.client.0.smithi060.stdout: 49933 ? 00:00:00 snaptest-double 2024-04-14T23:55:34.656 INFO:tasks.workunit.client.0.smithi060.stdout: 49951 ? 00:00:00 ps 2024-04-14T23:55:34.657 INFO:tasks.workunit.client.0.smithi060.stderr:+ rmdir a/.snap/one a/.snap/two 2024-04-14T23:55:34.680 INFO:tasks.workunit.client.0.smithi060.stderr:+ rm a/foo 2024-04-14T23:55:34.686 INFO:tasks.workunit.client.0.smithi060.stderr:+ rmdir a 2024-04-14T23:55:34.702 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 20` 2024-04-14T23:55:34.702 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir a 2024-04-14T23:55:34.709 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir a/.snap/one 2024-04-14T23:55:34.709 INFO:tasks.workunit.client.0.smithi060.stderr:+ cat 2024-04-14T23:55:34.746 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir a/.snap/two 2024-04-14T23:55:34.769 INFO:tasks.workunit.client.0.smithi060.stderr:+ chmod 777 a/foo 2024-04-14T23:55:34.772 INFO:tasks.workunit.client.0.smithi060.stderr:+ sync 2024-04-14T23:55:34.881 INFO:tasks.workunit.client.0.smithi060.stderr:+ ps 2024-04-14T23:55:34.905 INFO:tasks.workunit.client.0.smithi060.stdout: PID TTY TIME CMD 2024-04-14T23:55:34.905 INFO:tasks.workunit.client.0.smithi060.stdout: 4282 ? 00:00:00 systemd 2024-04-14T23:55:34.905 INFO:tasks.workunit.client.0.smithi060.stdout: 4284 ? 00:00:00 (sd-pam) 2024-04-14T23:55:34.905 INFO:tasks.workunit.client.0.smithi060.stdout: 22914 ? 00:00:02 sshd 2024-04-14T23:55:34.905 INFO:tasks.workunit.client.0.smithi060.stdout: 49915 ? 00:00:00 timeout 2024-04-14T23:55:34.905 INFO:tasks.workunit.client.0.smithi060.stdout: 49933 ? 00:00:00 snaptest-double 2024-04-14T23:55:34.905 INFO:tasks.workunit.client.0.smithi060.stdout: 49961 ? 00:00:00 ps 2024-04-14T23:55:34.907 INFO:tasks.workunit.client.0.smithi060.stderr:+ rmdir a/.snap/one a/.snap/two 2024-04-14T23:55:34.929 INFO:tasks.workunit.client.0.smithi060.stderr:+ rm a/foo 2024-04-14T23:55:34.937 INFO:tasks.workunit.client.0.smithi060.stderr:+ rmdir a 2024-04-14T23:55:34.951 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 20` 2024-04-14T23:55:34.951 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir a 2024-04-14T23:55:34.959 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir a/.snap/one 2024-04-14T23:55:34.959 INFO:tasks.workunit.client.0.smithi060.stderr:+ cat 2024-04-14T23:55:34.985 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir a/.snap/two 2024-04-14T23:55:35.008 INFO:tasks.workunit.client.0.smithi060.stderr:+ chmod 777 a/foo 2024-04-14T23:55:35.012 INFO:tasks.workunit.client.0.smithi060.stderr:+ sync 2024-04-14T23:55:35.081 INFO:tasks.workunit.client.0.smithi060.stderr:+ ps 2024-04-14T23:55:35.106 INFO:tasks.workunit.client.0.smithi060.stdout: PID TTY TIME CMD 2024-04-14T23:55:35.106 INFO:tasks.workunit.client.0.smithi060.stdout: 4282 ? 00:00:00 systemd 2024-04-14T23:55:35.106 INFO:tasks.workunit.client.0.smithi060.stdout: 4284 ? 00:00:00 (sd-pam) 2024-04-14T23:55:35.106 INFO:tasks.workunit.client.0.smithi060.stdout: 22914 ? 00:00:02 sshd 2024-04-14T23:55:35.106 INFO:tasks.workunit.client.0.smithi060.stdout: 49915 ? 00:00:00 timeout 2024-04-14T23:55:35.106 INFO:tasks.workunit.client.0.smithi060.stdout: 49933 ? 00:00:00 snaptest-double 2024-04-14T23:55:35.106 INFO:tasks.workunit.client.0.smithi060.stdout: 49971 ? 00:00:00 ps 2024-04-14T23:55:35.108 INFO:tasks.workunit.client.0.smithi060.stderr:+ rmdir a/.snap/one a/.snap/two 2024-04-14T23:55:35.133 INFO:tasks.workunit.client.0.smithi060.stderr:+ rm a/foo 2024-04-14T23:55:35.143 INFO:tasks.workunit.client.0.smithi060.stderr:+ rmdir a 2024-04-14T23:55:35.155 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 20` 2024-04-14T23:55:35.155 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir a 2024-04-14T23:55:35.161 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir a/.snap/one 2024-04-14T23:55:35.162 INFO:tasks.workunit.client.0.smithi060.stderr:+ cat 2024-04-14T23:55:35.190 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir a/.snap/two 2024-04-14T23:55:35.212 INFO:tasks.workunit.client.0.smithi060.stderr:+ chmod 777 a/foo 2024-04-14T23:55:35.217 INFO:tasks.workunit.client.0.smithi060.stderr:+ sync 2024-04-14T23:55:35.290 INFO:tasks.workunit.client.0.smithi060.stderr:+ ps 2024-04-14T23:55:35.313 INFO:tasks.workunit.client.0.smithi060.stdout: PID TTY TIME CMD 2024-04-14T23:55:35.314 INFO:tasks.workunit.client.0.smithi060.stdout: 4282 ? 00:00:00 systemd 2024-04-14T23:55:35.314 INFO:tasks.workunit.client.0.smithi060.stdout: 4284 ? 00:00:00 (sd-pam) 2024-04-14T23:55:35.314 INFO:tasks.workunit.client.0.smithi060.stdout: 22914 ? 00:00:02 sshd 2024-04-14T23:55:35.314 INFO:tasks.workunit.client.0.smithi060.stdout: 49915 ? 00:00:00 timeout 2024-04-14T23:55:35.314 INFO:tasks.workunit.client.0.smithi060.stdout: 49933 ? 00:00:00 snaptest-double 2024-04-14T23:55:35.314 INFO:tasks.workunit.client.0.smithi060.stdout: 49981 ? 00:00:00 ps 2024-04-14T23:55:35.316 INFO:tasks.workunit.client.0.smithi060.stderr:+ rmdir a/.snap/one a/.snap/two 2024-04-14T23:55:35.339 INFO:tasks.workunit.client.0.smithi060.stderr:+ rm a/foo 2024-04-14T23:55:35.347 INFO:tasks.workunit.client.0.smithi060.stderr:+ rmdir a 2024-04-14T23:55:35.363 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 20` 2024-04-14T23:55:35.364 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir a 2024-04-14T23:55:35.372 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir a/.snap/one 2024-04-14T23:55:35.372 INFO:tasks.workunit.client.0.smithi060.stderr:+ cat 2024-04-14T23:55:35.401 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir a/.snap/two 2024-04-14T23:55:35.426 INFO:tasks.workunit.client.0.smithi060.stderr:+ chmod 777 a/foo 2024-04-14T23:55:35.429 INFO:tasks.workunit.client.0.smithi060.stderr:+ sync 2024-04-14T23:55:35.502 INFO:tasks.workunit.client.0.smithi060.stderr:+ ps 2024-04-14T23:55:35.521 INFO:tasks.workunit.client.0.smithi060.stdout: PID TTY TIME CMD 2024-04-14T23:55:35.522 INFO:tasks.workunit.client.0.smithi060.stdout: 4282 ? 00:00:00 systemd 2024-04-14T23:55:35.522 INFO:tasks.workunit.client.0.smithi060.stdout: 4284 ? 00:00:00 (sd-pam) 2024-04-14T23:55:35.522 INFO:tasks.workunit.client.0.smithi060.stdout: 22914 ? 00:00:02 sshd 2024-04-14T23:55:35.522 INFO:tasks.workunit.client.0.smithi060.stdout: 49915 ? 00:00:00 timeout 2024-04-14T23:55:35.522 INFO:tasks.workunit.client.0.smithi060.stdout: 49933 ? 00:00:00 snaptest-double 2024-04-14T23:55:35.522 INFO:tasks.workunit.client.0.smithi060.stdout: 49991 ? 00:00:00 ps 2024-04-14T23:55:35.788 INFO:tasks.workunit.client.0.smithi060.stderr:+ rmdir a/.snap/one a/.snap/two 2024-04-14T23:55:35.815 INFO:tasks.workunit.client.0.smithi060.stderr:+ rm a/foo 2024-04-14T23:55:35.822 INFO:tasks.workunit.client.0.smithi060.stderr:+ rmdir a 2024-04-14T23:55:35.837 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 20` 2024-04-14T23:55:35.837 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir a 2024-04-14T23:55:35.844 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir a/.snap/one 2024-04-14T23:55:35.844 INFO:tasks.workunit.client.0.smithi060.stderr:+ cat 2024-04-14T23:55:35.877 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir a/.snap/two 2024-04-14T23:55:35.899 INFO:tasks.workunit.client.0.smithi060.stderr:+ chmod 777 a/foo 2024-04-14T23:55:35.903 INFO:tasks.workunit.client.0.smithi060.stderr:+ sync 2024-04-14T23:55:35.991 INFO:tasks.workunit.client.0.smithi060.stderr:+ ps 2024-04-14T23:55:36.015 INFO:tasks.workunit.client.0.smithi060.stdout: PID TTY TIME CMD 2024-04-14T23:55:36.015 INFO:tasks.workunit.client.0.smithi060.stdout: 4282 ? 00:00:00 systemd 2024-04-14T23:55:36.015 INFO:tasks.workunit.client.0.smithi060.stdout: 4284 ? 00:00:00 (sd-pam) 2024-04-14T23:55:36.015 INFO:tasks.workunit.client.0.smithi060.stdout: 22914 ? 00:00:02 sshd 2024-04-14T23:55:36.015 INFO:tasks.workunit.client.0.smithi060.stdout: 49915 ? 00:00:00 timeout 2024-04-14T23:55:36.015 INFO:tasks.workunit.client.0.smithi060.stdout: 49933 ? 00:00:00 snaptest-double 2024-04-14T23:55:36.015 INFO:tasks.workunit.client.0.smithi060.stdout: 50001 ? 00:00:00 ps 2024-04-14T23:55:36.285 INFO:tasks.workunit.client.0.smithi060.stderr:+ rmdir a/.snap/one a/.snap/two 2024-04-14T23:55:36.314 INFO:tasks.workunit.client.0.smithi060.stderr:+ rm a/foo 2024-04-14T23:55:36.321 INFO:tasks.workunit.client.0.smithi060.stderr:+ rmdir a 2024-04-14T23:55:36.334 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 20` 2024-04-14T23:55:36.335 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir a 2024-04-14T23:55:36.342 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir a/.snap/one 2024-04-14T23:55:36.342 INFO:tasks.workunit.client.0.smithi060.stderr:+ cat 2024-04-14T23:55:36.369 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir a/.snap/two 2024-04-14T23:55:36.391 INFO:tasks.workunit.client.0.smithi060.stderr:+ chmod 777 a/foo 2024-04-14T23:55:36.394 INFO:tasks.workunit.client.0.smithi060.stderr:+ sync 2024-04-14T23:55:36.474 INFO:tasks.workunit.client.0.smithi060.stderr:+ ps 2024-04-14T23:55:36.498 INFO:tasks.workunit.client.0.smithi060.stdout: PID TTY TIME CMD 2024-04-14T23:55:36.498 INFO:tasks.workunit.client.0.smithi060.stdout: 4282 ? 00:00:00 systemd 2024-04-14T23:55:36.498 INFO:tasks.workunit.client.0.smithi060.stdout: 4284 ? 00:00:00 (sd-pam) 2024-04-14T23:55:36.498 INFO:tasks.workunit.client.0.smithi060.stdout: 22914 ? 00:00:02 sshd 2024-04-14T23:55:36.499 INFO:tasks.workunit.client.0.smithi060.stdout: 49915 ? 00:00:00 timeout 2024-04-14T23:55:36.499 INFO:tasks.workunit.client.0.smithi060.stdout: 49933 ? 00:00:00 snaptest-double 2024-04-14T23:55:36.499 INFO:tasks.workunit.client.0.smithi060.stdout: 50011 ? 00:00:00 ps 2024-04-14T23:55:36.500 INFO:tasks.workunit.client.0.smithi060.stderr:+ rmdir a/.snap/one a/.snap/two 2024-04-14T23:55:36.526 INFO:tasks.workunit.client.0.smithi060.stderr:+ rm a/foo 2024-04-14T23:55:36.532 INFO:tasks.workunit.client.0.smithi060.stderr:+ rmdir a 2024-04-14T23:55:36.546 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 20` 2024-04-14T23:55:36.546 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir a 2024-04-14T23:55:36.553 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir a/.snap/one 2024-04-14T23:55:36.553 INFO:tasks.workunit.client.0.smithi060.stderr:+ cat 2024-04-14T23:55:36.603 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir a/.snap/two 2024-04-14T23:55:36.667 INFO:tasks.workunit.client.0.smithi060.stderr:+ chmod 777 a/foo 2024-04-14T23:55:36.671 INFO:tasks.workunit.client.0.smithi060.stderr:+ sync 2024-04-14T23:55:36.750 INFO:tasks.workunit.client.0.smithi060.stderr:+ ps 2024-04-14T23:55:36.774 INFO:tasks.workunit.client.0.smithi060.stdout: PID TTY TIME CMD 2024-04-14T23:55:36.774 INFO:tasks.workunit.client.0.smithi060.stdout: 4282 ? 00:00:00 systemd 2024-04-14T23:55:36.775 INFO:tasks.workunit.client.0.smithi060.stdout: 4284 ? 00:00:00 (sd-pam) 2024-04-14T23:55:36.775 INFO:tasks.workunit.client.0.smithi060.stdout: 22914 ? 00:00:02 sshd 2024-04-14T23:55:36.775 INFO:tasks.workunit.client.0.smithi060.stdout: 49915 ? 00:00:00 timeout 2024-04-14T23:55:36.775 INFO:tasks.workunit.client.0.smithi060.stdout: 49933 ? 00:00:00 snaptest-double 2024-04-14T23:55:36.775 INFO:tasks.workunit.client.0.smithi060.stdout: 50021 ? 00:00:00 ps 2024-04-14T23:55:36.776 INFO:tasks.workunit.client.0.smithi060.stderr:+ rmdir a/.snap/one a/.snap/two 2024-04-14T23:55:36.802 INFO:tasks.workunit.client.0.smithi060.stderr:+ rm a/foo 2024-04-14T23:55:36.809 INFO:tasks.workunit.client.0.smithi060.stderr:+ rmdir a 2024-04-14T23:55:36.824 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 20` 2024-04-14T23:55:36.824 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir a 2024-04-14T23:55:36.832 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir a/.snap/one 2024-04-14T23:55:36.832 INFO:tasks.workunit.client.0.smithi060.stderr:+ cat 2024-04-14T23:55:36.876 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir a/.snap/two 2024-04-14T23:55:36.912 INFO:tasks.workunit.client.0.smithi060.stderr:+ chmod 777 a/foo 2024-04-14T23:55:36.916 INFO:tasks.workunit.client.0.smithi060.stderr:+ sync 2024-04-14T23:55:36.983 INFO:tasks.workunit.client.0.smithi060.stderr:+ ps 2024-04-14T23:55:37.007 INFO:tasks.workunit.client.0.smithi060.stdout: PID TTY TIME CMD 2024-04-14T23:55:37.007 INFO:tasks.workunit.client.0.smithi060.stdout: 4282 ? 00:00:00 systemd 2024-04-14T23:55:37.007 INFO:tasks.workunit.client.0.smithi060.stdout: 4284 ? 00:00:00 (sd-pam) 2024-04-14T23:55:37.007 INFO:tasks.workunit.client.0.smithi060.stdout: 22914 ? 00:00:02 sshd 2024-04-14T23:55:37.007 INFO:tasks.workunit.client.0.smithi060.stdout: 49915 ? 00:00:00 timeout 2024-04-14T23:55:37.007 INFO:tasks.workunit.client.0.smithi060.stdout: 49933 ? 00:00:00 snaptest-double 2024-04-14T23:55:37.007 INFO:tasks.workunit.client.0.smithi060.stdout: 50031 ? 00:00:00 ps 2024-04-14T23:55:37.009 INFO:tasks.workunit.client.0.smithi060.stderr:+ rmdir a/.snap/one a/.snap/two 2024-04-14T23:55:37.032 INFO:tasks.workunit.client.0.smithi060.stderr:+ rm a/foo 2024-04-14T23:55:37.038 INFO:tasks.workunit.client.0.smithi060.stderr:+ rmdir a 2024-04-14T23:55:37.052 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 20` 2024-04-14T23:55:37.053 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir a 2024-04-14T23:55:37.061 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir a/.snap/one 2024-04-14T23:55:37.061 INFO:tasks.workunit.client.0.smithi060.stderr:+ cat 2024-04-14T23:55:37.100 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir a/.snap/two 2024-04-14T23:55:37.135 INFO:tasks.workunit.client.0.smithi060.stderr:+ chmod 777 a/foo 2024-04-14T23:55:37.138 INFO:tasks.workunit.client.0.smithi060.stderr:+ sync 2024-04-14T23:55:37.218 INFO:tasks.workunit.client.0.smithi060.stderr:+ ps 2024-04-14T23:55:37.241 INFO:tasks.workunit.client.0.smithi060.stdout: PID TTY TIME CMD 2024-04-14T23:55:37.241 INFO:tasks.workunit.client.0.smithi060.stdout: 4282 ? 00:00:00 systemd 2024-04-14T23:55:37.242 INFO:tasks.workunit.client.0.smithi060.stdout: 4284 ? 00:00:00 (sd-pam) 2024-04-14T23:55:37.242 INFO:tasks.workunit.client.0.smithi060.stdout: 22914 ? 00:00:02 sshd 2024-04-14T23:55:37.242 INFO:tasks.workunit.client.0.smithi060.stdout: 49915 ? 00:00:00 timeout 2024-04-14T23:55:37.242 INFO:tasks.workunit.client.0.smithi060.stdout: 49933 ? 00:00:00 snaptest-double 2024-04-14T23:55:37.242 INFO:tasks.workunit.client.0.smithi060.stdout: 50041 ? 00:00:00 ps 2024-04-14T23:55:37.243 INFO:tasks.workunit.client.0.smithi060.stderr:+ rmdir a/.snap/one a/.snap/two 2024-04-14T23:55:37.271 INFO:tasks.workunit.client.0.smithi060.stderr:+ rm a/foo 2024-04-14T23:55:37.279 INFO:tasks.workunit.client.0.smithi060.stderr:+ rmdir a 2024-04-14T23:55:37.300 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 20` 2024-04-14T23:55:37.301 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir a 2024-04-14T23:55:37.310 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir a/.snap/one 2024-04-14T23:55:37.310 INFO:tasks.workunit.client.0.smithi060.stderr:+ cat 2024-04-14T23:55:37.350 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir a/.snap/two 2024-04-14T23:55:37.384 INFO:tasks.workunit.client.0.smithi060.stderr:+ chmod 777 a/foo 2024-04-14T23:55:37.387 INFO:tasks.workunit.client.0.smithi060.stderr:+ sync 2024-04-14T23:55:37.459 INFO:tasks.workunit.client.0.smithi060.stderr:+ ps 2024-04-14T23:55:37.482 INFO:tasks.workunit.client.0.smithi060.stdout: PID TTY TIME CMD 2024-04-14T23:55:37.482 INFO:tasks.workunit.client.0.smithi060.stdout: 4282 ? 00:00:00 systemd 2024-04-14T23:55:37.482 INFO:tasks.workunit.client.0.smithi060.stdout: 4284 ? 00:00:00 (sd-pam) 2024-04-14T23:55:37.483 INFO:tasks.workunit.client.0.smithi060.stdout: 22914 ? 00:00:02 sshd 2024-04-14T23:55:37.483 INFO:tasks.workunit.client.0.smithi060.stdout: 49915 ? 00:00:00 timeout 2024-04-14T23:55:37.483 INFO:tasks.workunit.client.0.smithi060.stdout: 49933 ? 00:00:00 snaptest-double 2024-04-14T23:55:37.483 INFO:tasks.workunit.client.0.smithi060.stdout: 50051 ? 00:00:00 ps 2024-04-14T23:55:37.484 INFO:tasks.workunit.client.0.smithi060.stderr:+ rmdir a/.snap/one a/.snap/two 2024-04-14T23:55:37.514 INFO:tasks.workunit.client.0.smithi060.stderr:+ rm a/foo 2024-04-14T23:55:37.523 INFO:tasks.workunit.client.0.smithi060.stderr:+ rmdir a 2024-04-14T23:55:37.541 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 20` 2024-04-14T23:55:37.541 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir a 2024-04-14T23:55:37.550 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir a/.snap/one 2024-04-14T23:55:37.550 INFO:tasks.workunit.client.0.smithi060.stderr:+ cat 2024-04-14T23:55:37.594 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir a/.snap/two 2024-04-14T23:55:37.629 INFO:tasks.workunit.client.0.smithi060.stderr:+ chmod 777 a/foo 2024-04-14T23:55:37.633 INFO:tasks.workunit.client.0.smithi060.stderr:+ sync 2024-04-14T23:55:37.710 INFO:tasks.workunit.client.0.smithi060.stderr:+ ps 2024-04-14T23:55:37.734 INFO:tasks.workunit.client.0.smithi060.stdout: PID TTY TIME CMD 2024-04-14T23:55:37.734 INFO:tasks.workunit.client.0.smithi060.stdout: 4282 ? 00:00:00 systemd 2024-04-14T23:55:37.734 INFO:tasks.workunit.client.0.smithi060.stdout: 4284 ? 00:00:00 (sd-pam) 2024-04-14T23:55:37.734 INFO:tasks.workunit.client.0.smithi060.stdout: 22914 ? 00:00:02 sshd 2024-04-14T23:55:37.734 INFO:tasks.workunit.client.0.smithi060.stdout: 49915 ? 00:00:00 timeout 2024-04-14T23:55:37.734 INFO:tasks.workunit.client.0.smithi060.stdout: 49933 ? 00:00:00 snaptest-double 2024-04-14T23:55:37.734 INFO:tasks.workunit.client.0.smithi060.stdout: 50061 ? 00:00:00 ps 2024-04-14T23:55:37.736 INFO:tasks.workunit.client.0.smithi060.stderr:+ rmdir a/.snap/one a/.snap/two 2024-04-14T23:55:37.765 INFO:tasks.workunit.client.0.smithi060.stderr:+ rm a/foo 2024-04-14T23:55:37.774 INFO:tasks.workunit.client.0.smithi060.stderr:+ rmdir a 2024-04-14T23:55:37.932 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 20` 2024-04-14T23:55:37.932 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir a 2024-04-14T23:55:37.941 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir a/.snap/one 2024-04-14T23:55:37.941 INFO:tasks.workunit.client.0.smithi060.stderr:+ cat 2024-04-14T23:55:37.975 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir a/.snap/two 2024-04-14T23:55:38.000 INFO:tasks.workunit.client.0.smithi060.stderr:+ chmod 777 a/foo 2024-04-14T23:55:38.003 INFO:tasks.workunit.client.0.smithi060.stderr:+ sync 2024-04-14T23:55:38.077 INFO:tasks.workunit.client.0.smithi060.stderr:+ ps 2024-04-14T23:55:38.101 INFO:tasks.workunit.client.0.smithi060.stdout: PID TTY TIME CMD 2024-04-14T23:55:38.102 INFO:tasks.workunit.client.0.smithi060.stdout: 4282 ? 00:00:00 systemd 2024-04-14T23:55:38.102 INFO:tasks.workunit.client.0.smithi060.stdout: 4284 ? 00:00:00 (sd-pam) 2024-04-14T23:55:38.102 INFO:tasks.workunit.client.0.smithi060.stdout: 22914 ? 00:00:02 sshd 2024-04-14T23:55:38.102 INFO:tasks.workunit.client.0.smithi060.stdout: 49915 ? 00:00:00 timeout 2024-04-14T23:55:38.102 INFO:tasks.workunit.client.0.smithi060.stdout: 49933 ? 00:00:00 snaptest-double 2024-04-14T23:55:38.102 INFO:tasks.workunit.client.0.smithi060.stdout: 50071 ? 00:00:00 ps 2024-04-14T23:55:38.102 INFO:tasks.workunit.client.0.smithi060.stderr:+ rmdir a/.snap/one a/.snap/two 2024-04-14T23:55:38.125 INFO:tasks.workunit.client.0.smithi060.stderr:+ rm a/foo 2024-04-14T23:55:38.131 INFO:tasks.workunit.client.0.smithi060.stderr:+ rmdir a 2024-04-14T23:55:38.145 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 20` 2024-04-14T23:55:38.145 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir a 2024-04-14T23:55:38.153 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir a/.snap/one 2024-04-14T23:55:38.153 INFO:tasks.workunit.client.0.smithi060.stderr:+ cat 2024-04-14T23:55:38.178 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir a/.snap/two 2024-04-14T23:55:38.229 INFO:tasks.workunit.client.0.smithi060.stderr:+ chmod 777 a/foo 2024-04-14T23:55:38.233 INFO:tasks.workunit.client.0.smithi060.stderr:+ sync 2024-04-14T23:55:38.310 INFO:tasks.workunit.client.0.smithi060.stderr:+ ps 2024-04-14T23:55:38.333 INFO:tasks.workunit.client.0.smithi060.stdout: PID TTY TIME CMD 2024-04-14T23:55:38.333 INFO:tasks.workunit.client.0.smithi060.stdout: 4282 ? 00:00:00 systemd 2024-04-14T23:55:38.333 INFO:tasks.workunit.client.0.smithi060.stdout: 4284 ? 00:00:00 (sd-pam) 2024-04-14T23:55:38.333 INFO:tasks.workunit.client.0.smithi060.stdout: 22914 ? 00:00:02 sshd 2024-04-14T23:55:38.333 INFO:tasks.workunit.client.0.smithi060.stdout: 49915 ? 00:00:00 timeout 2024-04-14T23:55:38.333 INFO:tasks.workunit.client.0.smithi060.stdout: 49933 ? 00:00:00 snaptest-double 2024-04-14T23:55:38.333 INFO:tasks.workunit.client.0.smithi060.stdout: 50081 ? 00:00:00 ps 2024-04-14T23:55:38.335 INFO:tasks.workunit.client.0.smithi060.stderr:+ rmdir a/.snap/one a/.snap/two 2024-04-14T23:55:38.364 INFO:tasks.workunit.client.0.smithi060.stderr:+ rm a/foo 2024-04-14T23:55:38.371 INFO:tasks.workunit.client.0.smithi060.stderr:+ rmdir a 2024-04-14T23:55:38.386 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 20` 2024-04-14T23:55:38.387 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir a 2024-04-14T23:55:38.394 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir a/.snap/one 2024-04-14T23:55:38.394 INFO:tasks.workunit.client.0.smithi060.stderr:+ cat 2024-04-14T23:55:38.421 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir a/.snap/two 2024-04-14T23:55:38.444 INFO:tasks.workunit.client.0.smithi060.stderr:+ chmod 777 a/foo 2024-04-14T23:55:38.448 INFO:tasks.workunit.client.0.smithi060.stderr:+ sync 2024-04-14T23:55:38.528 INFO:tasks.workunit.client.0.smithi060.stderr:+ ps 2024-04-14T23:55:38.550 INFO:tasks.workunit.client.0.smithi060.stdout: PID TTY TIME CMD 2024-04-14T23:55:38.550 INFO:tasks.workunit.client.0.smithi060.stdout: 4282 ? 00:00:00 systemd 2024-04-14T23:55:38.550 INFO:tasks.workunit.client.0.smithi060.stdout: 4284 ? 00:00:00 (sd-pam) 2024-04-14T23:55:38.551 INFO:tasks.workunit.client.0.smithi060.stdout: 22914 ? 00:00:02 sshd 2024-04-14T23:55:38.551 INFO:tasks.workunit.client.0.smithi060.stdout: 49915 ? 00:00:00 timeout 2024-04-14T23:55:38.551 INFO:tasks.workunit.client.0.smithi060.stdout: 49933 ? 00:00:00 snaptest-double 2024-04-14T23:55:38.551 INFO:tasks.workunit.client.0.smithi060.stdout: 50091 ? 00:00:00 ps 2024-04-14T23:55:38.552 INFO:tasks.workunit.client.0.smithi060.stderr:+ rmdir a/.snap/one a/.snap/two 2024-04-14T23:55:38.575 INFO:tasks.workunit.client.0.smithi060.stderr:+ rm a/foo 2024-04-14T23:55:38.585 INFO:tasks.workunit.client.0.smithi060.stderr:+ rmdir a 2024-04-14T23:55:38.601 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 20` 2024-04-14T23:55:38.601 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir a 2024-04-14T23:55:38.608 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir a/.snap/one 2024-04-14T23:55:38.608 INFO:tasks.workunit.client.0.smithi060.stderr:+ cat 2024-04-14T23:55:38.634 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir a/.snap/two 2024-04-14T23:55:38.661 INFO:tasks.workunit.client.0.smithi060.stderr:+ chmod 777 a/foo 2024-04-14T23:55:38.666 INFO:tasks.workunit.client.0.smithi060.stderr:+ sync 2024-04-14T23:55:38.737 INFO:tasks.workunit.client.0.smithi060.stderr:+ ps 2024-04-14T23:55:38.774 INFO:tasks.workunit.client.0.smithi060.stdout: PID TTY TIME CMD 2024-04-14T23:55:38.774 INFO:tasks.workunit.client.0.smithi060.stdout: 4282 ? 00:00:00 systemd 2024-04-14T23:55:38.774 INFO:tasks.workunit.client.0.smithi060.stdout: 4284 ? 00:00:00 (sd-pam) 2024-04-14T23:55:38.774 INFO:tasks.workunit.client.0.smithi060.stdout: 22914 ? 00:00:02 sshd 2024-04-14T23:55:38.774 INFO:tasks.workunit.client.0.smithi060.stdout: 49915 ? 00:00:00 timeout 2024-04-14T23:55:38.774 INFO:tasks.workunit.client.0.smithi060.stdout: 49933 ? 00:00:00 snaptest-double 2024-04-14T23:55:38.774 INFO:tasks.workunit.client.0.smithi060.stdout: 50101 ? 00:00:00 ps 2024-04-14T23:55:38.777 INFO:tasks.workunit.client.0.smithi060.stderr:+ rmdir a/.snap/one a/.snap/two 2024-04-14T23:55:38.833 INFO:tasks.workunit.client.0.smithi060.stderr:+ rm a/foo 2024-04-14T23:55:38.843 INFO:tasks.workunit.client.0.smithi060.stderr:+ rmdir a 2024-04-14T23:55:38.863 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 20` 2024-04-14T23:55:38.863 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir a 2024-04-14T23:55:38.873 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir a/.snap/one 2024-04-14T23:55:38.873 INFO:tasks.workunit.client.0.smithi060.stderr:+ cat 2024-04-14T23:55:38.925 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir a/.snap/two 2024-04-14T23:55:38.951 INFO:tasks.workunit.client.0.smithi060.stderr:+ chmod 777 a/foo 2024-04-14T23:55:38.955 INFO:tasks.workunit.client.0.smithi060.stderr:+ sync 2024-04-14T23:55:39.421 INFO:tasks.workunit.client.0.smithi060.stderr:+ ps 2024-04-14T23:55:39.446 INFO:tasks.workunit.client.0.smithi060.stdout: PID TTY TIME CMD 2024-04-14T23:55:39.446 INFO:tasks.workunit.client.0.smithi060.stdout: 4282 ? 00:00:00 systemd 2024-04-14T23:55:39.446 INFO:tasks.workunit.client.0.smithi060.stdout: 4284 ? 00:00:00 (sd-pam) 2024-04-14T23:55:39.446 INFO:tasks.workunit.client.0.smithi060.stdout: 22914 ? 00:00:02 sshd 2024-04-14T23:55:39.446 INFO:tasks.workunit.client.0.smithi060.stdout: 49915 ? 00:00:00 timeout 2024-04-14T23:55:39.446 INFO:tasks.workunit.client.0.smithi060.stdout: 49933 ? 00:00:00 snaptest-double 2024-04-14T23:55:39.446 INFO:tasks.workunit.client.0.smithi060.stdout: 50111 ? 00:00:00 ps 2024-04-14T23:55:39.447 INFO:tasks.workunit.client.0.smithi060.stderr:+ rmdir a/.snap/one a/.snap/two 2024-04-14T23:55:39.469 INFO:tasks.workunit.client.0.smithi060.stderr:+ rm a/foo 2024-04-14T23:55:39.476 INFO:tasks.workunit.client.0.smithi060.stderr:+ rmdir a 2024-04-14T23:55:39.489 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 20` 2024-04-14T23:55:39.489 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir a 2024-04-14T23:55:39.497 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir a/.snap/one 2024-04-14T23:55:39.497 INFO:tasks.workunit.client.0.smithi060.stderr:+ cat 2024-04-14T23:55:39.523 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir a/.snap/two 2024-04-14T23:55:39.542 INFO:tasks.workunit.client.0.smithi060.stderr:+ chmod 777 a/foo 2024-04-14T23:55:39.546 INFO:tasks.workunit.client.0.smithi060.stderr:+ sync 2024-04-14T23:55:39.636 INFO:tasks.workunit.client.0.smithi060.stderr:+ ps 2024-04-14T23:55:39.660 INFO:tasks.workunit.client.0.smithi060.stdout: PID TTY TIME CMD 2024-04-14T23:55:39.660 INFO:tasks.workunit.client.0.smithi060.stdout: 4282 ? 00:00:00 systemd 2024-04-14T23:55:39.661 INFO:tasks.workunit.client.0.smithi060.stdout: 4284 ? 00:00:00 (sd-pam) 2024-04-14T23:55:39.661 INFO:tasks.workunit.client.0.smithi060.stdout: 22914 ? 00:00:02 sshd 2024-04-14T23:55:39.661 INFO:tasks.workunit.client.0.smithi060.stdout: 49915 ? 00:00:00 timeout 2024-04-14T23:55:39.661 INFO:tasks.workunit.client.0.smithi060.stdout: 49933 ? 00:00:00 snaptest-double 2024-04-14T23:55:39.661 INFO:tasks.workunit.client.0.smithi060.stdout: 50121 ? 00:00:00 ps 2024-04-14T23:55:39.662 INFO:tasks.workunit.client.0.smithi060.stderr:+ rmdir a/.snap/one a/.snap/two 2024-04-14T23:55:39.681 INFO:tasks.workunit.client.0.smithi060.stderr:+ rm a/foo 2024-04-14T23:55:39.689 INFO:tasks.workunit.client.0.smithi060.stderr:+ rmdir a 2024-04-14T23:55:39.705 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 20` 2024-04-14T23:55:39.706 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir a 2024-04-14T23:55:39.714 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir a/.snap/one 2024-04-14T23:55:39.714 INFO:tasks.workunit.client.0.smithi060.stderr:+ cat 2024-04-14T23:55:39.741 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir a/.snap/two 2024-04-14T23:55:39.764 INFO:tasks.workunit.client.0.smithi060.stderr:+ chmod 777 a/foo 2024-04-14T23:55:39.767 INFO:tasks.workunit.client.0.smithi060.stderr:+ sync 2024-04-14T23:55:39.836 INFO:tasks.workunit.client.0.smithi060.stderr:+ ps 2024-04-14T23:55:39.860 INFO:tasks.workunit.client.0.smithi060.stdout: PID TTY TIME CMD 2024-04-14T23:55:39.860 INFO:tasks.workunit.client.0.smithi060.stdout: 4282 ? 00:00:00 systemd 2024-04-14T23:55:39.860 INFO:tasks.workunit.client.0.smithi060.stdout: 4284 ? 00:00:00 (sd-pam) 2024-04-14T23:55:39.860 INFO:tasks.workunit.client.0.smithi060.stdout: 22914 ? 00:00:02 sshd 2024-04-14T23:55:39.860 INFO:tasks.workunit.client.0.smithi060.stdout: 49915 ? 00:00:00 timeout 2024-04-14T23:55:39.861 INFO:tasks.workunit.client.0.smithi060.stdout: 49933 ? 00:00:00 snaptest-double 2024-04-14T23:55:39.861 INFO:tasks.workunit.client.0.smithi060.stdout: 50131 ? 00:00:00 ps 2024-04-14T23:55:39.862 INFO:tasks.workunit.client.0.smithi060.stderr:+ rmdir a/.snap/one a/.snap/two 2024-04-14T23:55:39.886 INFO:tasks.workunit.client.0.smithi060.stderr:+ rm a/foo 2024-04-14T23:55:39.896 INFO:tasks.workunit.client.0.smithi060.stderr:+ rmdir a 2024-04-14T23:55:39.914 INFO:tasks.workunit.client.0.smithi060.stderr:+ echo OK 2024-04-14T23:55:39.914 INFO:tasks.workunit.client.0.smithi060.stdout:OK 2024-04-14T23:55:39.916 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-14T23:55:39.916 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-14T23:55:40.035 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-estale.sh... 2024-04-14T23:55:40.035 DEBUG:teuthology.orchestra.run.smithi060:workunit test fs/snaps/snaptest-estale.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a9a752df26c63acad72e1b3569fd79a515ca0765 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-estale.sh 2024-04-14T23:55:40.110 INFO:tasks.workunit.client.0.smithi060.stderr:+ mkdir .snap/foo 2024-04-14T23:55:40.133 INFO:tasks.workunit.client.0.smithi060.stdout:We want ENOENT, not ESTALE, here. 2024-04-14T23:55:40.133 INFO:tasks.workunit.client.0.smithi060.stderr:+ echo 'We want ENOENT, not ESTALE, here.' 2024-04-14T23:55:40.134 INFO:tasks.workunit.client.0.smithi060.stderr:++ seq 1 100 2024-04-14T23:55:40.138 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.141 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/1 2024-04-14T23:55:40.141 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.149 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/1': No such file or directory 2024-04-14T23:55:40.150 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.152 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/2 2024-04-14T23:55:40.152 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.156 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/2': No such file or directory 2024-04-14T23:55:40.157 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.158 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/3 2024-04-14T23:55:40.158 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.160 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/3': No such file or directory 2024-04-14T23:55:40.161 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.161 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/4 2024-04-14T23:55:40.162 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.164 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/4': No such file or directory 2024-04-14T23:55:40.164 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.165 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/5 2024-04-14T23:55:40.165 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.168 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/5': No such file or directory 2024-04-14T23:55:40.168 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.169 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/6 2024-04-14T23:55:40.170 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.174 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/6': No such file or directory 2024-04-14T23:55:40.174 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.175 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/7 2024-04-14T23:55:40.176 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.178 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/7': No such file or directory 2024-04-14T23:55:40.178 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.180 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/8 2024-04-14T23:55:40.180 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.182 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/8': No such file or directory 2024-04-14T23:55:40.182 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.183 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/9 2024-04-14T23:55:40.183 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.185 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/9': No such file or directory 2024-04-14T23:55:40.186 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.186 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/10 2024-04-14T23:55:40.187 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.189 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/10': No such file or directory 2024-04-14T23:55:40.189 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.190 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/11 2024-04-14T23:55:40.190 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.193 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/11': No such file or directory 2024-04-14T23:55:40.193 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.194 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/12 2024-04-14T23:55:40.194 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.196 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/12': No such file or directory 2024-04-14T23:55:40.197 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.197 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/13 2024-04-14T23:55:40.198 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.200 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/13': No such file or directory 2024-04-14T23:55:40.201 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.201 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/14 2024-04-14T23:55:40.201 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.204 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/14': No such file or directory 2024-04-14T23:55:40.204 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.205 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/15 2024-04-14T23:55:40.205 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.207 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/15': No such file or directory 2024-04-14T23:55:40.208 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.208 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/16 2024-04-14T23:55:40.209 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.211 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/16': No such file or directory 2024-04-14T23:55:40.211 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.212 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/17 2024-04-14T23:55:40.212 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.215 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/17': No such file or directory 2024-04-14T23:55:40.215 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.216 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/18 2024-04-14T23:55:40.217 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.219 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/18': No such file or directory 2024-04-14T23:55:40.219 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.220 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/19 2024-04-14T23:55:40.220 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.223 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/19': No such file or directory 2024-04-14T23:55:40.223 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.224 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/20 2024-04-14T23:55:40.224 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.227 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/20': No such file or directory 2024-04-14T23:55:40.227 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.228 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/21 2024-04-14T23:55:40.228 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.230 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/21': No such file or directory 2024-04-14T23:55:40.231 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.231 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/22 2024-04-14T23:55:40.232 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.234 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/22': No such file or directory 2024-04-14T23:55:40.235 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.235 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/23 2024-04-14T23:55:40.236 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.239 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/23': No such file or directory 2024-04-14T23:55:40.239 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.240 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/24 2024-04-14T23:55:40.240 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.242 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/24': No such file or directory 2024-04-14T23:55:40.243 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.244 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/25 2024-04-14T23:55:40.244 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.247 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/25': No such file or directory 2024-04-14T23:55:40.248 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.248 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/26 2024-04-14T23:55:40.249 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.251 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/26': No such file or directory 2024-04-14T23:55:40.252 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.252 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/27 2024-04-14T23:55:40.253 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.255 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/27': No such file or directory 2024-04-14T23:55:40.256 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.257 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/28 2024-04-14T23:55:40.257 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.260 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/28': No such file or directory 2024-04-14T23:55:40.260 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.261 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/29 2024-04-14T23:55:40.262 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.264 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/29': No such file or directory 2024-04-14T23:55:40.264 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.266 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/30 2024-04-14T23:55:40.266 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.268 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/30': No such file or directory 2024-04-14T23:55:40.269 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.269 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/31 2024-04-14T23:55:40.270 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.272 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/31': No such file or directory 2024-04-14T23:55:40.272 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.273 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/32 2024-04-14T23:55:40.274 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.276 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/32': No such file or directory 2024-04-14T23:55:40.276 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.278 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/33 2024-04-14T23:55:40.278 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.280 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/33': No such file or directory 2024-04-14T23:55:40.280 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.281 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/34 2024-04-14T23:55:40.282 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.284 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/34': No such file or directory 2024-04-14T23:55:40.284 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.285 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/35 2024-04-14T23:55:40.285 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.288 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/35': No such file or directory 2024-04-14T23:55:40.288 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.290 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/36 2024-04-14T23:55:40.290 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.292 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/36': No such file or directory 2024-04-14T23:55:40.292 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.293 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/37 2024-04-14T23:55:40.293 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.296 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/37': No such file or directory 2024-04-14T23:55:40.296 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.297 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/38 2024-04-14T23:55:40.297 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.300 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/38': No such file or directory 2024-04-14T23:55:40.300 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.301 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/39 2024-04-14T23:55:40.301 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.304 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/39': No such file or directory 2024-04-14T23:55:40.304 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.305 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/40 2024-04-14T23:55:40.305 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.308 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/40': No such file or directory 2024-04-14T23:55:40.308 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.309 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/41 2024-04-14T23:55:40.309 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.312 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/41': No such file or directory 2024-04-14T23:55:40.312 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.313 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/42 2024-04-14T23:55:40.314 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.316 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/42': No such file or directory 2024-04-14T23:55:40.316 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.317 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/43 2024-04-14T23:55:40.317 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.320 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/43': No such file or directory 2024-04-14T23:55:40.321 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.321 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/44 2024-04-14T23:55:40.322 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.324 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/44': No such file or directory 2024-04-14T23:55:40.325 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.325 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/45 2024-04-14T23:55:40.326 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.328 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/45': No such file or directory 2024-04-14T23:55:40.329 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.330 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/46 2024-04-14T23:55:40.330 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.332 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/46': No such file or directory 2024-04-14T23:55:40.333 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.333 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/47 2024-04-14T23:55:40.334 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.336 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/47': No such file or directory 2024-04-14T23:55:40.337 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.337 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/48 2024-04-14T23:55:40.338 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.340 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/48': No such file or directory 2024-04-14T23:55:40.340 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.341 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/49 2024-04-14T23:55:40.342 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.345 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/49': No such file or directory 2024-04-14T23:55:40.346 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.346 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/50 2024-04-14T23:55:40.346 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.348 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/50': No such file or directory 2024-04-14T23:55:40.348 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.349 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/51 2024-04-14T23:55:40.350 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.352 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/51': No such file or directory 2024-04-14T23:55:40.353 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.353 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/52 2024-04-14T23:55:40.354 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.356 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/52': No such file or directory 2024-04-14T23:55:40.357 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.358 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/53 2024-04-14T23:55:40.358 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.362 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/53': No such file or directory 2024-04-14T23:55:40.362 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.362 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/54 2024-04-14T23:55:40.362 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.365 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/54': No such file or directory 2024-04-14T23:55:40.365 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.366 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/55 2024-04-14T23:55:40.366 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.369 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/55': No such file or directory 2024-04-14T23:55:40.369 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.370 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/56 2024-04-14T23:55:40.371 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.373 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/56': No such file or directory 2024-04-14T23:55:40.374 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.374 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/57 2024-04-14T23:55:40.375 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.377 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/57': No such file or directory 2024-04-14T23:55:40.378 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.379 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/58 2024-04-14T23:55:40.379 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.382 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/58': No such file or directory 2024-04-14T23:55:40.383 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.383 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/59 2024-04-14T23:55:40.384 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.387 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/59': No such file or directory 2024-04-14T23:55:40.387 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.388 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/60 2024-04-14T23:55:40.388 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.391 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/60': No such file or directory 2024-04-14T23:55:40.391 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.392 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/61 2024-04-14T23:55:40.393 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.395 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/61': No such file or directory 2024-04-14T23:55:40.396 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.396 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/62 2024-04-14T23:55:40.397 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.399 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/62': No such file or directory 2024-04-14T23:55:40.400 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.401 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/63 2024-04-14T23:55:40.401 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.404 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/63': No such file or directory 2024-04-14T23:55:40.404 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.405 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/64 2024-04-14T23:55:40.405 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.408 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/64': No such file or directory 2024-04-14T23:55:40.408 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.409 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/65 2024-04-14T23:55:40.409 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.412 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/65': No such file or directory 2024-04-14T23:55:40.413 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.413 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/66 2024-04-14T23:55:40.414 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.417 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/66': No such file or directory 2024-04-14T23:55:40.417 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.418 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/67 2024-04-14T23:55:40.419 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.422 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/67': No such file or directory 2024-04-14T23:55:40.422 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.423 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/68 2024-04-14T23:55:40.423 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.426 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/68': No such file or directory 2024-04-14T23:55:40.427 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.428 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/69 2024-04-14T23:55:40.428 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.432 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/69': No such file or directory 2024-04-14T23:55:40.433 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.433 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/70 2024-04-14T23:55:40.434 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.437 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/70': No such file or directory 2024-04-14T23:55:40.437 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.438 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/71 2024-04-14T23:55:40.438 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.441 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/71': No such file or directory 2024-04-14T23:55:40.442 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.442 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/72 2024-04-14T23:55:40.443 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.446 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/72': No such file or directory 2024-04-14T23:55:40.446 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.447 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/73 2024-04-14T23:55:40.448 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.451 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/73': No such file or directory 2024-04-14T23:55:40.454 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.454 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.454 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/74 2024-04-14T23:55:40.459 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/74': No such file or directory 2024-04-14T23:55:40.460 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.462 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/75 2024-04-14T23:55:40.465 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.469 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/75': No such file or directory 2024-04-14T23:55:40.470 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.471 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/76 2024-04-14T23:55:40.472 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.476 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/76': No such file or directory 2024-04-14T23:55:40.476 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.476 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/77 2024-04-14T23:55:40.477 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.479 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/77': No such file or directory 2024-04-14T23:55:40.479 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.480 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/78 2024-04-14T23:55:40.480 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.482 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/78': No such file or directory 2024-04-14T23:55:40.483 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.483 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/79 2024-04-14T23:55:40.484 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.486 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/79': No such file or directory 2024-04-14T23:55:40.486 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.487 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/80 2024-04-14T23:55:40.487 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.489 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/80': No such file or directory 2024-04-14T23:55:40.490 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.490 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/81 2024-04-14T23:55:40.490 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.492 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/81': No such file or directory 2024-04-14T23:55:40.493 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.493 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/82 2024-04-14T23:55:40.494 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.496 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/82': No such file or directory 2024-04-14T23:55:40.496 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.497 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/83 2024-04-14T23:55:40.497 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.500 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/83': No such file or directory 2024-04-14T23:55:40.500 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.501 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/84 2024-04-14T23:55:40.501 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.503 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/84': No such file or directory 2024-04-14T23:55:40.504 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.504 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/85 2024-04-14T23:55:40.504 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.507 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/85': No such file or directory 2024-04-14T23:55:40.507 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.507 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/86 2024-04-14T23:55:40.508 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.510 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/86': No such file or directory 2024-04-14T23:55:40.510 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.511 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/87 2024-04-14T23:55:40.512 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.514 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/87': No such file or directory 2024-04-14T23:55:40.514 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.515 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/88 2024-04-14T23:55:40.515 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.517 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/88': No such file or directory 2024-04-14T23:55:40.518 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.518 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/89 2024-04-14T23:55:40.518 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.521 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/89': No such file or directory 2024-04-14T23:55:40.522 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.522 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/90 2024-04-14T23:55:40.522 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.524 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/90': No such file or directory 2024-04-14T23:55:40.524 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.525 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/91 2024-04-14T23:55:40.525 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.527 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/91': No such file or directory 2024-04-14T23:55:40.528 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.528 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/92 2024-04-14T23:55:40.528 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.531 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/92': No such file or directory 2024-04-14T23:55:40.531 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.532 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/93 2024-04-14T23:55:40.532 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.534 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/93': No such file or directory 2024-04-14T23:55:40.535 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.535 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/94 2024-04-14T23:55:40.536 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.538 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/94': No such file or directory 2024-04-14T23:55:40.538 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.539 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/95 2024-04-14T23:55:40.539 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.541 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/95': No such file or directory 2024-04-14T23:55:40.542 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.542 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/96 2024-04-14T23:55:40.543 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.545 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/96': No such file or directory 2024-04-14T23:55:40.546 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.546 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/97 2024-04-14T23:55:40.547 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.549 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/97': No such file or directory 2024-04-14T23:55:40.549 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.550 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/98 2024-04-14T23:55:40.550 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.552 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/98': No such file or directory 2024-04-14T23:55:40.553 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.553 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/99 2024-04-14T23:55:40.554 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.556 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/99': No such file or directory 2024-04-14T23:55:40.556 INFO:tasks.workunit.client.0.smithi060.stderr:+ for f in `seq 1 100` 2024-04-14T23:55:40.557 INFO:tasks.workunit.client.0.smithi060.stderr:+ stat .snap/foo/100 2024-04-14T23:55:40.557 INFO:tasks.workunit.client.0.smithi060.stderr:+ grep 'No such file' 2024-04-14T23:55:40.560 INFO:tasks.workunit.client.0.smithi060.stdout:stat: cannot statx '.snap/foo/100': No such file or directory 2024-04-14T23:55:40.560 INFO:tasks.workunit.client.0.smithi060.stderr:+ rmdir .snap/foo 2024-04-14T23:55:40.568 INFO:tasks.workunit.client.0.smithi060.stdout:OK 2024-04-14T23:55:40.568 INFO:tasks.workunit.client.0.smithi060.stderr:+ echo OK 2024-04-14T23:55:40.569 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-14T23:55:40.569 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-14T23:55:40.656 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-git-ceph.sh... 2024-04-14T23:55:40.656 DEBUG:teuthology.orchestra.run.smithi060:workunit test fs/snaps/snaptest-git-ceph.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a9a752df26c63acad72e1b3569fd79a515ca0765 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh 2024-04-14T23:55:40.725 INFO:tasks.workunit.client.0.smithi060.stderr:+ set -e 2024-04-14T23:55:40.725 INFO:tasks.workunit.client.0.smithi060.stderr:+ sudo git config --global http.sslVerify false 2024-04-14T23:55:40.776 INFO:tasks.workunit.client.0.smithi060.stderr:+ sudo git config --global http.postBuffer 1048576000 2024-04-14T23:55:40.820 INFO:tasks.workunit.client.0.smithi060.stderr:+ retried=false 2024-04-14T23:55:40.820 INFO:tasks.workunit.client.0.smithi060.stderr:+ trap -- retry EXIT 2024-04-14T23:55:40.820 INFO:tasks.workunit.client.0.smithi060.stderr:+ rm -rf ceph 2024-04-14T23:55:40.823 INFO:tasks.workunit.client.0.smithi060.stderr:+ timeout 1800 git clone https://git.ceph.com/ceph.git 2024-04-14T23:55:40.830 INFO:tasks.workunit.client.0.smithi060.stderr:Cloning into 'ceph'... 2024-04-14T23:55:45.808 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-14T23:55:45.817 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-14T23:56:13.028 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:56:13.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.237+0000 7f5c44a00640 1 Processor -- start 2024-04-14T23:56:13.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.238+0000 7f5c44a00640 1 -- start start 2024-04-14T23:56:13.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.239+0000 7f5c44a00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5c40154cf0 0x7f5c4015a940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:13.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.239+0000 7f5c44a00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c400c5520 con 0x7f5c4006c710 2024-04-14T23:56:13.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.239+0000 7f5c44a00640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c4015b040 con 0x7f5c4006c310 2024-04-14T23:56:13.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.239+0000 7f5c44a00640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f5c401550d0 con 0x7f5c40154cf0 2024-04-14T23:56:13.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.240+0000 7f5c3fe00640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f5c4006c710 0x7f5c400cbd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:42008/0 (socket says 172.21.15.60:42008) 2024-04-14T23:56:13.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.240+0000 7f5c3fe00640 1 -- 172.21.15.60:0/2890441705 learned_addr learned my addr 172.21.15.60:0/2890441705 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:56:13.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.240+0000 7f5c3f400640 1 --2- 172.21.15.60:0/2890441705 >> v2:172.21.15.60:3300/0 conn(0x7f5c40154cf0 0x7f5c4015a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:13.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.240+0000 7f5c3ea00640 1 --1- 172.21.15.60:0/2890441705 >> v1:172.21.15.186:6789/0 conn(0x7f5c4006c310 0x7f5c400cb120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.60:46200/0 (socket says 172.21.15.60:46200) 2024-04-14T23:56:13.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.240+0000 7f5c3f400640 1 -- 172.21.15.60:0/2890441705 >> v1:172.21.15.186:6789/0 conn(0x7f5c4006c310 legacy=0x7f5c400cb120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:13.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.240+0000 7f5c3e000640 1 -- 172.21.15.60:0/2890441705 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3155517291 0 0) 0x7f5c4015b040 con 0x7f5c4006c310 2024-04-14T23:56:13.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.240+0000 7f5c3f400640 1 -- 172.21.15.60:0/2890441705 >> v1:172.21.15.60:6789/0 conn(0x7f5c4006c710 legacy=0x7f5c400cbd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:13.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.240+0000 7f5c3f400640 1 -- 172.21.15.60:0/2890441705 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c4015b300 con 0x7f5c40154cf0 2024-04-14T23:56:13.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.241+0000 7f5c3f400640 1 --2- 172.21.15.60:0/2890441705 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5c40154cf0 0x7f5c4015a940 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5c300097a0 tx=0x7f5c3002b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f174ed732152460d server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:56:13.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.242+0000 7f5c3e000640 1 -- 172.21.15.60:0/2890441705 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c3002f810 con 0x7f5c40154cf0 2024-04-14T23:56:13.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.242+0000 7f5c3e000640 1 -- 172.21.15.60:0/2890441705 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c3002f9b0 con 0x7f5c40154cf0 2024-04-14T23:56:13.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.242+0000 7f5c3e000640 1 -- 172.21.15.60:0/2890441705 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c3002fd80 con 0x7f5c40154cf0 2024-04-14T23:56:13.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.242+0000 7f5c44a00640 1 -- 172.21.15.60:0/2890441705 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5c40154cf0 msgr2=0x7f5c4015a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:13.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.242+0000 7f5c44a00640 1 --2- 172.21.15.60:0/2890441705 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5c40154cf0 0x7f5c4015a940 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5c300097a0 tx=0x7f5c3002b9f0 comp rx=0 tx=0).stop 2024-04-14T23:56:13.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.242+0000 7f5c44a00640 1 -- 172.21.15.60:0/2890441705 shutdown_connections 2024-04-14T23:56:13.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.242+0000 7f5c44a00640 1 --2- 172.21.15.60:0/2890441705 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5c40154cf0 0x7f5c4015a940 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:13.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.242+0000 7f5c44a00640 1 -- 172.21.15.60:0/2890441705 >> 172.21.15.60:0/2890441705 conn(0x7f5c400670f0 msgr2=0x7f5c40067500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:56:13.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.242+0000 7f5c44a00640 1 -- 172.21.15.60:0/2890441705 shutdown_connections 2024-04-14T23:56:13.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.243+0000 7f5c44a00640 1 -- 172.21.15.60:0/2890441705 wait complete. 2024-04-14T23:56:13.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.244+0000 7f5c44a00640 1 Processor -- start 2024-04-14T23:56:13.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.244+0000 7f5c44a00640 1 -- start start 2024-04-14T23:56:13.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.245+0000 7f5c44a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c4006c310 0x7f5c400cec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:13.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.245+0000 7f5c44a00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5c4006c710 0x7f5c401f0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:13.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.245+0000 7f5c44a00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5c40154cf0 0x7f5c400b55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:13.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.245+0000 7f5c44a00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f5c4006bb10 con 0x7f5c4006c710 2024-04-14T23:56:13.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.245+0000 7f5c44a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5c4015b8c0 con 0x7f5c4006c310 2024-04-14T23:56:13.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.245+0000 7f5c44a00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f5c4006bdf0 con 0x7f5c40154cf0 2024-04-14T23:56:13.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.246+0000 7f5c3ea00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5c4006c710 0x7f5c401f0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:13.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.246+0000 7f5c3f400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c4006c310 0x7f5c400cec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:13.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.246+0000 7f5c3f400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c4006c310 0x7f5c400cec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:38456/0 (socket says 172.21.15.60:38456) 2024-04-14T23:56:13.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.246+0000 7f5c3f400640 1 -- 172.21.15.60:0/3276351850 learned_addr learned my addr 172.21.15.60:0/3276351850 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:56:13.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.246+0000 7f5c3fe00640 1 --2- 172.21.15.60:0/3276351850 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5c40154cf0 0x7f5c400b55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:13.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.246+0000 7f5c3ea00640 1 -- 172.21.15.60:0/3276351850 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5c40154cf0 msgr2=0x7f5c400b55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:13.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.246+0000 7f5c3ea00640 1 --2- 172.21.15.60:0/3276351850 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5c40154cf0 0x7f5c400b55f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:13.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.246+0000 7f5c3ea00640 1 -- 172.21.15.60:0/3276351850 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c4006c310 msgr2=0x7f5c400cec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:13.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.246+0000 7f5c3ea00640 1 --2- 172.21.15.60:0/3276351850 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c4006c310 0x7f5c400cec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:13.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.246+0000 7f5c3ea00640 1 -- 172.21.15.60:0/3276351850 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c400cf190 con 0x7f5c4006c710 2024-04-14T23:56:13.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.246+0000 7f5c3f400640 1 --2- 172.21.15.60:0/3276351850 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c4006c310 0x7f5c400cec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:56:13.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.247+0000 7f5c3ea00640 1 --2- 172.21.15.60:0/3276351850 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5c4006c710 0x7f5c401f0610 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f5c2800c410 tx=0x7f5c2800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:56:13.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.247+0000 7f5c37400640 1 -- 172.21.15.60:0/3276351850 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c280074e0 con 0x7f5c4006c710 2024-04-14T23:56:13.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.247+0000 7f5c37400640 1 -- 172.21.15.60:0/3276351850 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c280047e0 con 0x7f5c4006c710 2024-04-14T23:56:13.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.247+0000 7f5c44a00640 1 -- 172.21.15.60:0/3276351850 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c401f0e50 con 0x7f5c4006c710 2024-04-14T23:56:13.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.247+0000 7f5c37400640 1 -- 172.21.15.60:0/3276351850 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c28004bd0 con 0x7f5c4006c710 2024-04-14T23:56:13.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.247+0000 7f5c44a00640 1 -- 172.21.15.60:0/3276351850 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c401f1410 con 0x7f5c4006c710 2024-04-14T23:56:13.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.248+0000 7f5c44a00640 1 -- 172.21.15.60:0/3276351850 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c4004f540 con 0x7f5c4006c710 2024-04-14T23:56:13.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.249+0000 7f5c37400640 1 -- 172.21.15.60:0/3276351850 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f5c28011070 con 0x7f5c4006c710 2024-04-14T23:56:13.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.251+0000 7f5c37400640 1 --2- 172.21.15.60:0/3276351850 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f5c1c06b990 0x7f5c1c06de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:13.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.251+0000 7f5c37400640 1 -- 172.21.15.60:0/3276351850 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6127+0+0 (secure 0 0 0) 0x7f5c2808a4f0 con 0x7f5c4006c710 2024-04-14T23:56:13.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.251+0000 7f5c3f400640 1 --2- 172.21.15.60:0/3276351850 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f5c1c06b990 0x7f5c1c06de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:13.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.252+0000 7f5c3f400640 1 --2- 172.21.15.60:0/3276351850 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f5c1c06b990 0x7f5c1c06de50 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5c30009560 tx=0x7f5c30008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:56:13.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.255+0000 7f5c37400640 1 -- 172.21.15.60:0/3276351850 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f5c28058870 con 0x7f5c4006c710 2024-04-14T23:56:13.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.524+0000 7f5c44a00640 1 -- 172.21.15.60:0/3276351850 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5c4014ed70 con 0x7f5c4006c710 2024-04-14T23:56:13.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.526+0000 7f5c37400640 1 -- 172.21.15.60:0/3276351850 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6839 (secure 0 0 0) 0x7f5c2805c910 con 0x7f5c4006c710 2024-04-14T23:56:13.531 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:56:13.531 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5191,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/2356847578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2356847578},{"type":"v1","addr":"172.21.15.60:6835","nonce":2356847578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:55:13.553698+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4986},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4986":{"gid":4986,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":36,"addr":"172.21.15.60:6833/1971580590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1971580590},{"type":"v1","addr":"172.21.15.60:6833","nonce":1971580590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4986,"qdb_cluster":[4986]},"id":1}]} 2024-04-14T23:56:13.531 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 26 2024-04-14T23:56:13.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.528+0000 7f5c44a00640 1 -- 172.21.15.60:0/3276351850 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f5c1c06b990 msgr2=0x7f5c1c06de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:13.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.528+0000 7f5c44a00640 1 --2- 172.21.15.60:0/3276351850 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f5c1c06b990 0x7f5c1c06de50 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5c30009560 tx=0x7f5c30008030 comp rx=0 tx=0).stop 2024-04-14T23:56:13.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.528+0000 7f5c44a00640 1 -- 172.21.15.60:0/3276351850 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5c4006c710 msgr2=0x7f5c401f0610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:13.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.528+0000 7f5c44a00640 1 --2- 172.21.15.60:0/3276351850 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5c4006c710 0x7f5c401f0610 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f5c2800c410 tx=0x7f5c2800c8d0 comp rx=0 tx=0).stop 2024-04-14T23:56:13.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.528+0000 7f5c44a00640 1 -- 172.21.15.60:0/3276351850 shutdown_connections 2024-04-14T23:56:13.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.528+0000 7f5c44a00640 1 --2- 172.21.15.60:0/3276351850 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f5c1c06b990 0x7f5c1c06de50 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:13.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.528+0000 7f5c44a00640 1 --2- 172.21.15.60:0/3276351850 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5c40154cf0 0x7f5c400b55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:13.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.528+0000 7f5c44a00640 1 --2- 172.21.15.60:0/3276351850 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5c4006c710 0x7f5c401f0610 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:13.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.528+0000 7f5c44a00640 1 --2- 172.21.15.60:0/3276351850 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c4006c310 0x7f5c400cec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:13.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.528+0000 7f5c44a00640 1 -- 172.21.15.60:0/3276351850 >> 172.21.15.60:0/3276351850 conn(0x7f5c400670f0 msgr2=0x7f5c40159270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:56:13.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.528+0000 7f5c44a00640 1 -- 172.21.15.60:0/3276351850 shutdown_connections 2024-04-14T23:56:13.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.528+0000 7f5c44a00640 1 -- 172.21.15.60:0/3276351850 wait complete. 2024-04-14T23:56:13.552 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2024-04-14T23:56:13.552 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-14T23:56:13.552 INFO:teuthology.orchestra.run:waiting for 300 2024-04-14T23:56:13.582 INFO:tasks.ceph.mds.a:Stopped 2024-04-14T23:56:13.582 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:56:13.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.829+0000 7f74acc00640 1 Processor -- start 2024-04-14T23:56:13.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.829+0000 7f74acc00640 1 -- start start 2024-04-14T23:56:13.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.831+0000 7f74acc00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f74a815b800 0x7f74a80cd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:13.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.831+0000 7f74acc00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74a80cdb30 con 0x7f74a8154dd0 2024-04-14T23:56:13.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.831+0000 7f74acc00640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74a815e280 con 0x7f74a80cd6e0 2024-04-14T23:56:13.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.831+0000 7f74acc00640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f74a815f0e0 con 0x7f74a815b800 2024-04-14T23:56:13.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.831+0000 7f74a6a00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f74a815b800 0x7f74a80cd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:13.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.831+0000 7f74a7400640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f74a8154dd0 0x7f74a80cb2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:42024/0 (socket says 172.21.15.60:42024) 2024-04-14T23:56:13.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.831+0000 7f74a6a00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f74a815b800 0x7f74a80cd1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34088/0 (socket says 172.21.15.60:34088) 2024-04-14T23:56:13.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.831+0000 7f74a7400640 1 -- 172.21.15.60:0/1796440684 learned_addr learned my addr 172.21.15.60:0/1796440684 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:56:13.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.831+0000 7f74a6a00640 1 -- 172.21.15.60:0/1796440684 >> v1:172.21.15.186:6789/0 conn(0x7f74a80cd6e0 legacy=0x7f74a80b48f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-14T23:56:13.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.831+0000 7f74a6a00640 1 -- 172.21.15.60:0/1796440684 >> v1:172.21.15.60:6789/0 conn(0x7f74a8154dd0 legacy=0x7f74a80cb2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:13.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.831+0000 7f74a6000640 1 -- 172.21.15.60:0/1796440684 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 20172631 0 0) 0x7f74a80cdb30 con 0x7f74a8154dd0 2024-04-14T23:56:13.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.832+0000 7f74a6a00640 1 -- 172.21.15.60:0/1796440684 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74a815e560 con 0x7f74a815b800 2024-04-14T23:56:13.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.833+0000 7f74a6a00640 1 --2- 172.21.15.60:0/1796440684 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74a815b800 0x7f74a80cd1a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f74940097a0 tx=0x7f749402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d27db1cf2a55bb3c server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:56:13.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.833+0000 7f74a6000640 1 -- 172.21.15.60:0/1796440684 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7494035020 con 0x7f74a815b800 2024-04-14T23:56:13.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.834+0000 7f74a6000640 1 -- 172.21.15.60:0/1796440684 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7494031830 con 0x7f74a815b800 2024-04-14T23:56:13.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.834+0000 7f74a6000640 1 -- 172.21.15.60:0/1796440684 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7494031c00 con 0x7f74a815b800 2024-04-14T23:56:13.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.834+0000 7f74acc00640 1 -- 172.21.15.60:0/1796440684 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74a815b800 msgr2=0x7f74a80cd1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:13.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.834+0000 7f74acc00640 1 --2- 172.21.15.60:0/1796440684 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74a815b800 0x7f74a80cd1a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f74940097a0 tx=0x7f749402b8d0 comp rx=0 tx=0).stop 2024-04-14T23:56:13.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.834+0000 7f74acc00640 1 -- 172.21.15.60:0/1796440684 shutdown_connections 2024-04-14T23:56:13.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.834+0000 7f74acc00640 1 --2- 172.21.15.60:0/1796440684 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74a815b800 0x7f74a80cd1a0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:13.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.834+0000 7f74acc00640 1 -- 172.21.15.60:0/1796440684 >> 172.21.15.60:0/1796440684 conn(0x7f74a8067300 msgr2=0x7f74a8067710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:56:13.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.834+0000 7f74acc00640 1 -- 172.21.15.60:0/1796440684 shutdown_connections 2024-04-14T23:56:13.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.835+0000 7f74acc00640 1 -- 172.21.15.60:0/1796440684 wait complete. 2024-04-14T23:56:13.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.837+0000 7f74acc00640 1 Processor -- start 2024-04-14T23:56:13.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.837+0000 7f74acc00640 1 -- start start 2024-04-14T23:56:13.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.839+0000 7f74acc00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74a80cd6e0 0x7f74a81e7a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:13.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.839+0000 7f74acc00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74a8154dd0 0x7f74a81f0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:13.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.839+0000 7f74acc00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74a815b800 0x7f74a81567c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:13.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.839+0000 7f74acc00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f74a815ee00 con 0x7f74a815b800 2024-04-14T23:56:13.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.839+0000 7f74acc00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f74a815eb20 con 0x7f74a8154dd0 2024-04-14T23:56:13.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.839+0000 7f74acc00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f74a80cb9f0 con 0x7f74a80cd6e0 2024-04-14T23:56:13.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.839+0000 7f74a7400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74a80cd6e0 0x7f74a81e7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:13.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.839+0000 7f74a6a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74a8154dd0 0x7f74a81f0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:13.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.839+0000 7f74a7400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74a80cd6e0 0x7f74a81e7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.60:58944/0 (socket says 172.21.15.60:58944) 2024-04-14T23:56:13.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.839+0000 7f74a7400640 1 -- 172.21.15.60:0/653809869 learned_addr learned my addr 172.21.15.60:0/653809869 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:56:13.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.839+0000 7f74a6a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74a8154dd0 0x7f74a81f0620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:38460/0 (socket says 172.21.15.60:38460) 2024-04-14T23:56:13.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.839+0000 7f74a7e00640 1 --2- 172.21.15.60:0/653809869 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74a815b800 0x7f74a81567c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:13.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.840+0000 7f74a7400640 1 -- 172.21.15.60:0/653809869 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74a8154dd0 msgr2=0x7f74a81f0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:13.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.840+0000 7f74a7400640 1 --2- 172.21.15.60:0/653809869 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74a8154dd0 0x7f74a81f0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:13.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.840+0000 7f74a7400640 1 -- 172.21.15.60:0/653809869 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74a815b800 msgr2=0x7f74a81567c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:13.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.840+0000 7f74a7400640 1 --2- 172.21.15.60:0/653809869 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74a815b800 0x7f74a81567c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:13.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.840+0000 7f74a7400640 1 -- 172.21.15.60:0/653809869 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74a8152220 con 0x7f74a80cd6e0 2024-04-14T23:56:13.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.840+0000 7f74a7400640 1 --2- 172.21.15.60:0/653809869 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74a80cd6e0 0x7f74a81e7a10 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f749800c4d0 tx=0x7f749800c990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:56:13.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.840+0000 7f749f400640 1 -- 172.21.15.60:0/653809869 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7498018070 con 0x7f74a80cd6e0 2024-04-14T23:56:13.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.840+0000 7f749f400640 1 -- 172.21.15.60:0/653809869 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7498007500 con 0x7f74a80cd6e0 2024-04-14T23:56:13.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.840+0000 7f74acc00640 1 -- 172.21.15.60:0/653809869 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74a814fc30 con 0x7f74a80cd6e0 2024-04-14T23:56:13.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.840+0000 7f749f400640 1 -- 172.21.15.60:0/653809869 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7498004a20 con 0x7f74a80cd6e0 2024-04-14T23:56:13.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.840+0000 7f74acc00640 1 -- 172.21.15.60:0/653809869 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f74a8157790 con 0x7f74a80cd6e0 2024-04-14T23:56:13.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.842+0000 7f74acc00640 1 -- 172.21.15.60:0/653809869 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74a804f540 con 0x7f74a80cd6e0 2024-04-14T23:56:13.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.843+0000 7f749f400640 1 -- 172.21.15.60:0/653809869 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f749801b070 con 0x7f74a80cd6e0 2024-04-14T23:56:13.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.843+0000 7f749f400640 1 --2- 172.21.15.60:0/653809869 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f748806b990 0x7f748806de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:13.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.844+0000 7f749f400640 1 -- 172.21.15.60:0/653809869 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(43..43 src has 1..43) ==== 6127+0+0 (secure 0 0 0) 0x7f7498089ca0 con 0x7f74a80cd6e0 2024-04-14T23:56:13.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.844+0000 7f74a6a00640 1 --2- 172.21.15.60:0/653809869 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f748806b990 0x7f748806de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:13.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.844+0000 7f74a6a00640 1 --2- 172.21.15.60:0/653809869 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f748806b990 0x7f748806de50 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f7494036000 tx=0x7f749403d040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:56:13.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:13.848+0000 7f749f400640 1 -- 172.21.15.60:0/653809869 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f74980580b0 con 0x7f74a80cd6e0 2024-04-14T23:56:14.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:14.104+0000 7f74acc00640 1 -- 172.21.15.60:0/653809869 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f74a80cbe40 con 0x7f74a80cd6e0 2024-04-14T23:56:14.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:14.105+0000 7f749f400640 1 -- 172.21.15.60:0/653809869 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6839 (secure 0 0 0) 0x7f749805c150 con 0x7f74a80cd6e0 2024-04-14T23:56:14.110 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:56:14.110 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5191,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/2356847578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2356847578},{"type":"v1","addr":"172.21.15.60:6835","nonce":2356847578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:55:13.553698+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4986},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4986":{"gid":4986,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":36,"addr":"172.21.15.60:6833/1971580590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1971580590},{"type":"v1","addr":"172.21.15.60:6833","nonce":1971580590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4986,"qdb_cluster":[4986]},"id":1}]} 2024-04-14T23:56:14.111 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 26 2024-04-14T23:56:14.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:14.107+0000 7f74acc00640 1 -- 172.21.15.60:0/653809869 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f748806b990 msgr2=0x7f748806de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:14.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:14.107+0000 7f74acc00640 1 --2- 172.21.15.60:0/653809869 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f748806b990 0x7f748806de50 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f7494036000 tx=0x7f749403d040 comp rx=0 tx=0).stop 2024-04-14T23:56:14.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:14.107+0000 7f74acc00640 1 -- 172.21.15.60:0/653809869 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74a80cd6e0 msgr2=0x7f74a81e7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:14.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:14.107+0000 7f74acc00640 1 --2- 172.21.15.60:0/653809869 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74a80cd6e0 0x7f74a81e7a10 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f749800c4d0 tx=0x7f749800c990 comp rx=0 tx=0).stop 2024-04-14T23:56:14.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:14.108+0000 7f74acc00640 1 -- 172.21.15.60:0/653809869 shutdown_connections 2024-04-14T23:56:14.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:14.108+0000 7f74acc00640 1 --2- 172.21.15.60:0/653809869 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f748806b990 0x7f748806de50 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:14.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:14.108+0000 7f74acc00640 1 --2- 172.21.15.60:0/653809869 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f74a815b800 0x7f74a81567c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:14.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:14.108+0000 7f74acc00640 1 --2- 172.21.15.60:0/653809869 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74a8154dd0 0x7f74a81f0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:14.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:14.108+0000 7f74acc00640 1 --2- 172.21.15.60:0/653809869 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74a80cd6e0 0x7f74a81e7a10 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:14.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:14.108+0000 7f74acc00640 1 -- 172.21.15.60:0/653809869 >> 172.21.15.60:0/653809869 conn(0x7f74a8067300 msgr2=0x7f74a8158830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:56:14.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:14.108+0000 7f74acc00640 1 -- 172.21.15.60:0/653809869 shutdown_connections 2024-04-14T23:56:14.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:14.108+0000 7f74acc00640 1 -- 172.21.15.60:0/653809869 wait complete. 2024-04-14T23:56:14.131 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-14T23:56:15.942 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-14T23:56:15.950 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-14T23:56:16.132 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:56:16.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.354+0000 7faac5800640 1 Processor -- start 2024-04-14T23:56:16.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.355+0000 7faac5800640 1 -- start start 2024-04-14T23:56:16.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.356+0000 7faac5800640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7faac00c7080 0x7faac006b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:16.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.356+0000 7faac5800640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faac006b850 con 0x7faac015ba70 2024-04-14T23:56:16.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.356+0000 7faac5800640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faac006bb30 con 0x7faac00c6950 2024-04-14T23:56:16.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.356+0000 7faac5800640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7faac006c740 con 0x7faac00c7080 2024-04-14T23:56:16.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.356+0000 7faabfe00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7faac00c7080 0x7faac006b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:16.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.357+0000 7faac4e00640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7faac015ba70 0x7faac0070d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:42036/0 (socket says 172.21.15.60:42036) 2024-04-14T23:56:16.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.357+0000 7faabfe00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7faac00c7080 0x7faac006b1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34110/0 (socket says 172.21.15.60:34110) 2024-04-14T23:56:16.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.357+0000 7faac4e00640 1 -- 172.21.15.60:0/2329503145 learned_addr learned my addr 172.21.15.60:0/2329503145 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:56:16.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.357+0000 7faabfe00640 1 -- 172.21.15.60:0/2329503145 >> v1:172.21.15.186:6789/0 conn(0x7faac00c6950 legacy=0x7faac015b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:16.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.357+0000 7faabfe00640 1 -- 172.21.15.60:0/2329503145 >> v1:172.21.15.60:6789/0 conn(0x7faac015ba70 legacy=0x7faac0070d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:16.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.357+0000 7faabfe00640 1 -- 172.21.15.60:0/2329503145 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faac006be10 con 0x7faac00c7080 2024-04-14T23:56:16.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.358+0000 7faabfe00640 1 --2- 172.21.15.60:0/2329503145 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faac00c7080 0x7faac006b1c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7faab00097a0 tx=0x7faab002b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ae7d9350b2c9e39 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:56:16.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.358+0000 7faabf400640 1 -- 172.21.15.60:0/2329503145 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faab002f810 con 0x7faac00c7080 2024-04-14T23:56:16.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.359+0000 7faabf400640 1 -- 172.21.15.60:0/2329503145 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faab002f9b0 con 0x7faac00c7080 2024-04-14T23:56:16.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.359+0000 7faabf400640 1 -- 172.21.15.60:0/2329503145 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faab002fd80 con 0x7faac00c7080 2024-04-14T23:56:16.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.359+0000 7faac5800640 1 -- 172.21.15.60:0/2329503145 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faac00c7080 msgr2=0x7faac006b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:16.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.359+0000 7faac5800640 1 --2- 172.21.15.60:0/2329503145 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faac00c7080 0x7faac006b1c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7faab00097a0 tx=0x7faab002b9f0 comp rx=0 tx=0).stop 2024-04-14T23:56:16.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.359+0000 7faac5800640 1 -- 172.21.15.60:0/2329503145 shutdown_connections 2024-04-14T23:56:16.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.359+0000 7faac5800640 1 --2- 172.21.15.60:0/2329503145 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faac00c7080 0x7faac006b1c0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:16.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.359+0000 7faac5800640 1 -- 172.21.15.60:0/2329503145 >> 172.21.15.60:0/2329503145 conn(0x7faac00c8b50 msgr2=0x7faac00c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:56:16.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.359+0000 7faac5800640 1 -- 172.21.15.60:0/2329503145 shutdown_connections 2024-04-14T23:56:16.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.359+0000 7faac5800640 1 -- 172.21.15.60:0/2329503145 wait complete. 2024-04-14T23:56:16.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.361+0000 7faac5800640 1 Processor -- start 2024-04-14T23:56:16.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.362+0000 7faac5800640 1 -- start start 2024-04-14T23:56:16.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.362+0000 7faac5800640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faac00c6950 0x7faac0072020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:16.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.362+0000 7faac5800640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faac00c7080 0x7faac01f08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:16.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.363+0000 7faac5800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faac015ba70 0x7faac01f0e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:16.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.363+0000 7faac5800640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7faac00c7460 con 0x7faac00c7080 2024-04-14T23:56:16.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.363+0000 7faac5800640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7faac006c3d0 con 0x7faac015ba70 2024-04-14T23:56:16.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.363+0000 7faac5800640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7faac00c75e0 con 0x7faac00c6950 2024-04-14T23:56:16.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.363+0000 7faabfe00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faac00c6950 0x7faac0072020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:16.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.363+0000 7faab7e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faac00c7080 0x7faac01f08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:16.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.363+0000 7faac4e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faac015ba70 0x7faac01f0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:16.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.363+0000 7faabfe00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faac00c6950 0x7faac0072020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.60:58958/0 (socket says 172.21.15.60:58958) 2024-04-14T23:56:16.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.363+0000 7faabfe00640 1 -- 172.21.15.60:0/3234772119 learned_addr learned my addr 172.21.15.60:0/3234772119 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:56:16.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.363+0000 7faabfe00640 1 -- 172.21.15.60:0/3234772119 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faac015ba70 msgr2=0x7faac01f0e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:16.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.363+0000 7faabfe00640 1 --2- 172.21.15.60:0/3234772119 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faac015ba70 0x7faac01f0e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:16.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.363+0000 7faabfe00640 1 -- 172.21.15.60:0/3234772119 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faac00c7080 msgr2=0x7faac01f08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:16.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.363+0000 7faabfe00640 1 --2- 172.21.15.60:0/3234772119 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faac00c7080 0x7faac01f08c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:16.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.363+0000 7faabfe00640 1 -- 172.21.15.60:0/3234772119 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faac01f3db0 con 0x7faac00c6950 2024-04-14T23:56:16.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.363+0000 7faab7e00640 1 --2- 172.21.15.60:0/3234772119 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faac00c7080 0x7faac01f08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:56:16.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.364+0000 7faabfe00640 1 --2- 172.21.15.60:0/3234772119 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faac00c6950 0x7faac0072020 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7faab0009890 tx=0x7faab0004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:56:16.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.365+0000 7faabcc00640 1 -- 172.21.15.60:0/3234772119 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faab0046070 con 0x7faac00c6950 2024-04-14T23:56:16.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.365+0000 7faabcc00640 1 -- 172.21.15.60:0/3234772119 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faab0037c90 con 0x7faac00c6950 2024-04-14T23:56:16.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.365+0000 7faabcc00640 1 -- 172.21.15.60:0/3234772119 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faab002fe60 con 0x7faac00c6950 2024-04-14T23:56:16.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.365+0000 7faac5800640 1 -- 172.21.15.60:0/3234772119 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faac01f4090 con 0x7faac00c6950 2024-04-14T23:56:16.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.365+0000 7faac5800640 1 -- 172.21.15.60:0/3234772119 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7faac01f4650 con 0x7faac00c6950 2024-04-14T23:56:16.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.366+0000 7faac5800640 1 -- 172.21.15.60:0/3234772119 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faac0068bc0 con 0x7faac00c6950 2024-04-14T23:56:16.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.367+0000 7faabcc00640 1 -- 172.21.15.60:0/3234772119 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7faab0049070 con 0x7faac00c6950 2024-04-14T23:56:16.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.368+0000 7faabcc00640 1 --2- 172.21.15.60:0/3234772119 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7faa9c06b9c0 0x7faa9c06de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:16.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.368+0000 7faabcc00640 1 -- 172.21.15.60:0/3234772119 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(43..43 src has 1..43) ==== 6127+0+0 (secure 0 0 0) 0x7faab00ac440 con 0x7faac00c6950 2024-04-14T23:56:16.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.368+0000 7faab7e00640 1 --2- 172.21.15.60:0/3234772119 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7faa9c06b9c0 0x7faa9c06de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:16.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.369+0000 7faab7e00640 1 --2- 172.21.15.60:0/3234772119 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7faa9c06b9c0 0x7faa9c06de80 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7faac00c7de0 tx=0x7faaa8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:56:16.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.373+0000 7faabcc00640 1 -- 172.21.15.60:0/3234772119 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7faac0068bc0 con 0x7faac00c6950 2024-04-14T23:56:16.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.624+0000 7faac5800640 1 -- 172.21.15.60:0/3234772119 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faac004f540 con 0x7faac00c6950 2024-04-14T23:56:16.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.626+0000 7faabcc00640 1 -- 172.21.15.60:0/3234772119 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6839 (secure 0 0 0) 0x7faab0033030 con 0x7faac00c6950 2024-04-14T23:56:16.631 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:56:16.631 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5191,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/2356847578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2356847578},{"type":"v1","addr":"172.21.15.60:6835","nonce":2356847578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:55:13.553698+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4986},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4986":{"gid":4986,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":36,"addr":"172.21.15.60:6833/1971580590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1971580590},{"type":"v1","addr":"172.21.15.60:6833","nonce":1971580590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4986,"qdb_cluster":[4986]},"id":1}]} 2024-04-14T23:56:16.631 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 26 2024-04-14T23:56:16.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.628+0000 7faac5800640 1 -- 172.21.15.60:0/3234772119 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7faa9c06b9c0 msgr2=0x7faa9c06de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:16.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.628+0000 7faac5800640 1 --2- 172.21.15.60:0/3234772119 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7faa9c06b9c0 0x7faa9c06de80 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7faac00c7de0 tx=0x7faaa8009040 comp rx=0 tx=0).stop 2024-04-14T23:56:16.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.628+0000 7faac5800640 1 -- 172.21.15.60:0/3234772119 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faac00c6950 msgr2=0x7faac0072020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:16.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.628+0000 7faac5800640 1 --2- 172.21.15.60:0/3234772119 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faac00c6950 0x7faac0072020 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7faab0009890 tx=0x7faab0004730 comp rx=0 tx=0).stop 2024-04-14T23:56:16.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.629+0000 7faac5800640 1 -- 172.21.15.60:0/3234772119 shutdown_connections 2024-04-14T23:56:16.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.629+0000 7faac5800640 1 --2- 172.21.15.60:0/3234772119 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7faa9c06b9c0 0x7faa9c06de80 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:16.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.629+0000 7faac5800640 1 --2- 172.21.15.60:0/3234772119 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faac015ba70 0x7faac01f0e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:16.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.629+0000 7faac5800640 1 --2- 172.21.15.60:0/3234772119 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faac00c7080 0x7faac01f08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:16.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.629+0000 7faac5800640 1 --2- 172.21.15.60:0/3234772119 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faac00c6950 0x7faac0072020 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:16.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.629+0000 7faac5800640 1 -- 172.21.15.60:0/3234772119 >> 172.21.15.60:0/3234772119 conn(0x7faac00c8b50 msgr2=0x7faac00bf600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:56:16.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.633+0000 7faac5800640 1 -- 172.21.15.60:0/3234772119 shutdown_connections 2024-04-14T23:56:16.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:16.633+0000 7faac5800640 1 -- 172.21.15.60:0/3234772119 wait complete. 2024-04-14T23:56:16.658 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-14T23:56:18.660 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:56:18.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:18.887+0000 7f5d45c00640 1 Processor -- start 2024-04-14T23:56:18.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:18.887+0000 7f5d45c00640 1 -- start start 2024-04-14T23:56:18.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:18.889+0000 7f5d45c00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5d40154d90 0x7f5d400c82d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:18.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:18.889+0000 7f5d45c00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5d40155170 0x7f5d4015eb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:18.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:18.889+0000 7f5d45c00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d400c6690 con 0x7f5d400713c0 2024-04-14T23:56:18.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:18.889+0000 7f5d45c00640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f5d4006fd80 con 0x7f5d40155170 2024-04-14T23:56:18.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:18.889+0000 7f5d45c00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f5d4015e670 con 0x7f5d40154d90 2024-04-14T23:56:18.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:18.890+0000 7f5d3e000640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5d40155170 0x7f5d4015eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:18.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:18.890+0000 7f5d3ea00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5d40154d90 0x7f5d400c82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:18.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:18.890+0000 7f5d3e000640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5d40155170 0x7f5d4015eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34138/0 (socket says 172.21.15.60:34138) 2024-04-14T23:56:18.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:18.890+0000 7f5d3e000640 1 -- 172.21.15.60:0/2065219391 learned_addr learned my addr 172.21.15.60:0/2065219391 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:56:18.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:18.890+0000 7f5d3f400640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f5d400713c0 0x7f5d4006b680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:42048/0 (socket says 172.21.15.60:42048) 2024-04-14T23:56:18.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:18.890+0000 7f5d3ea00640 1 -- 172.21.15.60:0/2065219391 >> v2:172.21.15.60:3300/0 conn(0x7f5d40155170 msgr2=0x7f5d4015eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:56:18.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:18.890+0000 7f5d3ea00640 1 --2- 172.21.15.60:0/2065219391 >> v2:172.21.15.60:3300/0 conn(0x7f5d40155170 0x7f5d4015eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:18.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:18.890+0000 7f5d3ea00640 1 -- 172.21.15.60:0/2065219391 >> v1:172.21.15.60:6789/0 conn(0x7f5d400713c0 legacy=0x7f5d4006b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:18.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:18.890+0000 7f5d3ea00640 1 -- 172.21.15.60:0/2065219391 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d400c6970 con 0x7f5d40154d90 2024-04-14T23:56:18.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:18.891+0000 7f5d3ea00640 1 --2- 172.21.15.60:0/2065219391 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5d40154d90 0x7f5d400c82d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5d30009d90 tx=0x7f5d3002bf40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=614959d1a82bd718 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:56:18.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:18.892+0000 7f5d3d600640 1 -- 172.21.15.60:0/2065219391 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d30039070 con 0x7f5d40154d90 2024-04-14T23:56:18.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:18.892+0000 7f5d3d600640 1 -- 172.21.15.60:0/2065219391 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d3002ccf0 con 0x7f5d40154d90 2024-04-14T23:56:18.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:18.892+0000 7f5d3d600640 1 -- 172.21.15.60:0/2065219391 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d30034660 con 0x7f5d40154d90 2024-04-14T23:56:18.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:18.892+0000 7f5d45c00640 1 -- 172.21.15.60:0/2065219391 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5d40154d90 msgr2=0x7f5d400c82d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:18.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:18.892+0000 7f5d45c00640 1 --2- 172.21.15.60:0/2065219391 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5d40154d90 0x7f5d400c82d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5d30009d90 tx=0x7f5d3002bf40 comp rx=0 tx=0).stop 2024-04-14T23:56:18.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:18.892+0000 7f5d45c00640 1 -- 172.21.15.60:0/2065219391 shutdown_connections 2024-04-14T23:56:18.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:18.892+0000 7f5d45c00640 1 --2- 172.21.15.60:0/2065219391 >> v2:172.21.15.60:3300/0 conn(0x7f5d40155170 0x7f5d4015eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:18.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:18.892+0000 7f5d45c00640 1 --2- 172.21.15.60:0/2065219391 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5d40154d90 0x7f5d400c82d0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:18.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:18.892+0000 7f5d45c00640 1 -- 172.21.15.60:0/2065219391 >> 172.21.15.60:0/2065219391 conn(0x7f5d4006d370 msgr2=0x7f5d4006d780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:56:18.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:18.893+0000 7f5d45c00640 1 -- 172.21.15.60:0/2065219391 shutdown_connections 2024-04-14T23:56:18.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:18.893+0000 7f5d45c00640 1 -- 172.21.15.60:0/2065219391 wait complete. 2024-04-14T23:56:18.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:18.894+0000 7f5d45c00640 1 Processor -- start 2024-04-14T23:56:18.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:18.894+0000 7f5d45c00640 1 -- start start 2024-04-14T23:56:18.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:18.895+0000 7f5d45c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d400713c0 0x7f5d400c4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:18.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:18.895+0000 7f5d45c00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5d40154d90 0x7f5d400ca590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:18.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:18.895+0000 7f5d45c00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5d400caad0 0x7f5d401f44b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:18.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:18.895+0000 7f5d45c00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f5d4006bd90 con 0x7f5d400caad0 2024-04-14T23:56:18.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:18.895+0000 7f5d45c00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5d4015f040 con 0x7f5d400713c0 2024-04-14T23:56:18.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:18.895+0000 7f5d45c00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f5d4006bf10 con 0x7f5d40154d90 2024-04-14T23:56:18.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:18.896+0000 7f5d3f400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5d400caad0 0x7f5d401f44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:18.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:18.896+0000 7f5d3f400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5d400caad0 0x7f5d401f44b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34144/0 (socket says 172.21.15.60:34144) 2024-04-14T23:56:18.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:18.896+0000 7f5d3f400640 1 -- 172.21.15.60:0/1683481343 learned_addr learned my addr 172.21.15.60:0/1683481343 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:56:18.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:18.896+0000 7f5d3ea00640 1 --2- 172.21.15.60:0/1683481343 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d400713c0 0x7f5d400c4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:18.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:18.896+0000 7f5d3e000640 1 --2- 172.21.15.60:0/1683481343 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5d40154d90 0x7f5d400ca590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:18.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:18.896+0000 7f5d3f400640 1 -- 172.21.15.60:0/1683481343 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5d40154d90 msgr2=0x7f5d400ca590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:18.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:18.896+0000 7f5d3f400640 1 --2- 172.21.15.60:0/1683481343 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5d40154d90 0x7f5d400ca590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:18.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:18.896+0000 7f5d3f400640 1 -- 172.21.15.60:0/1683481343 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d400713c0 msgr2=0x7f5d400c4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:18.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:18.896+0000 7f5d3f400640 1 --2- 172.21.15.60:0/1683481343 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d400713c0 0x7f5d400c4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:18.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:18.896+0000 7f5d3f400640 1 -- 172.21.15.60:0/1683481343 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d40157520 con 0x7f5d400caad0 2024-04-14T23:56:18.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:18.897+0000 7f5d3f400640 1 --2- 172.21.15.60:0/1683481343 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5d400caad0 0x7f5d401f44b0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f5d3800c430 tx=0x7f5d3800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:56:18.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:18.897+0000 7f5d36a00640 1 -- 172.21.15.60:0/1683481343 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d380074e0 con 0x7f5d400caad0 2024-04-14T23:56:18.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:18.897+0000 7f5d36a00640 1 -- 172.21.15.60:0/1683481343 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d38007680 con 0x7f5d400caad0 2024-04-14T23:56:18.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:18.897+0000 7f5d36a00640 1 -- 172.21.15.60:0/1683481343 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d380049e0 con 0x7f5d400caad0 2024-04-14T23:56:18.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:18.897+0000 7f5d45c00640 1 -- 172.21.15.60:0/1683481343 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d401566e0 con 0x7f5d400caad0 2024-04-14T23:56:18.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:18.897+0000 7f5d45c00640 1 -- 172.21.15.60:0/1683481343 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d40156c80 con 0x7f5d400caad0 2024-04-14T23:56:18.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:18.898+0000 7f5d45c00640 1 -- 172.21.15.60:0/1683481343 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d40076970 con 0x7f5d400caad0 2024-04-14T23:56:18.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:18.899+0000 7f5d36a00640 1 -- 172.21.15.60:0/1683481343 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f5d3801a020 con 0x7f5d400caad0 2024-04-14T23:56:18.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:18.900+0000 7f5d36a00640 1 --2- 172.21.15.60:0/1683481343 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f5d1c06b990 0x7f5d1c06de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:18.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:18.900+0000 7f5d36a00640 1 -- 172.21.15.60:0/1683481343 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6127+0+0 (secure 0 0 0) 0x7f5d3808a190 con 0x7f5d400caad0 2024-04-14T23:56:18.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:18.900+0000 7f5d3ea00640 1 --2- 172.21.15.60:0/1683481343 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f5d1c06b990 0x7f5d1c06de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:18.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:18.901+0000 7f5d3ea00640 1 --2- 172.21.15.60:0/1683481343 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f5d1c06b990 0x7f5d1c06de50 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5d30036000 tx=0x7f5d30030000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:56:18.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:18.905+0000 7f5d36a00640 1 -- 172.21.15.60:0/1683481343 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f5d38058480 con 0x7f5d400caad0 2024-04-14T23:56:19.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:19.139+0000 7f5d45c00640 1 -- 172.21.15.60:0/1683481343 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5d400c60c0 con 0x7f5d400caad0 2024-04-14T23:56:19.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:19.140+0000 7f5d36a00640 1 -- 172.21.15.60:0/1683481343 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6839 (secure 0 0 0) 0x7f5d3805c520 con 0x7f5d400caad0 2024-04-14T23:56:19.146 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:56:19.146 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5191,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/2356847578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2356847578},{"type":"v1","addr":"172.21.15.60:6835","nonce":2356847578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:55:13.553698+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4986},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4986":{"gid":4986,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":36,"addr":"172.21.15.60:6833/1971580590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1971580590},{"type":"v1","addr":"172.21.15.60:6833","nonce":1971580590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4986,"qdb_cluster":[4986]},"id":1}]} 2024-04-14T23:56:19.146 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 26 2024-04-14T23:56:19.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:19.143+0000 7f5d45c00640 1 -- 172.21.15.60:0/1683481343 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f5d1c06b990 msgr2=0x7f5d1c06de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:19.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:19.143+0000 7f5d45c00640 1 --2- 172.21.15.60:0/1683481343 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f5d1c06b990 0x7f5d1c06de50 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5d30036000 tx=0x7f5d30030000 comp rx=0 tx=0).stop 2024-04-14T23:56:19.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:19.144+0000 7f5d45c00640 1 -- 172.21.15.60:0/1683481343 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5d400caad0 msgr2=0x7f5d401f44b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:19.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:19.144+0000 7f5d45c00640 1 --2- 172.21.15.60:0/1683481343 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5d400caad0 0x7f5d401f44b0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f5d3800c430 tx=0x7f5d3800c8f0 comp rx=0 tx=0).stop 2024-04-14T23:56:19.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:19.144+0000 7f5d45c00640 1 -- 172.21.15.60:0/1683481343 shutdown_connections 2024-04-14T23:56:19.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:19.144+0000 7f5d45c00640 1 --2- 172.21.15.60:0/1683481343 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f5d1c06b990 0x7f5d1c06de50 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:19.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:19.144+0000 7f5d45c00640 1 --2- 172.21.15.60:0/1683481343 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5d400caad0 0x7f5d401f44b0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:19.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:19.144+0000 7f5d45c00640 1 --2- 172.21.15.60:0/1683481343 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5d40154d90 0x7f5d400ca590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:19.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:19.144+0000 7f5d45c00640 1 --2- 172.21.15.60:0/1683481343 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d400713c0 0x7f5d400c4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:19.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:19.144+0000 7f5d45c00640 1 -- 172.21.15.60:0/1683481343 >> 172.21.15.60:0/1683481343 conn(0x7f5d4006d370 msgr2=0x7f5d40071860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:56:19.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:19.144+0000 7f5d45c00640 1 -- 172.21.15.60:0/1683481343 shutdown_connections 2024-04-14T23:56:19.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:19.144+0000 7f5d45c00640 1 -- 172.21.15.60:0/1683481343 wait complete. 2024-04-14T23:56:19.168 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-14T23:56:21.169 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:56:21.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.395+0000 7f1275800640 1 Processor -- start 2024-04-14T23:56:21.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.395+0000 7f1275800640 1 -- start start 2024-04-14T23:56:21.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.397+0000 7f1275800640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f127006f1f0 0x7f12700c72a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:21.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.397+0000 7f1275800640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f12700731e0 0x7f12700c77e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:21.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.397+0000 7f1275800640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f127015b0d0 con 0x7f127006f5d0 2024-04-14T23:56:21.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.397+0000 7f1275800640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f12700c6e10 con 0x7f12700731e0 2024-04-14T23:56:21.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.397+0000 7f1275800640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f127015b370 con 0x7f127006f1f0 2024-04-14T23:56:21.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.397+0000 7f126f400640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f12700731e0 0x7f12700c77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:21.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.397+0000 7f126e000640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f127006f5d0 0x7f1270154980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.60:46220/0 (socket says 172.21.15.60:46220) 2024-04-14T23:56:21.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.397+0000 7f126f400640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f12700731e0 0x7f12700c77e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34148/0 (socket says 172.21.15.60:34148) 2024-04-14T23:56:21.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.397+0000 7f126e000640 1 -- 172.21.15.60:0/2337718325 learned_addr learned my addr 172.21.15.60:0/2337718325 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:56:21.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.397+0000 7f126ea00640 1 --2- 172.21.15.60:0/2337718325 >> v2:172.21.15.186:3300/0 conn(0x7f127006f1f0 0x7f12700c72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:21.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.398+0000 7f126d600640 1 -- 172.21.15.60:0/2337718325 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1417839637 0 0) 0x7f127015b0d0 con 0x7f127006f5d0 2024-04-14T23:56:21.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.398+0000 7f126d600640 1 -- 172.21.15.60:0/2337718325 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1254003520 con 0x7f127006f5d0 2024-04-14T23:56:21.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.398+0000 7f126f400640 1 -- 172.21.15.60:0/2337718325 >> v2:172.21.15.186:3300/0 conn(0x7f127006f1f0 msgr2=0x7f12700c72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:56:21.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.398+0000 7f126f400640 1 --2- 172.21.15.60:0/2337718325 >> v2:172.21.15.186:3300/0 conn(0x7f127006f1f0 0x7f12700c72a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:21.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.398+0000 7f126f400640 1 -- 172.21.15.60:0/2337718325 >> v1:172.21.15.186:6789/0 conn(0x7f127006f5d0 legacy=0x7f1270154980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:21.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.398+0000 7f126f400640 1 -- 172.21.15.60:0/2337718325 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f127015b650 con 0x7f12700731e0 2024-04-14T23:56:21.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.399+0000 7f126f400640 1 --2- 172.21.15.60:0/2337718325 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12700731e0 0x7f12700c77e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f1270073940 tx=0x7f126802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4080b69502a44ed5 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:56:21.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.399+0000 7f126d600640 1 -- 172.21.15.60:0/2337718325 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1268039070 con 0x7f12700731e0 2024-04-14T23:56:21.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.399+0000 7f126d600640 1 -- 172.21.15.60:0/2337718325 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f126802cd00 con 0x7f12700731e0 2024-04-14T23:56:21.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.399+0000 7f126d600640 1 -- 172.21.15.60:0/2337718325 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1268034660 con 0x7f12700731e0 2024-04-14T23:56:21.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.400+0000 7f1275800640 1 -- 172.21.15.60:0/2337718325 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12700731e0 msgr2=0x7f12700c77e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:21.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.400+0000 7f1275800640 1 --2- 172.21.15.60:0/2337718325 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12700731e0 0x7f12700c77e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f1270073940 tx=0x7f126802bf50 comp rx=0 tx=0).stop 2024-04-14T23:56:21.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.400+0000 7f1275800640 1 -- 172.21.15.60:0/2337718325 shutdown_connections 2024-04-14T23:56:21.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.400+0000 7f1275800640 1 --2- 172.21.15.60:0/2337718325 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12700731e0 0x7f12700c77e0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:21.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.400+0000 7f1275800640 1 --2- 172.21.15.60:0/2337718325 >> v2:172.21.15.186:3300/0 conn(0x7f127006f1f0 0x7f12700c72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:21.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.400+0000 7f1275800640 1 -- 172.21.15.60:0/2337718325 >> 172.21.15.60:0/2337718325 conn(0x7f12700cc170 msgr2=0x7f12700cc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:56:21.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.400+0000 7f1275800640 1 -- 172.21.15.60:0/2337718325 shutdown_connections 2024-04-14T23:56:21.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.400+0000 7f1275800640 1 -- 172.21.15.60:0/2337718325 wait complete. 2024-04-14T23:56:21.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.402+0000 7f1275800640 1 Processor -- start 2024-04-14T23:56:21.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.402+0000 7f1275800640 1 -- start start 2024-04-14T23:56:21.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.403+0000 7f1275800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f127006f1f0 0x7f1270146540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:21.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.403+0000 7f1275800640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f127006f5d0 0x7f12700c5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:21.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.403+0000 7f1275800640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12700731e0 0x7f12701f03b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:21.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.403+0000 7f1275800640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f1270155090 con 0x7f127006f5d0 2024-04-14T23:56:21.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.403+0000 7f1275800640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1270071180 con 0x7f127006f1f0 2024-04-14T23:56:21.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.403+0000 7f1275800640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f1270155210 con 0x7f12700731e0 2024-04-14T23:56:21.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.403+0000 7f126e000640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f127006f5d0 0x7f12700c5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:21.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.403+0000 7f126ea00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f127006f1f0 0x7f1270146540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:21.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.403+0000 7f126f400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12700731e0 0x7f12701f03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:21.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.403+0000 7f126e000640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f127006f5d0 0x7f12700c5b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34156/0 (socket says 172.21.15.60:34156) 2024-04-14T23:56:21.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.403+0000 7f126e000640 1 -- 172.21.15.60:0/3450587848 learned_addr learned my addr 172.21.15.60:0/3450587848 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:56:21.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.404+0000 7f126e000640 1 -- 172.21.15.60:0/3450587848 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12700731e0 msgr2=0x7f12701f03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:21.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.404+0000 7f126e000640 1 --2- 172.21.15.60:0/3450587848 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12700731e0 0x7f12701f03b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:21.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.404+0000 7f126e000640 1 -- 172.21.15.60:0/3450587848 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f127006f1f0 msgr2=0x7f1270146540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:21.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.404+0000 7f126e000640 1 --2- 172.21.15.60:0/3450587848 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f127006f1f0 0x7f1270146540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:21.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.404+0000 7f126e000640 1 -- 172.21.15.60:0/3450587848 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12700ca4b0 con 0x7f127006f5d0 2024-04-14T23:56:21.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.404+0000 7f126ea00640 1 --2- 172.21.15.60:0/3450587848 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f127006f1f0 0x7f1270146540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:56:21.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.404+0000 7f126e000640 1 --2- 172.21.15.60:0/3450587848 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f127006f5d0 0x7f12700c5b70 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f125c00c390 tx=0x7f125c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:56:21.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.404+0000 7f1266a00640 1 -- 172.21.15.60:0/3450587848 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f125c017070 con 0x7f127006f5d0 2024-04-14T23:56:21.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.404+0000 7f1266a00640 1 -- 172.21.15.60:0/3450587848 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f125c005ce0 con 0x7f127006f5d0 2024-04-14T23:56:21.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.404+0000 7f1266a00640 1 -- 172.21.15.60:0/3450587848 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f125c0072f0 con 0x7f127006f5d0 2024-04-14T23:56:21.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.404+0000 7f1275800640 1 -- 172.21.15.60:0/3450587848 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12701f3b70 con 0x7f127006f5d0 2024-04-14T23:56:21.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.404+0000 7f1275800640 1 -- 172.21.15.60:0/3450587848 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12701f4130 con 0x7f127006f5d0 2024-04-14T23:56:21.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.406+0000 7f1275800640 1 -- 172.21.15.60:0/3450587848 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1270076970 con 0x7f127006f5d0 2024-04-14T23:56:21.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.407+0000 7f1266a00640 1 -- 172.21.15.60:0/3450587848 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f125c01a070 con 0x7f127006f5d0 2024-04-14T23:56:21.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.407+0000 7f1266a00640 1 --2- 172.21.15.60:0/3450587848 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f125006b990 0x7f125006de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:21.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.408+0000 7f1266a00640 1 -- 172.21.15.60:0/3450587848 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6127+0+0 (secure 0 0 0) 0x7f125c089d70 con 0x7f127006f5d0 2024-04-14T23:56:21.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.408+0000 7f126ea00640 1 --2- 172.21.15.60:0/3450587848 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f125006b990 0x7f125006de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:21.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.409+0000 7f126ea00640 1 --2- 172.21.15.60:0/3450587848 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f125006b990 0x7f125006de50 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f1260009e90 tx=0x7f1260009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:56:21.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.413+0000 7f1266a00640 1 -- 172.21.15.60:0/3450587848 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f125c0580f0 con 0x7f127006f5d0 2024-04-14T23:56:21.665 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.660+0000 7f1275800640 1 -- 172.21.15.60:0/3450587848 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1270149b20 con 0x7f127006f5d0 2024-04-14T23:56:21.666 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.661+0000 7f1266a00640 1 -- 172.21.15.60:0/3450587848 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6839 (secure 0 0 0) 0x7f125c05c190 con 0x7f127006f5d0 2024-04-14T23:56:21.667 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:56:21.667 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5191,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/2356847578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2356847578},{"type":"v1","addr":"172.21.15.60:6835","nonce":2356847578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:55:13.553698+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4986},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4986":{"gid":4986,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":36,"addr":"172.21.15.60:6833/1971580590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1971580590},{"type":"v1","addr":"172.21.15.60:6833","nonce":1971580590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4986,"qdb_cluster":[4986]},"id":1}]} 2024-04-14T23:56:21.667 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 26 2024-04-14T23:56:21.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.664+0000 7f1275800640 1 -- 172.21.15.60:0/3450587848 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f125006b990 msgr2=0x7f125006de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:21.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.664+0000 7f1275800640 1 --2- 172.21.15.60:0/3450587848 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f125006b990 0x7f125006de50 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f1260009e90 tx=0x7f1260009190 comp rx=0 tx=0).stop 2024-04-14T23:56:21.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.664+0000 7f1275800640 1 -- 172.21.15.60:0/3450587848 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f127006f5d0 msgr2=0x7f12700c5b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:21.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.664+0000 7f1275800640 1 --2- 172.21.15.60:0/3450587848 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f127006f5d0 0x7f12700c5b70 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f125c00c390 tx=0x7f125c00c850 comp rx=0 tx=0).stop 2024-04-14T23:56:21.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.665+0000 7f1275800640 1 -- 172.21.15.60:0/3450587848 shutdown_connections 2024-04-14T23:56:21.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.665+0000 7f1275800640 1 --2- 172.21.15.60:0/3450587848 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f125006b990 0x7f125006de50 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:21.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.665+0000 7f1275800640 1 --2- 172.21.15.60:0/3450587848 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12700731e0 0x7f12701f03b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:21.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.665+0000 7f1275800640 1 --2- 172.21.15.60:0/3450587848 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f127006f5d0 0x7f12700c5b70 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:21.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.665+0000 7f1275800640 1 --2- 172.21.15.60:0/3450587848 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f127006f1f0 0x7f1270146540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:21.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.665+0000 7f1275800640 1 -- 172.21.15.60:0/3450587848 >> 172.21.15.60:0/3450587848 conn(0x7f12700cc170 msgr2=0x7f127014a040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:56:21.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.665+0000 7f1275800640 1 -- 172.21.15.60:0/3450587848 shutdown_connections 2024-04-14T23:56:21.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:21.665+0000 7f1275800640 1 -- 172.21.15.60:0/3450587848 wait complete. 2024-04-14T23:56:21.690 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-14T23:56:23.691 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:56:23.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:23.917+0000 7f47d4c00640 1 Processor -- start 2024-04-14T23:56:23.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:23.917+0000 7f47d4c00640 1 -- start start 2024-04-14T23:56:23.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:23.919+0000 7f47d4c00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f47d00c63c0 0x7f47d00c6b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:23.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:23.919+0000 7f47d4c00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f47d015c9b0 0x7f47d00c70b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:23.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:23.919+0000 7f47d4c00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47d01530d0 con 0x7f47d0152cf0 2024-04-14T23:56:23.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:23.919+0000 7f47d4c00640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f47d00751f0 con 0x7f47d015c9b0 2024-04-14T23:56:23.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:23.919+0000 7f47d4c00640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f47d00c8370 con 0x7f47d00c63c0 2024-04-14T23:56:23.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:23.919+0000 7f47ce800640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f47d015c9b0 0x7f47d00c70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:23.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:23.920+0000 7f47ce800640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f47d015c9b0 0x7f47d00c70b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45236/0 (socket says 172.21.15.60:45236) 2024-04-14T23:56:23.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:23.920+0000 7f47ce800640 1 -- 172.21.15.60:0/1508149639 learned_addr learned my addr 172.21.15.60:0/1508149639 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:56:23.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:23.920+0000 7f47cde00640 1 --1- 172.21.15.60:0/1508149639 >> v1:172.21.15.60:6789/0 conn(0x7f47d0152cf0 0x7f47d015c2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:47434/0 (socket says 172.21.15.60:47434) 2024-04-14T23:56:23.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:23.920+0000 7f47cd400640 1 --2- 172.21.15.60:0/1508149639 >> v2:172.21.15.186:3300/0 conn(0x7f47d00c63c0 0x7f47d00c6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:23.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:23.920+0000 7f47cd400640 1 -- 172.21.15.60:0/1508149639 >> v2:172.21.15.60:3300/0 conn(0x7f47d015c9b0 msgr2=0x7f47d00c70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:56:23.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:23.920+0000 7f47cd400640 1 --2- 172.21.15.60:0/1508149639 >> v2:172.21.15.60:3300/0 conn(0x7f47d015c9b0 0x7f47d00c70b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:23.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:23.920+0000 7f47cd400640 1 -- 172.21.15.60:0/1508149639 >> v1:172.21.15.60:6789/0 conn(0x7f47d0152cf0 legacy=0x7f47d015c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:23.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:23.920+0000 7f47cd400640 1 -- 172.21.15.60:0/1508149639 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47d00c3810 con 0x7f47d00c63c0 2024-04-14T23:56:23.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:23.920+0000 7f47ce800640 1 --2- 172.21.15.60:0/1508149639 >> v2:172.21.15.60:3300/0 conn(0x7f47d015c9b0 0x7f47d00c70b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:56:23.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:23.921+0000 7f47cd400640 1 --2- 172.21.15.60:0/1508149639 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f47d00c63c0 0x7f47d00c6b70 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f47d0070e80 tx=0x7f47b802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=972cf262b309203b server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:56:23.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:23.922+0000 7f47cca00640 1 -- 172.21.15.60:0/1508149639 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47b8034070 con 0x7f47d00c63c0 2024-04-14T23:56:23.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:23.922+0000 7f47cca00640 1 -- 172.21.15.60:0/1508149639 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47b8030610 con 0x7f47d00c63c0 2024-04-14T23:56:23.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:23.922+0000 7f47cca00640 1 -- 172.21.15.60:0/1508149639 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47b80309e0 con 0x7f47d00c63c0 2024-04-14T23:56:23.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:23.922+0000 7f47d4c00640 1 -- 172.21.15.60:0/1508149639 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f47d00c63c0 msgr2=0x7f47d00c6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:23.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:23.922+0000 7f47d4c00640 1 --2- 172.21.15.60:0/1508149639 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f47d00c63c0 0x7f47d00c6b70 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f47d0070e80 tx=0x7f47b802b960 comp rx=0 tx=0).stop 2024-04-14T23:56:23.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:23.922+0000 7f47d4c00640 1 -- 172.21.15.60:0/1508149639 shutdown_connections 2024-04-14T23:56:23.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:23.922+0000 7f47d4c00640 1 --2- 172.21.15.60:0/1508149639 >> v2:172.21.15.60:3300/0 conn(0x7f47d015c9b0 0x7f47d00c70b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:23.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:23.922+0000 7f47d4c00640 1 --2- 172.21.15.60:0/1508149639 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f47d00c63c0 0x7f47d00c6b70 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:23.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:23.922+0000 7f47d4c00640 1 -- 172.21.15.60:0/1508149639 >> 172.21.15.60:0/1508149639 conn(0x7f47d00cc170 msgr2=0x7f47d00cc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:56:23.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:23.923+0000 7f47d4c00640 1 -- 172.21.15.60:0/1508149639 shutdown_connections 2024-04-14T23:56:23.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:23.923+0000 7f47d4c00640 1 -- 172.21.15.60:0/1508149639 wait complete. 2024-04-14T23:56:23.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:23.925+0000 7f47d4c00640 1 Processor -- start 2024-04-14T23:56:23.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:23.925+0000 7f47d4c00640 1 -- start start 2024-04-14T23:56:23.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:23.926+0000 7f47d4c00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f47d00c63c0 0x7f47d01ef280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:23.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:23.926+0000 7f47d4c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f47d0152cf0 0x7f47d01579a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:23.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:23.926+0000 7f47d4c00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f47d015c9b0 0x7f47d0146400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:23.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:23.926+0000 7f47d4c00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f47d00c88f0 con 0x7f47d015c9b0 2024-04-14T23:56:23.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:23.926+0000 7f47d4c00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f47d00c8610 con 0x7f47d0152cf0 2024-04-14T23:56:23.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:23.926+0000 7f47d4c00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f47d0159450 con 0x7f47d00c63c0 2024-04-14T23:56:23.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:23.926+0000 7f47ce800640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f47d015c9b0 0x7f47d0146400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:23.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:23.927+0000 7f47ce800640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f47d015c9b0 0x7f47d0146400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45238/0 (socket says 172.21.15.60:45238) 2024-04-14T23:56:23.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:23.927+0000 7f47cd400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f47d0152cf0 0x7f47d01579a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:23.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:23.927+0000 7f47ce800640 1 -- 172.21.15.60:0/1204390904 learned_addr learned my addr 172.21.15.60:0/1204390904 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:56:23.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:23.927+0000 7f47cde00640 1 --2- 172.21.15.60:0/1204390904 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f47d00c63c0 0x7f47d01ef280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:23.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:23.927+0000 7f47ce800640 1 -- 172.21.15.60:0/1204390904 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f47d00c63c0 msgr2=0x7f47d01ef280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:23.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:23.927+0000 7f47ce800640 1 --2- 172.21.15.60:0/1204390904 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f47d00c63c0 0x7f47d01ef280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:23.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:23.927+0000 7f47ce800640 1 -- 172.21.15.60:0/1204390904 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f47d0152cf0 msgr2=0x7f47d01579a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:23.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:23.927+0000 7f47ce800640 1 --2- 172.21.15.60:0/1204390904 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f47d0152cf0 0x7f47d01579a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:23.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:23.927+0000 7f47ce800640 1 -- 172.21.15.60:0/1204390904 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47d0159ff0 con 0x7f47d015c9b0 2024-04-14T23:56:23.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:23.927+0000 7f47ce800640 1 --2- 172.21.15.60:0/1204390904 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f47d015c9b0 0x7f47d0146400 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f47c8009e70 tx=0x7f47c800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:56:23.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:23.928+0000 7f47c6000640 1 -- 172.21.15.60:0/1204390904 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47c800cbe0 con 0x7f47d015c9b0 2024-04-14T23:56:23.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:23.928+0000 7f47c6000640 1 -- 172.21.15.60:0/1204390904 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47c800cd80 con 0x7f47d015c9b0 2024-04-14T23:56:23.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:23.928+0000 7f47c6000640 1 -- 172.21.15.60:0/1204390904 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47c8003bb0 con 0x7f47d015c9b0 2024-04-14T23:56:23.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:23.928+0000 7f47d4c00640 1 -- 172.21.15.60:0/1204390904 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47d015a290 con 0x7f47d015c9b0 2024-04-14T23:56:23.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:23.928+0000 7f47d4c00640 1 -- 172.21.15.60:0/1204390904 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47d0145640 con 0x7f47d015c9b0 2024-04-14T23:56:23.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:23.929+0000 7f47d4c00640 1 -- 172.21.15.60:0/1204390904 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47d0076970 con 0x7f47d015c9b0 2024-04-14T23:56:23.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:23.930+0000 7f47c6000640 1 -- 172.21.15.60:0/1204390904 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f47c801b030 con 0x7f47d015c9b0 2024-04-14T23:56:23.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:23.931+0000 7f47c6000640 1 --2- 172.21.15.60:0/1204390904 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f47ac06b9c0 0x7f47ac06de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:23.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:23.931+0000 7f47c6000640 1 -- 172.21.15.60:0/1204390904 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6127+0+0 (secure 0 0 0) 0x7f47c808acd0 con 0x7f47d015c9b0 2024-04-14T23:56:23.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:23.932+0000 7f47cde00640 1 --2- 172.21.15.60:0/1204390904 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f47ac06b9c0 0x7f47ac06de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:23.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:23.932+0000 7f47cde00640 1 --2- 172.21.15.60:0/1204390904 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f47ac06b9c0 0x7f47ac06de80 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f47c0007fd0 tx=0x7f47c0007e80 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:56:23.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:23.938+0000 7f47c6000640 1 -- 172.21.15.60:0/1204390904 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f47c8022020 con 0x7f47d015c9b0 2024-04-14T23:56:24.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:24.187+0000 7f47d4c00640 1 -- 172.21.15.60:0/1204390904 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f47d01562f0 con 0x7f47d015c9b0 2024-04-14T23:56:24.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:24.188+0000 7f47c6000640 1 -- 172.21.15.60:0/1204390904 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6839 (secure 0 0 0) 0x7f47c8010040 con 0x7f47d015c9b0 2024-04-14T23:56:24.194 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:56:24.194 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5191,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/2356847578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2356847578},{"type":"v1","addr":"172.21.15.60:6835","nonce":2356847578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:55:13.553698+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4986},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4986":{"gid":4986,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":36,"addr":"172.21.15.60:6833/1971580590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1971580590},{"type":"v1","addr":"172.21.15.60:6833","nonce":1971580590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4986,"qdb_cluster":[4986]},"id":1}]} 2024-04-14T23:56:24.194 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 26 2024-04-14T23:56:24.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:24.191+0000 7f47d4c00640 1 -- 172.21.15.60:0/1204390904 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f47ac06b9c0 msgr2=0x7f47ac06de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:24.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:24.191+0000 7f47d4c00640 1 --2- 172.21.15.60:0/1204390904 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f47ac06b9c0 0x7f47ac06de80 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f47c0007fd0 tx=0x7f47c0007e80 comp rx=0 tx=0).stop 2024-04-14T23:56:24.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:24.191+0000 7f47d4c00640 1 -- 172.21.15.60:0/1204390904 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f47d015c9b0 msgr2=0x7f47d0146400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:24.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:24.191+0000 7f47d4c00640 1 --2- 172.21.15.60:0/1204390904 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f47d015c9b0 0x7f47d0146400 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f47c8009e70 tx=0x7f47c800c490 comp rx=0 tx=0).stop 2024-04-14T23:56:24.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:24.191+0000 7f47d4c00640 1 -- 172.21.15.60:0/1204390904 shutdown_connections 2024-04-14T23:56:24.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:24.191+0000 7f47d4c00640 1 --2- 172.21.15.60:0/1204390904 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f47ac06b9c0 0x7f47ac06de80 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:24.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:24.191+0000 7f47d4c00640 1 --2- 172.21.15.60:0/1204390904 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f47d015c9b0 0x7f47d0146400 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:24.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:24.191+0000 7f47d4c00640 1 --2- 172.21.15.60:0/1204390904 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f47d0152cf0 0x7f47d01579a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:24.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:24.191+0000 7f47d4c00640 1 --2- 172.21.15.60:0/1204390904 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f47d00c63c0 0x7f47d01ef280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:24.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:24.191+0000 7f47d4c00640 1 -- 172.21.15.60:0/1204390904 >> 172.21.15.60:0/1204390904 conn(0x7f47d00cc170 msgr2=0x7f47d015c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:56:24.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:24.191+0000 7f47d4c00640 1 -- 172.21.15.60:0/1204390904 shutdown_connections 2024-04-14T23:56:24.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:24.191+0000 7f47d4c00640 1 -- 172.21.15.60:0/1204390904 wait complete. 2024-04-14T23:56:24.216 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-14T23:56:26.217 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:56:26.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.445+0000 7fb586000640 1 Processor -- start 2024-04-14T23:56:26.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.445+0000 7fb586000640 1 -- start start 2024-04-14T23:56:26.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.447+0000 7fb586000640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb5800b1e00 0x7fb5800c6950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:26.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.447+0000 7fb586000640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb580073320 0x7fb5800c7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:26.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.447+0000 7fb586000640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb580152fb0 con 0x7fb5800c84e0 2024-04-14T23:56:26.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.447+0000 7fb586000640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fb5800c6540 con 0x7fb580073320 2024-04-14T23:56:26.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.447+0000 7fb586000640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fb5800b21e0 con 0x7fb5800b1e00 2024-04-14T23:56:26.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.448+0000 7fb57fe00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb580073320 0x7fb5800c7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:26.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.448+0000 7fb57fe00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb580073320 0x7fb5800c7fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:38622/0 (socket says 172.21.15.60:38622) 2024-04-14T23:56:26.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.448+0000 7fb57fe00640 1 -- 172.21.15.60:0/1819789417 learned_addr learned my addr 172.21.15.60:0/1819789417 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:56:26.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.448+0000 7fb585600640 1 --1- 172.21.15.60:0/1819789417 >> v1:172.21.15.60:6789/0 conn(0x7fb5800c84e0 0x7fb5800c3d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:47448/0 (socket says 172.21.15.60:47448) 2024-04-14T23:56:26.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.448+0000 7fb584c00640 1 --2- 172.21.15.60:0/1819789417 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb5800b1e00 0x7fb5800c6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:26.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.448+0000 7fb57fe00640 1 -- 172.21.15.60:0/1819789417 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb5800b1e00 msgr2=0x7fb5800c6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:56:26.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.448+0000 7fb57fe00640 1 --2- 172.21.15.60:0/1819789417 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb5800b1e00 0x7fb5800c6950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:26.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.448+0000 7fb57fe00640 1 -- 172.21.15.60:0/1819789417 >> v1:172.21.15.60:6789/0 conn(0x7fb5800c84e0 legacy=0x7fb5800c3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:26.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.448+0000 7fb57fe00640 1 -- 172.21.15.60:0/1819789417 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb580159310 con 0x7fb580073320 2024-04-14T23:56:26.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.449+0000 7fb57fe00640 1 --2- 172.21.15.60:0/1819789417 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb580073320 0x7fb5800c7fa0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb56c009870 tx=0x7fb56c02ba10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a42cf8fe786ceaa9 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:56:26.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.450+0000 7fb57f400640 1 -- 172.21.15.60:0/1819789417 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb56c02f830 con 0x7fb580073320 2024-04-14T23:56:26.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.450+0000 7fb57f400640 1 -- 172.21.15.60:0/1819789417 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb56c02f9d0 con 0x7fb580073320 2024-04-14T23:56:26.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.450+0000 7fb57f400640 1 -- 172.21.15.60:0/1819789417 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb56c02fda0 con 0x7fb580073320 2024-04-14T23:56:26.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.450+0000 7fb586000640 1 -- 172.21.15.60:0/1819789417 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb580073320 msgr2=0x7fb5800c7fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:26.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.450+0000 7fb586000640 1 --2- 172.21.15.60:0/1819789417 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb580073320 0x7fb5800c7fa0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb56c009870 tx=0x7fb56c02ba10 comp rx=0 tx=0).stop 2024-04-14T23:56:26.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.450+0000 7fb586000640 1 -- 172.21.15.60:0/1819789417 shutdown_connections 2024-04-14T23:56:26.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.451+0000 7fb586000640 1 --2- 172.21.15.60:0/1819789417 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb580073320 0x7fb5800c7fa0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:26.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.451+0000 7fb586000640 1 --2- 172.21.15.60:0/1819789417 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb5800b1e00 0x7fb5800c6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:26.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.451+0000 7fb586000640 1 -- 172.21.15.60:0/1819789417 >> 172.21.15.60:0/1819789417 conn(0x7fb5800c8b50 msgr2=0x7fb5800c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:56:26.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.451+0000 7fb586000640 1 -- 172.21.15.60:0/1819789417 shutdown_connections 2024-04-14T23:56:26.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.451+0000 7fb586000640 1 -- 172.21.15.60:0/1819789417 wait complete. 2024-04-14T23:56:26.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.453+0000 7fb586000640 1 Processor -- start 2024-04-14T23:56:26.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.453+0000 7fb586000640 1 -- start start 2024-04-14T23:56:26.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.454+0000 7fb586000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb580073320 0x7fb5801f4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:26.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.454+0000 7fb586000640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5800b1e00 0x7fb5801f0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:26.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.454+0000 7fb586000640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb5800c84e0 0x7fb5800c1e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:26.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.454+0000 7fb586000640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb5800c38b0 con 0x7fb5800b1e00 2024-04-14T23:56:26.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.454+0000 7fb586000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb580159890 con 0x7fb580073320 2024-04-14T23:56:26.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.454+0000 7fb586000640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fb5800c5f80 con 0x7fb5800c84e0 2024-04-14T23:56:26.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.455+0000 7fb57fe00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5800b1e00 0x7fb5801f0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:26.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.455+0000 7fb584c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb580073320 0x7fb5801f4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:26.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.455+0000 7fb57fe00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5800b1e00 0x7fb5801f0b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45252/0 (socket says 172.21.15.60:45252) 2024-04-14T23:56:26.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.455+0000 7fb57fe00640 1 -- 172.21.15.60:0/2011177473 learned_addr learned my addr 172.21.15.60:0/2011177473 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:56:26.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.455+0000 7fb585600640 1 --2- 172.21.15.60:0/2011177473 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb5800c84e0 0x7fb5800c1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:26.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.455+0000 7fb584c00640 1 -- 172.21.15.60:0/2011177473 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb5800c84e0 msgr2=0x7fb5800c1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:26.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.455+0000 7fb584c00640 1 --2- 172.21.15.60:0/2011177473 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb5800c84e0 0x7fb5800c1e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:26.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.455+0000 7fb584c00640 1 -- 172.21.15.60:0/2011177473 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5800b1e00 msgr2=0x7fb5801f0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:26.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.455+0000 7fb584c00640 1 --2- 172.21.15.60:0/2011177473 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5800b1e00 0x7fb5801f0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:26.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.455+0000 7fb584c00640 1 -- 172.21.15.60:0/2011177473 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5800c23a0 con 0x7fb580073320 2024-04-14T23:56:26.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.455+0000 7fb57fe00640 1 --2- 172.21.15.60:0/2011177473 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5800b1e00 0x7fb5801f0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:56:26.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.455+0000 7fb584c00640 1 --2- 172.21.15.60:0/2011177473 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb580073320 0x7fb5801f4ba0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb57400d550 tx=0x7fb57400da10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:56:26.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.456+0000 7fb57cc00640 1 -- 172.21.15.60:0/2011177473 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5740125f0 con 0x7fb580073320 2024-04-14T23:56:26.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.456+0000 7fb57cc00640 1 -- 172.21.15.60:0/2011177473 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb574019070 con 0x7fb580073320 2024-04-14T23:56:26.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.456+0000 7fb586000640 1 -- 172.21.15.60:0/2011177473 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5800b1070 con 0x7fb580073320 2024-04-14T23:56:26.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.456+0000 7fb57cc00640 1 -- 172.21.15.60:0/2011177473 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5740129b0 con 0x7fb580073320 2024-04-14T23:56:26.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.456+0000 7fb586000640 1 -- 172.21.15.60:0/2011177473 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5800b1630 con 0x7fb580073320 2024-04-14T23:56:26.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.457+0000 7fb586000640 1 -- 172.21.15.60:0/2011177473 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb58004f540 con 0x7fb580073320 2024-04-14T23:56:26.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.458+0000 7fb57cc00640 1 -- 172.21.15.60:0/2011177473 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7fb574012b50 con 0x7fb580073320 2024-04-14T23:56:26.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.459+0000 7fb57cc00640 1 --2- 172.21.15.60:0/2011177473 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fb56006b9c0 0x7fb56006de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:26.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.460+0000 7fb57cc00640 1 -- 172.21.15.60:0/2011177473 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6127+0+0 (secure 0 0 0) 0x7fb57408ac00 con 0x7fb580073320 2024-04-14T23:56:26.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.460+0000 7fb57fe00640 1 --2- 172.21.15.60:0/2011177473 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fb56006b9c0 0x7fb56006de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:26.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.460+0000 7fb57fe00640 1 --2- 172.21.15.60:0/2011177473 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fb56006b9c0 0x7fb56006de80 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb56c009870 tx=0x7fb56c008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:56:26.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.464+0000 7fb57cc00640 1 -- 172.21.15.60:0/2011177473 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb574058f80 con 0x7fb580073320 2024-04-14T23:56:26.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.700+0000 7fb586000640 1 -- 172.21.15.60:0/2011177473 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb580156240 con 0x7fb580073320 2024-04-14T23:56:26.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.702+0000 7fb57cc00640 1 -- 172.21.15.60:0/2011177473 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6839 (secure 0 0 0) 0x7fb57405f030 con 0x7fb580073320 2024-04-14T23:56:26.707 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:56:26.707 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5191,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/2356847578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2356847578},{"type":"v1","addr":"172.21.15.60:6835","nonce":2356847578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:55:13.553698+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4986},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4986":{"gid":4986,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":36,"addr":"172.21.15.60:6833/1971580590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1971580590},{"type":"v1","addr":"172.21.15.60:6833","nonce":1971580590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4986,"qdb_cluster":[4986]},"id":1}]} 2024-04-14T23:56:26.707 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 26 2024-04-14T23:56:26.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.704+0000 7fb586000640 1 -- 172.21.15.60:0/2011177473 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fb56006b9c0 msgr2=0x7fb56006de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:26.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.704+0000 7fb586000640 1 --2- 172.21.15.60:0/2011177473 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fb56006b9c0 0x7fb56006de80 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb56c009870 tx=0x7fb56c008030 comp rx=0 tx=0).stop 2024-04-14T23:56:26.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.704+0000 7fb586000640 1 -- 172.21.15.60:0/2011177473 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb580073320 msgr2=0x7fb5801f4ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:26.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.704+0000 7fb586000640 1 --2- 172.21.15.60:0/2011177473 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb580073320 0x7fb5801f4ba0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb57400d550 tx=0x7fb57400da10 comp rx=0 tx=0).stop 2024-04-14T23:56:26.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.704+0000 7fb586000640 1 -- 172.21.15.60:0/2011177473 shutdown_connections 2024-04-14T23:56:26.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.704+0000 7fb586000640 1 --2- 172.21.15.60:0/2011177473 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fb56006b9c0 0x7fb56006de80 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:26.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.704+0000 7fb586000640 1 --2- 172.21.15.60:0/2011177473 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb5800c84e0 0x7fb5800c1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:26.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.704+0000 7fb586000640 1 --2- 172.21.15.60:0/2011177473 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5800b1e00 0x7fb5801f0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:26.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.705+0000 7fb586000640 1 --2- 172.21.15.60:0/2011177473 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb580073320 0x7fb5801f4ba0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:26.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.705+0000 7fb586000640 1 -- 172.21.15.60:0/2011177473 >> 172.21.15.60:0/2011177473 conn(0x7fb5800c8b50 msgr2=0x7fb5800cb1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:56:26.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.705+0000 7fb586000640 1 -- 172.21.15.60:0/2011177473 shutdown_connections 2024-04-14T23:56:26.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:26.705+0000 7fb586000640 1 -- 172.21.15.60:0/2011177473 wait complete. 2024-04-14T23:56:26.728 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-14T23:56:28.729 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:56:28.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:28.954+0000 7fddff200640 1 Processor -- start 2024-04-14T23:56:28.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:28.954+0000 7fddff200640 1 -- start start 2024-04-14T23:56:28.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:28.956+0000 7fddff200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fde000c86d0 0x7fde001625c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:28.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:28.956+0000 7fddff200640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fde000c6950 0x7fde00162b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:28.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:28.956+0000 7fddff200640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde00071bf0 con 0x7fde0015c8a0 2024-04-14T23:56:28.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:28.956+0000 7fddff200640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fde00163040 con 0x7fde000c6950 2024-04-14T23:56:28.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:28.956+0000 7fddff200640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fde0015cc80 con 0x7fde000c86d0 2024-04-14T23:56:28.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:28.956+0000 7fddfe800640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fde000c6950 0x7fde00162b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:28.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:28.956+0000 7fddfd400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fde000c86d0 0x7fde001625c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:28.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:28.956+0000 7fddfde00640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fde0015c8a0 0x7fde00072390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.60:34100/0 (socket says 172.21.15.60:34100) 2024-04-14T23:56:28.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:28.956+0000 7fddfde00640 1 -- 172.21.15.60:0/4081348065 learned_addr learned my addr 172.21.15.60:0/4081348065 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:56:28.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:28.957+0000 7fddfd400640 1 -- 172.21.15.60:0/4081348065 >> v2:172.21.15.60:3300/0 conn(0x7fde000c6950 msgr2=0x7fde00162b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:56:28.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:28.957+0000 7fddfd400640 1 --2- 172.21.15.60:0/4081348065 >> v2:172.21.15.60:3300/0 conn(0x7fde000c6950 0x7fde00162b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:28.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:28.957+0000 7fddfd400640 1 -- 172.21.15.60:0/4081348065 >> v1:172.21.15.186:6789/0 conn(0x7fde0015c8a0 legacy=0x7fde00072390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:28.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:28.957+0000 7fddfd400640 1 -- 172.21.15.60:0/4081348065 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde001632a0 con 0x7fde000c86d0 2024-04-14T23:56:28.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:28.957+0000 7fddfca00640 1 -- 172.21.15.60:0/4081348065 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3440347681 0 0) 0x7fddf0001110 con 0x7fde0015c8a0 2024-04-14T23:56:28.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:28.958+0000 7fddfd400640 1 --2- 172.21.15.60:0/4081348065 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fde000c86d0 0x7fde001625c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fddec0097a0 tx=0x7fddec02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3abbb85f099ca801 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:56:28.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:28.958+0000 7fddfca00640 1 -- 172.21.15.60:0/4081348065 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddec035020 con 0x7fde000c86d0 2024-04-14T23:56:28.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:28.958+0000 7fddfca00640 1 -- 172.21.15.60:0/4081348065 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fddec031810 con 0x7fde000c86d0 2024-04-14T23:56:28.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:28.958+0000 7fddfca00640 1 -- 172.21.15.60:0/4081348065 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddec031be0 con 0x7fde000c86d0 2024-04-14T23:56:28.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:28.959+0000 7fddff200640 1 -- 172.21.15.60:0/4081348065 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fde000c86d0 msgr2=0x7fde001625c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:28.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:28.959+0000 7fddff200640 1 --2- 172.21.15.60:0/4081348065 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fde000c86d0 0x7fde001625c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fddec0097a0 tx=0x7fddec02b8b0 comp rx=0 tx=0).stop 2024-04-14T23:56:28.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:28.959+0000 7fddff200640 1 -- 172.21.15.60:0/4081348065 shutdown_connections 2024-04-14T23:56:28.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:28.959+0000 7fddff200640 1 --2- 172.21.15.60:0/4081348065 >> v2:172.21.15.60:3300/0 conn(0x7fde000c6950 0x7fde00162b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:28.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:28.959+0000 7fddff200640 1 --2- 172.21.15.60:0/4081348065 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fde000c86d0 0x7fde001625c0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:28.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:28.959+0000 7fddff200640 1 -- 172.21.15.60:0/4081348065 >> 172.21.15.60:0/4081348065 conn(0x7fde000c8b50 msgr2=0x7fde000c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:56:28.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:28.959+0000 7fddff200640 1 -- 172.21.15.60:0/4081348065 shutdown_connections 2024-04-14T23:56:28.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:28.959+0000 7fddff200640 1 -- 172.21.15.60:0/4081348065 wait complete. 2024-04-14T23:56:28.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:28.961+0000 7fddff200640 1 Processor -- start 2024-04-14T23:56:28.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:28.961+0000 7fddff200640 1 -- start start 2024-04-14T23:56:28.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:28.962+0000 7fddff200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fde000c6950 0x7fde001f55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:28.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:28.962+0000 7fddff200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fde000c86d0 0x7fde001546b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:28.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:28.962+0000 7fddff200640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fde0015c8a0 0x7fde000cabb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:28.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:28.962+0000 7fddff200640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fde00071e70 con 0x7fde0015c8a0 2024-04-14T23:56:28.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:28.962+0000 7fddff200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fde000c7290 con 0x7fde000c86d0 2024-04-14T23:56:28.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:28.962+0000 7fddff200640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fde0006b1c0 con 0x7fde000c6950 2024-04-14T23:56:28.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:28.962+0000 7fddfe800640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fde0015c8a0 0x7fde000cabb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:28.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:28.962+0000 7fddfe800640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fde0015c8a0 0x7fde000cabb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45276/0 (socket says 172.21.15.60:45276) 2024-04-14T23:56:28.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:28.962+0000 7fddfe800640 1 -- 172.21.15.60:0/762707085 learned_addr learned my addr 172.21.15.60:0/762707085 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:56:28.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:28.962+0000 7fddfd400640 1 --2- 172.21.15.60:0/762707085 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fde000c86d0 0x7fde001546b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:28.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:28.962+0000 7fddfde00640 1 --2- 172.21.15.60:0/762707085 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fde000c6950 0x7fde001f55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:28.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:28.963+0000 7fddfe800640 1 -- 172.21.15.60:0/762707085 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fde000c6950 msgr2=0x7fde001f55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:28.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:28.963+0000 7fddfe800640 1 --2- 172.21.15.60:0/762707085 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fde000c6950 0x7fde001f55c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:28.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:28.963+0000 7fddfe800640 1 -- 172.21.15.60:0/762707085 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fde000c86d0 msgr2=0x7fde001546b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:28.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:28.963+0000 7fddfe800640 1 --2- 172.21.15.60:0/762707085 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fde000c86d0 0x7fde001546b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:28.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:28.963+0000 7fddfe800640 1 -- 172.21.15.60:0/762707085 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde001597a0 con 0x7fde0015c8a0 2024-04-14T23:56:28.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:28.963+0000 7fddfd400640 1 --2- 172.21.15.60:0/762707085 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fde000c86d0 0x7fde001546b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:56:28.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:28.963+0000 7fddfe800640 1 --2- 172.21.15.60:0/762707085 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fde0015c8a0 0x7fde000cabb0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fddf800efc0 tx=0x7fddf800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:56:28.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:28.963+0000 7fddf6000640 1 -- 172.21.15.60:0/762707085 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddf8018020 con 0x7fde0015c8a0 2024-04-14T23:56:28.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:28.963+0000 7fddf6000640 1 -- 172.21.15.60:0/762707085 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fddf8014480 con 0x7fde0015c8a0 2024-04-14T23:56:28.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:28.963+0000 7fddf6000640 1 -- 172.21.15.60:0/762707085 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddf8014870 con 0x7fde0015c8a0 2024-04-14T23:56:28.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:28.963+0000 7fddff200640 1 -- 172.21.15.60:0/762707085 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde00159a80 con 0x7fde0015c8a0 2024-04-14T23:56:28.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:28.964+0000 7fddff200640 1 -- 172.21.15.60:0/762707085 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde001583a0 con 0x7fde0015c8a0 2024-04-14T23:56:28.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:28.965+0000 7fddff200640 1 -- 172.21.15.60:0/762707085 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde0004f540 con 0x7fde0015c8a0 2024-04-14T23:56:28.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:28.966+0000 7fddf6000640 1 -- 172.21.15.60:0/762707085 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7fddf8005d10 con 0x7fde0015c8a0 2024-04-14T23:56:28.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:28.967+0000 7fddf6000640 1 --2- 172.21.15.60:0/762707085 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fdde006b990 0x7fdde006de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:28.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:28.967+0000 7fddf6000640 1 -- 172.21.15.60:0/762707085 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6127+0+0 (secure 0 0 0) 0x7fddf808b7d0 con 0x7fde0015c8a0 2024-04-14T23:56:28.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:28.968+0000 7fddfde00640 1 --2- 172.21.15.60:0/762707085 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fdde006b990 0x7fdde006de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:28.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:28.968+0000 7fddfde00640 1 --2- 172.21.15.60:0/762707085 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fdde006b990 0x7fdde006de50 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fddf0000f60 tx=0x7fddf0007e40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:56:28.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:28.972+0000 7fddf6000640 1 -- 172.21.15.60:0/762707085 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fddf8022030 con 0x7fde0015c8a0 2024-04-14T23:56:29.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.210+0000 7fddff200640 1 -- 172.21.15.60:0/762707085 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fde000c7970 con 0x7fde0015c8a0 2024-04-14T23:56:29.217 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.211+0000 7fddf6000640 1 -- 172.21.15.60:0/762707085 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+6006 (secure 0 0 0) 0x7fddf8010040 con 0x7fde0015c8a0 2024-04-14T23:56:29.217 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:56:29.217 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:56:28.972996+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5191},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5191":{"gid":5191,"name":"c","rank":0,"incarnation":27,"state":"up:replay","state_seq":1,"addr":"172.21.15.60:6835/2356847578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2356847578},{"type":"v1","addr":"172.21.15.60:6835","nonce":2356847578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-14T23:56:29.218 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 27 2024-04-14T23:56:29.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.213+0000 7fddff200640 1 -- 172.21.15.60:0/762707085 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fdde006b990 msgr2=0x7fdde006de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:29.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.213+0000 7fddff200640 1 --2- 172.21.15.60:0/762707085 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fdde006b990 0x7fdde006de50 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fddf0000f60 tx=0x7fddf0007e40 comp rx=0 tx=0).stop 2024-04-14T23:56:29.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.213+0000 7fddff200640 1 -- 172.21.15.60:0/762707085 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fde0015c8a0 msgr2=0x7fde000cabb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:29.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.213+0000 7fddff200640 1 --2- 172.21.15.60:0/762707085 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fde0015c8a0 0x7fde000cabb0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fddf800efc0 tx=0x7fddf800c490 comp rx=0 tx=0).stop 2024-04-14T23:56:29.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.214+0000 7fddff200640 1 -- 172.21.15.60:0/762707085 shutdown_connections 2024-04-14T23:56:29.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.214+0000 7fddff200640 1 --2- 172.21.15.60:0/762707085 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fdde006b990 0x7fdde006de50 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:29.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.214+0000 7fddff200640 1 --2- 172.21.15.60:0/762707085 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fde0015c8a0 0x7fde000cabb0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:29.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.214+0000 7fddff200640 1 --2- 172.21.15.60:0/762707085 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fde000c86d0 0x7fde001546b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:29.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.214+0000 7fddff200640 1 --2- 172.21.15.60:0/762707085 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fde000c6950 0x7fde001f55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:29.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.214+0000 7fddff200640 1 -- 172.21.15.60:0/762707085 >> 172.21.15.60:0/762707085 conn(0x7fde000c8b50 msgr2=0x7fde0015aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:56:29.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.214+0000 7fddff200640 1 -- 172.21.15.60:0/762707085 shutdown_connections 2024-04-14T23:56:29.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.214+0000 7fddff200640 1 -- 172.21.15.60:0/762707085 wait complete. 2024-04-14T23:56:29.241 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2024-04-14T23:56:29.241 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-04-14T23:56:29.241 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:56:29.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.468+0000 7f82a5200640 1 Processor -- start 2024-04-14T23:56:29.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.468+0000 7f82a5200640 1 -- start start 2024-04-14T23:56:29.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.470+0000 7f82a5200640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f82a0073320 0x7f82a00c6950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:29.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.470+0000 7f82a5200640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f82a00c3d90 0x7f82a00c41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:29.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.470+0000 7f82a5200640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82a0152fb0 con 0x7f82a00b1e00 2024-04-14T23:56:29.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.470+0000 7f82a5200640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f82a00c6540 con 0x7f82a00c3d90 2024-04-14T23:56:29.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.470+0000 7f82a5200640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f82a00b21e0 con 0x7f82a0073320 2024-04-14T23:56:29.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.470+0000 7f829fe00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f82a00c3d90 0x7f82a00c41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:29.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.470+0000 7f829f400640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f82a00b1e00 0x7f82a00c7fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:47456/0 (socket says 172.21.15.60:47456) 2024-04-14T23:56:29.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.470+0000 7f829ea00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f82a0073320 0x7f82a00c6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:29.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.470+0000 7f829f400640 1 -- 172.21.15.60:0/1461811670 learned_addr learned my addr 172.21.15.60:0/1461811670 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:56:29.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.471+0000 7f829fe00640 1 -- 172.21.15.60:0/1461811670 >> v2:172.21.15.186:3300/0 conn(0x7f82a0073320 msgr2=0x7f82a00c6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:56:29.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.471+0000 7f829fe00640 1 --2- 172.21.15.60:0/1461811670 >> v2:172.21.15.186:3300/0 conn(0x7f82a0073320 0x7f82a00c6950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:29.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.471+0000 7f829fe00640 1 -- 172.21.15.60:0/1461811670 >> v1:172.21.15.60:6789/0 conn(0x7f82a00b1e00 legacy=0x7f82a00c7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:29.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.471+0000 7f829fe00640 1 -- 172.21.15.60:0/1461811670 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82a0159310 con 0x7f82a00c3d90 2024-04-14T23:56:29.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.472+0000 7f829fe00640 1 --2- 172.21.15.60:0/1461811670 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82a00c3d90 0x7f82a00c41c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f82a006e450 tx=0x7f829802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29dafc2da858dbe2 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:56:29.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.472+0000 7f829e000640 1 -- 172.21.15.60:0/1461811670 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8298039070 con 0x7f82a00c3d90 2024-04-14T23:56:29.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.472+0000 7f829e000640 1 -- 172.21.15.60:0/1461811670 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f829802cd10 con 0x7f82a00c3d90 2024-04-14T23:56:29.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.472+0000 7f829e000640 1 -- 172.21.15.60:0/1461811670 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8298034660 con 0x7f82a00c3d90 2024-04-14T23:56:29.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.473+0000 7f82a5200640 1 -- 172.21.15.60:0/1461811670 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82a00c3d90 msgr2=0x7f82a00c41c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:29.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.473+0000 7f82a5200640 1 --2- 172.21.15.60:0/1461811670 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82a00c3d90 0x7f82a00c41c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f82a006e450 tx=0x7f829802bf60 comp rx=0 tx=0).stop 2024-04-14T23:56:29.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.473+0000 7f82a5200640 1 -- 172.21.15.60:0/1461811670 shutdown_connections 2024-04-14T23:56:29.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.473+0000 7f82a5200640 1 --2- 172.21.15.60:0/1461811670 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82a00c3d90 0x7f82a00c41c0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:29.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.473+0000 7f82a5200640 1 --2- 172.21.15.60:0/1461811670 >> v2:172.21.15.186:3300/0 conn(0x7f82a0073320 0x7f82a00c6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:29.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.473+0000 7f82a5200640 1 -- 172.21.15.60:0/1461811670 >> 172.21.15.60:0/1461811670 conn(0x7f82a00c8b50 msgr2=0x7f82a00c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:56:29.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.473+0000 7f82a5200640 1 -- 172.21.15.60:0/1461811670 shutdown_connections 2024-04-14T23:56:29.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.473+0000 7f82a5200640 1 -- 172.21.15.60:0/1461811670 wait complete. 2024-04-14T23:56:29.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.475+0000 7f82a5200640 1 Processor -- start 2024-04-14T23:56:29.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.475+0000 7f82a5200640 1 -- start start 2024-04-14T23:56:29.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.476+0000 7f82a5200640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82a0073320 0x7f82a01f4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:29.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.476+0000 7f82a5200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f82a00b1e00 0x7f82a00ca830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:29.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.476+0000 7f82a5200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f82a00c3d90 0x7f82a01f0e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:29.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.476+0000 7f82a5200640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f82a00c38b0 con 0x7f82a0073320 2024-04-14T23:56:29.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.476+0000 7f82a5200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f82a0159890 con 0x7f82a00c3d90 2024-04-14T23:56:29.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.476+0000 7f82a5200640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f82a01595b0 con 0x7f82a00b1e00 2024-04-14T23:56:29.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.476+0000 7f829f400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82a0073320 0x7f82a01f4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:29.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.476+0000 7f829fe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f82a00c3d90 0x7f82a01f0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:29.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.476+0000 7f829f400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82a0073320 0x7f82a01f4610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45304/0 (socket says 172.21.15.60:45304) 2024-04-14T23:56:29.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.476+0000 7f829f400640 1 -- 172.21.15.60:0/2883547345 learned_addr learned my addr 172.21.15.60:0/2883547345 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:56:29.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.476+0000 7f829ea00640 1 --2- 172.21.15.60:0/2883547345 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f82a00b1e00 0x7f82a00ca830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:29.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.477+0000 7f829fe00640 1 -- 172.21.15.60:0/2883547345 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f82a00b1e00 msgr2=0x7f82a00ca830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:29.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.477+0000 7f829fe00640 1 --2- 172.21.15.60:0/2883547345 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f82a00b1e00 0x7f82a00ca830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:29.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.477+0000 7f829fe00640 1 -- 172.21.15.60:0/2883547345 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82a0073320 msgr2=0x7f82a01f4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:29.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.477+0000 7f829fe00640 1 --2- 172.21.15.60:0/2883547345 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82a0073320 0x7f82a01f4610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:29.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.477+0000 7f829fe00640 1 -- 172.21.15.60:0/2883547345 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82a0157870 con 0x7f82a00c3d90 2024-04-14T23:56:29.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.477+0000 7f829f400640 1 --2- 172.21.15.60:0/2883547345 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82a0073320 0x7f82a01f4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:56:29.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.477+0000 7f829fe00640 1 --2- 172.21.15.60:0/2883547345 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f82a00c3d90 0x7f82a01f0e40 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f82a006df90 tx=0x7f8298004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:56:29.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.478+0000 7f8297400640 1 -- 172.21.15.60:0/2883547345 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8298045070 con 0x7f82a00c3d90 2024-04-14T23:56:29.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.478+0000 7f8297400640 1 -- 172.21.15.60:0/2883547345 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f829803a040 con 0x7f82a00c3d90 2024-04-14T23:56:29.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.478+0000 7f8297400640 1 -- 172.21.15.60:0/2883547345 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8298039050 con 0x7f82a00c3d90 2024-04-14T23:56:29.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.478+0000 7f82a5200640 1 -- 172.21.15.60:0/2883547345 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82a0072020 con 0x7f82a00c3d90 2024-04-14T23:56:29.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.478+0000 7f82a5200640 1 -- 172.21.15.60:0/2883547345 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82a01560d0 con 0x7f82a00c3d90 2024-04-14T23:56:29.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.479+0000 7f82a5200640 1 -- 172.21.15.60:0/2883547345 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82a004f540 con 0x7f82a00c3d90 2024-04-14T23:56:29.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.480+0000 7f8297400640 1 -- 172.21.15.60:0/2883547345 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f829803e890 con 0x7f82a00c3d90 2024-04-14T23:56:29.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.481+0000 7f8297400640 1 --2- 172.21.15.60:0/2883547345 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f828006b990 0x7f828006de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:29.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.481+0000 7f8297400640 1 -- 172.21.15.60:0/2883547345 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6169+0+0 (secure 0 0 0) 0x7f82980ac8e0 con 0x7f82a00c3d90 2024-04-14T23:56:29.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.481+0000 7f829f400640 1 --2- 172.21.15.60:0/2883547345 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f828006b990 0x7f828006de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:29.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.482+0000 7f829f400640 1 --2- 172.21.15.60:0/2883547345 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f828006b990 0x7f828006de50 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f8290004e60 tx=0x7f8290007f10 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:56:29.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.486+0000 7f8297400640 1 -- 172.21.15.60:0/2883547345 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f829807acc0 con 0x7f82a00c3d90 2024-04-14T23:56:29.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.726+0000 7f82a5200640 1 -- 172.21.15.60:0/2883547345 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f82a006c490 con 0x7f82a00c3d90 2024-04-14T23:56:29.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.727+0000 7f8297400640 1 -- 172.21.15.60:0/2883547345 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+6006 (secure 0 0 0) 0x7f829807ed60 con 0x7f82a00c3d90 2024-04-14T23:56:29.733 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:56:29.733 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:56:28.972996+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5191},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5191":{"gid":5191,"name":"c","rank":0,"incarnation":27,"state":"up:replay","state_seq":1,"addr":"172.21.15.60:6835/2356847578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2356847578},{"type":"v1","addr":"172.21.15.60:6835","nonce":2356847578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-14T23:56:29.733 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 27 2024-04-14T23:56:29.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.729+0000 7f82a5200640 1 -- 172.21.15.60:0/2883547345 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f828006b990 msgr2=0x7f828006de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:29.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.729+0000 7f82a5200640 1 --2- 172.21.15.60:0/2883547345 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f828006b990 0x7f828006de50 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f8290004e60 tx=0x7f8290007f10 comp rx=0 tx=0).stop 2024-04-14T23:56:29.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.729+0000 7f82a5200640 1 -- 172.21.15.60:0/2883547345 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f82a00c3d90 msgr2=0x7f82a01f0e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:29.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.729+0000 7f82a5200640 1 --2- 172.21.15.60:0/2883547345 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f82a00c3d90 0x7f82a01f0e40 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f82a006df90 tx=0x7f8298004a40 comp rx=0 tx=0).stop 2024-04-14T23:56:29.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.729+0000 7f82a5200640 1 -- 172.21.15.60:0/2883547345 shutdown_connections 2024-04-14T23:56:29.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.729+0000 7f82a5200640 1 --2- 172.21.15.60:0/2883547345 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f828006b990 0x7f828006de50 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:29.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.729+0000 7f82a5200640 1 --2- 172.21.15.60:0/2883547345 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f82a00c3d90 0x7f82a01f0e40 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:29.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.729+0000 7f82a5200640 1 --2- 172.21.15.60:0/2883547345 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f82a00b1e00 0x7f82a00ca830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:29.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.729+0000 7f82a5200640 1 --2- 172.21.15.60:0/2883547345 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82a0073320 0x7f82a01f4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:29.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.729+0000 7f82a5200640 1 -- 172.21.15.60:0/2883547345 >> 172.21.15.60:0/2883547345 conn(0x7f82a00c8b50 msgr2=0x7f82a0157ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:56:29.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.730+0000 7f82a5200640 1 -- 172.21.15.60:0/2883547345 shutdown_connections 2024-04-14T23:56:29.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:29.730+0000 7f82a5200640 1 -- 172.21.15.60:0/2883547345 wait complete. 2024-04-14T23:56:29.756 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 27, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-14T23:48:42.700188+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 27, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5191": { "addr": "172.21.15.60:6835/2356847578", "addrs": { "addrvec": [ { "addr": "172.21.15.60:6834", "nonce": 2356847578, "type": "v2" }, { "addr": "172.21.15.60:6835", "nonce": 2356847578, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5191, "incarnation": 27, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 44, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-14T23:56:28.972996+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5191 } } } ], "standbys": [ { "addr": "172.21.15.60:6837/1527815634", "addrs": { "addrvec": [ { "addr": "172.21.15.60:6836", "nonce": 1527815634, "type": "v2" }, { "addr": "172.21.15.60:6837", "nonce": 1527815634, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4197, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6838/3958319486", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6835", "nonce": 3958319486, "type": "v2" }, { "addr": "172.21.15.186:6838", "nonce": 3958319486, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4220, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6837/3910951185", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6834", "nonce": 3910951185, "type": "v2" }, { "addr": "172.21.15.186:6837", "nonce": 3910951185, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4264, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6839/2155020404", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6836", "nonce": 2155020404, "type": "v2" }, { "addr": "172.21.15.186:6839", "nonce": 2155020404, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4267, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-14T23:56:31.757 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:56:31.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:31.988+0000 7fddb3200640 1 Processor -- start 2024-04-14T23:56:31.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:31.988+0000 7fddb3200640 1 -- start start 2024-04-14T23:56:31.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:31.989+0000 7fddb3200640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fddb40c7080 0x7fddb406b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:31.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:31.989+0000 7fddb3200640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fddb40c6950 0x7fddb415b360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:31.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:31.990+0000 7fddb3200640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fddb406b850 con 0x7fddb40c7460 2024-04-14T23:56:31.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:31.990+0000 7fddb3200640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fddb406c740 con 0x7fddb40c6950 2024-04-14T23:56:31.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:31.990+0000 7fddb3200640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fddb406bb30 con 0x7fddb40c7080 2024-04-14T23:56:31.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:31.990+0000 7fddb1400640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fddb40c6950 0x7fddb415b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:31.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:31.990+0000 7fddb1400640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fddb40c6950 0x7fddb415b360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35650/0 (socket says 172.21.15.60:35650) 2024-04-14T23:56:31.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:31.990+0000 7fddb1400640 1 -- 172.21.15.60:0/2938508252 learned_addr learned my addr 172.21.15.60:0/2938508252 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:56:31.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:31.990+0000 7fddb1e00640 1 --2- 172.21.15.60:0/2938508252 >> v2:172.21.15.186:3300/0 conn(0x7fddb40c7080 0x7fddb406b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:31.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:31.990+0000 7fddb2800640 1 --1- 172.21.15.60:0/2938508252 >> v1:172.21.15.186:6789/0 conn(0x7fddb40c7460 0x7fddb4070d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.60:43990/0 (socket says 172.21.15.60:43990) 2024-04-14T23:56:31.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:31.990+0000 7fddb1400640 1 -- 172.21.15.60:0/2938508252 >> v2:172.21.15.186:3300/0 conn(0x7fddb40c7080 msgr2=0x7fddb406b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:56:31.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:31.990+0000 7fddb1400640 1 --2- 172.21.15.60:0/2938508252 >> v2:172.21.15.186:3300/0 conn(0x7fddb40c7080 0x7fddb406b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:31.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:31.990+0000 7fddb1400640 1 -- 172.21.15.60:0/2938508252 >> v1:172.21.15.186:6789/0 conn(0x7fddb40c7460 legacy=0x7fddb4070d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:31.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:31.990+0000 7fddb0a00640 1 -- 172.21.15.60:0/2938508252 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1309762032 0 0) 0x7fddb406b850 con 0x7fddb40c7460 2024-04-14T23:56:31.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:31.990+0000 7fddb1400640 1 -- 172.21.15.60:0/2938508252 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddb406be10 con 0x7fddb40c6950 2024-04-14T23:56:31.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:31.992+0000 7fddb1400640 1 --2- 172.21.15.60:0/2938508252 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fddb40c6950 0x7fddb415b360 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fddb40c9ab0 tx=0x7fdd9c02ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ed48dd5ee074c33 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:56:31.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:31.992+0000 7fddb0a00640 1 -- 172.21.15.60:0/2938508252 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd9c02f850 con 0x7fddb40c6950 2024-04-14T23:56:31.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:31.992+0000 7fddb0a00640 1 -- 172.21.15.60:0/2938508252 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd9c02f9f0 con 0x7fddb40c6950 2024-04-14T23:56:31.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:31.992+0000 7fddb0a00640 1 -- 172.21.15.60:0/2938508252 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd9c02fdc0 con 0x7fddb40c6950 2024-04-14T23:56:31.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:31.993+0000 7fddb3200640 1 -- 172.21.15.60:0/2938508252 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fddb40c6950 msgr2=0x7fddb415b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:31.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:31.993+0000 7fddb3200640 1 --2- 172.21.15.60:0/2938508252 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fddb40c6950 0x7fddb415b360 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fddb40c9ab0 tx=0x7fdd9c02ba30 comp rx=0 tx=0).stop 2024-04-14T23:56:31.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:31.993+0000 7fddb3200640 1 -- 172.21.15.60:0/2938508252 shutdown_connections 2024-04-14T23:56:31.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:31.993+0000 7fddb3200640 1 --2- 172.21.15.60:0/2938508252 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fddb40c6950 0x7fddb415b360 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:31.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:31.993+0000 7fddb3200640 1 --2- 172.21.15.60:0/2938508252 >> v2:172.21.15.186:3300/0 conn(0x7fddb40c7080 0x7fddb406b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:31.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:31.993+0000 7fddb3200640 1 -- 172.21.15.60:0/2938508252 >> 172.21.15.60:0/2938508252 conn(0x7fddb40c8b50 msgr2=0x7fddb40c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:56:31.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:31.993+0000 7fddb3200640 1 -- 172.21.15.60:0/2938508252 shutdown_connections 2024-04-14T23:56:31.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:31.993+0000 7fddb3200640 1 -- 172.21.15.60:0/2938508252 wait complete. 2024-04-14T23:56:31.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:31.995+0000 7fddb3200640 1 Processor -- start 2024-04-14T23:56:32.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:31.995+0000 7fddb3200640 1 -- start start 2024-04-14T23:56:32.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:31.996+0000 7fddb3200640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fddb40c6950 0x7fddb40b0c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:32.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:31.996+0000 7fddb3200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fddb40c7080 0x7fddb41e4290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:32.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:31.996+0000 7fddb3200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fddb40c7460 0x7fddb41f3d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:32.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:31.996+0000 7fddb3200640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fddb4071460 con 0x7fddb40c6950 2024-04-14T23:56:32.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:31.996+0000 7fddb3200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fddb406c3d0 con 0x7fddb40c7460 2024-04-14T23:56:32.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:31.996+0000 7fddb3200640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fddb40715e0 con 0x7fddb40c7080 2024-04-14T23:56:32.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:31.996+0000 7fddb1e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fddb40c6950 0x7fddb40b0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:32.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:31.996+0000 7fddb1e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fddb40c6950 0x7fddb40b0c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35660/0 (socket says 172.21.15.60:35660) 2024-04-14T23:56:32.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:31.996+0000 7fddb1e00640 1 -- 172.21.15.60:0/295698904 learned_addr learned my addr 172.21.15.60:0/295698904 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:56:32.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:31.996+0000 7fddb2800640 1 --2- 172.21.15.60:0/295698904 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fddb40c7460 0x7fddb41f3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:32.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:31.996+0000 7fddb1400640 1 --2- 172.21.15.60:0/295698904 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fddb40c7080 0x7fddb41e4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:32.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:31.996+0000 7fddb1e00640 1 -- 172.21.15.60:0/295698904 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fddb40c7080 msgr2=0x7fddb41e4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:32.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:31.996+0000 7fddb1e00640 1 --2- 172.21.15.60:0/295698904 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fddb40c7080 0x7fddb41e4290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:32.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:31.996+0000 7fddb1e00640 1 -- 172.21.15.60:0/295698904 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fddb40c7460 msgr2=0x7fddb41f3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:32.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:31.996+0000 7fddb1e00640 1 --2- 172.21.15.60:0/295698904 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fddb40c7460 0x7fddb41f3d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:32.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:31.996+0000 7fddb1e00640 1 -- 172.21.15.60:0/295698904 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddb40cb0c0 con 0x7fddb40c6950 2024-04-14T23:56:32.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:31.996+0000 7fddb1400640 1 --2- 172.21.15.60:0/295698904 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fddb40c7080 0x7fddb41e4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-14T23:56:32.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:31.997+0000 7fddb1e00640 1 --2- 172.21.15.60:0/295698904 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fddb40c6950 0x7fddb40b0c40 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fddac00efc0 tx=0x7fddac00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:56:32.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:31.997+0000 7fddaa000640 1 -- 172.21.15.60:0/295698904 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddac018020 con 0x7fddb40c6950 2024-04-14T23:56:32.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:31.997+0000 7fddaa000640 1 -- 172.21.15.60:0/295698904 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fddac014480 con 0x7fddb40c6950 2024-04-14T23:56:32.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:31.997+0000 7fddaa000640 1 -- 172.21.15.60:0/295698904 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddac014850 con 0x7fddb40c6950 2024-04-14T23:56:32.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:31.997+0000 7fddb3200640 1 -- 172.21.15.60:0/295698904 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddb41e0ef0 con 0x7fddb40c6950 2024-04-14T23:56:32.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:31.997+0000 7fddb3200640 1 -- 172.21.15.60:0/295698904 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fddb40cade0 con 0x7fddb40c6950 2024-04-14T23:56:32.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:31.999+0000 7fddb3200640 1 -- 172.21.15.60:0/295698904 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fddb404f540 con 0x7fddb40c6950 2024-04-14T23:56:32.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:31.999+0000 7fddaa000640 1 -- 172.21.15.60:0/295698904 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7fddac005d10 con 0x7fddb40c6950 2024-04-14T23:56:32.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:32.000+0000 7fddaa000640 1 --2- 172.21.15.60:0/295698904 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fdd9006b990 0x7fdd9006de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:32.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:32.000+0000 7fddaa000640 1 -- 172.21.15.60:0/295698904 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6169+0+0 (secure 0 0 0) 0x7fddac08b8f0 con 0x7fddb40c6950 2024-04-14T23:56:32.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:32.001+0000 7fddb1400640 1 --2- 172.21.15.60:0/295698904 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fdd9006b990 0x7fdd9006de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:32.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:32.002+0000 7fddb1400640 1 --2- 172.21.15.60:0/295698904 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fdd9006b990 0x7fdd9006de50 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fdd9c036000 tx=0x7fdd9c008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:56:32.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:32.005+0000 7fddaa000640 1 -- 172.21.15.60:0/295698904 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fddac022030 con 0x7fddb40c6950 2024-04-14T23:56:32.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:32.243+0000 7fddb3200640 1 -- 172.21.15.60:0/295698904 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fddb40c7970 con 0x7fddb40c6950 2024-04-14T23:56:32.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:32.244+0000 7fddaa000640 1 -- 172.21.15.60:0/295698904 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+6006 (secure 0 0 0) 0x7fddac010040 con 0x7fddb40c6950 2024-04-14T23:56:32.250 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:56:32.250 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:56:28.972996+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5191},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5191":{"gid":5191,"name":"c","rank":0,"incarnation":27,"state":"up:replay","state_seq":1,"addr":"172.21.15.60:6835/2356847578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2356847578},{"type":"v1","addr":"172.21.15.60:6835","nonce":2356847578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-14T23:56:32.250 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 27 2024-04-14T23:56:32.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:32.247+0000 7fddb3200640 1 -- 172.21.15.60:0/295698904 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fdd9006b990 msgr2=0x7fdd9006de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:32.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:32.247+0000 7fddb3200640 1 --2- 172.21.15.60:0/295698904 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fdd9006b990 0x7fdd9006de50 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fdd9c036000 tx=0x7fdd9c008030 comp rx=0 tx=0).stop 2024-04-14T23:56:32.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:32.247+0000 7fddb3200640 1 -- 172.21.15.60:0/295698904 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fddb40c6950 msgr2=0x7fddb40b0c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:32.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:32.247+0000 7fddb3200640 1 --2- 172.21.15.60:0/295698904 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fddb40c6950 0x7fddb40b0c40 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fddac00efc0 tx=0x7fddac00c490 comp rx=0 tx=0).stop 2024-04-14T23:56:32.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:32.248+0000 7fddb3200640 1 -- 172.21.15.60:0/295698904 shutdown_connections 2024-04-14T23:56:32.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:32.248+0000 7fddb3200640 1 --2- 172.21.15.60:0/295698904 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fdd9006b990 0x7fdd9006de50 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:32.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:32.248+0000 7fddb3200640 1 --2- 172.21.15.60:0/295698904 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fddb40c7460 0x7fddb41f3d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:32.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:32.248+0000 7fddb3200640 1 --2- 172.21.15.60:0/295698904 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fddb40c7080 0x7fddb41e4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:32.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:32.248+0000 7fddb3200640 1 --2- 172.21.15.60:0/295698904 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fddb40c6950 0x7fddb40b0c40 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:32.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:32.248+0000 7fddb3200640 1 -- 172.21.15.60:0/295698904 >> 172.21.15.60:0/295698904 conn(0x7fddb40c8b50 msgr2=0x7fddb40c45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:56:32.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:32.248+0000 7fddb3200640 1 -- 172.21.15.60:0/295698904 shutdown_connections 2024-04-14T23:56:32.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:32.248+0000 7fddb3200640 1 -- 172.21.15.60:0/295698904 wait complete. 2024-04-14T23:56:32.270 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-14T23:56:34.271 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:56:34.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.509+0000 7fc4b1a00640 1 Processor -- start 2024-04-14T23:56:34.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.509+0000 7fc4b1a00640 1 -- start start 2024-04-14T23:56:34.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.510+0000 7fc4b1a00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc4ac0cd540 0x7fc4ac15ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:34.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.510+0000 7fc4b1a00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4ac0b3f50 con 0x7fc4ac0cb750 2024-04-14T23:56:34.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.510+0000 7fc4b1a00640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4ac0b39b0 con 0x7fc4ac0cd160 2024-04-14T23:56:34.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.510+0000 7fc4b1a00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fc4ac15b370 con 0x7fc4ac0cd540 2024-04-14T23:56:34.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.511+0000 7fc4aaa00640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fc4ac0cd160 0x7fc4ac0b3190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.60:44000/0 (socket says 172.21.15.60:44000) 2024-04-14T23:56:34.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.511+0000 7fc4aaa00640 1 -- 172.21.15.60:0/3072567083 learned_addr learned my addr 172.21.15.60:0/3072567083 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:56:34.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.511+0000 7fc4aa000640 1 --2- 172.21.15.60:0/3072567083 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc4ac0cd540 0x7fc4ac15ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:34.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.511+0000 7fc4aa000640 1 -- 172.21.15.60:0/3072567083 >> v1:172.21.15.186:6789/0 conn(0x7fc4ac0cd160 legacy=0x7fc4ac0b3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:34.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.511+0000 7fc4aa000640 1 -- 172.21.15.60:0/3072567083 >> v1:172.21.15.60:6789/0 conn(0x7fc4ac0cb750 legacy=0x7fc4ac154b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:34.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.511+0000 7fc4aa000640 1 -- 172.21.15.60:0/3072567083 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4ac0b3c70 con 0x7fc4ac0cd540 2024-04-14T23:56:34.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.511+0000 7fc4a9600640 1 -- 172.21.15.60:0/3072567083 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1343793766 0 0) 0x7fc4ac0b39b0 con 0x7fc4ac0cd160 2024-04-14T23:56:34.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.513+0000 7fc4aa000640 1 --2- 172.21.15.60:0/3072567083 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc4ac0cd540 0x7fc4ac15ae30 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fc4980097d0 tx=0x7fc49802b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e027e01b333850de server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:56:34.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.513+0000 7fc4a9600640 1 -- 172.21.15.60:0/3072567083 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc498034070 con 0x7fc4ac0cd540 2024-04-14T23:56:34.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.513+0000 7fc4a9600640 1 -- 172.21.15.60:0/3072567083 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc498030800 con 0x7fc4ac0cd540 2024-04-14T23:56:34.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.513+0000 7fc4a9600640 1 -- 172.21.15.60:0/3072567083 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc498030bd0 con 0x7fc4ac0cd540 2024-04-14T23:56:34.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.514+0000 7fc4b1a00640 1 -- 172.21.15.60:0/3072567083 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc4ac0cd540 msgr2=0x7fc4ac15ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:34.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.514+0000 7fc4b1a00640 1 --2- 172.21.15.60:0/3072567083 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc4ac0cd540 0x7fc4ac15ae30 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fc4980097d0 tx=0x7fc49802b940 comp rx=0 tx=0).stop 2024-04-14T23:56:34.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.514+0000 7fc4b1a00640 1 -- 172.21.15.60:0/3072567083 shutdown_connections 2024-04-14T23:56:34.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.514+0000 7fc4b1a00640 1 --2- 172.21.15.60:0/3072567083 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc4ac0cd540 0x7fc4ac15ae30 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:34.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.514+0000 7fc4b1a00640 1 -- 172.21.15.60:0/3072567083 >> 172.21.15.60:0/3072567083 conn(0x7fc4ac066f10 msgr2=0x7fc4ac0672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:56:34.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.514+0000 7fc4b1a00640 1 -- 172.21.15.60:0/3072567083 shutdown_connections 2024-04-14T23:56:34.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.514+0000 7fc4b1a00640 1 -- 172.21.15.60:0/3072567083 wait complete. 2024-04-14T23:56:34.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.516+0000 7fc4b1a00640 1 Processor -- start 2024-04-14T23:56:34.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.516+0000 7fc4b1a00640 1 -- start start 2024-04-14T23:56:34.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.517+0000 7fc4b1a00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4ac0cb750 0x7fc4ac14d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:34.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.517+0000 7fc4b1a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc4ac0cd160 0x7fc4ac15a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:34.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.517+0000 7fc4b1a00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc4ac0cd540 0x7fc4ac0cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:34.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.517+0000 7fc4b1a00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ac15ea70 con 0x7fc4ac0cb750 2024-04-14T23:56:34.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.517+0000 7fc4b1a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ac15e790 con 0x7fc4ac0cd160 2024-04-14T23:56:34.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.517+0000 7fc4b1a00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fc4ac0ccb20 con 0x7fc4ac0cd540 2024-04-14T23:56:34.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.517+0000 7fc4ab400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc4ac0cd540 0x7fc4ac0cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:34.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.517+0000 7fc4ab400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc4ac0cd540 0x7fc4ac0cfa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.60:60280/0 (socket says 172.21.15.60:60280) 2024-04-14T23:56:34.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.517+0000 7fc4ab400640 1 -- 172.21.15.60:0/391779338 learned_addr learned my addr 172.21.15.60:0/391779338 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:56:34.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.518+0000 7fc4aa000640 1 --2- 172.21.15.60:0/391779338 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc4ac0cd160 0x7fc4ac15a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:34.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.518+0000 7fc4aaa00640 1 --2- 172.21.15.60:0/391779338 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4ac0cb750 0x7fc4ac14d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:34.525 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.518+0000 7fc4ab400640 1 -- 172.21.15.60:0/391779338 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc4ac0cd160 msgr2=0x7fc4ac15a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:34.525 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.518+0000 7fc4ab400640 1 --2- 172.21.15.60:0/391779338 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc4ac0cd160 0x7fc4ac15a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:34.525 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.518+0000 7fc4ab400640 1 -- 172.21.15.60:0/391779338 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4ac0cb750 msgr2=0x7fc4ac14d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:34.525 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.518+0000 7fc4ab400640 1 --2- 172.21.15.60:0/391779338 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4ac0cb750 0x7fc4ac14d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:34.525 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.518+0000 7fc4ab400640 1 -- 172.21.15.60:0/391779338 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4ac1f12c0 con 0x7fc4ac0cd540 2024-04-14T23:56:34.525 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.519+0000 7fc4ab400640 1 --2- 172.21.15.60:0/391779338 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc4ac0cd540 0x7fc4ac0cfa90 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fc4a400c730 tx=0x7fc4a400cb30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:56:34.525 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.519+0000 7fc4a2a00640 1 -- 172.21.15.60:0/391779338 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4a4003b70 con 0x7fc4ac0cd540 2024-04-14T23:56:34.525 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.519+0000 7fc4a2a00640 1 -- 172.21.15.60:0/391779338 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4a4003d30 con 0x7fc4ac0cd540 2024-04-14T23:56:34.525 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.519+0000 7fc4a2a00640 1 -- 172.21.15.60:0/391779338 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4a4005310 con 0x7fc4ac0cd540 2024-04-14T23:56:34.525 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.519+0000 7fc4b1a00640 1 -- 172.21.15.60:0/391779338 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4ac0d0cd0 con 0x7fc4ac0cd540 2024-04-14T23:56:34.525 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.520+0000 7fc4b1a00640 1 -- 172.21.15.60:0/391779338 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4ac0cf570 con 0x7fc4ac0cd540 2024-04-14T23:56:34.526 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.521+0000 7fc4b1a00640 1 -- 172.21.15.60:0/391779338 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4ac068bc0 con 0x7fc4ac0cd540 2024-04-14T23:56:34.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.522+0000 7fc4a2a00640 1 -- 172.21.15.60:0/391779338 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7fc4a40054b0 con 0x7fc4ac0cd540 2024-04-14T23:56:34.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.523+0000 7fc4a2a00640 1 --2- 172.21.15.60:0/391779338 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fc48c06ba10 0x7fc48c06ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:34.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.523+0000 7fc4a2a00640 1 -- 172.21.15.60:0/391779338 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(44..44 src has 1..44) ==== 6169+0+0 (secure 0 0 0) 0x7fc4a408a650 con 0x7fc4ac0cd540 2024-04-14T23:56:34.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.523+0000 7fc4aaa00640 1 --2- 172.21.15.60:0/391779338 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fc48c06ba10 0x7fc48c06ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:34.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.524+0000 7fc4aaa00640 1 --2- 172.21.15.60:0/391779338 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fc48c06ba10 0x7fc48c06ded0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc49c007fd0 tx=0x7fc49c007de0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:56:34.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.527+0000 7fc4a2a00640 1 -- 172.21.15.60:0/391779338 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc4ac068bc0 con 0x7fc4ac0cd540 2024-04-14T23:56:34.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.768+0000 7fc4b1a00640 1 -- 172.21.15.60:0/391779338 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc4ac076970 con 0x7fc4ac0cd540 2024-04-14T23:56:34.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.769+0000 7fc4a2a00640 1 -- 172.21.15.60:0/391779338 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+6006 (secure 0 0 0) 0x7fc4a4058ab0 con 0x7fc4ac0cd540 2024-04-14T23:56:34.775 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:56:34.775 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:56:28.972996+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5191},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5191":{"gid":5191,"name":"c","rank":0,"incarnation":27,"state":"up:replay","state_seq":1,"addr":"172.21.15.60:6835/2356847578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2356847578},{"type":"v1","addr":"172.21.15.60:6835","nonce":2356847578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-14T23:56:34.775 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 27 2024-04-14T23:56:34.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.772+0000 7fc4b1a00640 1 -- 172.21.15.60:0/391779338 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fc48c06ba10 msgr2=0x7fc48c06ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:34.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.772+0000 7fc4b1a00640 1 --2- 172.21.15.60:0/391779338 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fc48c06ba10 0x7fc48c06ded0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc49c007fd0 tx=0x7fc49c007de0 comp rx=0 tx=0).stop 2024-04-14T23:56:34.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.772+0000 7fc4b1a00640 1 -- 172.21.15.60:0/391779338 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc4ac0cd540 msgr2=0x7fc4ac0cfa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:34.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.772+0000 7fc4b1a00640 1 --2- 172.21.15.60:0/391779338 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc4ac0cd540 0x7fc4ac0cfa90 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fc4a400c730 tx=0x7fc4a400cb30 comp rx=0 tx=0).stop 2024-04-14T23:56:34.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.772+0000 7fc4b1a00640 1 -- 172.21.15.60:0/391779338 shutdown_connections 2024-04-14T23:56:34.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.772+0000 7fc4b1a00640 1 --2- 172.21.15.60:0/391779338 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fc48c06ba10 0x7fc48c06ded0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:34.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.772+0000 7fc4b1a00640 1 --2- 172.21.15.60:0/391779338 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc4ac0cd540 0x7fc4ac0cfa90 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:34.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.772+0000 7fc4b1a00640 1 --2- 172.21.15.60:0/391779338 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc4ac0cd160 0x7fc4ac15a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:34.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.772+0000 7fc4b1a00640 1 --2- 172.21.15.60:0/391779338 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4ac0cb750 0x7fc4ac14d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:34.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.772+0000 7fc4b1a00640 1 -- 172.21.15.60:0/391779338 >> 172.21.15.60:0/391779338 conn(0x7fc4ac066f10 msgr2=0x7fc4ac0b4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:56:34.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.773+0000 7fc4b1a00640 1 -- 172.21.15.60:0/391779338 shutdown_connections 2024-04-14T23:56:34.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:34.773+0000 7fc4b1a00640 1 -- 172.21.15.60:0/391779338 wait complete. 2024-04-14T23:56:34.795 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-14T23:56:36.797 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:56:37.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.008+0000 7fdf4ce00640 1 Processor -- start 2024-04-14T23:56:37.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.009+0000 7fdf4ce00640 1 -- start start 2024-04-14T23:56:37.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.010+0000 7fdf4ce00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fdf480731e0 0x7fdf481664b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:37.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.011+0000 7fdf4ce00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf4815b0d0 con 0x7fdf4806f5d0 2024-04-14T23:56:37.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.011+0000 7fdf4ce00640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf4815b370 con 0x7fdf4806f1f0 2024-04-14T23:56:37.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.011+0000 7fdf4ce00640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fdf480c6e10 con 0x7fdf480731e0 2024-04-14T23:56:37.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.011+0000 7fdf3fe00640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fdf4806f5d0 0x7fdf48154980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:50648/0 (socket says 172.21.15.60:50648) 2024-04-14T23:56:37.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.011+0000 7fdf3fe00640 1 -- 172.21.15.60:0/2024277975 learned_addr learned my addr 172.21.15.60:0/2024277975 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:56:37.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.011+0000 7fdf46800640 1 --2- 172.21.15.60:0/2024277975 >> v2:172.21.15.186:3300/0 conn(0x7fdf480731e0 0x7fdf481664b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:37.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.011+0000 7fdf45400640 1 -- 172.21.15.60:0/2024277975 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1693087642 0 0) 0x7fdf4815b370 con 0x7fdf4806f1f0 2024-04-14T23:56:37.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.011+0000 7fdf45400640 1 -- 172.21.15.60:0/2024277975 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf2c003520 con 0x7fdf4806f1f0 2024-04-14T23:56:37.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.011+0000 7fdf45400640 1 -- 172.21.15.60:0/2024277975 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2440933480 0 0) 0x7fdf4815b0d0 con 0x7fdf4806f5d0 2024-04-14T23:56:37.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.011+0000 7fdf46800640 1 -- 172.21.15.60:0/2024277975 >> v1:172.21.15.186:6789/0 conn(0x7fdf4806f1f0 legacy=0x7fdf480c72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:37.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.011+0000 7fdf46800640 1 -- 172.21.15.60:0/2024277975 >> v1:172.21.15.60:6789/0 conn(0x7fdf4806f5d0 legacy=0x7fdf48154980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:37.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.011+0000 7fdf46800640 1 -- 172.21.15.60:0/2024277975 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf4815b650 con 0x7fdf480731e0 2024-04-14T23:56:37.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.013+0000 7fdf46800640 1 --2- 172.21.15.60:0/2024277975 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf480731e0 0x7fdf481664b0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fdf40009d90 tx=0x7fdf4002bfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c49c26a63ced7825 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:56:37.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.013+0000 7fdf45400640 1 -- 172.21.15.60:0/2024277975 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf40039070 con 0x7fdf480731e0 2024-04-14T23:56:37.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.013+0000 7fdf45400640 1 -- 172.21.15.60:0/2024277975 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf4002cd80 con 0x7fdf480731e0 2024-04-14T23:56:37.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.013+0000 7fdf45400640 1 -- 172.21.15.60:0/2024277975 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf40034640 con 0x7fdf480731e0 2024-04-14T23:56:37.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.013+0000 7fdf4ce00640 1 -- 172.21.15.60:0/2024277975 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf480731e0 msgr2=0x7fdf481664b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:37.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.013+0000 7fdf4ce00640 1 --2- 172.21.15.60:0/2024277975 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf480731e0 0x7fdf481664b0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fdf40009d90 tx=0x7fdf4002bfd0 comp rx=0 tx=0).stop 2024-04-14T23:56:37.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.014+0000 7fdf4ce00640 1 -- 172.21.15.60:0/2024277975 shutdown_connections 2024-04-14T23:56:37.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.014+0000 7fdf4ce00640 1 --2- 172.21.15.60:0/2024277975 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf480731e0 0x7fdf481664b0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:37.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.014+0000 7fdf4ce00640 1 -- 172.21.15.60:0/2024277975 >> 172.21.15.60:0/2024277975 conn(0x7fdf480cc170 msgr2=0x7fdf480cc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:56:37.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.014+0000 7fdf4ce00640 1 -- 172.21.15.60:0/2024277975 shutdown_connections 2024-04-14T23:56:37.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.014+0000 7fdf4ce00640 1 -- 172.21.15.60:0/2024277975 wait complete. 2024-04-14T23:56:37.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.015+0000 7fdf4ce00640 1 Processor -- start 2024-04-14T23:56:37.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.016+0000 7fdf4ce00640 1 -- start start 2024-04-14T23:56:37.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.016+0000 7fdf4ce00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf4806f1f0 0x7fdf48145b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:37.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.016+0000 7fdf4ce00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf4806f5d0 0x7fdf481460c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:37.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.016+0000 7fdf4ce00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf480731e0 0x7fdf481404a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:37.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.016+0000 7fdf4ce00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fdf480c79b0 con 0x7fdf4806f5d0 2024-04-14T23:56:37.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.016+0000 7fdf4ce00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdf48071180 con 0x7fdf480731e0 2024-04-14T23:56:37.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.016+0000 7fdf4ce00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fdf480c7b30 con 0x7fdf4806f1f0 2024-04-14T23:56:37.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.017+0000 7fdf3fe00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf4806f5d0 0x7fdf481460c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:37.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.017+0000 7fdf3fe00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf4806f5d0 0x7fdf481460c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35676/0 (socket says 172.21.15.60:35676) 2024-04-14T23:56:37.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.017+0000 7fdf3fe00640 1 -- 172.21.15.60:0/526924734 learned_addr learned my addr 172.21.15.60:0/526924734 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:56:37.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.017+0000 7fdf46800640 1 --2- 172.21.15.60:0/526924734 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf480731e0 0x7fdf481404a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:37.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.017+0000 7fdf45e00640 1 --2- 172.21.15.60:0/526924734 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf4806f1f0 0x7fdf48145b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:37.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.017+0000 7fdf3fe00640 1 -- 172.21.15.60:0/526924734 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf4806f1f0 msgr2=0x7fdf48145b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:37.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.017+0000 7fdf3fe00640 1 --2- 172.21.15.60:0/526924734 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf4806f1f0 0x7fdf48145b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:37.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.017+0000 7fdf3fe00640 1 -- 172.21.15.60:0/526924734 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf480731e0 msgr2=0x7fdf481404a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:37.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.017+0000 7fdf3fe00640 1 --2- 172.21.15.60:0/526924734 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf480731e0 0x7fdf481404a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:37.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.017+0000 7fdf3fe00640 1 -- 172.21.15.60:0/526924734 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf480cbaf0 con 0x7fdf4806f5d0 2024-04-14T23:56:37.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.018+0000 7fdf3fe00640 1 --2- 172.21.15.60:0/526924734 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf4806f5d0 0x7fdf481460c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fdf3000c3e0 tx=0x7fdf3000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:56:37.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.018+0000 7fdf3e000640 1 -- 172.21.15.60:0/526924734 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf300074e0 con 0x7fdf4806f5d0 2024-04-14T23:56:37.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.018+0000 7fdf3e000640 1 -- 172.21.15.60:0/526924734 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf30007680 con 0x7fdf4806f5d0 2024-04-14T23:56:37.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.018+0000 7fdf4ce00640 1 -- 172.21.15.60:0/526924734 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf481f3510 con 0x7fdf4806f5d0 2024-04-14T23:56:37.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.018+0000 7fdf3e000640 1 -- 172.21.15.60:0/526924734 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf30004a20 con 0x7fdf4806f5d0 2024-04-14T23:56:37.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.018+0000 7fdf4ce00640 1 -- 172.21.15.60:0/526924734 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf481f3ad0 con 0x7fdf4806f5d0 2024-04-14T23:56:37.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.019+0000 7fdf4ce00640 1 -- 172.21.15.60:0/526924734 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf48076970 con 0x7fdf4806f5d0 2024-04-14T23:56:37.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.020+0000 7fdf3e000640 1 -- 172.21.15.60:0/526924734 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7fdf30011070 con 0x7fdf4806f5d0 2024-04-14T23:56:37.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.022+0000 7fdf3e000640 1 --2- 172.21.15.60:0/526924734 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fdf2406b990 0x7fdf2406de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:37.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.022+0000 7fdf3e000640 1 -- 172.21.15.60:0/526924734 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6169+0+0 (secure 0 0 0) 0x7fdf3008a5e0 con 0x7fdf4806f5d0 2024-04-14T23:56:37.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.022+0000 7fdf45e00640 1 --2- 172.21.15.60:0/526924734 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fdf2406b990 0x7fdf2406de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:37.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.023+0000 7fdf45e00640 1 --2- 172.21.15.60:0/526924734 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fdf2406b990 0x7fdf2406de50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fdf380011e0 tx=0x7fdf38009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:56:37.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.026+0000 7fdf3e000640 1 -- 172.21.15.60:0/526924734 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fdf300588a0 con 0x7fdf4806f5d0 2024-04-14T23:56:37.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.268+0000 7fdf4ce00640 1 -- 172.21.15.60:0/526924734 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdf480c4280 con 0x7fdf4806f5d0 2024-04-14T23:56:37.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.269+0000 7fdf3e000640 1 -- 172.21.15.60:0/526924734 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+6006 (secure 0 0 0) 0x7fdf3005c940 con 0x7fdf4806f5d0 2024-04-14T23:56:37.274 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:56:37.274 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:56:28.972996+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5191},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5191":{"gid":5191,"name":"c","rank":0,"incarnation":27,"state":"up:replay","state_seq":1,"addr":"172.21.15.60:6835/2356847578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2356847578},{"type":"v1","addr":"172.21.15.60:6835","nonce":2356847578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-14T23:56:37.275 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 27 2024-04-14T23:56:37.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.271+0000 7fdf4ce00640 1 -- 172.21.15.60:0/526924734 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fdf2406b990 msgr2=0x7fdf2406de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:37.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.271+0000 7fdf4ce00640 1 --2- 172.21.15.60:0/526924734 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fdf2406b990 0x7fdf2406de50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fdf380011e0 tx=0x7fdf38009040 comp rx=0 tx=0).stop 2024-04-14T23:56:37.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.271+0000 7fdf4ce00640 1 -- 172.21.15.60:0/526924734 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf4806f5d0 msgr2=0x7fdf481460c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:37.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.271+0000 7fdf4ce00640 1 --2- 172.21.15.60:0/526924734 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf4806f5d0 0x7fdf481460c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fdf3000c3e0 tx=0x7fdf3000c8a0 comp rx=0 tx=0).stop 2024-04-14T23:56:37.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.272+0000 7fdf4ce00640 1 -- 172.21.15.60:0/526924734 shutdown_connections 2024-04-14T23:56:37.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.272+0000 7fdf4ce00640 1 --2- 172.21.15.60:0/526924734 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fdf2406b990 0x7fdf2406de50 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:37.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.272+0000 7fdf4ce00640 1 --2- 172.21.15.60:0/526924734 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf480731e0 0x7fdf481404a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:37.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.272+0000 7fdf4ce00640 1 --2- 172.21.15.60:0/526924734 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf4806f5d0 0x7fdf481460c0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:37.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.272+0000 7fdf4ce00640 1 --2- 172.21.15.60:0/526924734 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf4806f1f0 0x7fdf48145b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:37.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.272+0000 7fdf4ce00640 1 -- 172.21.15.60:0/526924734 >> 172.21.15.60:0/526924734 conn(0x7fdf480cc170 msgr2=0x7fdf481499f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:56:37.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.272+0000 7fdf4ce00640 1 -- 172.21.15.60:0/526924734 shutdown_connections 2024-04-14T23:56:37.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:37.272+0000 7fdf4ce00640 1 -- 172.21.15.60:0/526924734 wait complete. 2024-04-14T23:56:37.295 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-14T23:56:39.296 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:56:39.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.525+0000 7f823b600640 1 Processor -- start 2024-04-14T23:56:39.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.525+0000 7f823b600640 1 -- start start 2024-04-14T23:56:39.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.527+0000 7f823b600640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f82340c63c0 0x7f82340c6b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:39.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.527+0000 7f823b600640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f823415c9b0 0x7f82340c70b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:39.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.527+0000 7f823b600640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82341530d0 con 0x7f8234152cf0 2024-04-14T23:56:39.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.527+0000 7f823b600640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f82340751f0 con 0x7f823415c9b0 2024-04-14T23:56:39.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.527+0000 7f823b600640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f82340c8370 con 0x7f82340c63c0 2024-04-14T23:56:39.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.527+0000 7f8239600640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f823415c9b0 0x7f82340c70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:39.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.527+0000 7f8238c00640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f8234152cf0 0x7f823415c2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:50658/0 (socket says 172.21.15.60:50658) 2024-04-14T23:56:39.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.527+0000 7f8238c00640 1 -- 172.21.15.60:0/3588555073 learned_addr learned my addr 172.21.15.60:0/3588555073 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:56:39.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.527+0000 7f8233e00640 1 --2- 172.21.15.60:0/3588555073 >> v2:172.21.15.186:3300/0 conn(0x7f82340c63c0 0x7f82340c6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:39.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.528+0000 7f8239600640 1 -- 172.21.15.60:0/3588555073 >> v2:172.21.15.186:3300/0 conn(0x7f82340c63c0 msgr2=0x7f82340c6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:56:39.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.528+0000 7f8239600640 1 --2- 172.21.15.60:0/3588555073 >> v2:172.21.15.186:3300/0 conn(0x7f82340c63c0 0x7f82340c6b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:39.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.528+0000 7f8239600640 1 -- 172.21.15.60:0/3588555073 >> v1:172.21.15.60:6789/0 conn(0x7f8234152cf0 legacy=0x7f823415c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:39.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.528+0000 7f8239600640 1 -- 172.21.15.60:0/3588555073 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82340c3810 con 0x7f823415c9b0 2024-04-14T23:56:39.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.529+0000 7f8239600640 1 --2- 172.21.15.60:0/3588555073 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f823415c9b0 0x7f82340c70b0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f8234070e80 tx=0x7f822c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b5ba4548d9684b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:56:39.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.529+0000 7f8233400640 1 -- 172.21.15.60:0/3588555073 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f822c039070 con 0x7f823415c9b0 2024-04-14T23:56:39.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.529+0000 7f8233400640 1 -- 172.21.15.60:0/3588555073 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f822c02cd10 con 0x7f823415c9b0 2024-04-14T23:56:39.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.529+0000 7f8233400640 1 -- 172.21.15.60:0/3588555073 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f822c034660 con 0x7f823415c9b0 2024-04-14T23:56:39.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.530+0000 7f823b600640 1 -- 172.21.15.60:0/3588555073 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f823415c9b0 msgr2=0x7f82340c70b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:39.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.530+0000 7f823b600640 1 --2- 172.21.15.60:0/3588555073 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f823415c9b0 0x7f82340c70b0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f8234070e80 tx=0x7f822c02bf60 comp rx=0 tx=0).stop 2024-04-14T23:56:39.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.530+0000 7f823b600640 1 -- 172.21.15.60:0/3588555073 shutdown_connections 2024-04-14T23:56:39.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.530+0000 7f823b600640 1 --2- 172.21.15.60:0/3588555073 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f823415c9b0 0x7f82340c70b0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:39.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.530+0000 7f823b600640 1 --2- 172.21.15.60:0/3588555073 >> v2:172.21.15.186:3300/0 conn(0x7f82340c63c0 0x7f82340c6b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:39.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.530+0000 7f823b600640 1 -- 172.21.15.60:0/3588555073 >> 172.21.15.60:0/3588555073 conn(0x7f82340cc170 msgr2=0x7f82340cc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:56:39.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.530+0000 7f823b600640 1 -- 172.21.15.60:0/3588555073 shutdown_connections 2024-04-14T23:56:39.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.530+0000 7f823b600640 1 -- 172.21.15.60:0/3588555073 wait complete. 2024-04-14T23:56:39.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.532+0000 7f823b600640 1 Processor -- start 2024-04-14T23:56:39.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.532+0000 7f823b600640 1 -- start start 2024-04-14T23:56:39.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.533+0000 7f823b600640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82340c63c0 0x7f8234151610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:39.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.533+0000 7f823b600640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8234152cf0 0x7f82341f42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:39.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.533+0000 7f823b600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f823415c9b0 0x7f82341f4830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:39.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.533+0000 7f823b600640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f82340c88f0 con 0x7f82340c63c0 2024-04-14T23:56:39.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.533+0000 7f823b600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f82340c8610 con 0x7f823415c9b0 2024-04-14T23:56:39.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.533+0000 7f823b600640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f8234159450 con 0x7f8234152cf0 2024-04-14T23:56:39.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.533+0000 7f8238c00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82340c63c0 0x7f8234151610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:39.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.533+0000 7f8233e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8234152cf0 0x7f82341f42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:39.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.533+0000 7f8233e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8234152cf0 0x7f82341f42f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.60:60294/0 (socket says 172.21.15.60:60294) 2024-04-14T23:56:39.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.533+0000 7f8238c00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82340c63c0 0x7f8234151610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35702/0 (socket says 172.21.15.60:35702) 2024-04-14T23:56:39.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.533+0000 7f8233e00640 1 -- 172.21.15.60:0/3420367562 learned_addr learned my addr 172.21.15.60:0/3420367562 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:56:39.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.533+0000 7f8239600640 1 --2- 172.21.15.60:0/3420367562 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f823415c9b0 0x7f82341f4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:39.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.534+0000 7f8233e00640 1 -- 172.21.15.60:0/3420367562 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f823415c9b0 msgr2=0x7f82341f4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:39.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.534+0000 7f8233e00640 1 --2- 172.21.15.60:0/3420367562 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f823415c9b0 0x7f82341f4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:39.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.534+0000 7f8233e00640 1 -- 172.21.15.60:0/3420367562 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82340c63c0 msgr2=0x7f8234151610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:39.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.534+0000 7f8233e00640 1 --2- 172.21.15.60:0/3420367562 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82340c63c0 0x7f8234151610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:39.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.534+0000 7f8233e00640 1 -- 172.21.15.60:0/3420367562 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8234152540 con 0x7f8234152cf0 2024-04-14T23:56:39.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.534+0000 7f8233e00640 1 --2- 172.21.15.60:0/3420367562 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8234152cf0 0x7f82341f42f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f822000e9b0 tx=0x7f822000ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:56:39.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.534+0000 7f8230c00640 1 -- 172.21.15.60:0/3420367562 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8220017070 con 0x7f8234152cf0 2024-04-14T23:56:39.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.534+0000 7f8230c00640 1 -- 172.21.15.60:0/3420367562 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f822000c9a0 con 0x7f8234152cf0 2024-04-14T23:56:39.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.534+0000 7f823b600640 1 -- 172.21.15.60:0/3420367562 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82341e1f60 con 0x7f8234152cf0 2024-04-14T23:56:39.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.534+0000 7f8230c00640 1 -- 172.21.15.60:0/3420367562 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f822000cd70 con 0x7f8234152cf0 2024-04-14T23:56:39.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.535+0000 7f823b600640 1 -- 172.21.15.60:0/3420367562 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f823415bcd0 con 0x7f8234152cf0 2024-04-14T23:56:39.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.536+0000 7f823b600640 1 -- 172.21.15.60:0/3420367562 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8234076970 con 0x7f8234152cf0 2024-04-14T23:56:39.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.537+0000 7f8230c00640 1 -- 172.21.15.60:0/3420367562 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f822001a070 con 0x7f8234152cf0 2024-04-14T23:56:39.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.538+0000 7f8230c00640 1 --2- 172.21.15.60:0/3420367562 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f821406b990 0x7f821406de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:39.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.538+0000 7f8230c00640 1 -- 172.21.15.60:0/3420367562 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(44..44 src has 1..44) ==== 6169+0+0 (secure 0 0 0) 0x7f822008aa90 con 0x7f8234152cf0 2024-04-14T23:56:39.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.538+0000 7f8238c00640 1 --2- 172.21.15.60:0/3420367562 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f821406b990 0x7f821406de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:39.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.539+0000 7f8238c00640 1 --2- 172.21.15.60:0/3420367562 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f821406b990 0x7f821406de50 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f8224007fd0 tx=0x7f8224007e80 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:56:39.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.543+0000 7f8230c00640 1 -- 172.21.15.60:0/3420367562 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8220058d50 con 0x7f8234152cf0 2024-04-14T23:56:39.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.781+0000 7f823b600640 1 -- 172.21.15.60:0/3420367562 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f82340c7f40 con 0x7f8234152cf0 2024-04-14T23:56:39.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.782+0000 7f8230c00640 1 -- 172.21.15.60:0/3420367562 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+6010 (secure 0 0 0) 0x7f822005cdf0 con 0x7f8234152cf0 2024-04-14T23:56:39.788 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:56:39.788 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:56:36.729400+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5191},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5191":{"gid":5191,"name":"c","rank":0,"incarnation":27,"state":"up:reconnect","state_seq":20,"addr":"172.21.15.60:6835/2356847578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2356847578},{"type":"v1","addr":"172.21.15.60:6835","nonce":2356847578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-14T23:56:39.788 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 28 2024-04-14T23:56:39.791 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.786+0000 7f823b600640 1 -- 172.21.15.60:0/3420367562 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f821406b990 msgr2=0x7f821406de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:39.791 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.786+0000 7f823b600640 1 --2- 172.21.15.60:0/3420367562 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f821406b990 0x7f821406de50 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f8224007fd0 tx=0x7f8224007e80 comp rx=0 tx=0).stop 2024-04-14T23:56:39.791 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.786+0000 7f823b600640 1 -- 172.21.15.60:0/3420367562 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8234152cf0 msgr2=0x7f82341f42f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:39.791 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.786+0000 7f823b600640 1 --2- 172.21.15.60:0/3420367562 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8234152cf0 0x7f82341f42f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f822000e9b0 tx=0x7f822000ee70 comp rx=0 tx=0).stop 2024-04-14T23:56:39.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.786+0000 7f823b600640 1 -- 172.21.15.60:0/3420367562 shutdown_connections 2024-04-14T23:56:39.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.786+0000 7f823b600640 1 --2- 172.21.15.60:0/3420367562 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f821406b990 0x7f821406de50 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:39.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.786+0000 7f823b600640 1 --2- 172.21.15.60:0/3420367562 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f823415c9b0 0x7f82341f4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:39.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.786+0000 7f823b600640 1 --2- 172.21.15.60:0/3420367562 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8234152cf0 0x7f82341f42f0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:39.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.786+0000 7f823b600640 1 --2- 172.21.15.60:0/3420367562 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82340c63c0 0x7f8234151610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:39.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.786+0000 7f823b600640 1 -- 172.21.15.60:0/3420367562 >> 172.21.15.60:0/3420367562 conn(0x7f82340cc170 msgr2=0x7f823415c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:56:39.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.786+0000 7f823b600640 1 -- 172.21.15.60:0/3420367562 shutdown_connections 2024-04-14T23:56:39.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:39.786+0000 7f823b600640 1 -- 172.21.15.60:0/3420367562 wait complete. 2024-04-14T23:56:39.812 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-14T23:56:41.812 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:56:42.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.036+0000 7f63e3600640 1 Processor -- start 2024-04-14T23:56:42.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.037+0000 7f63e3600640 1 -- start start 2024-04-14T23:56:42.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.038+0000 7f63e3600640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f63dc152cf0 0x7f63dc15c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:42.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.038+0000 7f63e3600640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f63dc159450 0x7f63dc15c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:42.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.038+0000 7f63e3600640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63dc1530d0 con 0x7f63dc0c63c0 2024-04-14T23:56:42.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.038+0000 7f63e3600640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f63dc0751f0 con 0x7f63dc152cf0 2024-04-14T23:56:42.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.038+0000 7f63e3600640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f63dc0c8370 con 0x7f63dc159450 2024-04-14T23:56:42.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.038+0000 7f63e0c00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f63dc152cf0 0x7f63dc15c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:42.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.039+0000 7f63dbe00640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f63dc0c63c0 0x7f63dc0c3f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:50956/0 (socket says 172.21.15.60:50956) 2024-04-14T23:56:42.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.039+0000 7f63e0c00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f63dc152cf0 0x7f63dc15c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54294/0 (socket says 172.21.15.60:54294) 2024-04-14T23:56:42.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.039+0000 7f63dbe00640 1 -- 172.21.15.60:0/4221182457 learned_addr learned my addr 172.21.15.60:0/4221182457 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:56:42.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.039+0000 7f63e1600640 1 --2- 172.21.15.60:0/4221182457 >> v2:172.21.15.186:3300/0 conn(0x7f63dc159450 0x7f63dc15c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:42.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.039+0000 7f63e0c00640 1 -- 172.21.15.60:0/4221182457 >> v2:172.21.15.186:3300/0 conn(0x7f63dc159450 msgr2=0x7f63dc15c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:56:42.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.039+0000 7f63e0c00640 1 --2- 172.21.15.60:0/4221182457 >> v2:172.21.15.186:3300/0 conn(0x7f63dc159450 0x7f63dc15c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:42.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.039+0000 7f63e0c00640 1 -- 172.21.15.60:0/4221182457 >> v1:172.21.15.60:6789/0 conn(0x7f63dc0c63c0 legacy=0x7f63dc0c3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:42.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.039+0000 7f63e0c00640 1 -- 172.21.15.60:0/4221182457 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63dc0c3810 con 0x7f63dc152cf0 2024-04-14T23:56:42.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.040+0000 7f63e0c00640 1 --2- 172.21.15.60:0/4221182457 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f63dc152cf0 0x7f63dc15c2a0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f63dc06c750 tx=0x7f63cc02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25a157b5f0862682 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:56:42.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.040+0000 7f63db400640 1 -- 172.21.15.60:0/4221182457 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63cc034070 con 0x7f63dc152cf0 2024-04-14T23:56:42.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.041+0000 7f63db400640 1 -- 172.21.15.60:0/4221182457 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63cc030610 con 0x7f63dc152cf0 2024-04-14T23:56:42.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.041+0000 7f63db400640 1 -- 172.21.15.60:0/4221182457 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63cc0309e0 con 0x7f63dc152cf0 2024-04-14T23:56:42.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.041+0000 7f63e3600640 1 -- 172.21.15.60:0/4221182457 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f63dc152cf0 msgr2=0x7f63dc15c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:42.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.041+0000 7f63e3600640 1 --2- 172.21.15.60:0/4221182457 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f63dc152cf0 0x7f63dc15c2a0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f63dc06c750 tx=0x7f63cc02b960 comp rx=0 tx=0).stop 2024-04-14T23:56:42.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.041+0000 7f63e3600640 1 -- 172.21.15.60:0/4221182457 shutdown_connections 2024-04-14T23:56:42.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.041+0000 7f63e3600640 1 --2- 172.21.15.60:0/4221182457 >> v2:172.21.15.186:3300/0 conn(0x7f63dc159450 0x7f63dc15c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:42.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.041+0000 7f63e3600640 1 --2- 172.21.15.60:0/4221182457 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f63dc152cf0 0x7f63dc15c2a0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:42.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.041+0000 7f63e3600640 1 -- 172.21.15.60:0/4221182457 >> 172.21.15.60:0/4221182457 conn(0x7f63dc0cc170 msgr2=0x7f63dc0cc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:56:42.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.041+0000 7f63e3600640 1 -- 172.21.15.60:0/4221182457 shutdown_connections 2024-04-14T23:56:42.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.041+0000 7f63e3600640 1 -- 172.21.15.60:0/4221182457 wait complete. 2024-04-14T23:56:42.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.043+0000 7f63e3600640 1 Processor -- start 2024-04-14T23:56:42.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.043+0000 7f63e3600640 1 -- start start 2024-04-14T23:56:42.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.044+0000 7f63e3600640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63dc0c63c0 0x7f63dc149020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:42.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.044+0000 7f63e3600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63dc152cf0 0x7f63dc1517d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:42.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.044+0000 7f63e3600640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f63dc159450 0x7f63dc071030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:42.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.044+0000 7f63e3600640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f63dc0c88f0 con 0x7f63dc159450 2024-04-14T23:56:42.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.044+0000 7f63e3600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f63dc0c8610 con 0x7f63dc152cf0 2024-04-14T23:56:42.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.044+0000 7f63e3600640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f63dc0c4690 con 0x7f63dc0c63c0 2024-04-14T23:56:42.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.044+0000 7f63e1600640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f63dc159450 0x7f63dc071030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:42.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.044+0000 7f63dbe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63dc152cf0 0x7f63dc1517d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:42.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.044+0000 7f63e0c00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63dc0c63c0 0x7f63dc149020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:42.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.044+0000 7f63dbe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63dc152cf0 0x7f63dc1517d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:33162/0 (socket says 172.21.15.60:33162) 2024-04-14T23:56:42.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.044+0000 7f63dbe00640 1 -- 172.21.15.60:0/3099002962 learned_addr learned my addr 172.21.15.60:0/3099002962 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:56:42.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.045+0000 7f63dbe00640 1 -- 172.21.15.60:0/3099002962 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63dc0c63c0 msgr2=0x7f63dc149020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:42.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.045+0000 7f63dbe00640 1 --2- 172.21.15.60:0/3099002962 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63dc0c63c0 0x7f63dc149020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:42.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.045+0000 7f63dbe00640 1 -- 172.21.15.60:0/3099002962 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f63dc159450 msgr2=0x7f63dc071030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:42.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.045+0000 7f63dbe00640 1 --2- 172.21.15.60:0/3099002962 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f63dc159450 0x7f63dc071030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:42.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.045+0000 7f63dbe00640 1 -- 172.21.15.60:0/3099002962 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63dc1f0e00 con 0x7f63dc152cf0 2024-04-14T23:56:42.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.045+0000 7f63e1600640 1 --2- 172.21.15.60:0/3099002962 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f63dc159450 0x7f63dc071030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:56:42.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.045+0000 7f63dbe00640 1 --2- 172.21.15.60:0/3099002962 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63dc152cf0 0x7f63dc1517d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f63c800c440 tx=0x7f63c8001e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:56:42.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.045+0000 7f63d8c00640 1 -- 172.21.15.60:0/3099002962 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63c8017070 con 0x7f63dc152cf0 2024-04-14T23:56:42.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.046+0000 7f63d8c00640 1 -- 172.21.15.60:0/3099002962 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63c8005ce0 con 0x7f63dc152cf0 2024-04-14T23:56:42.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.046+0000 7f63d8c00640 1 -- 172.21.15.60:0/3099002962 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63c80072f0 con 0x7f63dc152cf0 2024-04-14T23:56:42.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.046+0000 7f63e3600640 1 -- 172.21.15.60:0/3099002962 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63dc1473f0 con 0x7f63dc152cf0 2024-04-14T23:56:42.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.046+0000 7f63e3600640 1 -- 172.21.15.60:0/3099002962 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63dc1f1070 con 0x7f63dc152cf0 2024-04-14T23:56:42.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.047+0000 7f63e3600640 1 -- 172.21.15.60:0/3099002962 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63dc0c5be0 con 0x7f63dc152cf0 2024-04-14T23:56:42.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.048+0000 7f63d8c00640 1 -- 172.21.15.60:0/3099002962 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f63c801a070 con 0x7f63dc152cf0 2024-04-14T23:56:42.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.049+0000 7f63d8c00640 1 --2- 172.21.15.60:0/3099002962 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f63bc06b990 0x7f63bc06de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:42.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.049+0000 7f63d8c00640 1 -- 172.21.15.60:0/3099002962 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6169+0+0 (secure 0 0 0) 0x7f63c8089df0 con 0x7f63dc152cf0 2024-04-14T23:56:42.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.049+0000 7f63e0c00640 1 --2- 172.21.15.60:0/3099002962 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f63bc06b990 0x7f63bc06de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:42.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.050+0000 7f63e0c00640 1 --2- 172.21.15.60:0/3099002962 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f63bc06b990 0x7f63bc06de50 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f63cc036000 tx=0x7f63cc03b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:56:42.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.054+0000 7f63d8c00640 1 -- 172.21.15.60:0/3099002962 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f63dc0c5be0 con 0x7f63dc152cf0 2024-04-14T23:56:42.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.297+0000 7f63e3600640 1 -- 172.21.15.60:0/3099002962 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f63dc1562f0 con 0x7f63dc152cf0 2024-04-14T23:56:42.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.298+0000 7f63d8c00640 1 -- 172.21.15.60:0/3099002962 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+6010 (secure 0 0 0) 0x7f63c8058140 con 0x7f63dc152cf0 2024-04-14T23:56:42.303 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:56:42.303 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:56:36.729400+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5191},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5191":{"gid":5191,"name":"c","rank":0,"incarnation":27,"state":"up:reconnect","state_seq":20,"addr":"172.21.15.60:6835/2356847578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2356847578},{"type":"v1","addr":"172.21.15.60:6835","nonce":2356847578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-14T23:56:42.304 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 28 2024-04-14T23:56:42.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.301+0000 7f63e3600640 1 -- 172.21.15.60:0/3099002962 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f63bc06b990 msgr2=0x7f63bc06de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:42.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.301+0000 7f63e3600640 1 --2- 172.21.15.60:0/3099002962 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f63bc06b990 0x7f63bc06de50 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f63cc036000 tx=0x7f63cc03b040 comp rx=0 tx=0).stop 2024-04-14T23:56:42.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.301+0000 7f63e3600640 1 -- 172.21.15.60:0/3099002962 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63dc152cf0 msgr2=0x7f63dc1517d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:42.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.301+0000 7f63e3600640 1 --2- 172.21.15.60:0/3099002962 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63dc152cf0 0x7f63dc1517d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f63c800c440 tx=0x7f63c8001e70 comp rx=0 tx=0).stop 2024-04-14T23:56:42.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.301+0000 7f63e3600640 1 -- 172.21.15.60:0/3099002962 shutdown_connections 2024-04-14T23:56:42.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.301+0000 7f63e3600640 1 --2- 172.21.15.60:0/3099002962 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f63bc06b990 0x7f63bc06de50 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:42.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.301+0000 7f63e3600640 1 --2- 172.21.15.60:0/3099002962 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f63dc159450 0x7f63dc071030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:42.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.301+0000 7f63e3600640 1 --2- 172.21.15.60:0/3099002962 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63dc152cf0 0x7f63dc1517d0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:42.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.301+0000 7f63e3600640 1 --2- 172.21.15.60:0/3099002962 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63dc0c63c0 0x7f63dc149020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:42.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.301+0000 7f63e3600640 1 -- 172.21.15.60:0/3099002962 >> 172.21.15.60:0/3099002962 conn(0x7f63dc0cc170 msgr2=0x7f63dc15c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:56:42.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.301+0000 7f63e3600640 1 -- 172.21.15.60:0/3099002962 shutdown_connections 2024-04-14T23:56:42.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:42.301+0000 7f63e3600640 1 -- 172.21.15.60:0/3099002962 wait complete. 2024-04-14T23:56:42.325 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 28, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-14T23:48:42.700188+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 28, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5191": { "addr": "172.21.15.60:6835/2356847578", "addrs": { "addrvec": [ { "addr": "172.21.15.60:6834", "nonce": 2356847578, "type": "v2" }, { "addr": "172.21.15.60:6835", "nonce": 2356847578, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5191, "incarnation": 27, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:reconnect", "state_seq": 20 } }, "last_failure": 0, "last_failure_osd_epoch": 44, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-14T23:56:36.729400+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5191 } } } ], "standbys": [ { "addr": "172.21.15.60:6837/1527815634", "addrs": { "addrvec": [ { "addr": "172.21.15.60:6836", "nonce": 1527815634, "type": "v2" }, { "addr": "172.21.15.60:6837", "nonce": 1527815634, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4197, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6838/3958319486", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6835", "nonce": 3958319486, "type": "v2" }, { "addr": "172.21.15.186:6838", "nonce": 3958319486, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4220, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6837/3910951185", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6834", "nonce": 3910951185, "type": "v2" }, { "addr": "172.21.15.186:6837", "nonce": 3910951185, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4264, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6839/2155020404", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6836", "nonce": 2155020404, "type": "v2" }, { "addr": "172.21.15.186:6839", "nonce": 2155020404, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4267, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-14T23:56:44.326 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:56:44.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.556+0000 7fb9c6a00640 1 Processor -- start 2024-04-14T23:56:44.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.556+0000 7fb9c6a00640 1 -- start start 2024-04-14T23:56:44.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.557+0000 7fb9c6a00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb9c00b1e00 0x7fb9c00c6950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:44.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.557+0000 7fb9c6a00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb9c0073320 0x7fb9c00c7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:44.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.558+0000 7fb9c6a00640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9c0152fb0 con 0x7fb9c00c84e0 2024-04-14T23:56:44.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.558+0000 7fb9c6a00640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fb9c00c6540 con 0x7fb9c0073320 2024-04-14T23:56:44.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.558+0000 7fb9c6a00640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fb9c00b21e0 con 0x7fb9c00b1e00 2024-04-14T23:56:44.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.558+0000 7fb9c4c00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb9c0073320 0x7fb9c00c7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:44.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.558+0000 7fb9c4c00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb9c0073320 0x7fb9c00c7fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54320/0 (socket says 172.21.15.60:54320) 2024-04-14T23:56:44.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.558+0000 7fb9c4c00640 1 -- 172.21.15.60:0/4225800009 learned_addr learned my addr 172.21.15.60:0/4225800009 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:56:44.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.558+0000 7fb9c5600640 1 --2- 172.21.15.60:0/4225800009 >> v2:172.21.15.186:3300/0 conn(0x7fb9c00b1e00 0x7fb9c00c6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:44.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.558+0000 7fb9c4c00640 1 -- 172.21.15.60:0/4225800009 >> v2:172.21.15.186:3300/0 conn(0x7fb9c00b1e00 msgr2=0x7fb9c00c6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:56:44.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.558+0000 7fb9c4c00640 1 --2- 172.21.15.60:0/4225800009 >> v2:172.21.15.186:3300/0 conn(0x7fb9c00b1e00 0x7fb9c00c6950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:44.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.558+0000 7fb9c4c00640 1 -- 172.21.15.60:0/4225800009 >> v1:172.21.15.186:6789/0 conn(0x7fb9c00c84e0 legacy=0x7fb9c00c3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:44.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.558+0000 7fb9c4c00640 1 -- 172.21.15.60:0/4225800009 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9c0159310 con 0x7fb9c0073320 2024-04-14T23:56:44.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.560+0000 7fb9c4c00640 1 --2- 172.21.15.60:0/4225800009 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9c0073320 0x7fb9c00c7fa0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fb9c00c36e0 tx=0x7fb9ac02ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee8e7d1bdf3944ee server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:56:44.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.560+0000 7fb9bfe00640 1 -- 172.21.15.60:0/4225800009 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9ac02f850 con 0x7fb9c0073320 2024-04-14T23:56:44.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.560+0000 7fb9bfe00640 1 -- 172.21.15.60:0/4225800009 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9ac02f9f0 con 0x7fb9c0073320 2024-04-14T23:56:44.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.560+0000 7fb9bfe00640 1 -- 172.21.15.60:0/4225800009 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9ac02fdc0 con 0x7fb9c0073320 2024-04-14T23:56:44.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.560+0000 7fb9c6a00640 1 -- 172.21.15.60:0/4225800009 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9c0073320 msgr2=0x7fb9c00c7fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:44.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.560+0000 7fb9c6a00640 1 --2- 172.21.15.60:0/4225800009 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9c0073320 0x7fb9c00c7fa0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fb9c00c36e0 tx=0x7fb9ac02ba30 comp rx=0 tx=0).stop 2024-04-14T23:56:44.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.561+0000 7fb9c6a00640 1 -- 172.21.15.60:0/4225800009 shutdown_connections 2024-04-14T23:56:44.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.561+0000 7fb9c6a00640 1 --2- 172.21.15.60:0/4225800009 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9c0073320 0x7fb9c00c7fa0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:44.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.561+0000 7fb9c6a00640 1 --2- 172.21.15.60:0/4225800009 >> v2:172.21.15.186:3300/0 conn(0x7fb9c00b1e00 0x7fb9c00c6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:44.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.561+0000 7fb9c6a00640 1 -- 172.21.15.60:0/4225800009 >> 172.21.15.60:0/4225800009 conn(0x7fb9c00c8b50 msgr2=0x7fb9c00c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:56:44.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.561+0000 7fb9c6a00640 1 -- 172.21.15.60:0/4225800009 shutdown_connections 2024-04-14T23:56:44.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.561+0000 7fb9c6a00640 1 -- 172.21.15.60:0/4225800009 wait complete. 2024-04-14T23:56:44.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.562+0000 7fb9c6a00640 1 Processor -- start 2024-04-14T23:56:44.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.563+0000 7fb9c6a00640 1 -- start start 2024-04-14T23:56:44.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.563+0000 7fb9c6a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb9c0073320 0x7fb9c00b0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:44.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.563+0000 7fb9c6a00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9c00b1e00 0x7fb9c00ca810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:44.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.563+0000 7fb9c6a00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb9c00c84e0 0x7fb9c0146210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:44.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.564+0000 7fb9c6a00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb9c00c38b0 con 0x7fb9c00b1e00 2024-04-14T23:56:44.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.564+0000 7fb9c6a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb9c0159890 con 0x7fb9c0073320 2024-04-14T23:56:44.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.564+0000 7fb9c6a00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fb9c00c5f80 con 0x7fb9c00c84e0 2024-04-14T23:56:44.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.564+0000 7fb9c4c00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9c00b1e00 0x7fb9c00ca810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:44.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.564+0000 7fb9c4c00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9c00b1e00 0x7fb9c00ca810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54322/0 (socket says 172.21.15.60:54322) 2024-04-14T23:56:44.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.564+0000 7fb9c4c00640 1 -- 172.21.15.60:0/92303276 learned_addr learned my addr 172.21.15.60:0/92303276 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:56:44.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.564+0000 7fb9c5600640 1 --2- 172.21.15.60:0/92303276 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb9c0073320 0x7fb9c00b0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:44.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.564+0000 7fb9c6000640 1 --2- 172.21.15.60:0/92303276 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb9c00c84e0 0x7fb9c0146210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:44.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.564+0000 7fb9c4c00640 1 -- 172.21.15.60:0/92303276 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb9c00c84e0 msgr2=0x7fb9c0146210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:44.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.564+0000 7fb9c4c00640 1 --2- 172.21.15.60:0/92303276 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb9c00c84e0 0x7fb9c0146210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:44.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.564+0000 7fb9c4c00640 1 -- 172.21.15.60:0/92303276 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb9c0073320 msgr2=0x7fb9c00b0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:44.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.564+0000 7fb9c4c00640 1 --2- 172.21.15.60:0/92303276 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb9c0073320 0x7fb9c00b0bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:44.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.564+0000 7fb9c4c00640 1 -- 172.21.15.60:0/92303276 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9c00b10f0 con 0x7fb9c00b1e00 2024-04-14T23:56:44.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.565+0000 7fb9c4c00640 1 --2- 172.21.15.60:0/92303276 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9c00b1e00 0x7fb9c00ca810 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fb9ac02ba70 tx=0x7fb9ac004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:56:44.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.565+0000 7fb9bd600640 1 -- 172.21.15.60:0/92303276 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9ac037c40 con 0x7fb9c00b1e00 2024-04-14T23:56:44.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.565+0000 7fb9bd600640 1 -- 172.21.15.60:0/92303276 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9ac046070 con 0x7fb9c00b1e00 2024-04-14T23:56:44.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.565+0000 7fb9bd600640 1 -- 172.21.15.60:0/92303276 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9ac041400 con 0x7fb9c00b1e00 2024-04-14T23:56:44.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.565+0000 7fb9c6a00640 1 -- 172.21.15.60:0/92303276 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9c00b1680 con 0x7fb9c00b1e00 2024-04-14T23:56:44.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.565+0000 7fb9c6a00640 1 -- 172.21.15.60:0/92303276 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9c00ae970 con 0x7fb9c00b1e00 2024-04-14T23:56:44.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.566+0000 7fb9c6a00640 1 -- 172.21.15.60:0/92303276 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9c004f540 con 0x7fb9c00b1e00 2024-04-14T23:56:44.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.568+0000 7fb9bd600640 1 -- 172.21.15.60:0/92303276 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7fb9ac040070 con 0x7fb9c00b1e00 2024-04-14T23:56:44.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.568+0000 7fb9bd600640 1 --2- 172.21.15.60:0/92303276 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fb9a006b990 0x7fb9a006de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:44.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.569+0000 7fb9bd600640 1 -- 172.21.15.60:0/92303276 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6169+0+0 (secure 0 0 0) 0x7fb9ac0acfe0 con 0x7fb9c00b1e00 2024-04-14T23:56:44.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.569+0000 7fb9c5600640 1 --2- 172.21.15.60:0/92303276 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fb9a006b990 0x7fb9a006de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:44.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.569+0000 7fb9c5600640 1 --2- 172.21.15.60:0/92303276 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fb9a006b990 0x7fb9a006de50 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb9b4009b40 tx=0x7fb9b4009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:56:44.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.573+0000 7fb9bd600640 1 -- 172.21.15.60:0/92303276 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb9ac033030 con 0x7fb9c00b1e00 2024-04-14T23:56:44.817 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.812+0000 7fb9c6a00640 1 -- 172.21.15.60:0/92303276 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb9c006c050 con 0x7fb9c00b1e00 2024-04-14T23:56:44.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.813+0000 7fb9bd600640 1 -- 172.21.15.60:0/92303276 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+6007 (secure 0 0 0) 0x7fb9ac07b450 con 0x7fb9c00b1e00 2024-04-14T23:56:44.819 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:56:44.819 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:56:42.366646+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5191},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5191":{"gid":5191,"name":"c","rank":0,"incarnation":27,"state":"up:rejoin","state_seq":22,"addr":"172.21.15.60:6835/2356847578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2356847578},{"type":"v1","addr":"172.21.15.60:6835","nonce":2356847578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-14T23:56:44.819 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 29 2024-04-14T23:56:44.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.816+0000 7fb9c6a00640 1 -- 172.21.15.60:0/92303276 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fb9a006b990 msgr2=0x7fb9a006de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:44.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.816+0000 7fb9c6a00640 1 --2- 172.21.15.60:0/92303276 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fb9a006b990 0x7fb9a006de50 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb9b4009b40 tx=0x7fb9b4009190 comp rx=0 tx=0).stop 2024-04-14T23:56:44.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.816+0000 7fb9c6a00640 1 -- 172.21.15.60:0/92303276 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9c00b1e00 msgr2=0x7fb9c00ca810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:44.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.816+0000 7fb9c6a00640 1 --2- 172.21.15.60:0/92303276 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9c00b1e00 0x7fb9c00ca810 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fb9ac02ba70 tx=0x7fb9ac004730 comp rx=0 tx=0).stop 2024-04-14T23:56:44.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.816+0000 7fb9c6a00640 1 -- 172.21.15.60:0/92303276 shutdown_connections 2024-04-14T23:56:44.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.816+0000 7fb9c6a00640 1 --2- 172.21.15.60:0/92303276 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fb9a006b990 0x7fb9a006de50 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:44.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.816+0000 7fb9c6a00640 1 --2- 172.21.15.60:0/92303276 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb9c00c84e0 0x7fb9c0146210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:44.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.816+0000 7fb9c6a00640 1 --2- 172.21.15.60:0/92303276 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9c00b1e00 0x7fb9c00ca810 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:44.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.816+0000 7fb9c6a00640 1 --2- 172.21.15.60:0/92303276 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb9c0073320 0x7fb9c00b0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:44.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.816+0000 7fb9c6a00640 1 -- 172.21.15.60:0/92303276 >> 172.21.15.60:0/92303276 conn(0x7fb9c00c8b50 msgr2=0x7fb9c0157f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:56:44.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.816+0000 7fb9c6a00640 1 -- 172.21.15.60:0/92303276 shutdown_connections 2024-04-14T23:56:44.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:44.816+0000 7fb9c6a00640 1 -- 172.21.15.60:0/92303276 wait complete. 2024-04-14T23:56:44.841 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-14T23:56:46.082 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-14T23:56:46.091 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-14T23:56:46.842 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:56:47.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.069+0000 7fb720a00640 1 Processor -- start 2024-04-14T23:56:47.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.069+0000 7fb720a00640 1 -- start start 2024-04-14T23:56:47.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.071+0000 7fb720a00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb71c0c63c0 0x7fb71c0c6b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:47.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.071+0000 7fb720a00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb71c15c9b0 0x7fb71c0c70b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:47.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.071+0000 7fb720a00640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb71c1530d0 con 0x7fb71c152cf0 2024-04-14T23:56:47.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.071+0000 7fb720a00640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fb71c0751f0 con 0x7fb71c15c9b0 2024-04-14T23:56:47.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.071+0000 7fb720a00640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fb71c0c8370 con 0x7fb71c0c63c0 2024-04-14T23:56:47.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.071+0000 7fb71a800640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb71c15c9b0 0x7fb71c0c70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:47.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.071+0000 7fb719400640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb71c0c63c0 0x7fb71c0c6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:47.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.071+0000 7fb71a800640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb71c15c9b0 0x7fb71c0c70b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54336/0 (socket says 172.21.15.60:54336) 2024-04-14T23:56:47.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.071+0000 7fb71a800640 1 -- 172.21.15.60:0/1385488641 learned_addr learned my addr 172.21.15.60:0/1385488641 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:56:47.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.072+0000 7fb71a800640 1 -- 172.21.15.60:0/1385488641 >> v2:172.21.15.186:3300/0 conn(0x7fb71c0c63c0 msgr2=0x7fb71c0c6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:56:47.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.072+0000 7fb71a800640 1 --2- 172.21.15.60:0/1385488641 >> v2:172.21.15.186:3300/0 conn(0x7fb71c0c63c0 0x7fb71c0c6b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:47.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.072+0000 7fb71a800640 1 -- 172.21.15.60:0/1385488641 >> v1:172.21.15.186:6789/0 conn(0x7fb71c152cf0 legacy=0x7fb71c15c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:47.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.072+0000 7fb71a800640 1 -- 172.21.15.60:0/1385488641 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb71c0c3810 con 0x7fb71c15c9b0 2024-04-14T23:56:47.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.073+0000 7fb71a800640 1 --2- 172.21.15.60:0/1385488641 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb71c15c9b0 0x7fb71c0c70b0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb71c070e80 tx=0x7fb71402c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=747d25b5b67a86c server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:56:47.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.073+0000 7fb718a00640 1 -- 172.21.15.60:0/1385488641 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb71402cbb0 con 0x7fb71c15c9b0 2024-04-14T23:56:47.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.073+0000 7fb718a00640 1 -- 172.21.15.60:0/1385488641 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb71402cd50 con 0x7fb71c15c9b0 2024-04-14T23:56:47.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.073+0000 7fb718a00640 1 -- 172.21.15.60:0/1385488641 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb714034610 con 0x7fb71c15c9b0 2024-04-14T23:56:47.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.074+0000 7fb720a00640 1 -- 172.21.15.60:0/1385488641 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb71c15c9b0 msgr2=0x7fb71c0c70b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:47.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.074+0000 7fb720a00640 1 --2- 172.21.15.60:0/1385488641 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb71c15c9b0 0x7fb71c0c70b0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb71c070e80 tx=0x7fb71402c010 comp rx=0 tx=0).stop 2024-04-14T23:56:47.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.074+0000 7fb720a00640 1 -- 172.21.15.60:0/1385488641 shutdown_connections 2024-04-14T23:56:47.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.074+0000 7fb720a00640 1 --2- 172.21.15.60:0/1385488641 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb71c15c9b0 0x7fb71c0c70b0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:47.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.074+0000 7fb720a00640 1 --2- 172.21.15.60:0/1385488641 >> v2:172.21.15.186:3300/0 conn(0x7fb71c0c63c0 0x7fb71c0c6b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:47.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.074+0000 7fb720a00640 1 -- 172.21.15.60:0/1385488641 >> 172.21.15.60:0/1385488641 conn(0x7fb71c0cc170 msgr2=0x7fb71c0cc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:56:47.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.074+0000 7fb720a00640 1 -- 172.21.15.60:0/1385488641 shutdown_connections 2024-04-14T23:56:47.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.074+0000 7fb720a00640 1 -- 172.21.15.60:0/1385488641 wait complete. 2024-04-14T23:56:47.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.076+0000 7fb720a00640 1 Processor -- start 2024-04-14T23:56:47.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.076+0000 7fb720a00640 1 -- start start 2024-04-14T23:56:47.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.077+0000 7fb720a00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb71c0c63c0 0x7fb71c15b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:47.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.077+0000 7fb720a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb71c152cf0 0x7fb71c1541b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:47.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.077+0000 7fb720a00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb71c15c9b0 0x7fb71c0cb360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:47.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.077+0000 7fb720a00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb71c0c88f0 con 0x7fb71c0c63c0 2024-04-14T23:56:47.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.077+0000 7fb720a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb71c0c8610 con 0x7fb71c152cf0 2024-04-14T23:56:47.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.077+0000 7fb720a00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fb71c159450 con 0x7fb71c15c9b0 2024-04-14T23:56:47.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.077+0000 7fb719e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb71c0c63c0 0x7fb71c15b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:47.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.077+0000 7fb719e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb71c0c63c0 0x7fb71c15b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54352/0 (socket says 172.21.15.60:54352) 2024-04-14T23:56:47.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.077+0000 7fb719e00640 1 -- 172.21.15.60:0/1729876686 learned_addr learned my addr 172.21.15.60:0/1729876686 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:56:47.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.077+0000 7fb719400640 1 --2- 172.21.15.60:0/1729876686 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb71c152cf0 0x7fb71c1541b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:47.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.077+0000 7fb71a800640 1 --2- 172.21.15.60:0/1729876686 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb71c15c9b0 0x7fb71c0cb360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:47.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.078+0000 7fb719e00640 1 -- 172.21.15.60:0/1729876686 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb71c15c9b0 msgr2=0x7fb71c0cb360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:47.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.078+0000 7fb719e00640 1 --2- 172.21.15.60:0/1729876686 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb71c15c9b0 0x7fb71c0cb360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:47.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.078+0000 7fb719e00640 1 -- 172.21.15.60:0/1729876686 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb71c152cf0 msgr2=0x7fb71c1541b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:47.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.078+0000 7fb719e00640 1 --2- 172.21.15.60:0/1729876686 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb71c152cf0 0x7fb71c1541b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:47.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.078+0000 7fb719e00640 1 -- 172.21.15.60:0/1729876686 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb71c153440 con 0x7fb71c0c63c0 2024-04-14T23:56:47.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.078+0000 7fb719e00640 1 --2- 172.21.15.60:0/1729876686 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb71c0c63c0 0x7fb71c15b730 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fb70c00c480 tx=0x7fb70c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:56:47.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.078+0000 7fb712000640 1 -- 172.21.15.60:0/1729876686 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb70c0052d0 con 0x7fb71c0c63c0 2024-04-14T23:56:47.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.078+0000 7fb712000640 1 -- 172.21.15.60:0/1729876686 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb70c005470 con 0x7fb71c0c63c0 2024-04-14T23:56:47.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.078+0000 7fb712000640 1 -- 172.21.15.60:0/1729876686 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb70c012870 con 0x7fb71c0c63c0 2024-04-14T23:56:47.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.078+0000 7fb720a00640 1 -- 172.21.15.60:0/1729876686 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb71c1536e0 con 0x7fb71c0c63c0 2024-04-14T23:56:47.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.078+0000 7fb720a00640 1 -- 172.21.15.60:0/1729876686 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb71c1f2680 con 0x7fb71c0c63c0 2024-04-14T23:56:47.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.080+0000 7fb720a00640 1 -- 172.21.15.60:0/1729876686 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb71c1f2c90 con 0x7fb71c0c63c0 2024-04-14T23:56:47.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.081+0000 7fb712000640 1 -- 172.21.15.60:0/1729876686 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7fb70c012a10 con 0x7fb71c0c63c0 2024-04-14T23:56:47.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.082+0000 7fb712000640 1 --2- 172.21.15.60:0/1729876686 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fb6fc06b990 0x7fb6fc06de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:47.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.082+0000 7fb712000640 1 -- 172.21.15.60:0/1729876686 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6169+0+0 (secure 0 0 0) 0x7fb70c08a540 con 0x7fb71c0c63c0 2024-04-14T23:56:47.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.082+0000 7fb719400640 1 --2- 172.21.15.60:0/1729876686 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fb6fc06b990 0x7fb6fc06de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:47.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.083+0000 7fb719400640 1 --2- 172.21.15.60:0/1729876686 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fb6fc06b990 0x7fb6fc06de50 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb71c0cbba0 tx=0x7fb708009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:56:47.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.087+0000 7fb712000640 1 -- 172.21.15.60:0/1729876686 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb70c058920 con 0x7fb71c0c63c0 2024-04-14T23:56:47.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.333+0000 7fb720a00640 1 -- 172.21.15.60:0/1729876686 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb71c076970 con 0x7fb71c0c63c0 2024-04-14T23:56:47.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.334+0000 7fb712000640 1 -- 172.21.15.60:0/1729876686 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+6007 (secure 0 0 0) 0x7fb70c05c9c0 con 0x7fb71c0c63c0 2024-04-14T23:56:47.339 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:56:47.339 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:56:42.366646+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5191},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5191":{"gid":5191,"name":"c","rank":0,"incarnation":27,"state":"up:rejoin","state_seq":22,"addr":"172.21.15.60:6835/2356847578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2356847578},{"type":"v1","addr":"172.21.15.60:6835","nonce":2356847578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-14T23:56:47.340 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 29 2024-04-14T23:56:47.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.336+0000 7fb720a00640 1 -- 172.21.15.60:0/1729876686 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fb6fc06b990 msgr2=0x7fb6fc06de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:47.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.337+0000 7fb720a00640 1 --2- 172.21.15.60:0/1729876686 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fb6fc06b990 0x7fb6fc06de50 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb71c0cbba0 tx=0x7fb708009190 comp rx=0 tx=0).stop 2024-04-14T23:56:47.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.337+0000 7fb720a00640 1 -- 172.21.15.60:0/1729876686 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb71c0c63c0 msgr2=0x7fb71c15b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:47.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.337+0000 7fb720a00640 1 --2- 172.21.15.60:0/1729876686 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb71c0c63c0 0x7fb71c15b730 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fb70c00c480 tx=0x7fb70c00c940 comp rx=0 tx=0).stop 2024-04-14T23:56:47.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.337+0000 7fb720a00640 1 -- 172.21.15.60:0/1729876686 shutdown_connections 2024-04-14T23:56:47.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.337+0000 7fb720a00640 1 --2- 172.21.15.60:0/1729876686 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fb6fc06b990 0x7fb6fc06de50 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:47.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.337+0000 7fb720a00640 1 --2- 172.21.15.60:0/1729876686 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb71c15c9b0 0x7fb71c0cb360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:47.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.337+0000 7fb720a00640 1 --2- 172.21.15.60:0/1729876686 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb71c152cf0 0x7fb71c1541b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:47.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.337+0000 7fb720a00640 1 --2- 172.21.15.60:0/1729876686 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb71c0c63c0 0x7fb71c15b730 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:47.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.337+0000 7fb720a00640 1 -- 172.21.15.60:0/1729876686 >> 172.21.15.60:0/1729876686 conn(0x7fb71c0cc170 msgr2=0x7fb71c15c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:56:47.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.337+0000 7fb720a00640 1 -- 172.21.15.60:0/1729876686 shutdown_connections 2024-04-14T23:56:47.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:47.337+0000 7fb720a00640 1 -- 172.21.15.60:0/1729876686 wait complete. 2024-04-14T23:56:47.363 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-14T23:56:49.363 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:56:49.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.597+0000 7efd4f200640 1 Processor -- start 2024-04-14T23:56:49.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.598+0000 7efd4f200640 1 -- start start 2024-04-14T23:56:49.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.599+0000 7efd4f200640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7efd500c8b70 0x7efd500709a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:49.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.599+0000 7efd4f200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efd500c9bb0 0x7efd50070ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:49.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.599+0000 7efd4f200640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd500cbb60 con 0x7efd501517c0 2024-04-14T23:56:49.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.599+0000 7efd4f200640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7efd5015e250 con 0x7efd500c8b70 2024-04-14T23:56:49.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.599+0000 7efd4f200640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7efd50151ba0 con 0x7efd500c9bb0 2024-04-14T23:56:49.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.600+0000 7efd4de00640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7efd501517c0 0x7efd5015e6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.60:57558/0 (socket says 172.21.15.60:57558) 2024-04-14T23:56:49.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.600+0000 7efd4de00640 1 -- 172.21.15.60:0/246876053 learned_addr learned my addr 172.21.15.60:0/246876053 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:56:49.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.600+0000 7efd4e800640 1 --2- 172.21.15.60:0/246876053 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efd500c9bb0 0x7efd50070ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:49.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.600+0000 7efd4d400640 1 --2- 172.21.15.60:0/246876053 >> v2:172.21.15.186:3300/0 conn(0x7efd500c8b70 0x7efd500709a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:49.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.600+0000 7efd4ca00640 1 -- 172.21.15.60:0/246876053 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3806265077 0 0) 0x7efd500cbb60 con 0x7efd501517c0 2024-04-14T23:56:49.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.600+0000 7efd4ca00640 1 -- 172.21.15.60:0/246876053 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd30003520 con 0x7efd501517c0 2024-04-14T23:56:49.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.600+0000 7efd4e800640 1 -- 172.21.15.60:0/246876053 >> v2:172.21.15.186:3300/0 conn(0x7efd500c8b70 msgr2=0x7efd500709a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:56:49.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.600+0000 7efd4e800640 1 --2- 172.21.15.60:0/246876053 >> v2:172.21.15.186:3300/0 conn(0x7efd500c8b70 0x7efd500709a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:49.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.600+0000 7efd4e800640 1 -- 172.21.15.60:0/246876053 >> v1:172.21.15.186:6789/0 conn(0x7efd501517c0 legacy=0x7efd5015e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:49.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.600+0000 7efd4e800640 1 -- 172.21.15.60:0/246876053 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd500cb880 con 0x7efd500c9bb0 2024-04-14T23:56:49.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.601+0000 7efd4d400640 1 --2- 172.21.15.60:0/246876053 >> v2:172.21.15.186:3300/0 conn(0x7efd500c8b70 0x7efd500709a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:56:49.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.602+0000 7efd4e800640 1 --2- 172.21.15.60:0/246876053 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efd500c9bb0 0x7efd50070ee0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7efd480097a0 tx=0x7efd4802b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2dc7cfbcfb1075e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:56:49.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.602+0000 7efd4ca00640 1 -- 172.21.15.60:0/246876053 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd48035020 con 0x7efd500c9bb0 2024-04-14T23:56:49.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.602+0000 7efd4ca00640 1 -- 172.21.15.60:0/246876053 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd48031810 con 0x7efd500c9bb0 2024-04-14T23:56:49.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.602+0000 7efd4ca00640 1 -- 172.21.15.60:0/246876053 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd48031bc0 con 0x7efd500c9bb0 2024-04-14T23:56:49.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.603+0000 7efd4f200640 1 -- 172.21.15.60:0/246876053 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efd500c9bb0 msgr2=0x7efd50070ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:49.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.603+0000 7efd4f200640 1 --2- 172.21.15.60:0/246876053 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efd500c9bb0 0x7efd50070ee0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7efd480097a0 tx=0x7efd4802b8b0 comp rx=0 tx=0).stop 2024-04-14T23:56:49.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.603+0000 7efd4f200640 1 -- 172.21.15.60:0/246876053 shutdown_connections 2024-04-14T23:56:49.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.603+0000 7efd4f200640 1 --2- 172.21.15.60:0/246876053 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efd500c9bb0 0x7efd50070ee0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:49.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.603+0000 7efd4f200640 1 --2- 172.21.15.60:0/246876053 >> v2:172.21.15.186:3300/0 conn(0x7efd500c8b70 0x7efd500709a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:49.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.603+0000 7efd4f200640 1 -- 172.21.15.60:0/246876053 >> 172.21.15.60:0/246876053 conn(0x7efd50066f10 msgr2=0x7efd500672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:56:49.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.603+0000 7efd4f200640 1 -- 172.21.15.60:0/246876053 shutdown_connections 2024-04-14T23:56:49.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.603+0000 7efd4f200640 1 -- 172.21.15.60:0/246876053 wait complete. 2024-04-14T23:56:49.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.605+0000 7efd4f200640 1 Processor -- start 2024-04-14T23:56:49.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.605+0000 7efd4f200640 1 -- start start 2024-04-14T23:56:49.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.606+0000 7efd4f200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efd500c8b70 0x7efd500ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:49.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.606+0000 7efd4f200640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd500c9bb0 0x7efd5015c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:49.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.606+0000 7efd4f200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efd501517c0 0x7efd500cdde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:49.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.606+0000 7efd4f200640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7efd5015dee0 con 0x7efd500c9bb0 2024-04-14T23:56:49.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.606+0000 7efd4f200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7efd5015b7c0 con 0x7efd501517c0 2024-04-14T23:56:49.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.606+0000 7efd4f200640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7efd5015edf0 con 0x7efd500c8b70 2024-04-14T23:56:49.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.606+0000 7efd4d400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd500c9bb0 0x7efd5015c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:49.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.606+0000 7efd4e800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efd501517c0 0x7efd500cdde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:49.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.606+0000 7efd4de00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efd500c8b70 0x7efd500ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:49.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.606+0000 7efd4e800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efd501517c0 0x7efd500cdde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:33220/0 (socket says 172.21.15.60:33220) 2024-04-14T23:56:49.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.606+0000 7efd4e800640 1 -- 172.21.15.60:0/3846736982 learned_addr learned my addr 172.21.15.60:0/3846736982 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:56:49.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.607+0000 7efd4e800640 1 -- 172.21.15.60:0/3846736982 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efd500c8b70 msgr2=0x7efd500ceb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:49.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.607+0000 7efd4e800640 1 --2- 172.21.15.60:0/3846736982 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efd500c8b70 0x7efd500ceb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:49.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.607+0000 7efd4e800640 1 -- 172.21.15.60:0/3846736982 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd500c9bb0 msgr2=0x7efd5015c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:49.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.607+0000 7efd4e800640 1 --2- 172.21.15.60:0/3846736982 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd500c9bb0 0x7efd5015c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:49.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.607+0000 7efd4e800640 1 -- 172.21.15.60:0/3846736982 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd500cd060 con 0x7efd501517c0 2024-04-14T23:56:49.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.607+0000 7efd4e800640 1 --2- 172.21.15.60:0/3846736982 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efd501517c0 0x7efd500cdde0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7efd48034040 tx=0x7efd480046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:56:49.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.607+0000 7efd46000640 1 -- 172.21.15.60:0/3846736982 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd48040070 con 0x7efd501517c0 2024-04-14T23:56:49.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.607+0000 7efd46000640 1 -- 172.21.15.60:0/3846736982 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd48039cf0 con 0x7efd501517c0 2024-04-14T23:56:49.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.607+0000 7efd4f200640 1 -- 172.21.15.60:0/3846736982 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd501f1200 con 0x7efd501517c0 2024-04-14T23:56:49.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.607+0000 7efd46000640 1 -- 172.21.15.60:0/3846736982 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd48035040 con 0x7efd501517c0 2024-04-14T23:56:49.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.608+0000 7efd4f200640 1 -- 172.21.15.60:0/3846736982 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd500cd5e0 con 0x7efd501517c0 2024-04-14T23:56:49.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.609+0000 7efd4f200640 1 -- 172.21.15.60:0/3846736982 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd5004f540 con 0x7efd501517c0 2024-04-14T23:56:49.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.610+0000 7efd46000640 1 -- 172.21.15.60:0/3846736982 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7efd48038020 con 0x7efd501517c0 2024-04-14T23:56:49.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.610+0000 7efd46000640 1 --2- 172.21.15.60:0/3846736982 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7efd2c06b990 0x7efd2c06de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:49.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.611+0000 7efd46000640 1 -- 172.21.15.60:0/3846736982 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6169+0+0 (secure 0 0 0) 0x7efd480ac480 con 0x7efd501517c0 2024-04-14T23:56:49.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.611+0000 7efd4de00640 1 --2- 172.21.15.60:0/3846736982 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7efd2c06b990 0x7efd2c06de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:49.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.611+0000 7efd4de00640 1 --2- 172.21.15.60:0/3846736982 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7efd2c06b990 0x7efd2c06de50 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7efd40005880 tx=0x7efd40009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:56:49.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.615+0000 7efd46000640 1 -- 172.21.15.60:0/3846736982 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7efd4807a740 con 0x7efd501517c0 2024-04-14T23:56:49.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.866+0000 7efd4f200640 1 -- 172.21.15.60:0/3846736982 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efd5015a4b0 con 0x7efd501517c0 2024-04-14T23:56:49.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.867+0000 7efd46000640 1 -- 172.21.15.60:0/3846736982 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+6014 (secure 0 0 0) 0x7efd4807e7e0 con 0x7efd501517c0 2024-04-14T23:56:49.876 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:56:49.876 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:56:48.398092+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5191},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5191":{"gid":5191,"name":"c","rank":0,"incarnation":27,"state":"up:active","state_seq":24,"addr":"172.21.15.60:6835/2356847578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2356847578},{"type":"v1","addr":"172.21.15.60:6835","nonce":2356847578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5191,"qdb_cluster":[5191]},"id":1}]} 2024-04-14T23:56:49.878 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 30 2024-04-14T23:56:49.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.870+0000 7efd4f200640 1 -- 172.21.15.60:0/3846736982 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7efd2c06b990 msgr2=0x7efd2c06de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:49.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.870+0000 7efd4f200640 1 --2- 172.21.15.60:0/3846736982 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7efd2c06b990 0x7efd2c06de50 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7efd40005880 tx=0x7efd40009040 comp rx=0 tx=0).stop 2024-04-14T23:56:49.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.870+0000 7efd4f200640 1 -- 172.21.15.60:0/3846736982 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efd501517c0 msgr2=0x7efd500cdde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:49.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.870+0000 7efd4f200640 1 --2- 172.21.15.60:0/3846736982 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efd501517c0 0x7efd500cdde0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7efd48034040 tx=0x7efd480046c0 comp rx=0 tx=0).stop 2024-04-14T23:56:49.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.870+0000 7efd4f200640 1 -- 172.21.15.60:0/3846736982 shutdown_connections 2024-04-14T23:56:49.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.870+0000 7efd4f200640 1 --2- 172.21.15.60:0/3846736982 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7efd2c06b990 0x7efd2c06de50 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:49.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.870+0000 7efd4f200640 1 --2- 172.21.15.60:0/3846736982 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efd501517c0 0x7efd500cdde0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:49.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.870+0000 7efd4f200640 1 --2- 172.21.15.60:0/3846736982 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd500c9bb0 0x7efd5015c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:49.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.871+0000 7efd4f200640 1 --2- 172.21.15.60:0/3846736982 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efd500c8b70 0x7efd500ceb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:49.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.871+0000 7efd4f200640 1 -- 172.21.15.60:0/3846736982 >> 172.21.15.60:0/3846736982 conn(0x7efd50066f10 msgr2=0x7efd50070790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:56:49.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.871+0000 7efd4f200640 1 -- 172.21.15.60:0/3846736982 shutdown_connections 2024-04-14T23:56:49.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:49.871+0000 7efd4f200640 1 -- 172.21.15.60:0/3846736982 wait complete. 2024-04-14T23:56:49.898 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c has gained rank=0, replacing gid=4986 2024-04-14T23:56:49.898 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 1 secs before reviving mds.a 2024-04-14T23:56:50.899 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2024-04-14T23:56:50.899 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-14T23:56:50.899 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-14T23:56:50.907 INFO:tasks.ceph.mds.a:Started 2024-04-14T23:56:50.908 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:56:51.077 INFO:tasks.ceph.mds.a.smithi060.stdout:starting mds.a at 2024-04-14T23:56:51.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.105+0000 7f2c9ca00640 1 Processor -- start 2024-04-14T23:56:51.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.105+0000 7f2c9ca00640 1 -- start start 2024-04-14T23:56:51.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.107+0000 7f2c9ca00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2c980670e0 0x7f2c980c8280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:51.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.107+0000 7f2c9ca00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2c980c87c0 0x7f2c9815e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:51.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.107+0000 7f2c9ca00640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c9806e510 con 0x7f2c98154e70 2024-04-14T23:56:51.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.107+0000 7f2c9ca00640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f2c9815f030 con 0x7f2c980c87c0 2024-04-14T23:56:51.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.107+0000 7f2c9ca00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f2c980c7df0 con 0x7f2c980670e0 2024-04-14T23:56:51.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.107+0000 7f2c95e00640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f2c98154e70 0x7f2c9806dc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.60:57562/0 (socket says 172.21.15.60:57562) 2024-04-14T23:56:51.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.107+0000 7f2c95e00640 1 -- 172.21.15.60:0/3318416490 learned_addr learned my addr 172.21.15.60:0/3318416490 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:56:51.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.107+0000 7f2c95400640 1 --2- 172.21.15.60:0/3318416490 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2c980670e0 0x7f2c980c8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:51.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.107+0000 7f2c96800640 1 --2- 172.21.15.60:0/3318416490 >> v2:172.21.15.186:3300/0 conn(0x7f2c980c87c0 0x7f2c9815e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:51.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.108+0000 7f2c94a00640 1 -- 172.21.15.60:0/3318416490 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4099896108 0 0) 0x7f2c9806e510 con 0x7f2c98154e70 2024-04-14T23:56:51.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.108+0000 7f2c94a00640 1 -- 172.21.15.60:0/3318416490 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2c7c003520 con 0x7f2c98154e70 2024-04-14T23:56:51.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.108+0000 7f2c95400640 1 -- 172.21.15.60:0/3318416490 >> v2:172.21.15.186:3300/0 conn(0x7f2c980c87c0 msgr2=0x7f2c9815e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:56:51.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.108+0000 7f2c95400640 1 --2- 172.21.15.60:0/3318416490 >> v2:172.21.15.186:3300/0 conn(0x7f2c980c87c0 0x7f2c9815e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:51.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.108+0000 7f2c95400640 1 -- 172.21.15.60:0/3318416490 >> v1:172.21.15.186:6789/0 conn(0x7f2c98154e70 legacy=0x7f2c9806dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:51.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.108+0000 7f2c95400640 1 -- 172.21.15.60:0/3318416490 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c980c74c0 con 0x7f2c980670e0 2024-04-14T23:56:51.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.108+0000 7f2c96800640 1 --2- 172.21.15.60:0/3318416490 >> v2:172.21.15.186:3300/0 conn(0x7f2c980c87c0 0x7f2c9815e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:56:51.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.109+0000 7f2c95400640 1 --2- 172.21.15.60:0/3318416490 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2c980670e0 0x7f2c980c8280 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2c840097a0 tx=0x7f2c8402b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42b7aa2fc52ecb43 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:56:51.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.109+0000 7f2c94a00640 1 -- 172.21.15.60:0/3318416490 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c84034070 con 0x7f2c980670e0 2024-04-14T23:56:51.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.109+0000 7f2c94a00640 1 -- 172.21.15.60:0/3318416490 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c84030800 con 0x7f2c980670e0 2024-04-14T23:56:51.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.110+0000 7f2c94a00640 1 -- 172.21.15.60:0/3318416490 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c84030bd0 con 0x7f2c980670e0 2024-04-14T23:56:51.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.110+0000 7f2c9ca00640 1 -- 172.21.15.60:0/3318416490 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2c980670e0 msgr2=0x7f2c980c8280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:51.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.110+0000 7f2c9ca00640 1 --2- 172.21.15.60:0/3318416490 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2c980670e0 0x7f2c980c8280 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2c840097a0 tx=0x7f2c8402b940 comp rx=0 tx=0).stop 2024-04-14T23:56:51.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.110+0000 7f2c9ca00640 1 -- 172.21.15.60:0/3318416490 shutdown_connections 2024-04-14T23:56:51.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.110+0000 7f2c9ca00640 1 --2- 172.21.15.60:0/3318416490 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2c980670e0 0x7f2c980c8280 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:51.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.110+0000 7f2c9ca00640 1 --2- 172.21.15.60:0/3318416490 >> v2:172.21.15.186:3300/0 conn(0x7f2c980c87c0 0x7f2c9815e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:51.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.110+0000 7f2c9ca00640 1 -- 172.21.15.60:0/3318416490 >> 172.21.15.60:0/3318416490 conn(0x7f2c980cc8b0 msgr2=0x7f2c980cccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:56:51.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.110+0000 7f2c9ca00640 1 -- 172.21.15.60:0/3318416490 shutdown_connections 2024-04-14T23:56:51.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.110+0000 7f2c9ca00640 1 -- 172.21.15.60:0/3318416490 wait complete. 2024-04-14T23:56:51.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.112+0000 7f2c9ca00640 1 Processor -- start 2024-04-14T23:56:51.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.112+0000 7f2c9ca00640 1 -- start start 2024-04-14T23:56:51.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.113+0000 7f2c9ca00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2c980670e0 0x7f2c9814f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:51.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.113+0000 7f2c9ca00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c980c87c0 0x7f2c980cb900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:51.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.113+0000 7f2c9ca00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c98154e70 0x7f2c9814db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:51.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.113+0000 7f2c9ca00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f2c980ca6b0 con 0x7f2c980c87c0 2024-04-14T23:56:51.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.113+0000 7f2c9ca00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2c980c7a80 con 0x7f2c98154e70 2024-04-14T23:56:51.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.113+0000 7f2c9ca00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f2c980674e0 con 0x7f2c980670e0 2024-04-14T23:56:51.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.114+0000 7f2c95e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2c980670e0 0x7f2c9814f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:51.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.114+0000 7f2c95400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c980c87c0 0x7f2c980cb900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:51.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.114+0000 7f2c96800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c98154e70 0x7f2c9814db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:51.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.114+0000 7f2c95400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c980c87c0 0x7f2c980cb900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54390/0 (socket says 172.21.15.60:54390) 2024-04-14T23:56:51.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.114+0000 7f2c95e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2c980670e0 0x7f2c9814f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.60:54288/0 (socket says 172.21.15.60:54288) 2024-04-14T23:56:51.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.114+0000 7f2c95400640 1 -- 172.21.15.60:0/1652878277 learned_addr learned my addr 172.21.15.60:0/1652878277 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:56:51.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.114+0000 7f2c96800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c98154e70 0x7f2c9814db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:33262/0 (socket says 172.21.15.60:33262) 2024-04-14T23:56:51.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.114+0000 7f2c95e00640 1 -- 172.21.15.60:0/1652878277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c98154e70 msgr2=0x7f2c9814db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:51.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.114+0000 7f2c95e00640 1 --2- 172.21.15.60:0/1652878277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c98154e70 0x7f2c9814db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:51.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.114+0000 7f2c95e00640 1 -- 172.21.15.60:0/1652878277 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c980c87c0 msgr2=0x7f2c980cb900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:51.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.114+0000 7f2c95e00640 1 --2- 172.21.15.60:0/1652878277 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c980c87c0 0x7f2c980cb900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:51.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.114+0000 7f2c95e00640 1 -- 172.21.15.60:0/1652878277 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c981567f0 con 0x7f2c980670e0 2024-04-14T23:56:51.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.114+0000 7f2c96800640 1 --2- 172.21.15.60:0/1652878277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c98154e70 0x7f2c9814db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:56:51.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.114+0000 7f2c95e00640 1 --2- 172.21.15.60:0/1652878277 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2c980670e0 0x7f2c9814f110 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f2c8800c3c0 tx=0x7f2c8800c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:56:51.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.115+0000 7f2c8e000640 1 -- 172.21.15.60:0/1652878277 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c88017070 con 0x7f2c980670e0 2024-04-14T23:56:51.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.115+0000 7f2c8e000640 1 -- 172.21.15.60:0/1652878277 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c88005ce0 con 0x7f2c980670e0 2024-04-14T23:56:51.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.115+0000 7f2c8e000640 1 -- 172.21.15.60:0/1652878277 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c880072d0 con 0x7f2c980670e0 2024-04-14T23:56:51.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.115+0000 7f2c9ca00640 1 -- 172.21.15.60:0/1652878277 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c98156ad0 con 0x7f2c980670e0 2024-04-14T23:56:51.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.115+0000 7f2c9ca00640 1 -- 172.21.15.60:0/1652878277 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c98157090 con 0x7f2c980670e0 2024-04-14T23:56:51.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.116+0000 7f2c9ca00640 1 -- 172.21.15.60:0/1652878277 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c98076970 con 0x7f2c980670e0 2024-04-14T23:56:51.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.117+0000 7f2c8e000640 1 -- 172.21.15.60:0/1652878277 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f2c8801a070 con 0x7f2c980670e0 2024-04-14T23:56:51.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.118+0000 7f2c8e000640 1 --2- 172.21.15.60:0/1652878277 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f2c7806b990 0x7f2c7806de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:51.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.118+0000 7f2c8e000640 1 -- 172.21.15.60:0/1652878277 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(44..44 src has 1..44) ==== 6169+0+0 (secure 0 0 0) 0x7f2c88089d70 con 0x7f2c980670e0 2024-04-14T23:56:51.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.118+0000 7f2c95400640 1 --2- 172.21.15.60:0/1652878277 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f2c7806b990 0x7f2c7806de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:51.125 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.119+0000 7f2c95400640 1 --2- 172.21.15.60:0/1652878277 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f2c7806b990 0x7f2c7806de50 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f2c980cc360 tx=0x7f2c8403b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:56:51.129 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.123+0000 7f2c8e000640 1 -- 172.21.15.60:0/1652878277 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2c88058030 con 0x7f2c980670e0 2024-04-14T23:56:51.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.371+0000 7f2c9ca00640 1 -- 172.21.15.60:0/1652878277 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2c98068940 con 0x7f2c980670e0 2024-04-14T23:56:51.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.372+0000 7f2c8e000640 1 -- 172.21.15.60:0/1652878277 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+6014 (secure 0 0 0) 0x7f2c8805c0d0 con 0x7f2c980670e0 2024-04-14T23:56:51.379 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:56:51.379 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:56:48.398092+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5191},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5191":{"gid":5191,"name":"c","rank":0,"incarnation":27,"state":"up:active","state_seq":24,"addr":"172.21.15.60:6835/2356847578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2356847578},{"type":"v1","addr":"172.21.15.60:6835","nonce":2356847578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5191,"qdb_cluster":[5191]},"id":1}]} 2024-04-14T23:56:51.379 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 30 2024-04-14T23:56:51.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.375+0000 7f2c9ca00640 1 -- 172.21.15.60:0/1652878277 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f2c7806b990 msgr2=0x7f2c7806de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:51.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.375+0000 7f2c9ca00640 1 --2- 172.21.15.60:0/1652878277 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f2c7806b990 0x7f2c7806de50 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f2c980cc360 tx=0x7f2c8403b040 comp rx=0 tx=0).stop 2024-04-14T23:56:51.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.375+0000 7f2c9ca00640 1 -- 172.21.15.60:0/1652878277 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2c980670e0 msgr2=0x7f2c9814f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:51.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.375+0000 7f2c9ca00640 1 --2- 172.21.15.60:0/1652878277 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2c980670e0 0x7f2c9814f110 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f2c8800c3c0 tx=0x7f2c8800c880 comp rx=0 tx=0).stop 2024-04-14T23:56:51.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.375+0000 7f2c9ca00640 1 -- 172.21.15.60:0/1652878277 shutdown_connections 2024-04-14T23:56:51.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.375+0000 7f2c9ca00640 1 --2- 172.21.15.60:0/1652878277 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f2c7806b990 0x7f2c7806de50 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:51.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.375+0000 7f2c9ca00640 1 --2- 172.21.15.60:0/1652878277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c98154e70 0x7f2c9814db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:51.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.375+0000 7f2c9ca00640 1 --2- 172.21.15.60:0/1652878277 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c980c87c0 0x7f2c980cb900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:51.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.375+0000 7f2c9ca00640 1 --2- 172.21.15.60:0/1652878277 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2c980670e0 0x7f2c9814f110 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:51.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.375+0000 7f2c9ca00640 1 -- 172.21.15.60:0/1652878277 >> 172.21.15.60:0/1652878277 conn(0x7f2c980cc8b0 msgr2=0x7f2c980c6900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:56:51.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.375+0000 7f2c9ca00640 1 -- 172.21.15.60:0/1652878277 shutdown_connections 2024-04-14T23:56:51.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:51.375+0000 7f2c9ca00640 1 -- 172.21.15.60:0/1652878277 wait complete. 2024-04-14T23:56:51.399 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2024-04-14T23:56:53.400 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:56:53.623 INFO:tasks.workunit.client.0.smithi060.stderr:error: RPC failed; curl 56 OpenSSL SSL_read: error:0A000126:SSL routines::unexpected eof while reading, errno 0 2024-04-14T23:56:53.623 INFO:tasks.workunit.client.0.smithi060.stderr:fetch-pack: unexpected disconnect while reading sideband packet 2024-04-14T23:56:53.623 INFO:tasks.workunit.client.0.smithi060.stderr:fatal: early EOF 2024-04-14T23:56:53.633 INFO:tasks.workunit.client.0.smithi060.stderr:fatal: fetch-pack: invalid index-pack output 2024-04-14T23:56:53.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.634+0000 7f679a000640 1 Processor -- start 2024-04-14T23:56:53.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.634+0000 7f679a000640 1 -- start start 2024-04-14T23:56:53.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.636+0000 7f679a000640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6794152cf0 0x7f679415c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:53.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.636+0000 7f679a000640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f67940c63c0 0x7f679415c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:53.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.636+0000 7f679a000640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67941530d0 con 0x7f6794159450 2024-04-14T23:56:53.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.636+0000 7f679a000640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f67940751f0 con 0x7f6794152cf0 2024-04-14T23:56:53.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.636+0000 7f679a000640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f67940c8370 con 0x7f67940c63c0 2024-04-14T23:56:53.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.636+0000 7f6793400640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6794152cf0 0x7f679415c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:53.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.636+0000 7f6793e00640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f6794159450 0x7f67940c3f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:41560/0 (socket says 172.21.15.60:41560) 2024-04-14T23:56:53.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.636+0000 7f6793e00640 1 -- 172.21.15.60:0/2127678319 learned_addr learned my addr 172.21.15.60:0/2127678319 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:56:53.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.636+0000 7f6792a00640 1 --2- 172.21.15.60:0/2127678319 >> v2:172.21.15.186:3300/0 conn(0x7f67940c63c0 0x7f679415c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:53.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.636+0000 7f6793400640 1 -- 172.21.15.60:0/2127678319 >> v2:172.21.15.186:3300/0 conn(0x7f67940c63c0 msgr2=0x7f679415c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:56:53.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.636+0000 7f6793400640 1 --2- 172.21.15.60:0/2127678319 >> v2:172.21.15.186:3300/0 conn(0x7f67940c63c0 0x7f679415c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:53.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.637+0000 7f6793400640 1 -- 172.21.15.60:0/2127678319 >> v1:172.21.15.60:6789/0 conn(0x7f6794159450 legacy=0x7f67940c3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:53.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.637+0000 7f6793400640 1 -- 172.21.15.60:0/2127678319 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67940c3810 con 0x7f6794152cf0 2024-04-14T23:56:53.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.637+0000 7f6792000640 1 -- 172.21.15.60:0/2127678319 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4046057559 0 0) 0x7f67941530d0 con 0x7f6794159450 2024-04-14T23:56:53.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.638+0000 7f6793400640 1 --2- 172.21.15.60:0/2127678319 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6794152cf0 0x7f679415c2a0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f6794070e80 tx=0x7f678402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae26b2d524b57065 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:56:53.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.638+0000 7f6792000640 1 -- 172.21.15.60:0/2127678319 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6784034070 con 0x7f6794152cf0 2024-04-14T23:56:53.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.638+0000 7f6792000640 1 -- 172.21.15.60:0/2127678319 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6784030610 con 0x7f6794152cf0 2024-04-14T23:56:53.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.638+0000 7f6792000640 1 -- 172.21.15.60:0/2127678319 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67840309e0 con 0x7f6794152cf0 2024-04-14T23:56:53.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.639+0000 7f679a000640 1 -- 172.21.15.60:0/2127678319 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6794152cf0 msgr2=0x7f679415c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:53.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.639+0000 7f679a000640 1 --2- 172.21.15.60:0/2127678319 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6794152cf0 0x7f679415c2a0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f6794070e80 tx=0x7f678402b960 comp rx=0 tx=0).stop 2024-04-14T23:56:53.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.639+0000 7f679a000640 1 -- 172.21.15.60:0/2127678319 shutdown_connections 2024-04-14T23:56:53.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.639+0000 7f679a000640 1 --2- 172.21.15.60:0/2127678319 >> v2:172.21.15.186:3300/0 conn(0x7f67940c63c0 0x7f679415c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:53.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.639+0000 7f679a000640 1 --2- 172.21.15.60:0/2127678319 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6794152cf0 0x7f679415c2a0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:53.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.639+0000 7f679a000640 1 -- 172.21.15.60:0/2127678319 >> 172.21.15.60:0/2127678319 conn(0x7f67940cc170 msgr2=0x7f67940cc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:56:53.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.639+0000 7f679a000640 1 -- 172.21.15.60:0/2127678319 shutdown_connections 2024-04-14T23:56:53.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.639+0000 7f679a000640 1 -- 172.21.15.60:0/2127678319 wait complete. 2024-04-14T23:56:53.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.641+0000 7f679a000640 1 Processor -- start 2024-04-14T23:56:53.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.641+0000 7f679a000640 1 -- start start 2024-04-14T23:56:53.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.642+0000 7f679a000640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67940c63c0 0x7f67941488e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:53.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.642+0000 7f679a000640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6794152cf0 0x7f67941519a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:53.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.642+0000 7f679a000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6794159450 0x7f67941f3f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:53.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.642+0000 7f679a000640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f67940c88f0 con 0x7f67940c63c0 2024-04-14T23:56:53.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.642+0000 7f679a000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f67940c8610 con 0x7f6794159450 2024-04-14T23:56:53.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.642+0000 7f679a000640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f67940c4690 con 0x7f6794152cf0 2024-04-14T23:56:53.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.642+0000 7f6793400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67940c63c0 0x7f67941488e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:53.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.642+0000 7f6793400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67940c63c0 0x7f67941488e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49868/0 (socket says 172.21.15.60:49868) 2024-04-14T23:56:53.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.642+0000 7f6793400640 1 -- 172.21.15.60:0/2527388925 learned_addr learned my addr 172.21.15.60:0/2527388925 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:56:53.650 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.642+0000 7f6793e00640 1 --2- 172.21.15.60:0/2527388925 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6794159450 0x7f67941f3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:53.650 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.642+0000 7f6792a00640 1 --2- 172.21.15.60:0/2527388925 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6794152cf0 0x7f67941519a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:53.650 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.643+0000 7f6793400640 1 -- 172.21.15.60:0/2527388925 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6794152cf0 msgr2=0x7f67941519a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:53.650 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.643+0000 7f6793400640 1 --2- 172.21.15.60:0/2527388925 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6794152cf0 0x7f67941519a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:53.650 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.643+0000 7f6793400640 1 -- 172.21.15.60:0/2527388925 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6794159450 msgr2=0x7f67941f3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:53.650 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.643+0000 7f6793400640 1 --2- 172.21.15.60:0/2527388925 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6794159450 0x7f67941f3f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:53.650 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.643+0000 7f6793400640 1 -- 172.21.15.60:0/2527388925 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6794140200 con 0x7f67940c63c0 2024-04-14T23:56:53.650 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.643+0000 7f6793400640 1 --2- 172.21.15.60:0/2527388925 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67940c63c0 0x7f67941488e0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f6784000c00 tx=0x7f67840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:56:53.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.643+0000 7f678b400640 1 -- 172.21.15.60:0/2527388925 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6784034040 con 0x7f67940c63c0 2024-04-14T23:56:53.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.643+0000 7f678b400640 1 -- 172.21.15.60:0/2527388925 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6784030410 con 0x7f67940c63c0 2024-04-14T23:56:53.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.643+0000 7f678b400640 1 -- 172.21.15.60:0/2527388925 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6784039c50 con 0x7f67940c63c0 2024-04-14T23:56:53.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.643+0000 7f679a000640 1 -- 172.21.15.60:0/2527388925 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67940ca340 con 0x7f67940c63c0 2024-04-14T23:56:53.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.644+0000 7f679a000640 1 -- 172.21.15.60:0/2527388925 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6794072ab0 con 0x7f67940c63c0 2024-04-14T23:56:53.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.645+0000 7f679a000640 1 -- 172.21.15.60:0/2527388925 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6794076970 con 0x7f67940c63c0 2024-04-14T23:56:53.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.646+0000 7f678b400640 1 -- 172.21.15.60:0/2527388925 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f6784037070 con 0x7f67940c63c0 2024-04-14T23:56:53.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.647+0000 7f678b400640 1 --2- 172.21.15.60:0/2527388925 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f677406b9c0 0x7f677406de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:56:53.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.647+0000 7f678b400640 1 -- 172.21.15.60:0/2527388925 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6169+0+0 (secure 0 0 0) 0x7f67840ac940 con 0x7f67940c63c0 2024-04-14T23:56:53.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.648+0000 7f6792a00640 1 --2- 172.21.15.60:0/2527388925 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f677406b9c0 0x7f677406de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:56:53.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.648+0000 7f6792a00640 1 --2- 172.21.15.60:0/2527388925 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f677406b9c0 0x7f677406de80 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f679415ca40 tx=0x7f6780009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:56:53.658 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.652+0000 7f678b400640 1 -- 172.21.15.60:0/2527388925 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f678407ac90 con 0x7f67940c63c0 2024-04-14T23:56:53.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.901+0000 7f679a000640 1 -- 172.21.15.60:0/2527388925 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f67941562f0 con 0x7f67940c63c0 2024-04-14T23:56:53.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.902+0000 7f678b400640 1 -- 172.21.15.60:0/2527388925 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6839 (secure 0 0 0) 0x7f678407ed30 con 0x7f67940c63c0 2024-04-14T23:56:53.909 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:56:53.909 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5286,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/3281566884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3281566884},{"type":"v1","addr":"172.21.15.60:6833","nonce":3281566884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:56:48.398092+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5191},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5191":{"gid":5191,"name":"c","rank":0,"incarnation":27,"state":"up:active","state_seq":24,"addr":"172.21.15.60:6835/2356847578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2356847578},{"type":"v1","addr":"172.21.15.60:6835","nonce":2356847578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5191,"qdb_cluster":[5191]},"id":1}]} 2024-04-14T23:56:53.909 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 31 2024-04-14T23:56:53.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.905+0000 7f679a000640 1 -- 172.21.15.60:0/2527388925 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f677406b9c0 msgr2=0x7f677406de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:53.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.905+0000 7f679a000640 1 --2- 172.21.15.60:0/2527388925 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f677406b9c0 0x7f677406de80 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f679415ca40 tx=0x7f6780009190 comp rx=0 tx=0).stop 2024-04-14T23:56:53.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.905+0000 7f679a000640 1 -- 172.21.15.60:0/2527388925 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67940c63c0 msgr2=0x7f67941488e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:56:53.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.905+0000 7f679a000640 1 --2- 172.21.15.60:0/2527388925 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67940c63c0 0x7f67941488e0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f6784000c00 tx=0x7f67840046c0 comp rx=0 tx=0).stop 2024-04-14T23:56:53.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.906+0000 7f679a000640 1 -- 172.21.15.60:0/2527388925 shutdown_connections 2024-04-14T23:56:53.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.906+0000 7f679a000640 1 --2- 172.21.15.60:0/2527388925 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f677406b9c0 0x7f677406de80 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:53.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.906+0000 7f679a000640 1 --2- 172.21.15.60:0/2527388925 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6794159450 0x7f67941f3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:53.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.906+0000 7f679a000640 1 --2- 172.21.15.60:0/2527388925 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6794152cf0 0x7f67941519a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:53.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.906+0000 7f679a000640 1 --2- 172.21.15.60:0/2527388925 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67940c63c0 0x7f67941488e0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:56:53.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.906+0000 7f679a000640 1 -- 172.21.15.60:0/2527388925 >> 172.21.15.60:0/2527388925 conn(0x7f67940cc170 msgr2=0x7f6794158c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:56:53.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.906+0000 7f679a000640 1 -- 172.21.15.60:0/2527388925 shutdown_connections 2024-04-14T23:56:53.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:56:53.906+0000 7f679a000640 1 -- 172.21.15.60:0/2527388925 wait complete. 2024-04-14T23:56:53.932 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2024-04-14T23:56:53.932 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 33 secs before thrashing 2024-04-14T23:56:54.710 INFO:tasks.workunit.client.0.smithi060.stderr:+ retry 2024-04-14T23:56:54.710 INFO:tasks.workunit.client.0.smithi060.stderr:+ rm -rf ceph 2024-04-14T23:56:54.715 INFO:tasks.workunit.client.0.smithi060.stderr:+ timeout 3600 git clone https://git.ceph.com/ceph.git 2024-04-14T23:56:54.725 INFO:tasks.workunit.client.0.smithi060.stderr:Cloning into 'ceph'... 2024-04-14T23:56:57.608 INFO:tasks.ceph.mds.c.smithi060.stderr:2024-04-14T23:56:57.602+0000 7efe12400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-14T23:57:16.214 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-14T23:57:16.223 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-14T23:57:26.933 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:57:27.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.143+0000 7f6a54a00640 1 Processor -- start 2024-04-14T23:57:27.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.143+0000 7f6a54a00640 1 -- start start 2024-04-14T23:57:27.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.145+0000 7f6a54a00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6a50070d50 0x7f6a5015a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:27.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.145+0000 7f6a54a00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a5006b4b0 con 0x7f6a500c5380 2024-04-14T23:57:27.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.145+0000 7f6a54a00640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a5006b790 con 0x7f6a500c4fa0 2024-04-14T23:57:27.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.145+0000 7f6a54a00640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f6a500c69f0 con 0x7f6a50070d50 2024-04-14T23:57:27.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.146+0000 7f6a4fe00640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f6a500c5380 0x7f6a5015a7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54066/0 (socket says 172.21.15.60:54066) 2024-04-14T23:57:27.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.146+0000 7f6a4fe00640 1 -- 172.21.15.60:0/9573995 learned_addr learned my addr 172.21.15.60:0/9573995 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:57:27.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.146+0000 7f6a4ea00640 1 --2- 172.21.15.60:0/9573995 >> v2:172.21.15.186:3300/0 conn(0x7f6a50070d50 0x7f6a5015a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:27.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.146+0000 7f6a4ea00640 1 -- 172.21.15.60:0/9573995 >> v1:172.21.15.186:6789/0 conn(0x7f6a500c4fa0 legacy=0x7f6a500c77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:27.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.146+0000 7f6a4e000640 1 -- 172.21.15.60:0/9573995 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 130637976 0 0) 0x7f6a5006b4b0 con 0x7f6a500c5380 2024-04-14T23:57:27.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.146+0000 7f6a4ea00640 1 -- 172.21.15.60:0/9573995 >> v1:172.21.15.60:6789/0 conn(0x7f6a500c5380 legacy=0x7f6a5015a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:27.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.146+0000 7f6a4ea00640 1 -- 172.21.15.60:0/9573995 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a500c6cd0 con 0x7f6a50070d50 2024-04-14T23:57:27.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.147+0000 7f6a4ea00640 1 --2- 172.21.15.60:0/9573995 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a50070d50 0x7f6a5015a270 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c009720 tx=0x7f6a3c02b730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5bd75c59c0b76e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:57:27.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.148+0000 7f6a4e000640 1 -- 172.21.15.60:0/9573995 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a3c037070 con 0x7f6a50070d50 2024-04-14T23:57:27.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.148+0000 7f6a4e000640 1 -- 172.21.15.60:0/9573995 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a3c030710 con 0x7f6a50070d50 2024-04-14T23:57:27.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.148+0000 7f6a4e000640 1 -- 172.21.15.60:0/9573995 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a3c030ae0 con 0x7f6a50070d50 2024-04-14T23:57:27.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.148+0000 7f6a54a00640 1 -- 172.21.15.60:0/9573995 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a50070d50 msgr2=0x7f6a5015a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:27.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.148+0000 7f6a54a00640 1 --2- 172.21.15.60:0/9573995 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a50070d50 0x7f6a5015a270 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c009720 tx=0x7f6a3c02b730 comp rx=0 tx=0).stop 2024-04-14T23:57:27.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.148+0000 7f6a54a00640 1 -- 172.21.15.60:0/9573995 shutdown_connections 2024-04-14T23:57:27.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.148+0000 7f6a54a00640 1 --2- 172.21.15.60:0/9573995 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a50070d50 0x7f6a5015a270 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:27.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.148+0000 7f6a54a00640 1 -- 172.21.15.60:0/9573995 >> 172.21.15.60:0/9573995 conn(0x7f6a500c8b50 msgr2=0x7f6a500c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:57:27.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.148+0000 7f6a54a00640 1 -- 172.21.15.60:0/9573995 shutdown_connections 2024-04-14T23:57:27.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.149+0000 7f6a54a00640 1 -- 172.21.15.60:0/9573995 wait complete. 2024-04-14T23:57:27.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.150+0000 7f6a54a00640 1 Processor -- start 2024-04-14T23:57:27.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.150+0000 7f6a54a00640 1 -- start start 2024-04-14T23:57:27.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.151+0000 7f6a54a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a50070d50 0x7f6a501f1d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:27.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.151+0000 7f6a54a00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a500c4fa0 0x7f6a501f4080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:27.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.151+0000 7f6a54a00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6a500c5380 0x7f6a501f45c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:27.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.151+0000 7f6a54a00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f6a5006b1c0 con 0x7f6a500c4fa0 2024-04-14T23:57:27.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.151+0000 7f6a54a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6a500c7290 con 0x7f6a50070d50 2024-04-14T23:57:27.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.151+0000 7f6a54a00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f6a500c99d0 con 0x7f6a500c5380 2024-04-14T23:57:27.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.152+0000 7f6a4ea00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a500c4fa0 0x7f6a501f4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:27.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.152+0000 7f6a4ea00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a500c4fa0 0x7f6a501f4080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51130/0 (socket says 172.21.15.60:51130) 2024-04-14T23:57:27.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.152+0000 7f6a4ea00640 1 -- 172.21.15.60:0/3214810658 learned_addr learned my addr 172.21.15.60:0/3214810658 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:57:27.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.152+0000 7f6a4fe00640 1 --2- 172.21.15.60:0/3214810658 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6a500c5380 0x7f6a501f45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:27.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.152+0000 7f6a4f400640 1 --2- 172.21.15.60:0/3214810658 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a50070d50 0x7f6a501f1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:27.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.152+0000 7f6a4ea00640 1 -- 172.21.15.60:0/3214810658 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6a500c5380 msgr2=0x7f6a501f45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:27.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.152+0000 7f6a4ea00640 1 --2- 172.21.15.60:0/3214810658 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6a500c5380 0x7f6a501f45c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:27.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.152+0000 7f6a4ea00640 1 -- 172.21.15.60:0/3214810658 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a50070d50 msgr2=0x7f6a501f1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:27.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.152+0000 7f6a4ea00640 1 --2- 172.21.15.60:0/3214810658 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a50070d50 0x7f6a501f1d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:27.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.152+0000 7f6a4ea00640 1 -- 172.21.15.60:0/3214810658 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a501dffc0 con 0x7f6a500c4fa0 2024-04-14T23:57:27.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.152+0000 7f6a4f400640 1 --2- 172.21.15.60:0/3214810658 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a50070d50 0x7f6a501f1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:57:27.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.153+0000 7f6a4ea00640 1 --2- 172.21.15.60:0/3214810658 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a500c4fa0 0x7f6a501f4080 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c035040 tx=0x7f6a3c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:57:27.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.153+0000 7f6a47400640 1 -- 172.21.15.60:0/3214810658 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a3c037040 con 0x7f6a500c4fa0 2024-04-14T23:57:27.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.153+0000 7f6a47400640 1 -- 172.21.15.60:0/3214810658 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a3c004030 con 0x7f6a500c4fa0 2024-04-14T23:57:27.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.153+0000 7f6a47400640 1 -- 172.21.15.60:0/3214810658 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a3c004420 con 0x7f6a500c4fa0 2024-04-14T23:57:27.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.153+0000 7f6a54a00640 1 -- 172.21.15.60:0/3214810658 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a501e1770 con 0x7f6a500c4fa0 2024-04-14T23:57:27.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.153+0000 7f6a54a00640 1 -- 172.21.15.60:0/3214810658 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a501e1d30 con 0x7f6a500c4fa0 2024-04-14T23:57:27.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.154+0000 7f6a54a00640 1 -- 172.21.15.60:0/3214810658 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a5004f540 con 0x7f6a500c4fa0 2024-04-14T23:57:27.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.156+0000 7f6a47400640 1 -- 172.21.15.60:0/3214810658 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f6a3c032940 con 0x7f6a500c4fa0 2024-04-14T23:57:27.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.157+0000 7f6a47400640 1 --2- 172.21.15.60:0/3214810658 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f6a3006b990 0x7f6a3006de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:27.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.157+0000 7f6a47400640 1 -- 172.21.15.60:0/3214810658 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6169+0+0 (secure 0 0 0) 0x7f6a3c0ac880 con 0x7f6a500c4fa0 2024-04-14T23:57:27.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.157+0000 7f6a4f400640 1 --2- 172.21.15.60:0/3214810658 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f6a3006b990 0x7f6a3006de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:27.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.158+0000 7f6a4f400640 1 --2- 172.21.15.60:0/3214810658 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f6a3006b990 0x7f6a3006de50 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f6a40000f60 tx=0x7f6a40009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:57:27.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.162+0000 7f6a47400640 1 -- 172.21.15.60:0/3214810658 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6a3c042020 con 0x7f6a500c4fa0 2024-04-14T23:57:27.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.404+0000 7f6a54a00640 1 -- 172.21.15.60:0/3214810658 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6a500c9fd0 con 0x7f6a500c4fa0 2024-04-14T23:57:27.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.404+0000 7f6a47400640 1 -- 172.21.15.60:0/3214810658 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6839 (secure 0 0 0) 0x7f6a3c07ab40 con 0x7f6a500c4fa0 2024-04-14T23:57:27.411 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:57:27.411 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5286,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/3281566884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3281566884},{"type":"v1","addr":"172.21.15.60:6833","nonce":3281566884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:56:48.398092+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5191},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5191":{"gid":5191,"name":"c","rank":0,"incarnation":27,"state":"up:active","state_seq":24,"addr":"172.21.15.60:6835/2356847578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2356847578},{"type":"v1","addr":"172.21.15.60:6835","nonce":2356847578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5191,"qdb_cluster":[5191]},"id":1}]} 2024-04-14T23:57:27.411 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 31 2024-04-14T23:57:27.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.407+0000 7f6a54a00640 1 -- 172.21.15.60:0/3214810658 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f6a3006b990 msgr2=0x7f6a3006de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:27.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.407+0000 7f6a54a00640 1 --2- 172.21.15.60:0/3214810658 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f6a3006b990 0x7f6a3006de50 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f6a40000f60 tx=0x7f6a40009040 comp rx=0 tx=0).stop 2024-04-14T23:57:27.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.407+0000 7f6a54a00640 1 -- 172.21.15.60:0/3214810658 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a500c4fa0 msgr2=0x7f6a501f4080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:27.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.407+0000 7f6a54a00640 1 --2- 172.21.15.60:0/3214810658 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a500c4fa0 0x7f6a501f4080 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c035040 tx=0x7f6a3c0046c0 comp rx=0 tx=0).stop 2024-04-14T23:57:27.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.407+0000 7f6a54a00640 1 -- 172.21.15.60:0/3214810658 shutdown_connections 2024-04-14T23:57:27.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.407+0000 7f6a54a00640 1 --2- 172.21.15.60:0/3214810658 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f6a3006b990 0x7f6a3006de50 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:27.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.407+0000 7f6a54a00640 1 --2- 172.21.15.60:0/3214810658 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6a500c5380 0x7f6a501f45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:27.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.407+0000 7f6a54a00640 1 --2- 172.21.15.60:0/3214810658 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a500c4fa0 0x7f6a501f4080 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:27.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.407+0000 7f6a54a00640 1 --2- 172.21.15.60:0/3214810658 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a50070d50 0x7f6a501f1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:27.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.407+0000 7f6a54a00640 1 -- 172.21.15.60:0/3214810658 >> 172.21.15.60:0/3214810658 conn(0x7f6a500c8b50 msgr2=0x7f6a50071a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:57:27.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.407+0000 7f6a54a00640 1 -- 172.21.15.60:0/3214810658 shutdown_connections 2024-04-14T23:57:27.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.407+0000 7f6a54a00640 1 -- 172.21.15.60:0/3214810658 wait complete. 2024-04-14T23:57:27.437 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=0) 2024-04-14T23:57:27.437 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-14T23:57:27.437 INFO:teuthology.orchestra.run:waiting for 300 2024-04-14T23:57:27.463 INFO:tasks.ceph.mds.c:Stopped 2024-04-14T23:57:27.464 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:57:27.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.699+0000 7fba64c00640 1 Processor -- start 2024-04-14T23:57:27.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.699+0000 7fba64c00640 1 -- start start 2024-04-14T23:57:27.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.701+0000 7fba64c00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fba60157100 0x7fba6015acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:27.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.701+0000 7fba64c00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fba6006d290 0x7fba6015b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:27.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.701+0000 7fba64c00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba60157590 con 0x7fba600c8820 2024-04-14T23:57:27.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.701+0000 7fba64c00640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fba601580b0 con 0x7fba6006d290 2024-04-14T23:57:27.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.701+0000 7fba64c00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fba60157830 con 0x7fba60157100 2024-04-14T23:57:27.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.702+0000 7fba5e800640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fba6006d290 0x7fba6015b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:27.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.702+0000 7fba5de00640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fba600c8820 0x7fba60150fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54078/0 (socket says 172.21.15.60:54078) 2024-04-14T23:57:27.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.702+0000 7fba5de00640 1 -- 172.21.15.60:0/1448070527 learned_addr learned my addr 172.21.15.60:0/1448070527 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:57:27.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.702+0000 7fba5d400640 1 --2- 172.21.15.60:0/1448070527 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fba60157100 0x7fba6015acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:27.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.702+0000 7fba5ca00640 1 -- 172.21.15.60:0/1448070527 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3179021021 0 0) 0x7fba60157590 con 0x7fba600c8820 2024-04-14T23:57:27.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.702+0000 7fba5e800640 1 -- 172.21.15.60:0/1448070527 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fba60157100 msgr2=0x7fba6015acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:57:27.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.702+0000 7fba5e800640 1 --2- 172.21.15.60:0/1448070527 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fba60157100 0x7fba6015acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:27.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.702+0000 7fba5e800640 1 -- 172.21.15.60:0/1448070527 >> v1:172.21.15.60:6789/0 conn(0x7fba600c8820 legacy=0x7fba60150fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:27.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.702+0000 7fba5e800640 1 -- 172.21.15.60:0/1448070527 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba60157af0 con 0x7fba6006d290 2024-04-14T23:57:27.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.704+0000 7fba5e800640 1 --2- 172.21.15.60:0/1448070527 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba6006d290 0x7fba6015b200 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fba58009d90 tx=0x7fba5802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26632f8a5a000690 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:57:27.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.704+0000 7fba5ca00640 1 -- 172.21.15.60:0/1448070527 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba58039070 con 0x7fba6006d290 2024-04-14T23:57:27.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.704+0000 7fba5ca00640 1 -- 172.21.15.60:0/1448070527 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba5802cc50 con 0x7fba6006d290 2024-04-14T23:57:27.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.704+0000 7fba5ca00640 1 -- 172.21.15.60:0/1448070527 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba58034440 con 0x7fba6006d290 2024-04-14T23:57:27.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.705+0000 7fba64c00640 1 -- 172.21.15.60:0/1448070527 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba6006d290 msgr2=0x7fba6015b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:27.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.705+0000 7fba64c00640 1 --2- 172.21.15.60:0/1448070527 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba6006d290 0x7fba6015b200 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fba58009d90 tx=0x7fba5802bea0 comp rx=0 tx=0).stop 2024-04-14T23:57:27.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.705+0000 7fba64c00640 1 -- 172.21.15.60:0/1448070527 shutdown_connections 2024-04-14T23:57:27.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.705+0000 7fba64c00640 1 --2- 172.21.15.60:0/1448070527 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba6006d290 0x7fba6015b200 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:27.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.705+0000 7fba64c00640 1 --2- 172.21.15.60:0/1448070527 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fba60157100 0x7fba6015acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:27.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.705+0000 7fba64c00640 1 -- 172.21.15.60:0/1448070527 >> 172.21.15.60:0/1448070527 conn(0x7fba600c93b0 msgr2=0x7fba600c9790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:57:27.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.705+0000 7fba64c00640 1 -- 172.21.15.60:0/1448070527 shutdown_connections 2024-04-14T23:57:27.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.705+0000 7fba64c00640 1 -- 172.21.15.60:0/1448070527 wait complete. 2024-04-14T23:57:27.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.706+0000 7fba64c00640 1 Processor -- start 2024-04-14T23:57:27.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.707+0000 7fba64c00640 1 -- start start 2024-04-14T23:57:27.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.707+0000 7fba64c00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fba6006d290 0x7fba6014b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:27.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.707+0000 7fba64c00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba600c8820 0x7fba60148400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:27.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.707+0000 7fba64c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fba6014a3d0 0x7fba60148940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:27.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.707+0000 7fba64c00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fba600c39b0 con 0x7fba600c8820 2024-04-14T23:57:27.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.707+0000 7fba64c00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fba6015a7a0 con 0x7fba6014a3d0 2024-04-14T23:57:27.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.707+0000 7fba64c00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fba600c3b30 con 0x7fba6006d290 2024-04-14T23:57:27.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.708+0000 7fba5d400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba600c8820 0x7fba60148400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:27.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.708+0000 7fba5d400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba600c8820 0x7fba60148400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51140/0 (socket says 172.21.15.60:51140) 2024-04-14T23:57:27.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.708+0000 7fba5d400640 1 -- 172.21.15.60:0/1821389540 learned_addr learned my addr 172.21.15.60:0/1821389540 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:57:27.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.708+0000 7fba5de00640 1 --2- 172.21.15.60:0/1821389540 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fba6006d290 0x7fba6014b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:27.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.708+0000 7fba5e800640 1 --2- 172.21.15.60:0/1821389540 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fba6014a3d0 0x7fba60148940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:27.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.708+0000 7fba5d400640 1 -- 172.21.15.60:0/1821389540 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fba6006d290 msgr2=0x7fba6014b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:27.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.708+0000 7fba5d400640 1 --2- 172.21.15.60:0/1821389540 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fba6006d290 0x7fba6014b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:27.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.708+0000 7fba5d400640 1 -- 172.21.15.60:0/1821389540 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fba6014a3d0 msgr2=0x7fba60148940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:27.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.708+0000 7fba5d400640 1 --2- 172.21.15.60:0/1821389540 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fba6014a3d0 0x7fba60148940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:27.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.708+0000 7fba5d400640 1 -- 172.21.15.60:0/1821389540 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba60158490 con 0x7fba600c8820 2024-04-14T23:57:27.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.708+0000 7fba5d400640 1 --2- 172.21.15.60:0/1821389540 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba600c8820 0x7fba60148400 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fba4c00e870 tx=0x7fba4c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:57:27.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.709+0000 7fba56000640 1 -- 172.21.15.60:0/1821389540 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba4c017070 con 0x7fba600c8820 2024-04-14T23:57:27.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.709+0000 7fba56000640 1 -- 172.21.15.60:0/1821389540 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba4c00c940 con 0x7fba600c8820 2024-04-14T23:57:27.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.709+0000 7fba56000640 1 -- 172.21.15.60:0/1821389540 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba4c00cd30 con 0x7fba600c8820 2024-04-14T23:57:27.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.709+0000 7fba64c00640 1 -- 172.21.15.60:0/1821389540 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba601470f0 con 0x7fba600c8820 2024-04-14T23:57:27.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.709+0000 7fba64c00640 1 -- 172.21.15.60:0/1821389540 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba6014afe0 con 0x7fba600c8820 2024-04-14T23:57:27.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.710+0000 7fba64c00640 1 -- 172.21.15.60:0/1821389540 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba600684b0 con 0x7fba600c8820 2024-04-14T23:57:27.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.711+0000 7fba56000640 1 -- 172.21.15.60:0/1821389540 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7fba4c01a070 con 0x7fba600c8820 2024-04-14T23:57:27.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.712+0000 7fba56000640 1 --2- 172.21.15.60:0/1821389540 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fba4006b990 0x7fba4006de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:27.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.712+0000 7fba56000640 1 -- 172.21.15.60:0/1821389540 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6169+0+0 (secure 0 0 0) 0x7fba4c08abf0 con 0x7fba600c8820 2024-04-14T23:57:27.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.713+0000 7fba5de00640 1 --2- 172.21.15.60:0/1821389540 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fba4006b990 0x7fba4006de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:27.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.713+0000 7fba5de00640 1 --2- 172.21.15.60:0/1821389540 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fba4006b990 0x7fba4006de50 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fba50000f60 tx=0x7fba50009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:57:27.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.717+0000 7fba56000640 1 -- 172.21.15.60:0/1821389540 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fba4c058fd0 con 0x7fba600c8820 2024-04-14T23:57:27.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.958+0000 7fba64c00640 1 -- 172.21.15.60:0/1821389540 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fba600c8390 con 0x7fba600c8820 2024-04-14T23:57:27.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.959+0000 7fba56000640 1 -- 172.21.15.60:0/1821389540 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6839 (secure 0 0 0) 0x7fba4c05d070 con 0x7fba600c8820 2024-04-14T23:57:27.965 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:57:27.965 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5286,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/3281566884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3281566884},{"type":"v1","addr":"172.21.15.60:6833","nonce":3281566884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:56:48.398092+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5191},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5191":{"gid":5191,"name":"c","rank":0,"incarnation":27,"state":"up:active","state_seq":24,"addr":"172.21.15.60:6835/2356847578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2356847578},{"type":"v1","addr":"172.21.15.60:6835","nonce":2356847578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5191,"qdb_cluster":[5191]},"id":1}]} 2024-04-14T23:57:27.966 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 31 2024-04-14T23:57:27.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.962+0000 7fba64c00640 1 -- 172.21.15.60:0/1821389540 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fba4006b990 msgr2=0x7fba4006de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:27.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.962+0000 7fba64c00640 1 --2- 172.21.15.60:0/1821389540 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fba4006b990 0x7fba4006de50 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fba50000f60 tx=0x7fba50009040 comp rx=0 tx=0).stop 2024-04-14T23:57:27.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.962+0000 7fba64c00640 1 -- 172.21.15.60:0/1821389540 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba600c8820 msgr2=0x7fba60148400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:27.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.962+0000 7fba64c00640 1 --2- 172.21.15.60:0/1821389540 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba600c8820 0x7fba60148400 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fba4c00e870 tx=0x7fba4c00ed30 comp rx=0 tx=0).stop 2024-04-14T23:57:27.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.962+0000 7fba64c00640 1 -- 172.21.15.60:0/1821389540 shutdown_connections 2024-04-14T23:57:27.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.962+0000 7fba64c00640 1 --2- 172.21.15.60:0/1821389540 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fba4006b990 0x7fba4006de50 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:27.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.962+0000 7fba64c00640 1 --2- 172.21.15.60:0/1821389540 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fba6014a3d0 0x7fba60148940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:27.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.962+0000 7fba64c00640 1 --2- 172.21.15.60:0/1821389540 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba600c8820 0x7fba60148400 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:27.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.962+0000 7fba64c00640 1 --2- 172.21.15.60:0/1821389540 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fba6006d290 0x7fba6014b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:27.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.962+0000 7fba64c00640 1 -- 172.21.15.60:0/1821389540 >> 172.21.15.60:0/1821389540 conn(0x7fba600c93b0 msgr2=0x7fba600c7b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:57:27.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.962+0000 7fba64c00640 1 -- 172.21.15.60:0/1821389540 shutdown_connections 2024-04-14T23:57:27.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:27.962+0000 7fba64c00640 1 -- 172.21.15.60:0/1821389540 wait complete. 2024-04-14T23:57:27.987 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-14T23:57:29.988 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:57:30.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.204+0000 7f5452800640 1 Processor -- start 2024-04-14T23:57:30.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.204+0000 7f5452800640 1 -- start start 2024-04-14T23:57:30.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.206+0000 7f5452800640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f544c073320 0x7f544c0c6950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:30.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.206+0000 7f5452800640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f544c152fb0 con 0x7f544c0c3d90 2024-04-14T23:57:30.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.206+0000 7f5452800640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f544c159310 con 0x7f544c0b1e00 2024-04-14T23:57:30.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.206+0000 7f5452800640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f544c0c6540 con 0x7f544c073320 2024-04-14T23:57:30.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.206+0000 7f5451400640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f544c0b1e00 0x7f544c0c7fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.60:44076/0 (socket says 172.21.15.60:44076) 2024-04-14T23:57:30.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.206+0000 7f5451400640 1 -- 172.21.15.60:0/1244003925 learned_addr learned my addr 172.21.15.60:0/1244003925 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:57:30.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.206+0000 7f5450a00640 1 --2- 172.21.15.60:0/1244003925 >> v2:172.21.15.186:3300/0 conn(0x7f544c073320 0x7f544c0c6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:30.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.207+0000 7f5450a00640 1 -- 172.21.15.60:0/1244003925 >> v1:172.21.15.186:6789/0 conn(0x7f544c0b1e00 legacy=0x7f544c0c7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:30.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.207+0000 7f5450a00640 1 -- 172.21.15.60:0/1244003925 >> v1:172.21.15.60:6789/0 conn(0x7f544c0c3d90 legacy=0x7f544c0c41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:30.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.207+0000 7f544be00640 1 -- 172.21.15.60:0/1244003925 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 167126333 0 0) 0x7f544c152fb0 con 0x7f544c0c3d90 2024-04-14T23:57:30.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.207+0000 7f5450a00640 1 -- 172.21.15.60:0/1244003925 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f544c0c38b0 con 0x7f544c073320 2024-04-14T23:57:30.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.208+0000 7f5450a00640 1 --2- 172.21.15.60:0/1244003925 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f544c073320 0x7f544c0c6950 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f544c1594e0 tx=0x7f543802b9d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d2b6d3fc776cb14 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:57:30.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.209+0000 7f544be00640 1 -- 172.21.15.60:0/1244003925 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5438034070 con 0x7f544c073320 2024-04-14T23:57:30.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.209+0000 7f544be00640 1 -- 172.21.15.60:0/1244003925 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5438030890 con 0x7f544c073320 2024-04-14T23:57:30.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.209+0000 7f544be00640 1 -- 172.21.15.60:0/1244003925 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5438030c60 con 0x7f544c073320 2024-04-14T23:57:30.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.209+0000 7f5452800640 1 -- 172.21.15.60:0/1244003925 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f544c073320 msgr2=0x7f544c0c6950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:30.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.209+0000 7f5452800640 1 --2- 172.21.15.60:0/1244003925 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f544c073320 0x7f544c0c6950 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f544c1594e0 tx=0x7f543802b9d0 comp rx=0 tx=0).stop 2024-04-14T23:57:30.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.209+0000 7f5452800640 1 -- 172.21.15.60:0/1244003925 shutdown_connections 2024-04-14T23:57:30.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.209+0000 7f5452800640 1 --2- 172.21.15.60:0/1244003925 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f544c073320 0x7f544c0c6950 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:30.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.209+0000 7f5452800640 1 -- 172.21.15.60:0/1244003925 >> 172.21.15.60:0/1244003925 conn(0x7f544c0c8b50 msgr2=0x7f544c0c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:57:30.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.210+0000 7f5452800640 1 -- 172.21.15.60:0/1244003925 shutdown_connections 2024-04-14T23:57:30.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.210+0000 7f5452800640 1 -- 172.21.15.60:0/1244003925 wait complete. 2024-04-14T23:57:30.217 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.212+0000 7f5452800640 1 Processor -- start 2024-04-14T23:57:30.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.212+0000 7f5452800640 1 -- start start 2024-04-14T23:57:30.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.213+0000 7f5452800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f544c073320 0x7f544c155bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:30.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.213+0000 7f5452800640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f544c0b1e00 0x7f544c1e56f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:30.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.213+0000 7f5452800640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f544c0c3d90 0x7f544c1e5c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:30.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.213+0000 7f5452800640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f544c159890 con 0x7f544c0b1e00 2024-04-14T23:57:30.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.213+0000 7f5452800640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f544c0c5f80 con 0x7f544c073320 2024-04-14T23:57:30.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.213+0000 7f5452800640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f544c0c6260 con 0x7f544c0c3d90 2024-04-14T23:57:30.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.213+0000 7f5450a00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f544c0b1e00 0x7f544c1e56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:30.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.213+0000 7f5451400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f544c073320 0x7f544c155bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:30.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.213+0000 7f5451e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f544c0c3d90 0x7f544c1e5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:30.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.213+0000 7f5450a00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f544c0b1e00 0x7f544c1e56f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51154/0 (socket says 172.21.15.60:51154) 2024-04-14T23:57:30.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.213+0000 7f5450a00640 1 -- 172.21.15.60:0/3733704758 learned_addr learned my addr 172.21.15.60:0/3733704758 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:57:30.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.214+0000 7f5451400640 1 -- 172.21.15.60:0/3733704758 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f544c0c3d90 msgr2=0x7f544c1e5c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:30.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.214+0000 7f5451400640 1 --2- 172.21.15.60:0/3733704758 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f544c0c3d90 0x7f544c1e5c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:30.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.214+0000 7f5451400640 1 -- 172.21.15.60:0/3733704758 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f544c0b1e00 msgr2=0x7f544c1e56f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:30.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.214+0000 7f5451400640 1 --2- 172.21.15.60:0/3733704758 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f544c0b1e00 0x7f544c1e56f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:30.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.214+0000 7f5451400640 1 -- 172.21.15.60:0/3733704758 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f544c15c660 con 0x7f544c073320 2024-04-14T23:57:30.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.214+0000 7f5451400640 1 --2- 172.21.15.60:0/3733704758 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f544c073320 0x7f544c155bf0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f5440000f60 tx=0x7f544000c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:57:30.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.214+0000 7f5449600640 1 -- 172.21.15.60:0/3733704758 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5440017070 con 0x7f544c073320 2024-04-14T23:57:30.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.214+0000 7f5449600640 1 -- 172.21.15.60:0/3733704758 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5440005ce0 con 0x7f544c073320 2024-04-14T23:57:30.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.214+0000 7f5452800640 1 -- 172.21.15.60:0/3733704758 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f544c1ee9c0 con 0x7f544c073320 2024-04-14T23:57:30.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.215+0000 7f5449600640 1 -- 172.21.15.60:0/3733704758 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5440007310 con 0x7f544c073320 2024-04-14T23:57:30.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.215+0000 7f5452800640 1 -- 172.21.15.60:0/3733704758 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f544c153290 con 0x7f544c073320 2024-04-14T23:57:30.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.216+0000 7f5452800640 1 -- 172.21.15.60:0/3733704758 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f544c1580a0 con 0x7f544c073320 2024-04-14T23:57:30.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.217+0000 7f5449600640 1 -- 172.21.15.60:0/3733704758 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f544001a070 con 0x7f544c073320 2024-04-14T23:57:30.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.218+0000 7f5449600640 1 --2- 172.21.15.60:0/3733704758 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f542c06b9c0 0x7f542c06de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:30.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.218+0000 7f5449600640 1 -- 172.21.15.60:0/3733704758 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6169+0+0 (secure 0 0 0) 0x7f5440089d00 con 0x7f544c073320 2024-04-14T23:57:30.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.218+0000 7f5450a00640 1 --2- 172.21.15.60:0/3733704758 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f542c06b9c0 0x7f542c06de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:30.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.219+0000 7f5450a00640 1 --2- 172.21.15.60:0/3733704758 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f542c06b9c0 0x7f542c06de80 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f544c15c5f0 tx=0x7f5438008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:57:30.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.222+0000 7f5449600640 1 -- 172.21.15.60:0/3733704758 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f544c1580a0 con 0x7f544c073320 2024-04-14T23:57:30.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.594+0000 7f5452800640 1 -- 172.21.15.60:0/3733704758 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f544c0c6c50 con 0x7f544c073320 2024-04-14T23:57:30.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.595+0000 7f5449600640 1 -- 172.21.15.60:0/3733704758 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6839 (secure 0 0 0) 0x7f544c0c6c50 con 0x7f544c073320 2024-04-14T23:57:30.601 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:57:30.602 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5286,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/3281566884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3281566884},{"type":"v1","addr":"172.21.15.60:6833","nonce":3281566884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:56:48.398092+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5191},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5191":{"gid":5191,"name":"c","rank":0,"incarnation":27,"state":"up:active","state_seq":24,"addr":"172.21.15.60:6835/2356847578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2356847578},{"type":"v1","addr":"172.21.15.60:6835","nonce":2356847578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5191,"qdb_cluster":[5191]},"id":1}]} 2024-04-14T23:57:30.602 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 31 2024-04-14T23:57:30.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.599+0000 7f5452800640 1 -- 172.21.15.60:0/3733704758 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f542c06b9c0 msgr2=0x7f542c06de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:30.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.599+0000 7f5452800640 1 --2- 172.21.15.60:0/3733704758 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f542c06b9c0 0x7f542c06de80 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f544c15c5f0 tx=0x7f5438008030 comp rx=0 tx=0).stop 2024-04-14T23:57:30.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.599+0000 7f5452800640 1 -- 172.21.15.60:0/3733704758 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f544c073320 msgr2=0x7f544c155bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:30.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.599+0000 7f5452800640 1 --2- 172.21.15.60:0/3733704758 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f544c073320 0x7f544c155bf0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f5440000f60 tx=0x7f544000c800 comp rx=0 tx=0).stop 2024-04-14T23:57:30.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.599+0000 7f5452800640 1 -- 172.21.15.60:0/3733704758 shutdown_connections 2024-04-14T23:57:30.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.599+0000 7f5452800640 1 --2- 172.21.15.60:0/3733704758 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f542c06b9c0 0x7f542c06de80 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:30.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.599+0000 7f5452800640 1 --2- 172.21.15.60:0/3733704758 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f544c0c3d90 0x7f544c1e5c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:30.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.599+0000 7f5452800640 1 --2- 172.21.15.60:0/3733704758 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f544c0b1e00 0x7f544c1e56f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:30.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.599+0000 7f5452800640 1 --2- 172.21.15.60:0/3733704758 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f544c073320 0x7f544c155bf0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:30.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.599+0000 7f5452800640 1 -- 172.21.15.60:0/3733704758 >> 172.21.15.60:0/3733704758 conn(0x7f544c0c8b50 msgr2=0x7f544c157990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:57:30.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.599+0000 7f5452800640 1 -- 172.21.15.60:0/3733704758 shutdown_connections 2024-04-14T23:57:30.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:30.599+0000 7f5452800640 1 -- 172.21.15.60:0/3733704758 wait complete. 2024-04-14T23:57:30.623 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-14T23:57:32.624 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:57:32.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:32.844+0000 7f726fe00640 1 Processor -- start 2024-04-14T23:57:32.851 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:32.845+0000 7f726fe00640 1 -- start start 2024-04-14T23:57:32.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:32.846+0000 7f726fe00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f727015c8a0 0x7f7270072390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:32.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:32.846+0000 7f726fe00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f72700c86d0 0x7f72701625c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:32.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:32.846+0000 7f726fe00640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7270071bf0 con 0x7f72700c6950 2024-04-14T23:57:32.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:32.846+0000 7f726fe00640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f727015cc80 con 0x7f72700c86d0 2024-04-14T23:57:32.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:32.846+0000 7f726fe00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f72701632a0 con 0x7f727015c8a0 2024-04-14T23:57:32.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:32.847+0000 7f726f400640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f72700c6950 0x7f727015f050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.60:36886/0 (socket says 172.21.15.60:36886) 2024-04-14T23:57:32.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:32.847+0000 7f726ea00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f727015c8a0 0x7f7270072390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:32.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:32.847+0000 7f726f400640 1 -- 172.21.15.60:0/2736440703 learned_addr learned my addr 172.21.15.60:0/2736440703 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:57:32.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:32.847+0000 7f726e000640 1 --2- 172.21.15.60:0/2736440703 >> v2:172.21.15.60:3300/0 conn(0x7f72700c86d0 0x7f72701625c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:32.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:32.847+0000 7f726ea00640 1 -- 172.21.15.60:0/2736440703 >> v2:172.21.15.60:3300/0 conn(0x7f72700c86d0 msgr2=0x7f72701625c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:57:32.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:32.847+0000 7f726ea00640 1 --2- 172.21.15.60:0/2736440703 >> v2:172.21.15.60:3300/0 conn(0x7f72700c86d0 0x7f72701625c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:32.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:32.847+0000 7f726d600640 1 -- 172.21.15.60:0/2736440703 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 346132370 0 0) 0x7f7268001490 con 0x7f72700c6950 2024-04-14T23:57:32.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:32.847+0000 7f726ea00640 1 -- 172.21.15.60:0/2736440703 >> v1:172.21.15.186:6789/0 conn(0x7f72700c6950 legacy=0x7f727015f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:32.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:32.847+0000 7f726ea00640 1 -- 172.21.15.60:0/2736440703 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72700c6fb0 con 0x7f727015c8a0 2024-04-14T23:57:32.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:32.848+0000 7f726ea00640 1 --2- 172.21.15.60:0/2736440703 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f727015c8a0 0x7f7270072390 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f72600097a0 tx=0x7f726002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef741d342285d4c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:57:32.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:32.849+0000 7f726d600640 1 -- 172.21.15.60:0/2736440703 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7260035020 con 0x7f727015c8a0 2024-04-14T23:57:32.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:32.849+0000 7f726d600640 1 -- 172.21.15.60:0/2736440703 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7260031830 con 0x7f727015c8a0 2024-04-14T23:57:32.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:32.849+0000 7f726d600640 1 -- 172.21.15.60:0/2736440703 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7260031c00 con 0x7f727015c8a0 2024-04-14T23:57:32.856 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:32.849+0000 7f726fe00640 1 -- 172.21.15.60:0/2736440703 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f727015c8a0 msgr2=0x7f7270072390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:32.856 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:32.849+0000 7f726fe00640 1 --2- 172.21.15.60:0/2736440703 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f727015c8a0 0x7f7270072390 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f72600097a0 tx=0x7f726002b8d0 comp rx=0 tx=0).stop 2024-04-14T23:57:32.856 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:32.850+0000 7f726fe00640 1 -- 172.21.15.60:0/2736440703 shutdown_connections 2024-04-14T23:57:32.856 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:32.850+0000 7f726fe00640 1 --2- 172.21.15.60:0/2736440703 >> v2:172.21.15.60:3300/0 conn(0x7f72700c86d0 0x7f72701625c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:32.856 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:32.850+0000 7f726fe00640 1 --2- 172.21.15.60:0/2736440703 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f727015c8a0 0x7f7270072390 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:32.856 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:32.850+0000 7f726fe00640 1 -- 172.21.15.60:0/2736440703 >> 172.21.15.60:0/2736440703 conn(0x7f72700c8b50 msgr2=0x7f72700c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:57:32.856 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:32.850+0000 7f726fe00640 1 -- 172.21.15.60:0/2736440703 shutdown_connections 2024-04-14T23:57:32.856 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:32.850+0000 7f726fe00640 1 -- 172.21.15.60:0/2736440703 wait complete. 2024-04-14T23:57:32.857 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:32.851+0000 7f726fe00640 1 Processor -- start 2024-04-14T23:57:32.857 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:32.852+0000 7f726fe00640 1 -- start start 2024-04-14T23:57:32.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:32.852+0000 7f726fe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72700c6950 0x7f72701f8180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:32.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:32.852+0000 7f726fe00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f72700c86d0 0x7f7270155270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:32.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:32.852+0000 7f726fe00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f727015c8a0 0x7f7270158c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:32.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:32.853+0000 7f726fe00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f72700728d0 con 0x7f727015c8a0 2024-04-14T23:57:32.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:32.853+0000 7f726fe00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7270071e70 con 0x7f72700c6950 2024-04-14T23:57:32.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:32.853+0000 7f726fe00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f727006b1c0 con 0x7f72700c86d0 2024-04-14T23:57:32.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:32.853+0000 7f726f400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f727015c8a0 0x7f7270158c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:32.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:32.853+0000 7f726f400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f727015c8a0 0x7f7270158c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43436/0 (socket says 172.21.15.60:43436) 2024-04-14T23:57:32.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:32.853+0000 7f726f400640 1 -- 172.21.15.60:0/1174045603 learned_addr learned my addr 172.21.15.60:0/1174045603 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:57:32.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:32.853+0000 7f726ea00640 1 --2- 172.21.15.60:0/1174045603 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72700c6950 0x7f72701f8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:32.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:32.853+0000 7f726e000640 1 --2- 172.21.15.60:0/1174045603 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f72700c86d0 0x7f7270155270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:32.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:32.853+0000 7f726f400640 1 -- 172.21.15.60:0/1174045603 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f72700c86d0 msgr2=0x7f7270155270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:32.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:32.853+0000 7f726f400640 1 --2- 172.21.15.60:0/1174045603 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f72700c86d0 0x7f7270155270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:32.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:32.853+0000 7f726f400640 1 -- 172.21.15.60:0/1174045603 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72700c6950 msgr2=0x7f72701f8180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:32.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:32.853+0000 7f726f400640 1 --2- 172.21.15.60:0/1174045603 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72700c6950 0x7f72701f8180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:32.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:32.853+0000 7f726f400640 1 -- 172.21.15.60:0/1174045603 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f727015d040 con 0x7f727015c8a0 2024-04-14T23:57:32.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:32.854+0000 7f726f400640 1 --2- 172.21.15.60:0/1174045603 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f727015c8a0 0x7f7270158c80 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f726800c700 tx=0x7f726800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:57:32.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:32.854+0000 7f7266a00640 1 -- 172.21.15.60:0/1174045603 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7268018070 con 0x7f727015c8a0 2024-04-14T23:57:32.862 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:32.854+0000 7f7266a00640 1 -- 172.21.15.60:0/1174045603 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f726800ce10 con 0x7f727015c8a0 2024-04-14T23:57:32.862 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:32.854+0000 7f7266a00640 1 -- 172.21.15.60:0/1174045603 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f726800a880 con 0x7f727015c8a0 2024-04-14T23:57:32.862 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:32.854+0000 7f726fe00640 1 -- 172.21.15.60:0/1174045603 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f727015d320 con 0x7f727015c8a0 2024-04-14T23:57:32.862 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:32.854+0000 7f726fe00640 1 -- 172.21.15.60:0/1174045603 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f727015f060 con 0x7f727015c8a0 2024-04-14T23:57:32.862 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:32.855+0000 7f726fe00640 1 -- 172.21.15.60:0/1174045603 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f727004f540 con 0x7f727015c8a0 2024-04-14T23:57:32.863 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:32.857+0000 7f7266a00640 1 -- 172.21.15.60:0/1174045603 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f726800aa20 con 0x7f727015c8a0 2024-04-14T23:57:32.864 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:32.858+0000 7f7266a00640 1 --2- 172.21.15.60:0/1174045603 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f725006b990 0x7f725006de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:32.864 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:32.858+0000 7f7266a00640 1 -- 172.21.15.60:0/1174045603 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6169+0+0 (secure 0 0 0) 0x7f7268089f00 con 0x7f727015c8a0 2024-04-14T23:57:32.864 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:32.858+0000 7f726ea00640 1 --2- 172.21.15.60:0/1174045603 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f725006b990 0x7f725006de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:32.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:32.859+0000 7f726ea00640 1 --2- 172.21.15.60:0/1174045603 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f725006b990 0x7f725006de50 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f7260036000 tx=0x7f7260008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:57:32.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:32.863+0000 7f7266a00640 1 -- 172.21.15.60:0/1174045603 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7270071bf0 con 0x7f727015c8a0 2024-04-14T23:57:33.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:33.104+0000 7f726fe00640 1 -- 172.21.15.60:0/1174045603 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f727015c040 con 0x7f727015c8a0 2024-04-14T23:57:33.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:33.106+0000 7f7266a00640 1 -- 172.21.15.60:0/1174045603 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6839 (secure 0 0 0) 0x7f72680582e0 con 0x7f727015c8a0 2024-04-14T23:57:33.112 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:57:33.112 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5286,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/3281566884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3281566884},{"type":"v1","addr":"172.21.15.60:6833","nonce":3281566884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:56:48.398092+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5191},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5191":{"gid":5191,"name":"c","rank":0,"incarnation":27,"state":"up:active","state_seq":24,"addr":"172.21.15.60:6835/2356847578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2356847578},{"type":"v1","addr":"172.21.15.60:6835","nonce":2356847578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5191,"qdb_cluster":[5191]},"id":1}]} 2024-04-14T23:57:33.112 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 31 2024-04-14T23:57:33.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:33.109+0000 7f726fe00640 1 -- 172.21.15.60:0/1174045603 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f725006b990 msgr2=0x7f725006de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:33.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:33.109+0000 7f726fe00640 1 --2- 172.21.15.60:0/1174045603 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f725006b990 0x7f725006de50 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f7260036000 tx=0x7f7260008030 comp rx=0 tx=0).stop 2024-04-14T23:57:33.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:33.109+0000 7f726fe00640 1 -- 172.21.15.60:0/1174045603 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f727015c8a0 msgr2=0x7f7270158c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:33.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:33.109+0000 7f726fe00640 1 --2- 172.21.15.60:0/1174045603 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f727015c8a0 0x7f7270158c80 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f726800c700 tx=0x7f726800cbc0 comp rx=0 tx=0).stop 2024-04-14T23:57:33.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:33.109+0000 7f726fe00640 1 -- 172.21.15.60:0/1174045603 shutdown_connections 2024-04-14T23:57:33.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:33.109+0000 7f726fe00640 1 --2- 172.21.15.60:0/1174045603 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f725006b990 0x7f725006de50 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:33.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:33.109+0000 7f726fe00640 1 --2- 172.21.15.60:0/1174045603 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f727015c8a0 0x7f7270158c80 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:33.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:33.109+0000 7f726fe00640 1 --2- 172.21.15.60:0/1174045603 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f72700c86d0 0x7f7270155270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:33.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:33.109+0000 7f726fe00640 1 --2- 172.21.15.60:0/1174045603 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72700c6950 0x7f72701f8180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:33.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:33.109+0000 7f726fe00640 1 -- 172.21.15.60:0/1174045603 >> 172.21.15.60:0/1174045603 conn(0x7f72700c8b50 msgr2=0x7f727006d770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:57:33.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:33.109+0000 7f726fe00640 1 -- 172.21.15.60:0/1174045603 shutdown_connections 2024-04-14T23:57:33.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:33.109+0000 7f726fe00640 1 -- 172.21.15.60:0/1174045603 wait complete. 2024-04-14T23:57:33.136 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-14T23:57:35.137 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:57:35.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.363+0000 7f37efe00640 1 Processor -- start 2024-04-14T23:57:35.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.364+0000 7f37efe00640 1 -- start start 2024-04-14T23:57:35.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.365+0000 7f37efe00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f37f00b1e00 0x7f37f00c6950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:35.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.365+0000 7f37efe00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37f0152fb0 con 0x7f37f0073320 2024-04-14T23:57:35.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.365+0000 7f37efe00640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37f0159310 con 0x7f37f00c3d90 2024-04-14T23:57:35.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.365+0000 7f37efe00640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f37f00c6540 con 0x7f37f00b1e00 2024-04-14T23:57:35.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.366+0000 7f37eea00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f37f00b1e00 0x7f37f00c6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:35.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.366+0000 7f37ee000640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f37f0073320 0x7f37f00c7fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:35800/0 (socket says 172.21.15.60:35800) 2024-04-14T23:57:35.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.366+0000 7f37eea00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f37f00b1e00 0x7f37f00c6950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43442/0 (socket says 172.21.15.60:43442) 2024-04-14T23:57:35.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.366+0000 7f37ee000640 1 -- 172.21.15.60:0/2068737792 learned_addr learned my addr 172.21.15.60:0/2068737792 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:57:35.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.366+0000 7f37eea00640 1 -- 172.21.15.60:0/2068737792 >> v1:172.21.15.186:6789/0 conn(0x7f37f00c3d90 legacy=0x7f37f00c41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:35.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.366+0000 7f37eea00640 1 -- 172.21.15.60:0/2068737792 >> v1:172.21.15.60:6789/0 conn(0x7f37f0073320 legacy=0x7f37f00c7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:35.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.366+0000 7f37eea00640 1 -- 172.21.15.60:0/2068737792 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37f00c38b0 con 0x7f37f00b1e00 2024-04-14T23:57:35.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.367+0000 7f37eea00640 1 --2- 172.21.15.60:0/2068737792 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37f00b1e00 0x7f37f00c6950 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f37f01594e0 tx=0x7f37e002b9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc31f433366a63be server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:57:35.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.368+0000 7f37ed600640 1 -- 172.21.15.60:0/2068737792 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e002f800 con 0x7f37f00b1e00 2024-04-14T23:57:35.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.368+0000 7f37ed600640 1 -- 172.21.15.60:0/2068737792 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37e002f9a0 con 0x7f37f00b1e00 2024-04-14T23:57:35.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.368+0000 7f37ed600640 1 -- 172.21.15.60:0/2068737792 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e002fd70 con 0x7f37f00b1e00 2024-04-14T23:57:35.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.368+0000 7f37efe00640 1 -- 172.21.15.60:0/2068737792 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37f00b1e00 msgr2=0x7f37f00c6950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:35.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.368+0000 7f37efe00640 1 --2- 172.21.15.60:0/2068737792 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37f00b1e00 0x7f37f00c6950 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f37f01594e0 tx=0x7f37e002b9e0 comp rx=0 tx=0).stop 2024-04-14T23:57:35.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.369+0000 7f37efe00640 1 -- 172.21.15.60:0/2068737792 shutdown_connections 2024-04-14T23:57:35.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.369+0000 7f37efe00640 1 --2- 172.21.15.60:0/2068737792 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37f00b1e00 0x7f37f00c6950 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:35.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.369+0000 7f37efe00640 1 -- 172.21.15.60:0/2068737792 >> 172.21.15.60:0/2068737792 conn(0x7f37f00c8b50 msgr2=0x7f37f00c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:57:35.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.369+0000 7f37efe00640 1 -- 172.21.15.60:0/2068737792 shutdown_connections 2024-04-14T23:57:35.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.369+0000 7f37efe00640 1 -- 172.21.15.60:0/2068737792 wait complete. 2024-04-14T23:57:35.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.370+0000 7f37efe00640 1 Processor -- start 2024-04-14T23:57:35.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.371+0000 7f37efe00640 1 -- start start 2024-04-14T23:57:35.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.372+0000 7f37efe00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f37f0073320 0x7f37f0153180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:35.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.372+0000 7f37efe00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37f00c3d90 0x7f37f00c81a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:35.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.372+0000 7f37efe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f37f01df9d0 0x7f37f01552d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:35.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.372+0000 7f37efe00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f37f0159890 con 0x7f37f00c3d90 2024-04-14T23:57:35.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.372+0000 7f37efe00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f37f00c5f80 con 0x7f37f01df9d0 2024-04-14T23:57:35.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.372+0000 7f37efe00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f37f00c6260 con 0x7f37f0073320 2024-04-14T23:57:35.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.372+0000 7f37ee000640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37f00c3d90 0x7f37f00c81a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:35.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.372+0000 7f37ee000640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37f00c3d90 0x7f37f00c81a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43452/0 (socket says 172.21.15.60:43452) 2024-04-14T23:57:35.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.372+0000 7f37ee000640 1 -- 172.21.15.60:0/2754635522 learned_addr learned my addr 172.21.15.60:0/2754635522 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:57:35.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.372+0000 7f37eea00640 1 --2- 172.21.15.60:0/2754635522 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f37f0073320 0x7f37f0153180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:35.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.372+0000 7f37ef400640 1 --2- 172.21.15.60:0/2754635522 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f37f01df9d0 0x7f37f01552d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:35.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.372+0000 7f37ee000640 1 -- 172.21.15.60:0/2754635522 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f37f0073320 msgr2=0x7f37f0153180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:35.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.372+0000 7f37ee000640 1 --2- 172.21.15.60:0/2754635522 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f37f0073320 0x7f37f0153180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:35.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.372+0000 7f37ee000640 1 -- 172.21.15.60:0/2754635522 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f37f01df9d0 msgr2=0x7f37f01552d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:35.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.372+0000 7f37ee000640 1 --2- 172.21.15.60:0/2754635522 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f37f01df9d0 0x7f37f01552d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:35.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.372+0000 7f37ee000640 1 -- 172.21.15.60:0/2754635522 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37f01536c0 con 0x7f37f00c3d90 2024-04-14T23:57:35.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.373+0000 7f37ee000640 1 --2- 172.21.15.60:0/2754635522 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37f00c3d90 0x7f37f00c81a0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f37d800cb10 tx=0x7f37d800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:57:35.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.373+0000 7f37e6a00640 1 -- 172.21.15.60:0/2754635522 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37d8017070 con 0x7f37f00c3d90 2024-04-14T23:57:35.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.373+0000 7f37e6a00640 1 -- 172.21.15.60:0/2754635522 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37d800ac70 con 0x7f37f00c3d90 2024-04-14T23:57:35.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.373+0000 7f37e6a00640 1 -- 172.21.15.60:0/2754635522 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37d800ae60 con 0x7f37f00c3d90 2024-04-14T23:57:35.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.373+0000 7f37efe00640 1 -- 172.21.15.60:0/2754635522 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37f0154760 con 0x7f37f00c3d90 2024-04-14T23:57:35.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.373+0000 7f37efe00640 1 -- 172.21.15.60:0/2754635522 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37f01e0d50 con 0x7f37f00c3d90 2024-04-14T23:57:35.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.374+0000 7f37efe00640 1 -- 172.21.15.60:0/2754635522 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37f00673d0 con 0x7f37f00c3d90 2024-04-14T23:57:35.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.375+0000 7f37e6a00640 1 -- 172.21.15.60:0/2754635522 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f37d801a070 con 0x7f37f00c3d90 2024-04-14T23:57:35.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.376+0000 7f37e6a00640 1 --2- 172.21.15.60:0/2754635522 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f37cc06b990 0x7f37cc06de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:35.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.376+0000 7f37e6a00640 1 -- 172.21.15.60:0/2754635522 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6169+0+0 (secure 0 0 0) 0x7f37d808a7e0 con 0x7f37f00c3d90 2024-04-14T23:57:35.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.376+0000 7f37eea00640 1 --2- 172.21.15.60:0/2754635522 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f37cc06b990 0x7f37cc06de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:35.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.377+0000 7f37eea00640 1 --2- 172.21.15.60:0/2754635522 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f37cc06b990 0x7f37cc06de50 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f37e0036000 tx=0x7f37e0008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:57:35.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.381+0000 7f37e6a00640 1 -- 172.21.15.60:0/2754635522 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f37f00673d0 con 0x7f37f00c3d90 2024-04-14T23:57:35.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.626+0000 7f37efe00640 1 -- 172.21.15.60:0/2754635522 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f37f004f540 con 0x7f37f00c3d90 2024-04-14T23:57:35.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.627+0000 7f37e6a00640 1 -- 172.21.15.60:0/2754635522 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6839 (secure 0 0 0) 0x7f37d8058b30 con 0x7f37f00c3d90 2024-04-14T23:57:35.634 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:57:35.634 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5286,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/3281566884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3281566884},{"type":"v1","addr":"172.21.15.60:6833","nonce":3281566884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:56:48.398092+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5191},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5191":{"gid":5191,"name":"c","rank":0,"incarnation":27,"state":"up:active","state_seq":24,"addr":"172.21.15.60:6835/2356847578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2356847578},{"type":"v1","addr":"172.21.15.60:6835","nonce":2356847578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5191,"qdb_cluster":[5191]},"id":1}]} 2024-04-14T23:57:35.634 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 31 2024-04-14T23:57:35.636 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.630+0000 7f37efe00640 1 -- 172.21.15.60:0/2754635522 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f37cc06b990 msgr2=0x7f37cc06de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:35.636 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.630+0000 7f37efe00640 1 --2- 172.21.15.60:0/2754635522 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f37cc06b990 0x7f37cc06de50 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f37e0036000 tx=0x7f37e0008030 comp rx=0 tx=0).stop 2024-04-14T23:57:35.636 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.630+0000 7f37efe00640 1 -- 172.21.15.60:0/2754635522 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37f00c3d90 msgr2=0x7f37f00c81a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:35.636 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.630+0000 7f37efe00640 1 --2- 172.21.15.60:0/2754635522 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37f00c3d90 0x7f37f00c81a0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f37d800cb10 tx=0x7f37d800a060 comp rx=0 tx=0).stop 2024-04-14T23:57:35.636 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.630+0000 7f37efe00640 1 -- 172.21.15.60:0/2754635522 shutdown_connections 2024-04-14T23:57:35.636 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.630+0000 7f37efe00640 1 --2- 172.21.15.60:0/2754635522 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f37cc06b990 0x7f37cc06de50 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:35.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.630+0000 7f37efe00640 1 --2- 172.21.15.60:0/2754635522 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f37f01df9d0 0x7f37f01552d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:35.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.630+0000 7f37efe00640 1 --2- 172.21.15.60:0/2754635522 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37f00c3d90 0x7f37f00c81a0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:35.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.630+0000 7f37efe00640 1 --2- 172.21.15.60:0/2754635522 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f37f0073320 0x7f37f0153180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:35.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.630+0000 7f37efe00640 1 -- 172.21.15.60:0/2754635522 >> 172.21.15.60:0/2754635522 conn(0x7f37f00c8b50 msgr2=0x7f37f01524e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:57:35.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.630+0000 7f37efe00640 1 -- 172.21.15.60:0/2754635522 shutdown_connections 2024-04-14T23:57:35.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:35.630+0000 7f37efe00640 1 -- 172.21.15.60:0/2754635522 wait complete. 2024-04-14T23:57:35.656 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-14T23:57:37.657 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:57:37.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:37.879+0000 7f2859400640 1 Processor -- start 2024-04-14T23:57:37.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:37.880+0000 7f2859400640 1 -- start start 2024-04-14T23:57:37.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:37.881+0000 7f2859400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2854154cf0 0x7f285415a940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:37.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:37.881+0000 7f2859400640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f285406c310 0x7f285406c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:37.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:37.882+0000 7f2859400640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28540c5520 con 0x7f28540cb120 2024-04-14T23:57:37.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:37.882+0000 7f2859400640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f28541550d0 con 0x7f285406c310 2024-04-14T23:57:37.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:37.882+0000 7f2859400640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f285415b040 con 0x7f2854154cf0 2024-04-14T23:57:37.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:37.882+0000 7f284be00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f285406c310 0x7f285406c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:37.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:37.882+0000 7f2853e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2854154cf0 0x7f285415a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:37.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:37.882+0000 7f284be00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f285406c310 0x7f285406c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43468/0 (socket says 172.21.15.60:43468) 2024-04-14T23:57:37.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:37.882+0000 7f284be00640 1 -- 172.21.15.60:0/3583417958 learned_addr learned my addr 172.21.15.60:0/3583417958 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:57:37.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:37.882+0000 7f284be00640 1 -- 172.21.15.60:0/3583417958 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2854154cf0 msgr2=0x7f285415a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:57:37.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:37.882+0000 7f284be00640 1 --2- 172.21.15.60:0/3583417958 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2854154cf0 0x7f285415a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:37.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:37.882+0000 7f284be00640 1 -- 172.21.15.60:0/3583417958 >> v1:172.21.15.186:6789/0 conn(0x7f28540cb120 legacy=0x7f28540cbd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:37.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:37.882+0000 7f284be00640 1 -- 172.21.15.60:0/3583417958 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f285415b300 con 0x7f285406c310 2024-04-14T23:57:37.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:37.882+0000 7f2853400640 1 -- 172.21.15.60:0/3583417958 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1872232182 0 0) 0x7f28540c5520 con 0x7f28540cb120 2024-04-14T23:57:37.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:37.884+0000 7f284be00640 1 --2- 172.21.15.60:0/3583417958 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f285406c310 0x7f285406c710 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f283c009870 tx=0x7f283c02ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90586b1df861b047 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:57:37.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:37.884+0000 7f2853400640 1 -- 172.21.15.60:0/3583417958 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f283c02f840 con 0x7f285406c310 2024-04-14T23:57:37.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:37.884+0000 7f2853400640 1 -- 172.21.15.60:0/3583417958 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f283c02f9e0 con 0x7f285406c310 2024-04-14T23:57:37.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:37.884+0000 7f2853400640 1 -- 172.21.15.60:0/3583417958 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f283c02fdb0 con 0x7f285406c310 2024-04-14T23:57:37.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:37.884+0000 7f2859400640 1 -- 172.21.15.60:0/3583417958 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f285406c310 msgr2=0x7f285406c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:37.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:37.884+0000 7f2859400640 1 --2- 172.21.15.60:0/3583417958 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f285406c310 0x7f285406c710 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f283c009870 tx=0x7f283c02ba20 comp rx=0 tx=0).stop 2024-04-14T23:57:37.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:37.885+0000 7f2859400640 1 -- 172.21.15.60:0/3583417958 shutdown_connections 2024-04-14T23:57:37.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:37.885+0000 7f2859400640 1 --2- 172.21.15.60:0/3583417958 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f285406c310 0x7f285406c710 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:37.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:37.885+0000 7f2859400640 1 --2- 172.21.15.60:0/3583417958 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2854154cf0 0x7f285415a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:37.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:37.885+0000 7f2859400640 1 -- 172.21.15.60:0/3583417958 >> 172.21.15.60:0/3583417958 conn(0x7f28540670f0 msgr2=0x7f2854067500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:57:37.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:37.885+0000 7f2859400640 1 -- 172.21.15.60:0/3583417958 shutdown_connections 2024-04-14T23:57:37.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:37.885+0000 7f2859400640 1 -- 172.21.15.60:0/3583417958 wait complete. 2024-04-14T23:57:37.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:37.887+0000 7f2859400640 1 Processor -- start 2024-04-14T23:57:37.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:37.887+0000 7f2859400640 1 -- start start 2024-04-14T23:57:37.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:37.888+0000 7f2859400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f285406c310 0x7f28540b5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:37.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:37.888+0000 7f2859400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f28540cb120 0x7f28541591a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:37.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:37.888+0000 7f2859400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2854154cf0 0x7f28541f29a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:37.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:37.888+0000 7f2859400640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f285406bb10 con 0x7f2854154cf0 2024-04-14T23:57:37.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:37.888+0000 7f2859400640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f285415b8c0 con 0x7f285406c310 2024-04-14T23:57:37.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:37.888+0000 7f2859400640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f285406bdf0 con 0x7f28540cb120 2024-04-14T23:57:37.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:37.888+0000 7f2858a00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2854154cf0 0x7f28541f29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:37.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:37.888+0000 7f2858a00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2854154cf0 0x7f28541f29a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43484/0 (socket says 172.21.15.60:43484) 2024-04-14T23:57:37.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:37.888+0000 7f2858a00640 1 -- 172.21.15.60:0/1485278626 learned_addr learned my addr 172.21.15.60:0/1485278626 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:57:37.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:37.888+0000 7f2853e00640 1 --2- 172.21.15.60:0/1485278626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f285406c310 0x7f28540b5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:37.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:37.888+0000 7f284be00640 1 --2- 172.21.15.60:0/1485278626 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f28540cb120 0x7f28541591a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:37.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:37.889+0000 7f2858a00640 1 -- 172.21.15.60:0/1485278626 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f28540cb120 msgr2=0x7f28541591a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:37.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:37.889+0000 7f2858a00640 1 --2- 172.21.15.60:0/1485278626 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f28540cb120 0x7f28541591a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:37.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:37.889+0000 7f2858a00640 1 -- 172.21.15.60:0/1485278626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f285406c310 msgr2=0x7f28540b5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:37.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:37.889+0000 7f2858a00640 1 --2- 172.21.15.60:0/1485278626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f285406c310 0x7f28540b5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:37.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:37.889+0000 7f2858a00640 1 -- 172.21.15.60:0/1485278626 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f285415a780 con 0x7f2854154cf0 2024-04-14T23:57:37.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:37.889+0000 7f2858a00640 1 --2- 172.21.15.60:0/1485278626 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2854154cf0 0x7f28541f29a0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f284c00c760 tx=0x7f284c00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:57:37.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:37.889+0000 7f2850c00640 1 -- 172.21.15.60:0/1485278626 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f284c003da0 con 0x7f2854154cf0 2024-04-14T23:57:37.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:37.889+0000 7f2850c00640 1 -- 172.21.15.60:0/1485278626 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f284c005070 con 0x7f2854154cf0 2024-04-14T23:57:37.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:37.889+0000 7f2850c00640 1 -- 172.21.15.60:0/1485278626 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f284c005460 con 0x7f2854154cf0 2024-04-14T23:57:37.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:37.889+0000 7f2859400640 1 -- 172.21.15.60:0/1485278626 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f285415a4a0 con 0x7f2854154cf0 2024-04-14T23:57:37.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:37.889+0000 7f2859400640 1 -- 172.21.15.60:0/1485278626 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28541f1ec0 con 0x7f2854154cf0 2024-04-14T23:57:37.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:37.891+0000 7f2859400640 1 -- 172.21.15.60:0/1485278626 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28541f24b0 con 0x7f2854154cf0 2024-04-14T23:57:37.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:37.892+0000 7f2850c00640 1 -- 172.21.15.60:0/1485278626 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f284c005600 con 0x7f2854154cf0 2024-04-14T23:57:37.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:37.892+0000 7f2850c00640 1 --2- 172.21.15.60:0/1485278626 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f283006b990 0x7f283006de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:37.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:37.893+0000 7f2850c00640 1 -- 172.21.15.60:0/1485278626 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6169+0+0 (secure 0 0 0) 0x7f284c08a910 con 0x7f2854154cf0 2024-04-14T23:57:37.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:37.893+0000 7f2853e00640 1 --2- 172.21.15.60:0/1485278626 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f283006b990 0x7f283006de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:37.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:37.894+0000 7f2853e00640 1 --2- 172.21.15.60:0/1485278626 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f283006b990 0x7f283006de50 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2844009aa0 tx=0x7f2844009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:57:37.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:37.897+0000 7f2850c00640 1 -- 172.21.15.60:0/1485278626 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f284c058bd0 con 0x7f2854154cf0 2024-04-14T23:57:38.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:38.133+0000 7f2859400640 1 -- 172.21.15.60:0/1485278626 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f285404f540 con 0x7f2854154cf0 2024-04-14T23:57:38.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:38.134+0000 7f2850c00640 1 -- 172.21.15.60:0/1485278626 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6839 (secure 0 0 0) 0x7f284c05cc70 con 0x7f2854154cf0 2024-04-14T23:57:38.141 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:57:38.141 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5286,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/3281566884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3281566884},{"type":"v1","addr":"172.21.15.60:6833","nonce":3281566884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:56:48.398092+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5191},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5191":{"gid":5191,"name":"c","rank":0,"incarnation":27,"state":"up:active","state_seq":24,"addr":"172.21.15.60:6835/2356847578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2356847578},{"type":"v1","addr":"172.21.15.60:6835","nonce":2356847578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5191,"qdb_cluster":[5191]},"id":1}]} 2024-04-14T23:57:38.141 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 31 2024-04-14T23:57:38.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:38.136+0000 7f2859400640 1 -- 172.21.15.60:0/1485278626 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f283006b990 msgr2=0x7f283006de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:38.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:38.136+0000 7f2859400640 1 --2- 172.21.15.60:0/1485278626 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f283006b990 0x7f283006de50 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2844009aa0 tx=0x7f2844009190 comp rx=0 tx=0).stop 2024-04-14T23:57:38.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:38.137+0000 7f2859400640 1 -- 172.21.15.60:0/1485278626 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2854154cf0 msgr2=0x7f28541f29a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:38.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:38.137+0000 7f2859400640 1 --2- 172.21.15.60:0/1485278626 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2854154cf0 0x7f28541f29a0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f284c00c760 tx=0x7f284c00cc20 comp rx=0 tx=0).stop 2024-04-14T23:57:38.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:38.137+0000 7f2859400640 1 -- 172.21.15.60:0/1485278626 shutdown_connections 2024-04-14T23:57:38.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:38.137+0000 7f2859400640 1 --2- 172.21.15.60:0/1485278626 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f283006b990 0x7f283006de50 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:38.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:38.137+0000 7f2859400640 1 --2- 172.21.15.60:0/1485278626 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2854154cf0 0x7f28541f29a0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:38.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:38.137+0000 7f2859400640 1 --2- 172.21.15.60:0/1485278626 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f28540cb120 0x7f28541591a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:38.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:38.137+0000 7f2859400640 1 --2- 172.21.15.60:0/1485278626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f285406c310 0x7f28540b5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:38.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:38.137+0000 7f2859400640 1 -- 172.21.15.60:0/1485278626 >> 172.21.15.60:0/1485278626 conn(0x7f28540670f0 msgr2=0x7f28540cd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:57:38.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:38.137+0000 7f2859400640 1 -- 172.21.15.60:0/1485278626 shutdown_connections 2024-04-14T23:57:38.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:38.137+0000 7f2859400640 1 -- 172.21.15.60:0/1485278626 wait complete. 2024-04-14T23:57:38.161 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-14T23:57:40.162 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:57:40.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.388+0000 7f131d000640 1 Processor -- start 2024-04-14T23:57:40.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.389+0000 7f131d000640 1 -- start start 2024-04-14T23:57:40.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.390+0000 7f131d000640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f13180b1e00 0x7f13180c6950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:40.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.390+0000 7f131d000640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f13180c3d90 0x7f13180c41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:40.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.390+0000 7f131d000640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1318152fb0 con 0x7f1318073320 2024-04-14T23:57:40.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.390+0000 7f131d000640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f13180c6540 con 0x7f13180b1e00 2024-04-14T23:57:40.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.390+0000 7f131d000640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f13180b21e0 con 0x7f13180c3d90 2024-04-14T23:57:40.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.391+0000 7f1317400640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f13180b1e00 0x7f13180c6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:40.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.391+0000 7f1317400640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f13180b1e00 0x7f13180c6950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43494/0 (socket says 172.21.15.60:43494) 2024-04-14T23:57:40.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.391+0000 7f1317400640 1 -- 172.21.15.60:0/235299279 learned_addr learned my addr 172.21.15.60:0/235299279 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:57:40.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.391+0000 7f1317e00640 1 --2- 172.21.15.60:0/235299279 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f13180c3d90 0x7f13180c41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:40.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.391+0000 7f1317400640 1 -- 172.21.15.60:0/235299279 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f13180c3d90 msgr2=0x7f13180c41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:57:40.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.391+0000 7f1317400640 1 --2- 172.21.15.60:0/235299279 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f13180c3d90 0x7f13180c41c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:40.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.391+0000 7f1317400640 1 -- 172.21.15.60:0/235299279 >> v1:172.21.15.186:6789/0 conn(0x7f1318073320 legacy=0x7f13180c7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:40.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.391+0000 7f1317400640 1 -- 172.21.15.60:0/235299279 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1318159310 con 0x7f13180b1e00 2024-04-14T23:57:40.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.391+0000 7f1317e00640 1 --2- 172.21.15.60:0/235299279 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f13180c3d90 0x7f13180c41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:57:40.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.393+0000 7f1317400640 1 --2- 172.21.15.60:0/235299279 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13180b1e00 0x7f13180c6950 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f1308009870 tx=0x7f130802ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fff40563db6fa1b server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:57:40.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.393+0000 7f1316a00640 1 -- 172.21.15.60:0/235299279 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f130802f840 con 0x7f13180b1e00 2024-04-14T23:57:40.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.394+0000 7f1316a00640 1 -- 172.21.15.60:0/235299279 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f130802f9e0 con 0x7f13180b1e00 2024-04-14T23:57:40.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.394+0000 7f1316a00640 1 -- 172.21.15.60:0/235299279 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f130802fdb0 con 0x7f13180b1e00 2024-04-14T23:57:40.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.394+0000 7f131d000640 1 -- 172.21.15.60:0/235299279 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13180b1e00 msgr2=0x7f13180c6950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:40.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.394+0000 7f131d000640 1 --2- 172.21.15.60:0/235299279 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13180b1e00 0x7f13180c6950 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f1308009870 tx=0x7f130802ba20 comp rx=0 tx=0).stop 2024-04-14T23:57:40.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.394+0000 7f131d000640 1 -- 172.21.15.60:0/235299279 shutdown_connections 2024-04-14T23:57:40.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.394+0000 7f131d000640 1 --2- 172.21.15.60:0/235299279 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f13180c3d90 0x7f13180c41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:40.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.394+0000 7f131d000640 1 --2- 172.21.15.60:0/235299279 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13180b1e00 0x7f13180c6950 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:40.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.394+0000 7f131d000640 1 -- 172.21.15.60:0/235299279 >> 172.21.15.60:0/235299279 conn(0x7f13180c8b50 msgr2=0x7f13180c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:57:40.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.394+0000 7f131d000640 1 -- 172.21.15.60:0/235299279 shutdown_connections 2024-04-14T23:57:40.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.394+0000 7f131d000640 1 -- 172.21.15.60:0/235299279 wait complete. 2024-04-14T23:57:40.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.396+0000 7f131d000640 1 Processor -- start 2024-04-14T23:57:40.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.396+0000 7f131d000640 1 -- start start 2024-04-14T23:57:40.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.397+0000 7f131d000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1318073320 0x7f131814f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:40.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.397+0000 7f131d000640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13180b1e00 0x7f1318071da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:40.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.397+0000 7f131d000640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f13180c3d90 0x7f13181f4320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:40.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.397+0000 7f131d000640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f13180c38b0 con 0x7f13180b1e00 2024-04-14T23:57:40.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.397+0000 7f131d000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1318159890 con 0x7f1318073320 2024-04-14T23:57:40.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.397+0000 7f131d000640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f13181595b0 con 0x7f13180c3d90 2024-04-14T23:57:40.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.398+0000 7f1317400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1318073320 0x7f131814f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:40.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.398+0000 7f130fe00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13180b1e00 0x7f1318071da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:40.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.398+0000 7f1317400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1318073320 0x7f131814f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:38648/0 (socket says 172.21.15.60:38648) 2024-04-14T23:57:40.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.398+0000 7f1317400640 1 -- 172.21.15.60:0/1499340510 learned_addr learned my addr 172.21.15.60:0/1499340510 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:57:40.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.398+0000 7f1317e00640 1 --2- 172.21.15.60:0/1499340510 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f13180c3d90 0x7f13181f4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:40.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.398+0000 7f1317400640 1 -- 172.21.15.60:0/1499340510 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f13180c3d90 msgr2=0x7f13181f4320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:40.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.398+0000 7f1317400640 1 --2- 172.21.15.60:0/1499340510 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f13180c3d90 0x7f13181f4320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:40.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.398+0000 7f1317400640 1 -- 172.21.15.60:0/1499340510 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13180b1e00 msgr2=0x7f1318071da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:40.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.398+0000 7f1317400640 1 --2- 172.21.15.60:0/1499340510 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13180b1e00 0x7f1318071da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:40.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.398+0000 7f1317400640 1 -- 172.21.15.60:0/1499340510 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13181542e0 con 0x7f1318073320 2024-04-14T23:57:40.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.399+0000 7f1317400640 1 --2- 172.21.15.60:0/1499340510 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1318073320 0x7f131814f660 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f1308009990 tx=0x7f1308004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:57:40.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.399+0000 7f130f400640 1 -- 172.21.15.60:0/1499340510 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1308047070 con 0x7f1318073320 2024-04-14T23:57:40.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.399+0000 7f130f400640 1 -- 172.21.15.60:0/1499340510 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1308037be0 con 0x7f1318073320 2024-04-14T23:57:40.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.399+0000 7f130f400640 1 -- 172.21.15.60:0/1499340510 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1308042480 con 0x7f1318073320 2024-04-14T23:57:40.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.399+0000 7f131d000640 1 -- 172.21.15.60:0/1499340510 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13181f1780 con 0x7f1318073320 2024-04-14T23:57:40.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.399+0000 7f131d000640 1 -- 172.21.15.60:0/1499340510 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1318153320 con 0x7f1318073320 2024-04-14T23:57:40.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.400+0000 7f131d000640 1 -- 172.21.15.60:0/1499340510 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f131804f540 con 0x7f1318073320 2024-04-14T23:57:40.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.401+0000 7f130f400640 1 -- 172.21.15.60:0/1499340510 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f130804a070 con 0x7f1318073320 2024-04-14T23:57:40.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.402+0000 7f130f400640 1 --2- 172.21.15.60:0/1499340510 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f12f406b990 0x7f12f406de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:40.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.402+0000 7f130f400640 1 -- 172.21.15.60:0/1499340510 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6255+0+0 (secure 0 0 0) 0x7f13080acbe0 con 0x7f1318073320 2024-04-14T23:57:40.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.403+0000 7f130fe00640 1 --2- 172.21.15.60:0/1499340510 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f12f406b990 0x7f12f406de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:40.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.403+0000 7f130fe00640 1 --2- 172.21.15.60:0/1499340510 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f12f406b990 0x7f12f406de50 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f13181e4800 tx=0x7f1300009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:57:40.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.407+0000 7f130f400640 1 -- 172.21.15.60:0/1499340510 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1308033030 con 0x7f1318073320 2024-04-14T23:57:40.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.650+0000 7f131d000640 1 -- 172.21.15.60:0/1499340510 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1318156ef0 con 0x7f1318073320 2024-04-14T23:57:40.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.651+0000 7f130f400640 1 -- 172.21.15.60:0/1499340510 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+6006 (secure 0 0 0) 0x7f130807b090 con 0x7f1318073320 2024-04-14T23:57:40.658 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:57:40.658 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:57:39.002319+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"a","rank":0,"incarnation":32,"state":"up:replay","state_seq":1,"addr":"172.21.15.60:6833/3281566884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3281566884},{"type":"v1","addr":"172.21.15.60:6833","nonce":3281566884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-14T23:57:40.658 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 32 2024-04-14T23:57:40.660 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.654+0000 7f131d000640 1 -- 172.21.15.60:0/1499340510 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f12f406b990 msgr2=0x7f12f406de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:40.660 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.654+0000 7f131d000640 1 --2- 172.21.15.60:0/1499340510 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f12f406b990 0x7f12f406de50 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f13181e4800 tx=0x7f1300009040 comp rx=0 tx=0).stop 2024-04-14T23:57:40.660 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.654+0000 7f131d000640 1 -- 172.21.15.60:0/1499340510 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1318073320 msgr2=0x7f131814f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:40.660 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.654+0000 7f131d000640 1 --2- 172.21.15.60:0/1499340510 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1318073320 0x7f131814f660 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f1308009990 tx=0x7f1308004730 comp rx=0 tx=0).stop 2024-04-14T23:57:40.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.654+0000 7f131d000640 1 -- 172.21.15.60:0/1499340510 shutdown_connections 2024-04-14T23:57:40.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.654+0000 7f131d000640 1 --2- 172.21.15.60:0/1499340510 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f12f406b990 0x7f12f406de50 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:40.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.654+0000 7f131d000640 1 --2- 172.21.15.60:0/1499340510 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f13180c3d90 0x7f13181f4320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:40.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.654+0000 7f131d000640 1 --2- 172.21.15.60:0/1499340510 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13180b1e00 0x7f1318071da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:40.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.654+0000 7f131d000640 1 --2- 172.21.15.60:0/1499340510 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1318073320 0x7f131814f660 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:40.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.654+0000 7f131d000640 1 -- 172.21.15.60:0/1499340510 >> 172.21.15.60:0/1499340510 conn(0x7f13180c8b50 msgr2=0x7f13181579d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:57:40.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.654+0000 7f131d000640 1 -- 172.21.15.60:0/1499340510 shutdown_connections 2024-04-14T23:57:40.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.654+0000 7f131d000640 1 -- 172.21.15.60:0/1499340510 wait complete. 2024-04-14T23:57:40.680 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c down, removed from mdsmap 2024-04-14T23:57:40.680 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-04-14T23:57:40.680 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:57:40.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.897+0000 7f098e400640 1 Processor -- start 2024-04-14T23:57:40.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.898+0000 7f098e400640 1 -- start start 2024-04-14T23:57:40.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.899+0000 7f098e400640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f09880b1e00 0x7f09880c6950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:40.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.899+0000 7f098e400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f09880c3d90 0x7f09880c41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:40.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.899+0000 7f098e400640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0988152fb0 con 0x7f0988073320 2024-04-14T23:57:40.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.899+0000 7f098e400640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f09880c6540 con 0x7f09880b1e00 2024-04-14T23:57:40.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.899+0000 7f098e400640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f09880b21e0 con 0x7f09880c3d90 2024-04-14T23:57:40.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.899+0000 7f0987e00640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f0988073320 0x7f09880c7fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:35814/0 (socket says 172.21.15.60:35814) 2024-04-14T23:57:40.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.899+0000 7f0987e00640 1 -- 172.21.15.60:0/68635553 learned_addr learned my addr 172.21.15.60:0/68635553 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:57:40.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.899+0000 7f098d000640 1 --2- 172.21.15.60:0/68635553 >> v2:172.21.15.186:3300/0 conn(0x7f09880b1e00 0x7f09880c6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:40.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.899+0000 7f098da00640 1 --2- 172.21.15.60:0/68635553 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f09880c3d90 0x7f09880c41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:40.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.900+0000 7f0987400640 1 -- 172.21.15.60:0/68635553 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 95648812 0 0) 0x7f0988152fb0 con 0x7f0988073320 2024-04-14T23:57:40.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.900+0000 7f0987400640 1 -- 172.21.15.60:0/68635553 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0970003520 con 0x7f0988073320 2024-04-14T23:57:40.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.900+0000 7f098da00640 1 -- 172.21.15.60:0/68635553 >> v2:172.21.15.186:3300/0 conn(0x7f09880b1e00 msgr2=0x7f09880c6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:57:40.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.900+0000 7f098da00640 1 --2- 172.21.15.60:0/68635553 >> v2:172.21.15.186:3300/0 conn(0x7f09880b1e00 0x7f09880c6950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:40.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.900+0000 7f098da00640 1 -- 172.21.15.60:0/68635553 >> v1:172.21.15.60:6789/0 conn(0x7f0988073320 legacy=0x7f09880c7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:40.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.900+0000 7f098da00640 1 -- 172.21.15.60:0/68635553 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0988159310 con 0x7f09880c3d90 2024-04-14T23:57:40.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.901+0000 7f098da00640 1 --2- 172.21.15.60:0/68635553 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f09880c3d90 0x7f09880c41c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0980009d90 tx=0x7f098002bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=13b59b15a0db2a95 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:57:40.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.901+0000 7f0987400640 1 -- 172.21.15.60:0/68635553 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0980039070 con 0x7f09880c3d90 2024-04-14T23:57:40.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.902+0000 7f0987400640 1 -- 172.21.15.60:0/68635553 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f098002cce0 con 0x7f09880c3d90 2024-04-14T23:57:40.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.902+0000 7f0987400640 1 -- 172.21.15.60:0/68635553 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0980034660 con 0x7f09880c3d90 2024-04-14T23:57:40.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.902+0000 7f098e400640 1 -- 172.21.15.60:0/68635553 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f09880c3d90 msgr2=0x7f09880c41c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:40.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.902+0000 7f098e400640 1 --2- 172.21.15.60:0/68635553 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f09880c3d90 0x7f09880c41c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0980009d90 tx=0x7f098002bf30 comp rx=0 tx=0).stop 2024-04-14T23:57:40.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.902+0000 7f098e400640 1 -- 172.21.15.60:0/68635553 shutdown_connections 2024-04-14T23:57:40.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.902+0000 7f098e400640 1 --2- 172.21.15.60:0/68635553 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f09880c3d90 0x7f09880c41c0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:40.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.902+0000 7f098e400640 1 --2- 172.21.15.60:0/68635553 >> v2:172.21.15.186:3300/0 conn(0x7f09880b1e00 0x7f09880c6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:40.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.902+0000 7f098e400640 1 -- 172.21.15.60:0/68635553 >> 172.21.15.60:0/68635553 conn(0x7f09880c8b50 msgr2=0x7f09880c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:57:40.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.902+0000 7f098e400640 1 -- 172.21.15.60:0/68635553 shutdown_connections 2024-04-14T23:57:40.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.902+0000 7f098e400640 1 -- 172.21.15.60:0/68635553 wait complete. 2024-04-14T23:57:40.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.904+0000 7f098e400640 1 Processor -- start 2024-04-14T23:57:40.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.904+0000 7f098e400640 1 -- start start 2024-04-14T23:57:40.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.905+0000 7f098e400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0988073320 0x7f09881f1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:40.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.905+0000 7f098e400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09880b1e00 0x7f09881e2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:40.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.905+0000 7f098e400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f09880c3d90 0x7f09881e3430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:40.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.905+0000 7f098e400640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f09880c38b0 con 0x7f0988073320 2024-04-14T23:57:40.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.905+0000 7f098e400640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0988159890 con 0x7f09880b1e00 2024-04-14T23:57:40.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.905+0000 7f098e400640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f09881595b0 con 0x7f09880c3d90 2024-04-14T23:57:40.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.905+0000 7f0987e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09880b1e00 0x7f09881e2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:40.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.905+0000 7f098da00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f09880c3d90 0x7f09881e3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:40.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.905+0000 7f098d000640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0988073320 0x7f09881f1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:40.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.906+0000 7f0987e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09880b1e00 0x7f09881e2ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:38672/0 (socket says 172.21.15.60:38672) 2024-04-14T23:57:40.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.906+0000 7f0987e00640 1 -- 172.21.15.60:0/774323911 learned_addr learned my addr 172.21.15.60:0/774323911 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:57:40.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.906+0000 7f098da00640 1 --2- 172.21.15.60:0/774323911 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f09880c3d90 0x7f09881e3430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.60:47704/0 (socket says 172.21.15.60:47704) 2024-04-14T23:57:40.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.906+0000 7f0987e00640 1 -- 172.21.15.60:0/774323911 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f09880c3d90 msgr2=0x7f09881e3430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:40.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.906+0000 7f0987e00640 1 --2- 172.21.15.60:0/774323911 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f09880c3d90 0x7f09881e3430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:40.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.906+0000 7f0987e00640 1 -- 172.21.15.60:0/774323911 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0988073320 msgr2=0x7f09881f1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:40.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.906+0000 7f0987e00640 1 --2- 172.21.15.60:0/774323911 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0988073320 0x7f09881f1bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:40.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.906+0000 7f0987e00640 1 -- 172.21.15.60:0/774323911 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09881e4180 con 0x7f09880b1e00 2024-04-14T23:57:40.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.906+0000 7f098da00640 1 --2- 172.21.15.60:0/774323911 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f09880c3d90 0x7f09881e3430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:57:40.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.906+0000 7f098d000640 1 --2- 172.21.15.60:0/774323911 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0988073320 0x7f09881f1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:57:40.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.906+0000 7f0987e00640 1 --2- 172.21.15.60:0/774323911 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09880b1e00 0x7f09881e2ef0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f097400c390 tx=0x7f097400c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:57:40.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.907+0000 7f0984c00640 1 -- 172.21.15.60:0/774323911 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0974017070 con 0x7f09880b1e00 2024-04-14T23:57:40.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.907+0000 7f0984c00640 1 -- 172.21.15.60:0/774323911 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0974005ce0 con 0x7f09880b1e00 2024-04-14T23:57:40.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.907+0000 7f0984c00640 1 -- 172.21.15.60:0/774323911 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09740072f0 con 0x7f09880b1e00 2024-04-14T23:57:40.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.907+0000 7f098e400640 1 -- 172.21.15.60:0/774323911 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f098814eb60 con 0x7f09880b1e00 2024-04-14T23:57:40.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.907+0000 7f098e400640 1 -- 172.21.15.60:0/774323911 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09881f2490 con 0x7f09880b1e00 2024-04-14T23:57:40.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.908+0000 7f098e400640 1 -- 172.21.15.60:0/774323911 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f098804f540 con 0x7f09880b1e00 2024-04-14T23:57:40.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.909+0000 7f0984c00640 1 -- 172.21.15.60:0/774323911 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f097401a070 con 0x7f09880b1e00 2024-04-14T23:57:40.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.910+0000 7f0984c00640 1 --2- 172.21.15.60:0/774323911 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f096806b9c0 0x7f096806de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:40.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.910+0000 7f0984c00640 1 -- 172.21.15.60:0/774323911 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6255+0+0 (secure 0 0 0) 0x7f0974089df0 con 0x7f09880b1e00 2024-04-14T23:57:40.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.910+0000 7f098d000640 1 --2- 172.21.15.60:0/774323911 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f096806b9c0 0x7f096806de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:40.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.911+0000 7f098d000640 1 --2- 172.21.15.60:0/774323911 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f096806b9c0 0x7f096806de80 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f097c009f30 tx=0x7f097c009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:57:40.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:40.914+0000 7f0984c00640 1 -- 172.21.15.60:0/774323911 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f09740580f0 con 0x7f09880b1e00 2024-04-14T23:57:41.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:41.160+0000 7f098e400640 1 -- 172.21.15.60:0/774323911 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f09881568a0 con 0x7f09880b1e00 2024-04-14T23:57:41.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:41.161+0000 7f0984c00640 1 -- 172.21.15.60:0/774323911 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+6006 (secure 0 0 0) 0x7f097405c190 con 0x7f09880b1e00 2024-04-14T23:57:41.167 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:57:41.167 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:57:39.002319+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"a","rank":0,"incarnation":32,"state":"up:replay","state_seq":1,"addr":"172.21.15.60:6833/3281566884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3281566884},{"type":"v1","addr":"172.21.15.60:6833","nonce":3281566884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-14T23:57:41.168 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 32 2024-04-14T23:57:41.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:41.164+0000 7f098e400640 1 -- 172.21.15.60:0/774323911 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f096806b9c0 msgr2=0x7f096806de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:41.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:41.164+0000 7f098e400640 1 --2- 172.21.15.60:0/774323911 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f096806b9c0 0x7f096806de80 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f097c009f30 tx=0x7f097c009190 comp rx=0 tx=0).stop 2024-04-14T23:57:41.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:41.164+0000 7f098e400640 1 -- 172.21.15.60:0/774323911 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09880b1e00 msgr2=0x7f09881e2ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:41.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:41.164+0000 7f098e400640 1 --2- 172.21.15.60:0/774323911 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09880b1e00 0x7f09881e2ef0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f097400c390 tx=0x7f097400c850 comp rx=0 tx=0).stop 2024-04-14T23:57:41.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:41.164+0000 7f098e400640 1 -- 172.21.15.60:0/774323911 shutdown_connections 2024-04-14T23:57:41.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:41.164+0000 7f098e400640 1 --2- 172.21.15.60:0/774323911 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f096806b9c0 0x7f096806de80 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:41.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:41.164+0000 7f098e400640 1 --2- 172.21.15.60:0/774323911 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f09880c3d90 0x7f09881e3430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:41.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:41.164+0000 7f098e400640 1 --2- 172.21.15.60:0/774323911 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09880b1e00 0x7f09881e2ef0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:41.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:41.164+0000 7f098e400640 1 --2- 172.21.15.60:0/774323911 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0988073320 0x7f09881f1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:41.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:41.164+0000 7f098e400640 1 -- 172.21.15.60:0/774323911 >> 172.21.15.60:0/774323911 conn(0x7f09880c8b50 msgr2=0x7f09880cb290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:57:41.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:41.164+0000 7f098e400640 1 -- 172.21.15.60:0/774323911 shutdown_connections 2024-04-14T23:57:41.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:41.164+0000 7f098e400640 1 -- 172.21.15.60:0/774323911 wait complete. 2024-04-14T23:57:41.191 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 32, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-14T23:48:42.700188+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 32, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5286": { "addr": "172.21.15.60:6833/3281566884", "addrs": { "addrvec": [ { "addr": "172.21.15.60:6832", "nonce": 3281566884, "type": "v2" }, { "addr": "172.21.15.60:6833", "nonce": 3281566884, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5286, "incarnation": 32, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 45, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-14T23:57:39.002319+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5286 } } } ], "standbys": [ { "addr": "172.21.15.60:6837/1527815634", "addrs": { "addrvec": [ { "addr": "172.21.15.60:6836", "nonce": 1527815634, "type": "v2" }, { "addr": "172.21.15.60:6837", "nonce": 1527815634, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4197, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6838/3958319486", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6835", "nonce": 3958319486, "type": "v2" }, { "addr": "172.21.15.186:6838", "nonce": 3958319486, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4220, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6837/3910951185", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6834", "nonce": 3910951185, "type": "v2" }, { "addr": "172.21.15.186:6837", "nonce": 3910951185, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4264, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6839/2155020404", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6836", "nonce": 2155020404, "type": "v2" }, { "addr": "172.21.15.186:6839", "nonce": 2155020404, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4267, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-14T23:57:43.192 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:57:43.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.419+0000 7f1940c00640 1 Processor -- start 2024-04-14T23:57:43.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.419+0000 7f1940c00640 1 -- start start 2024-04-14T23:57:43.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.421+0000 7f1940c00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f193c0b1e00 0x7f193c0c6950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:43.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.421+0000 7f1940c00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f193c0c3d90 0x7f193c0c41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:43.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.421+0000 7f1940c00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f193c152fb0 con 0x7f193c073320 2024-04-14T23:57:43.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.421+0000 7f1940c00640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f193c0c6540 con 0x7f193c0b1e00 2024-04-14T23:57:43.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.421+0000 7f1940c00640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f193c0b21e0 con 0x7f193c0c3d90 2024-04-14T23:57:43.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.421+0000 7f193b400640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f193c0b1e00 0x7f193c0c6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:43.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.421+0000 7f1933e00640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f193c073320 0x7f193c0c7fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:48488/0 (socket says 172.21.15.60:48488) 2024-04-14T23:57:43.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.421+0000 7f1933e00640 1 -- 172.21.15.60:0/1151619622 learned_addr learned my addr 172.21.15.60:0/1151619622 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:57:43.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.421+0000 7f193be00640 1 --2- 172.21.15.60:0/1151619622 >> v2:172.21.15.186:3300/0 conn(0x7f193c0c3d90 0x7f193c0c41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:43.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.421+0000 7f193b400640 1 -- 172.21.15.60:0/1151619622 >> v2:172.21.15.186:3300/0 conn(0x7f193c0c3d90 msgr2=0x7f193c0c41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:57:43.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.421+0000 7f193b400640 1 --2- 172.21.15.60:0/1151619622 >> v2:172.21.15.186:3300/0 conn(0x7f193c0c3d90 0x7f193c0c41c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:43.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.421+0000 7f193b400640 1 -- 172.21.15.60:0/1151619622 >> v1:172.21.15.60:6789/0 conn(0x7f193c073320 legacy=0x7f193c0c7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:43.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.421+0000 7f193b400640 1 -- 172.21.15.60:0/1151619622 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f193c159310 con 0x7f193c0b1e00 2024-04-14T23:57:43.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.422+0000 7f193b400640 1 --2- 172.21.15.60:0/1151619622 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f193c0b1e00 0x7f193c0c6950 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f193c0c7c70 tx=0x7f192c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29ce644632f97c6a server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:57:43.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.422+0000 7f193aa00640 1 -- 172.21.15.60:0/1151619622 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f192c034070 con 0x7f193c0b1e00 2024-04-14T23:57:43.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.422+0000 7f193aa00640 1 -- 172.21.15.60:0/1151619622 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f192c030610 con 0x7f193c0b1e00 2024-04-14T23:57:43.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.422+0000 7f193aa00640 1 -- 172.21.15.60:0/1151619622 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f192c0309e0 con 0x7f193c0b1e00 2024-04-14T23:57:43.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.423+0000 7f1940c00640 1 -- 172.21.15.60:0/1151619622 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f193c0b1e00 msgr2=0x7f193c0c6950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:43.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.423+0000 7f1940c00640 1 --2- 172.21.15.60:0/1151619622 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f193c0b1e00 0x7f193c0c6950 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f193c0c7c70 tx=0x7f192c02b960 comp rx=0 tx=0).stop 2024-04-14T23:57:43.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.423+0000 7f1940c00640 1 -- 172.21.15.60:0/1151619622 shutdown_connections 2024-04-14T23:57:43.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.423+0000 7f1940c00640 1 --2- 172.21.15.60:0/1151619622 >> v2:172.21.15.186:3300/0 conn(0x7f193c0c3d90 0x7f193c0c41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:43.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.423+0000 7f1940c00640 1 --2- 172.21.15.60:0/1151619622 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f193c0b1e00 0x7f193c0c6950 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:43.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.423+0000 7f1940c00640 1 -- 172.21.15.60:0/1151619622 >> 172.21.15.60:0/1151619622 conn(0x7f193c0c8b50 msgr2=0x7f193c0c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:57:43.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.424+0000 7f1940c00640 1 -- 172.21.15.60:0/1151619622 shutdown_connections 2024-04-14T23:57:43.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.424+0000 7f1940c00640 1 -- 172.21.15.60:0/1151619622 wait complete. 2024-04-14T23:57:43.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.425+0000 7f1940c00640 1 Processor -- start 2024-04-14T23:57:43.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.426+0000 7f1940c00640 1 -- start start 2024-04-14T23:57:43.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.426+0000 7f1940c00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f193c073320 0x7f193c1f47f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:43.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.427+0000 7f1940c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f193c0b1e00 0x7f193c1f07f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:43.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.427+0000 7f1940c00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f193c0c3d90 0x7f193c0725b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:43.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.427+0000 7f1940c00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f193c0c38b0 con 0x7f193c073320 2024-04-14T23:57:43.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.427+0000 7f1940c00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f193c159890 con 0x7f193c0b1e00 2024-04-14T23:57:43.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.427+0000 7f1940c00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f193c1595b0 con 0x7f193c0c3d90 2024-04-14T23:57:43.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.427+0000 7f193b400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f193c073320 0x7f193c1f47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:43.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.427+0000 7f193b400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f193c073320 0x7f193c1f47f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51766/0 (socket says 172.21.15.60:51766) 2024-04-14T23:57:43.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.427+0000 7f193b400640 1 -- 172.21.15.60:0/2562705081 learned_addr learned my addr 172.21.15.60:0/2562705081 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:57:43.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.427+0000 7f1933e00640 1 --2- 172.21.15.60:0/2562705081 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f193c0b1e00 0x7f193c1f07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:43.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.427+0000 7f193be00640 1 --2- 172.21.15.60:0/2562705081 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f193c0c3d90 0x7f193c0725b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:43.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.427+0000 7f193b400640 1 -- 172.21.15.60:0/2562705081 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f193c0c3d90 msgr2=0x7f193c0725b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:43.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.427+0000 7f193b400640 1 --2- 172.21.15.60:0/2562705081 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f193c0c3d90 0x7f193c0725b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:43.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.427+0000 7f193b400640 1 -- 172.21.15.60:0/2562705081 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f193c0b1e00 msgr2=0x7f193c1f07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:43.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.427+0000 7f193b400640 1 --2- 172.21.15.60:0/2562705081 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f193c0b1e00 0x7f193c1f07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:43.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.427+0000 7f193b400640 1 -- 172.21.15.60:0/2562705081 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f193c1f1030 con 0x7f193c073320 2024-04-14T23:57:43.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.428+0000 7f193b400640 1 --2- 172.21.15.60:0/2562705081 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f193c073320 0x7f193c1f47f0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f192c009890 tx=0x7f192c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:57:43.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.428+0000 7f1933400640 1 -- 172.21.15.60:0/2562705081 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f192c048070 con 0x7f193c073320 2024-04-14T23:57:43.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.428+0000 7f1933400640 1 -- 172.21.15.60:0/2562705081 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f192c030410 con 0x7f193c073320 2024-04-14T23:57:43.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.428+0000 7f1933400640 1 -- 172.21.15.60:0/2562705081 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f192c034040 con 0x7f193c073320 2024-04-14T23:57:43.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.428+0000 7f1940c00640 1 -- 172.21.15.60:0/2562705081 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f193c1f1310 con 0x7f193c073320 2024-04-14T23:57:43.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.428+0000 7f1940c00640 1 -- 172.21.15.60:0/2562705081 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f193c1f3ff0 con 0x7f193c073320 2024-04-14T23:57:43.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.429+0000 7f1940c00640 1 -- 172.21.15.60:0/2562705081 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f193c04f540 con 0x7f193c073320 2024-04-14T23:57:43.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.430+0000 7f1933400640 1 -- 172.21.15.60:0/2562705081 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f192c04b070 con 0x7f193c073320 2024-04-14T23:57:43.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.431+0000 7f1933400640 1 --2- 172.21.15.60:0/2562705081 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f191806b9c0 0x7f191806de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:43.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.431+0000 7f1933400640 1 -- 172.21.15.60:0/2562705081 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6255+0+0 (secure 0 0 0) 0x7f192c0ac410 con 0x7f193c073320 2024-04-14T23:57:43.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.432+0000 7f1933e00640 1 --2- 172.21.15.60:0/2562705081 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f191806b9c0 0x7f191806de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:43.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.434+0000 7f1933e00640 1 --2- 172.21.15.60:0/2562705081 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f191806b9c0 0x7f191806de80 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f193c0adc50 tx=0x7f1924007e80 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:57:43.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.436+0000 7f1933400640 1 -- 172.21.15.60:0/2562705081 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f192c07a680 con 0x7f193c073320 2024-04-14T23:57:43.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.681+0000 7f1940c00640 1 -- 172.21.15.60:0/2562705081 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f193c0c2fe0 con 0x7f193c073320 2024-04-14T23:57:43.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.683+0000 7f1933400640 1 -- 172.21.15.60:0/2562705081 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+6006 (secure 0 0 0) 0x7f192c07e720 con 0x7f193c073320 2024-04-14T23:57:43.689 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:57:43.689 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:57:39.002319+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"a","rank":0,"incarnation":32,"state":"up:replay","state_seq":1,"addr":"172.21.15.60:6833/3281566884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3281566884},{"type":"v1","addr":"172.21.15.60:6833","nonce":3281566884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-14T23:57:43.689 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 32 2024-04-14T23:57:43.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.685+0000 7f1940c00640 1 -- 172.21.15.60:0/2562705081 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f191806b9c0 msgr2=0x7f191806de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:43.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.685+0000 7f1940c00640 1 --2- 172.21.15.60:0/2562705081 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f191806b9c0 0x7f191806de80 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f193c0adc50 tx=0x7f1924007e80 comp rx=0 tx=0).stop 2024-04-14T23:57:43.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.685+0000 7f1940c00640 1 -- 172.21.15.60:0/2562705081 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f193c073320 msgr2=0x7f193c1f47f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:43.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.685+0000 7f1940c00640 1 --2- 172.21.15.60:0/2562705081 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f193c073320 0x7f193c1f47f0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f192c009890 tx=0x7f192c0046c0 comp rx=0 tx=0).stop 2024-04-14T23:57:43.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.685+0000 7f1940c00640 1 -- 172.21.15.60:0/2562705081 shutdown_connections 2024-04-14T23:57:43.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.685+0000 7f1940c00640 1 --2- 172.21.15.60:0/2562705081 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f191806b9c0 0x7f191806de80 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:43.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.685+0000 7f1940c00640 1 --2- 172.21.15.60:0/2562705081 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f193c0c3d90 0x7f193c0725b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:43.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.685+0000 7f1940c00640 1 --2- 172.21.15.60:0/2562705081 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f193c0b1e00 0x7f193c1f07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:43.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.685+0000 7f1940c00640 1 --2- 172.21.15.60:0/2562705081 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f193c073320 0x7f193c1f47f0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:43.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.685+0000 7f1940c00640 1 -- 172.21.15.60:0/2562705081 >> 172.21.15.60:0/2562705081 conn(0x7f193c0c8b50 msgr2=0x7f193c152690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:57:43.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.686+0000 7f1940c00640 1 -- 172.21.15.60:0/2562705081 shutdown_connections 2024-04-14T23:57:43.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:43.686+0000 7f1940c00640 1 -- 172.21.15.60:0/2562705081 wait complete. 2024-04-14T23:57:43.713 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-14T23:57:45.714 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:57:45.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:45.905+0000 7f2989c00640 1 Processor -- start 2024-04-14T23:57:45.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:45.905+0000 7f2989c00640 1 -- start start 2024-04-14T23:57:45.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:45.907+0000 7f2989c00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2984154cf0 0x7f298415a940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:45.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:45.907+0000 7f2989c00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f298406c710 0x7f29840cbd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:45.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:45.908+0000 7f2989c00640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29840c5520 con 0x7f298406c310 2024-04-14T23:57:45.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:45.908+0000 7f2989c00640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f29841550d0 con 0x7f2984154cf0 2024-04-14T23:57:45.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:45.908+0000 7f2989c00640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f298415b040 con 0x7f298406c710 2024-04-14T23:57:45.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:45.908+0000 7f2983e00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2984154cf0 0x7f298415a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:45.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:45.908+0000 7f2983e00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2984154cf0 0x7f298415a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51780/0 (socket says 172.21.15.60:51780) 2024-04-14T23:57:45.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:45.908+0000 7f2983e00640 1 -- 172.21.15.60:0/360983629 learned_addr learned my addr 172.21.15.60:0/360983629 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:57:45.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:45.908+0000 7f2989200640 1 --2- 172.21.15.60:0/360983629 >> v2:172.21.15.186:3300/0 conn(0x7f298406c710 0x7f29840cbd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:45.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:45.908+0000 7f2983e00640 1 -- 172.21.15.60:0/360983629 >> v2:172.21.15.186:3300/0 conn(0x7f298406c710 msgr2=0x7f29840cbd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:57:45.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:45.908+0000 7f2983e00640 1 --2- 172.21.15.60:0/360983629 >> v2:172.21.15.186:3300/0 conn(0x7f298406c710 0x7f29840cbd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:45.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:45.908+0000 7f2983e00640 1 -- 172.21.15.60:0/360983629 >> v1:172.21.15.186:6789/0 conn(0x7f298406c310 legacy=0x7f29840cb120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:45.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:45.908+0000 7f2983e00640 1 -- 172.21.15.60:0/360983629 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f298415b300 con 0x7f2984154cf0 2024-04-14T23:57:45.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:45.910+0000 7f2983e00640 1 --2- 172.21.15.60:0/360983629 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2984154cf0 0x7f298415a940 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f29840b2d80 tx=0x7f297402ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0d1560c3b8ecc9f server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:57:45.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:45.910+0000 7f2983400640 1 -- 172.21.15.60:0/360983629 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f297402f850 con 0x7f2984154cf0 2024-04-14T23:57:45.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:45.910+0000 7f2983400640 1 -- 172.21.15.60:0/360983629 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f297402f9f0 con 0x7f2984154cf0 2024-04-14T23:57:45.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:45.910+0000 7f2983400640 1 -- 172.21.15.60:0/360983629 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f297402fdc0 con 0x7f2984154cf0 2024-04-14T23:57:45.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:45.911+0000 7f2989c00640 1 -- 172.21.15.60:0/360983629 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2984154cf0 msgr2=0x7f298415a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:45.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:45.911+0000 7f2989c00640 1 --2- 172.21.15.60:0/360983629 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2984154cf0 0x7f298415a940 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f29840b2d80 tx=0x7f297402ba30 comp rx=0 tx=0).stop 2024-04-14T23:57:45.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:45.911+0000 7f2989c00640 1 -- 172.21.15.60:0/360983629 shutdown_connections 2024-04-14T23:57:45.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:45.911+0000 7f2989c00640 1 --2- 172.21.15.60:0/360983629 >> v2:172.21.15.186:3300/0 conn(0x7f298406c710 0x7f29840cbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:45.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:45.911+0000 7f2989c00640 1 --2- 172.21.15.60:0/360983629 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2984154cf0 0x7f298415a940 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:45.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:45.911+0000 7f2989c00640 1 -- 172.21.15.60:0/360983629 >> 172.21.15.60:0/360983629 conn(0x7f29840670f0 msgr2=0x7f2984067500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:57:45.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:45.911+0000 7f2989c00640 1 -- 172.21.15.60:0/360983629 shutdown_connections 2024-04-14T23:57:45.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:45.911+0000 7f2989c00640 1 -- 172.21.15.60:0/360983629 wait complete. 2024-04-14T23:57:45.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:45.912+0000 7f2989c00640 1 Processor -- start 2024-04-14T23:57:45.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:45.913+0000 7f2989c00640 1 -- start start 2024-04-14T23:57:45.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:45.913+0000 7f2989c00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f298406c310 0x7f29841455b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:45.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:45.913+0000 7f2989c00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f298406c710 0x7f298415de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:45.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:45.913+0000 7f2989c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2984154cf0 0x7f2984141620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:45.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:45.914+0000 7f2989c00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f298406bb10 con 0x7f298406c710 2024-04-14T23:57:45.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:45.914+0000 7f2989c00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f298415b8c0 con 0x7f2984154cf0 2024-04-14T23:57:45.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:45.914+0000 7f2989c00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f298406bdf0 con 0x7f298406c310 2024-04-14T23:57:45.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:45.914+0000 7f297b400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f298406c710 0x7f298415de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:45.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:45.914+0000 7f2983e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f298406c310 0x7f29841455b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:45.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:45.914+0000 7f2989200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2984154cf0 0x7f2984141620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:45.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:45.914+0000 7f2983e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f298406c310 0x7f29841455b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.60:36980/0 (socket says 172.21.15.60:36980) 2024-04-14T23:57:45.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:45.914+0000 7f297b400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f298406c710 0x7f298415de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51796/0 (socket says 172.21.15.60:51796) 2024-04-14T23:57:45.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:45.914+0000 7f2983e00640 1 -- 172.21.15.60:0/1592625672 learned_addr learned my addr 172.21.15.60:0/1592625672 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:57:45.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:45.914+0000 7f2983e00640 1 -- 172.21.15.60:0/1592625672 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2984154cf0 msgr2=0x7f2984141620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:45.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:45.914+0000 7f2983e00640 1 --2- 172.21.15.60:0/1592625672 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2984154cf0 0x7f2984141620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:45.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:45.914+0000 7f2983e00640 1 -- 172.21.15.60:0/1592625672 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f298406c710 msgr2=0x7f298415de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:45.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:45.914+0000 7f2983e00640 1 --2- 172.21.15.60:0/1592625672 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f298406c710 0x7f298415de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:45.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:45.914+0000 7f2983e00640 1 -- 172.21.15.60:0/1592625672 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f298415bba0 con 0x7f298406c310 2024-04-14T23:57:45.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:45.914+0000 7f2989200640 1 --2- 172.21.15.60:0/1592625672 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2984154cf0 0x7f2984141620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:57:45.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:45.915+0000 7f2983e00640 1 --2- 172.21.15.60:0/1592625672 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f298406c310 0x7f29841455b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2974000c00 tx=0x7f2974004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:57:45.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:45.915+0000 7f2980c00640 1 -- 172.21.15.60:0/1592625672 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2974047070 con 0x7f298406c310 2024-04-14T23:57:45.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:45.915+0000 7f2980c00640 1 -- 172.21.15.60:0/1592625672 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2974037d30 con 0x7f298406c310 2024-04-14T23:57:45.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:45.915+0000 7f2980c00640 1 -- 172.21.15.60:0/1592625672 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29740426c0 con 0x7f298406c310 2024-04-14T23:57:45.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:45.915+0000 7f2989c00640 1 -- 172.21.15.60:0/1592625672 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f298415e740 con 0x7f298406c310 2024-04-14T23:57:45.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:45.915+0000 7f2989c00640 1 -- 172.21.15.60:0/1592625672 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f298415c120 con 0x7f298406c310 2024-04-14T23:57:45.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:45.916+0000 7f2989c00640 1 -- 172.21.15.60:0/1592625672 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f298404f540 con 0x7f298406c310 2024-04-14T23:57:45.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:45.917+0000 7f2980c00640 1 -- 172.21.15.60:0/1592625672 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f297404a070 con 0x7f298406c310 2024-04-14T23:57:45.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:45.919+0000 7f2980c00640 1 --2- 172.21.15.60:0/1592625672 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f296006b990 0x7f296006de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:45.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:45.919+0000 7f2980c00640 1 -- 172.21.15.60:0/1592625672 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(45..45 src has 1..45) ==== 6255+0+0 (secure 0 0 0) 0x7f29740acdb0 con 0x7f298406c310 2024-04-14T23:57:45.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:45.919+0000 7f297b400640 1 --2- 172.21.15.60:0/1592625672 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f296006b990 0x7f296006de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:45.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:45.920+0000 7f297b400640 1 --2- 172.21.15.60:0/1592625672 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f296006b990 0x7f296006de50 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f298414b2c0 tx=0x7f296c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:57:45.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:45.923+0000 7f2980c00640 1 -- 172.21.15.60:0/1592625672 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f29740b1070 con 0x7f298406c310 2024-04-14T23:57:46.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:46.161+0000 7f2989c00640 1 -- 172.21.15.60:0/1592625672 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f298414eb40 con 0x7f298406c310 2024-04-14T23:57:46.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:46.162+0000 7f2980c00640 1 -- 172.21.15.60:0/1592625672 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+6006 (secure 0 0 0) 0x7f2974033030 con 0x7f298406c310 2024-04-14T23:57:46.167 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:57:46.167 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:57:39.002319+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"a","rank":0,"incarnation":32,"state":"up:replay","state_seq":1,"addr":"172.21.15.60:6833/3281566884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3281566884},{"type":"v1","addr":"172.21.15.60:6833","nonce":3281566884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-14T23:57:46.168 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 32 2024-04-14T23:57:46.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:46.165+0000 7f2989c00640 1 -- 172.21.15.60:0/1592625672 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f296006b990 msgr2=0x7f296006de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:46.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:46.165+0000 7f2989c00640 1 --2- 172.21.15.60:0/1592625672 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f296006b990 0x7f296006de50 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f298414b2c0 tx=0x7f296c009040 comp rx=0 tx=0).stop 2024-04-14T23:57:46.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:46.165+0000 7f2989c00640 1 -- 172.21.15.60:0/1592625672 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f298406c310 msgr2=0x7f29841455b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:46.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:46.165+0000 7f2989c00640 1 --2- 172.21.15.60:0/1592625672 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f298406c310 0x7f29841455b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2974000c00 tx=0x7f2974004730 comp rx=0 tx=0).stop 2024-04-14T23:57:46.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:46.165+0000 7f2989c00640 1 -- 172.21.15.60:0/1592625672 shutdown_connections 2024-04-14T23:57:46.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:46.165+0000 7f2989c00640 1 --2- 172.21.15.60:0/1592625672 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f296006b990 0x7f296006de50 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:46.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:46.165+0000 7f2989c00640 1 --2- 172.21.15.60:0/1592625672 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2984154cf0 0x7f2984141620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:46.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:46.165+0000 7f2989c00640 1 --2- 172.21.15.60:0/1592625672 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f298406c710 0x7f298415de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:46.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:46.165+0000 7f2989c00640 1 --2- 172.21.15.60:0/1592625672 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f298406c310 0x7f29841455b0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:46.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:46.165+0000 7f2989c00640 1 -- 172.21.15.60:0/1592625672 >> 172.21.15.60:0/1592625672 conn(0x7f29840670f0 msgr2=0x7f29840caf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:57:46.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:46.165+0000 7f2989c00640 1 -- 172.21.15.60:0/1592625672 shutdown_connections 2024-04-14T23:57:46.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:46.165+0000 7f2989c00640 1 -- 172.21.15.60:0/1592625672 wait complete. 2024-04-14T23:57:46.187 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-14T23:57:46.356 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-14T23:57:46.362 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-14T23:57:48.188 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:57:48.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.417+0000 7f60ed400640 1 Processor -- start 2024-04-14T23:57:48.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.418+0000 7f60ed400640 1 -- start start 2024-04-14T23:57:48.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.419+0000 7f60ed400640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f60e8154cf0 0x7f60e815a940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:48.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.419+0000 7f60ed400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f60e806c310 0x7f60e806c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:48.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.419+0000 7f60ed400640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60e80c5520 con 0x7f60e80cb120 2024-04-14T23:57:48.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.419+0000 7f60ed400640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f60e81550d0 con 0x7f60e8154cf0 2024-04-14T23:57:48.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.419+0000 7f60ed400640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f60e815b040 con 0x7f60e806c310 2024-04-14T23:57:48.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.420+0000 7f60e7e00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f60e8154cf0 0x7f60e815a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:48.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.420+0000 7f60df400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f60e806c310 0x7f60e806c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:48.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.420+0000 7f60e7e00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f60e8154cf0 0x7f60e815a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51812/0 (socket says 172.21.15.60:51812) 2024-04-14T23:57:48.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.420+0000 7f60e7e00640 1 -- 172.21.15.60:0/2780209195 learned_addr learned my addr 172.21.15.60:0/2780209195 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:57:48.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.420+0000 7f60eca00640 1 --1- 172.21.15.60:0/2780209195 >> v1:172.21.15.186:6789/0 conn(0x7f60e80cb120 0x7f60e80cbd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.60:58782/0 (socket says 172.21.15.60:58782) 2024-04-14T23:57:48.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.420+0000 7f60e7e00640 1 -- 172.21.15.60:0/2780209195 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f60e806c310 msgr2=0x7f60e806c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:57:48.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.420+0000 7f60e7e00640 1 --2- 172.21.15.60:0/2780209195 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f60e806c310 0x7f60e806c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:48.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.420+0000 7f60e7400640 1 -- 172.21.15.60:0/2780209195 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3918985616 0 0) 0x7f60e80c5520 con 0x7f60e80cb120 2024-04-14T23:57:48.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.420+0000 7f60e7e00640 1 -- 172.21.15.60:0/2780209195 >> v1:172.21.15.186:6789/0 conn(0x7f60e80cb120 legacy=0x7f60e80cbd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:48.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.420+0000 7f60e7e00640 1 -- 172.21.15.60:0/2780209195 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60e815b300 con 0x7f60e8154cf0 2024-04-14T23:57:48.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.421+0000 7f60e7e00640 1 --2- 172.21.15.60:0/2780209195 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f60e8154cf0 0x7f60e815a940 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f60d8009e50 tx=0x7f60d802bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c9f14d26da5846c server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:57:48.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.422+0000 7f60e7400640 1 -- 172.21.15.60:0/2780209195 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60d802cb90 con 0x7f60e8154cf0 2024-04-14T23:57:48.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.422+0000 7f60e7400640 1 -- 172.21.15.60:0/2780209195 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60d802cd30 con 0x7f60e8154cf0 2024-04-14T23:57:48.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.422+0000 7f60e7400640 1 -- 172.21.15.60:0/2780209195 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60d8034610 con 0x7f60e8154cf0 2024-04-14T23:57:48.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.422+0000 7f60ed400640 1 -- 172.21.15.60:0/2780209195 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f60e8154cf0 msgr2=0x7f60e815a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:48.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.422+0000 7f60ed400640 1 --2- 172.21.15.60:0/2780209195 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f60e8154cf0 0x7f60e815a940 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f60d8009e50 tx=0x7f60d802bff0 comp rx=0 tx=0).stop 2024-04-14T23:57:48.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.422+0000 7f60ed400640 1 -- 172.21.15.60:0/2780209195 shutdown_connections 2024-04-14T23:57:48.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.423+0000 7f60ed400640 1 --2- 172.21.15.60:0/2780209195 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f60e806c310 0x7f60e806c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:48.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.423+0000 7f60ed400640 1 --2- 172.21.15.60:0/2780209195 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f60e8154cf0 0x7f60e815a940 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:48.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.423+0000 7f60ed400640 1 -- 172.21.15.60:0/2780209195 >> 172.21.15.60:0/2780209195 conn(0x7f60e80670f0 msgr2=0x7f60e8067500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:57:48.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.423+0000 7f60ed400640 1 -- 172.21.15.60:0/2780209195 shutdown_connections 2024-04-14T23:57:48.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.423+0000 7f60ed400640 1 -- 172.21.15.60:0/2780209195 wait complete. 2024-04-14T23:57:48.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.424+0000 7f60ed400640 1 Processor -- start 2024-04-14T23:57:48.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.425+0000 7f60ed400640 1 -- start start 2024-04-14T23:57:48.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.426+0000 7f60ed400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60e806c310 0x7f60e8157180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:48.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.426+0000 7f60ed400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f60e80cb120 0x7f60e815e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:48.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.426+0000 7f60ed400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f60e8154cf0 0x7f60e814ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:48.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.426+0000 7f60ed400640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f60e806bb10 con 0x7f60e80cb120 2024-04-14T23:57:48.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.426+0000 7f60ed400640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f60e815b8c0 con 0x7f60e806c310 2024-04-14T23:57:48.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.426+0000 7f60ed400640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f60e806bdf0 con 0x7f60e8154cf0 2024-04-14T23:57:48.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.426+0000 7f60df400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f60e80cb120 0x7f60e815e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:48.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.426+0000 7f60df400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f60e80cb120 0x7f60e815e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51824/0 (socket says 172.21.15.60:51824) 2024-04-14T23:57:48.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.426+0000 7f60df400640 1 -- 172.21.15.60:0/576237952 learned_addr learned my addr 172.21.15.60:0/576237952 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:57:48.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.426+0000 7f60eca00640 1 --2- 172.21.15.60:0/576237952 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f60e8154cf0 0x7f60e814ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:48.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.426+0000 7f60e7e00640 1 --2- 172.21.15.60:0/576237952 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60e806c310 0x7f60e8157180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:48.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.426+0000 7f60df400640 1 -- 172.21.15.60:0/576237952 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f60e8154cf0 msgr2=0x7f60e814ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:48.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.426+0000 7f60df400640 1 --2- 172.21.15.60:0/576237952 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f60e8154cf0 0x7f60e814ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:48.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.426+0000 7f60df400640 1 -- 172.21.15.60:0/576237952 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60e806c310 msgr2=0x7f60e8157180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:48.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.426+0000 7f60df400640 1 --2- 172.21.15.60:0/576237952 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60e806c310 0x7f60e8157180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:48.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.426+0000 7f60df400640 1 -- 172.21.15.60:0/576237952 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60e815bf10 con 0x7f60e80cb120 2024-04-14T23:57:48.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.427+0000 7f60df400640 1 --2- 172.21.15.60:0/576237952 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f60e80cb120 0x7f60e815e710 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f60d000e9b0 tx=0x7f60d000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:57:48.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.427+0000 7f60e4c00640 1 -- 172.21.15.60:0/576237952 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60d0017070 con 0x7f60e80cb120 2024-04-14T23:57:48.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.427+0000 7f60e4c00640 1 -- 172.21.15.60:0/576237952 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60d000c9a0 con 0x7f60e80cb120 2024-04-14T23:57:48.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.427+0000 7f60e4c00640 1 -- 172.21.15.60:0/576237952 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60d000cd90 con 0x7f60e80cb120 2024-04-14T23:57:48.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.427+0000 7f60ed400640 1 -- 172.21.15.60:0/576237952 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60e8157d60 con 0x7f60e80cb120 2024-04-14T23:57:48.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.427+0000 7f60ed400640 1 -- 172.21.15.60:0/576237952 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60e8155360 con 0x7f60e80cb120 2024-04-14T23:57:48.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.429+0000 7f60ed400640 1 -- 172.21.15.60:0/576237952 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60e804f540 con 0x7f60e80cb120 2024-04-14T23:57:48.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.429+0000 7f60e4c00640 1 -- 172.21.15.60:0/576237952 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f60d001a070 con 0x7f60e80cb120 2024-04-14T23:57:48.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.431+0000 7f60e4c00640 1 --2- 172.21.15.60:0/576237952 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f60c406b9c0 0x7f60c406de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:48.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.431+0000 7f60e4c00640 1 -- 172.21.15.60:0/576237952 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6255+0+0 (secure 0 0 0) 0x7f60d008adf0 con 0x7f60e80cb120 2024-04-14T23:57:48.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.431+0000 7f60e7e00640 1 --2- 172.21.15.60:0/576237952 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f60c406b9c0 0x7f60c406de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:48.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.432+0000 7f60e7e00640 1 --2- 172.21.15.60:0/576237952 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f60c406b9c0 0x7f60c406de80 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f60d8035000 tx=0x7f60d8030000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:57:48.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.436+0000 7f60e4c00640 1 -- 172.21.15.60:0/576237952 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f60d0059100 con 0x7f60e80cb120 2024-04-14T23:57:48.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.674+0000 7f60ed400640 1 -- 172.21.15.60:0/576237952 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f60e80cbb10 con 0x7f60e80cb120 2024-04-14T23:57:48.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.676+0000 7f60e4c00640 1 -- 172.21.15.60:0/576237952 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+6010 (secure 0 0 0) 0x7f60d005d1a0 con 0x7f60e80cb120 2024-04-14T23:57:48.681 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:57:48.681 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:57:46.452759+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"a","rank":0,"incarnation":32,"state":"up:reconnect","state_seq":15,"addr":"172.21.15.60:6833/3281566884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3281566884},{"type":"v1","addr":"172.21.15.60:6833","nonce":3281566884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-14T23:57:48.681 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 33 2024-04-14T23:57:48.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.678+0000 7f60ed400640 1 -- 172.21.15.60:0/576237952 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f60c406b9c0 msgr2=0x7f60c406de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:48.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.678+0000 7f60ed400640 1 --2- 172.21.15.60:0/576237952 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f60c406b9c0 0x7f60c406de80 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f60d8035000 tx=0x7f60d8030000 comp rx=0 tx=0).stop 2024-04-14T23:57:48.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.678+0000 7f60ed400640 1 -- 172.21.15.60:0/576237952 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f60e80cb120 msgr2=0x7f60e815e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:48.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.678+0000 7f60ed400640 1 --2- 172.21.15.60:0/576237952 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f60e80cb120 0x7f60e815e710 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f60d000e9b0 tx=0x7f60d000ee70 comp rx=0 tx=0).stop 2024-04-14T23:57:48.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.678+0000 7f60ed400640 1 -- 172.21.15.60:0/576237952 shutdown_connections 2024-04-14T23:57:48.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.678+0000 7f60ed400640 1 --2- 172.21.15.60:0/576237952 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f60c406b9c0 0x7f60c406de80 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:48.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.678+0000 7f60ed400640 1 --2- 172.21.15.60:0/576237952 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f60e8154cf0 0x7f60e814ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:48.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.678+0000 7f60ed400640 1 --2- 172.21.15.60:0/576237952 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f60e80cb120 0x7f60e815e710 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:48.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.678+0000 7f60ed400640 1 --2- 172.21.15.60:0/576237952 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60e806c310 0x7f60e8157180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:48.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.678+0000 7f60ed400640 1 -- 172.21.15.60:0/576237952 >> 172.21.15.60:0/576237952 conn(0x7f60e80670f0 msgr2=0x7f60e815a770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:57:48.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.678+0000 7f60ed400640 1 -- 172.21.15.60:0/576237952 shutdown_connections 2024-04-14T23:57:48.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:48.679+0000 7f60ed400640 1 -- 172.21.15.60:0/576237952 wait complete. 2024-04-14T23:57:48.701 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-14T23:57:50.701 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:57:50.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:50.930+0000 7f3181200640 1 Processor -- start 2024-04-14T23:57:50.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:50.931+0000 7f3181200640 1 -- start start 2024-04-14T23:57:50.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:50.932+0000 7f3181200640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f317c15c8a0 0x7f317c072390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:50.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:50.932+0000 7f3181200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f317c0c86d0 0x7f317c1625c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:50.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:50.932+0000 7f3181200640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f317c071bf0 con 0x7f317c0c6950 2024-04-14T23:57:50.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:50.932+0000 7f3181200640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f317c15cc80 con 0x7f317c15c8a0 2024-04-14T23:57:50.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:50.932+0000 7f3181200640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f317c1632a0 con 0x7f317c0c86d0 2024-04-14T23:57:50.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:50.933+0000 7f317b400640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f317c15c8a0 0x7f317c072390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:50.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:50.933+0000 7f317be00640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f317c0c6950 0x7f317c15f050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:48496/0 (socket says 172.21.15.60:48496) 2024-04-14T23:57:50.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:50.933+0000 7f317be00640 1 -- 172.21.15.60:0/3931954589 learned_addr learned my addr 172.21.15.60:0/3931954589 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:57:50.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:50.933+0000 7f317aa00640 1 --2- 172.21.15.60:0/3931954589 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f317c0c86d0 0x7f317c1625c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:50.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:50.933+0000 7f317aa00640 1 -- 172.21.15.60:0/3931954589 >> v2:172.21.15.60:3300/0 conn(0x7f317c15c8a0 msgr2=0x7f317c072390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:57:50.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:50.933+0000 7f317aa00640 1 --2- 172.21.15.60:0/3931954589 >> v2:172.21.15.60:3300/0 conn(0x7f317c15c8a0 0x7f317c072390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:50.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:50.933+0000 7f317aa00640 1 -- 172.21.15.60:0/3931954589 >> v1:172.21.15.60:6789/0 conn(0x7f317c0c6950 legacy=0x7f317c15f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:50.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:50.933+0000 7f317aa00640 1 -- 172.21.15.60:0/3931954589 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f317c0c6fb0 con 0x7f317c0c86d0 2024-04-14T23:57:50.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:50.935+0000 7f317aa00640 1 --2- 172.21.15.60:0/3931954589 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f317c0c86d0 0x7f317c1625c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f31680097a0 tx=0x7f316802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=997660dbefcb3e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:57:50.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:50.935+0000 7f317a000640 1 -- 172.21.15.60:0/3931954589 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3168035020 con 0x7f317c0c86d0 2024-04-14T23:57:50.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:50.935+0000 7f317a000640 1 -- 172.21.15.60:0/3931954589 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3168031830 con 0x7f317c0c86d0 2024-04-14T23:57:50.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:50.935+0000 7f317a000640 1 -- 172.21.15.60:0/3931954589 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3168031c00 con 0x7f317c0c86d0 2024-04-14T23:57:50.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:50.936+0000 7f3181200640 1 -- 172.21.15.60:0/3931954589 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f317c0c86d0 msgr2=0x7f317c1625c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:50.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:50.936+0000 7f3181200640 1 --2- 172.21.15.60:0/3931954589 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f317c0c86d0 0x7f317c1625c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f31680097a0 tx=0x7f316802b8d0 comp rx=0 tx=0).stop 2024-04-14T23:57:50.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:50.936+0000 7f3181200640 1 -- 172.21.15.60:0/3931954589 shutdown_connections 2024-04-14T23:57:50.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:50.936+0000 7f3181200640 1 --2- 172.21.15.60:0/3931954589 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f317c0c86d0 0x7f317c1625c0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:50.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:50.936+0000 7f3181200640 1 --2- 172.21.15.60:0/3931954589 >> v2:172.21.15.60:3300/0 conn(0x7f317c15c8a0 0x7f317c072390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:50.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:50.936+0000 7f3181200640 1 -- 172.21.15.60:0/3931954589 >> 172.21.15.60:0/3931954589 conn(0x7f317c0c8b50 msgr2=0x7f317c0c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:57:50.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:50.936+0000 7f3181200640 1 -- 172.21.15.60:0/3931954589 shutdown_connections 2024-04-14T23:57:50.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:50.936+0000 7f3181200640 1 -- 172.21.15.60:0/3931954589 wait complete. 2024-04-14T23:57:50.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:50.938+0000 7f3181200640 1 Processor -- start 2024-04-14T23:57:50.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:50.938+0000 7f3181200640 1 -- start start 2024-04-14T23:57:50.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:50.939+0000 7f3181200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f317c0c6950 0x7f317c1f8e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:50.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:50.939+0000 7f3181200640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f317c0c86d0 0x7f317c1539c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:50.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:50.939+0000 7f3181200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f317c15c8a0 0x7f317c1ea280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:50.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:50.939+0000 7f3181200640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f317c0728d0 con 0x7f317c0c86d0 2024-04-14T23:57:50.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:50.939+0000 7f3181200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f317c071e70 con 0x7f317c0c6950 2024-04-14T23:57:50.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:50.939+0000 7f3181200640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f317c06b1c0 con 0x7f317c15c8a0 2024-04-14T23:57:50.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:50.939+0000 7f317aa00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f317c0c86d0 0x7f317c1539c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:50.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:50.939+0000 7f317aa00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f317c0c86d0 0x7f317c1539c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51840/0 (socket says 172.21.15.60:51840) 2024-04-14T23:57:50.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:50.939+0000 7f317aa00640 1 -- 172.21.15.60:0/2169646940 learned_addr learned my addr 172.21.15.60:0/2169646940 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:57:50.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:50.939+0000 7f317be00640 1 --2- 172.21.15.60:0/2169646940 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f317c15c8a0 0x7f317c1ea280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:50.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:50.939+0000 7f317b400640 1 --2- 172.21.15.60:0/2169646940 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f317c0c6950 0x7f317c1f8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:50.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:50.939+0000 7f317aa00640 1 -- 172.21.15.60:0/2169646940 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f317c15c8a0 msgr2=0x7f317c1ea280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:50.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:50.939+0000 7f317aa00640 1 --2- 172.21.15.60:0/2169646940 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f317c15c8a0 0x7f317c1ea280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:50.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:50.939+0000 7f317aa00640 1 -- 172.21.15.60:0/2169646940 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f317c0c6950 msgr2=0x7f317c1f8e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:50.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:50.939+0000 7f317aa00640 1 --2- 172.21.15.60:0/2169646940 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f317c0c6950 0x7f317c1f8e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:50.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:50.939+0000 7f317aa00640 1 -- 172.21.15.60:0/2169646940 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f317c161db0 con 0x7f317c0c86d0 2024-04-14T23:57:50.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:50.940+0000 7f317aa00640 1 --2- 172.21.15.60:0/2169646940 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f317c0c86d0 0x7f317c1539c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f31680098c0 tx=0x7f31680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:57:50.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:50.940+0000 7f3173400640 1 -- 172.21.15.60:0/2169646940 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3168041070 con 0x7f317c0c86d0 2024-04-14T23:57:50.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:50.940+0000 7f3173400640 1 -- 172.21.15.60:0/2169646940 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f316803b920 con 0x7f317c0c86d0 2024-04-14T23:57:50.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:50.940+0000 7f3173400640 1 -- 172.21.15.60:0/2169646940 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3168035040 con 0x7f317c0c86d0 2024-04-14T23:57:50.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:50.940+0000 7f3181200640 1 -- 172.21.15.60:0/2169646940 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f317c162090 con 0x7f317c0c86d0 2024-04-14T23:57:50.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:50.940+0000 7f3181200640 1 -- 172.21.15.60:0/2169646940 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f317c1e9a80 con 0x7f317c0c86d0 2024-04-14T23:57:50.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:50.941+0000 7f3181200640 1 -- 172.21.15.60:0/2169646940 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f317c04f540 con 0x7f317c0c86d0 2024-04-14T23:57:50.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:50.942+0000 7f3173400640 1 -- 172.21.15.60:0/2169646940 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f316804c060 con 0x7f317c0c86d0 2024-04-14T23:57:50.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:50.943+0000 7f3173400640 1 --2- 172.21.15.60:0/2169646940 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f315c06b9c0 0x7f315c06de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:50.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:50.943+0000 7f3173400640 1 -- 172.21.15.60:0/2169646940 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6255+0+0 (secure 0 0 0) 0x7f31680ac230 con 0x7f317c0c86d0 2024-04-14T23:57:50.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:50.944+0000 7f317b400640 1 --2- 172.21.15.60:0/2169646940 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f315c06b9c0 0x7f315c06de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:50.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:50.945+0000 7f317b400640 1 --2- 172.21.15.60:0/2169646940 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f315c06b9c0 0x7f315c06de80 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f316c009b40 tx=0x7f316c009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:57:50.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:50.949+0000 7f3173400640 1 -- 172.21.15.60:0/2169646940 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3168031800 con 0x7f317c0c86d0 2024-04-14T23:57:51.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:51.185+0000 7f3181200640 1 -- 172.21.15.60:0/2169646940 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f317c1623c0 con 0x7f317c0c86d0 2024-04-14T23:57:51.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:51.186+0000 7f3173400640 1 -- 172.21.15.60:0/2169646940 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+6010 (secure 0 0 0) 0x7f316807a530 con 0x7f317c0c86d0 2024-04-14T23:57:51.192 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:57:51.192 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:57:46.452759+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"a","rank":0,"incarnation":32,"state":"up:reconnect","state_seq":15,"addr":"172.21.15.60:6833/3281566884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3281566884},{"type":"v1","addr":"172.21.15.60:6833","nonce":3281566884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-14T23:57:51.192 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 33 2024-04-14T23:57:51.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:51.189+0000 7f3181200640 1 -- 172.21.15.60:0/2169646940 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f315c06b9c0 msgr2=0x7f315c06de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:51.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:51.189+0000 7f3181200640 1 --2- 172.21.15.60:0/2169646940 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f315c06b9c0 0x7f315c06de80 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f316c009b40 tx=0x7f316c009190 comp rx=0 tx=0).stop 2024-04-14T23:57:51.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:51.189+0000 7f3181200640 1 -- 172.21.15.60:0/2169646940 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f317c0c86d0 msgr2=0x7f317c1539c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:51.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:51.189+0000 7f3181200640 1 --2- 172.21.15.60:0/2169646940 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f317c0c86d0 0x7f317c1539c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f31680098c0 tx=0x7f31680046c0 comp rx=0 tx=0).stop 2024-04-14T23:57:51.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:51.189+0000 7f3181200640 1 -- 172.21.15.60:0/2169646940 shutdown_connections 2024-04-14T23:57:51.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:51.189+0000 7f3181200640 1 --2- 172.21.15.60:0/2169646940 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f315c06b9c0 0x7f315c06de80 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:51.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:51.189+0000 7f3181200640 1 --2- 172.21.15.60:0/2169646940 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f317c15c8a0 0x7f317c1ea280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:51.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:51.189+0000 7f3181200640 1 --2- 172.21.15.60:0/2169646940 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f317c0c86d0 0x7f317c1539c0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:51.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:51.189+0000 7f3181200640 1 --2- 172.21.15.60:0/2169646940 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f317c0c6950 0x7f317c1f8e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:51.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:51.189+0000 7f3181200640 1 -- 172.21.15.60:0/2169646940 >> 172.21.15.60:0/2169646940 conn(0x7f317c0c8b50 msgr2=0x7f317c0c3c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:57:51.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:51.189+0000 7f3181200640 1 -- 172.21.15.60:0/2169646940 shutdown_connections 2024-04-14T23:57:51.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:51.190+0000 7f3181200640 1 -- 172.21.15.60:0/2169646940 wait complete. 2024-04-14T23:57:51.212 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-14T23:57:53.213 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:57:53.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.438+0000 7f9250c00640 1 Processor -- start 2024-04-14T23:57:53.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.438+0000 7f9250c00640 1 -- start start 2024-04-14T23:57:53.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.439+0000 7f9250c00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f924c154cf0 0x7f924c15a940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:53.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.440+0000 7f9250c00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f924c06c710 0x7f924c0cbd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:53.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.440+0000 7f9250c00640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f924c0c5520 con 0x7f924c06c310 2024-04-14T23:57:53.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.440+0000 7f9250c00640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f924c1550d0 con 0x7f924c06c710 2024-04-14T23:57:53.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.440+0000 7f9250c00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f924c15b040 con 0x7f924c154cf0 2024-04-14T23:57:53.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.440+0000 7f924be00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f924c06c710 0x7f924c0cbd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:53.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.440+0000 7f924aa00640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f924c06c310 0x7f924c0cb120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.60:34812/0 (socket says 172.21.15.60:34812) 2024-04-14T23:57:53.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.440+0000 7f924aa00640 1 -- 172.21.15.60:0/1971522854 learned_addr learned my addr 172.21.15.60:0/1971522854 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:57:53.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.440+0000 7f924b400640 1 --2- 172.21.15.60:0/1971522854 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f924c154cf0 0x7f924c15a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:53.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.441+0000 7f924be00640 1 -- 172.21.15.60:0/1971522854 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f924c154cf0 msgr2=0x7f924c15a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:57:53.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.441+0000 7f924be00640 1 --2- 172.21.15.60:0/1971522854 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f924c154cf0 0x7f924c15a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:53.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.441+0000 7f924be00640 1 -- 172.21.15.60:0/1971522854 >> v1:172.21.15.186:6789/0 conn(0x7f924c06c310 legacy=0x7f924c0cb120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:53.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.441+0000 7f924be00640 1 -- 172.21.15.60:0/1971522854 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f924c15b300 con 0x7f924c06c710 2024-04-14T23:57:53.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.441+0000 7f924a000640 1 -- 172.21.15.60:0/1971522854 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2251579416 0 0) 0x7f924c0c5520 con 0x7f924c06c310 2024-04-14T23:57:53.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.441+0000 7f924b400640 1 --2- 172.21.15.60:0/1971522854 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f924c154cf0 0x7f924c15a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:57:53.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.442+0000 7f924be00640 1 --2- 172.21.15.60:0/1971522854 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f924c06c710 0x7f924c0cbd30 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f9244009d90 tx=0x7f924402bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b17b1c364d08ce7f server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:57:53.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.442+0000 7f924a000640 1 -- 172.21.15.60:0/1971522854 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9244039070 con 0x7f924c06c710 2024-04-14T23:57:53.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.442+0000 7f924a000640 1 -- 172.21.15.60:0/1971522854 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f924402cce0 con 0x7f924c06c710 2024-04-14T23:57:53.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.442+0000 7f924a000640 1 -- 172.21.15.60:0/1971522854 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9244034660 con 0x7f924c06c710 2024-04-14T23:57:53.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.443+0000 7f9250c00640 1 -- 172.21.15.60:0/1971522854 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f924c06c710 msgr2=0x7f924c0cbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:53.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.443+0000 7f9250c00640 1 --2- 172.21.15.60:0/1971522854 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f924c06c710 0x7f924c0cbd30 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f9244009d90 tx=0x7f924402bf30 comp rx=0 tx=0).stop 2024-04-14T23:57:53.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.443+0000 7f9250c00640 1 -- 172.21.15.60:0/1971522854 shutdown_connections 2024-04-14T23:57:53.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.443+0000 7f9250c00640 1 --2- 172.21.15.60:0/1971522854 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f924c06c710 0x7f924c0cbd30 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:53.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.443+0000 7f9250c00640 1 --2- 172.21.15.60:0/1971522854 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f924c154cf0 0x7f924c15a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:53.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.443+0000 7f9250c00640 1 -- 172.21.15.60:0/1971522854 >> 172.21.15.60:0/1971522854 conn(0x7f924c0670f0 msgr2=0x7f924c067500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:57:53.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.443+0000 7f9250c00640 1 -- 172.21.15.60:0/1971522854 shutdown_connections 2024-04-14T23:57:53.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.443+0000 7f9250c00640 1 -- 172.21.15.60:0/1971522854 wait complete. 2024-04-14T23:57:53.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.445+0000 7f9250c00640 1 Processor -- start 2024-04-14T23:57:53.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.445+0000 7f9250c00640 1 -- start start 2024-04-14T23:57:53.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.446+0000 7f9250c00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f924c06c310 0x7f924c150160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:53.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.446+0000 7f9250c00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f924c06c710 0x7f924c1f2960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:53.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.446+0000 7f9250c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f924c154cf0 0x7f924c1f2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:53.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.446+0000 7f9250c00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f924c06bb10 con 0x7f924c06c710 2024-04-14T23:57:53.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.446+0000 7f9250c00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f924c15b8c0 con 0x7f924c154cf0 2024-04-14T23:57:53.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.446+0000 7f9250c00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f924c06bdf0 con 0x7f924c06c310 2024-04-14T23:57:53.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.446+0000 7f924aa00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f924c06c710 0x7f924c1f2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:53.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.446+0000 7f924be00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f924c154cf0 0x7f924c1f2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:53.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.446+0000 7f924aa00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f924c06c710 0x7f924c1f2960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41612/0 (socket says 172.21.15.60:41612) 2024-04-14T23:57:53.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.446+0000 7f924aa00640 1 -- 172.21.15.60:0/2147232209 learned_addr learned my addr 172.21.15.60:0/2147232209 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:57:53.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.446+0000 7f924b400640 1 --2- 172.21.15.60:0/2147232209 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f924c06c310 0x7f924c150160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:53.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.447+0000 7f924be00640 1 -- 172.21.15.60:0/2147232209 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f924c06c310 msgr2=0x7f924c150160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:53.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.447+0000 7f924be00640 1 --2- 172.21.15.60:0/2147232209 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f924c06c310 0x7f924c150160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:53.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.447+0000 7f924be00640 1 -- 172.21.15.60:0/2147232209 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f924c06c710 msgr2=0x7f924c1f2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:53.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.447+0000 7f924be00640 1 --2- 172.21.15.60:0/2147232209 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f924c06c710 0x7f924c1f2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:53.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.447+0000 7f924be00640 1 -- 172.21.15.60:0/2147232209 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f924c15ebb0 con 0x7f924c154cf0 2024-04-14T23:57:53.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.447+0000 7f924b400640 1 --2- 172.21.15.60:0/2147232209 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f924c06c310 0x7f924c150160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:57:53.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.447+0000 7f924be00640 1 --2- 172.21.15.60:0/2147232209 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f924c154cf0 0x7f924c1f2ea0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f924c0d13c0 tx=0x7f9244004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:57:53.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.447+0000 7f9243400640 1 -- 172.21.15.60:0/2147232209 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9244039050 con 0x7f924c154cf0 2024-04-14T23:57:53.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.448+0000 7f9243400640 1 -- 172.21.15.60:0/2147232209 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9244045070 con 0x7f924c154cf0 2024-04-14T23:57:53.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.448+0000 7f9243400640 1 -- 172.21.15.60:0/2147232209 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9244009450 con 0x7f924c154cf0 2024-04-14T23:57:53.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.448+0000 7f9250c00640 1 -- 172.21.15.60:0/2147232209 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f924c14bb50 con 0x7f924c154cf0 2024-04-14T23:57:53.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.448+0000 7f9250c00640 1 -- 172.21.15.60:0/2147232209 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f924c14faa0 con 0x7f924c154cf0 2024-04-14T23:57:53.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.449+0000 7f9250c00640 1 -- 172.21.15.60:0/2147232209 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f924c04f540 con 0x7f924c154cf0 2024-04-14T23:57:53.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.450+0000 7f9243400640 1 -- 172.21.15.60:0/2147232209 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f924403e850 con 0x7f924c154cf0 2024-04-14T23:57:53.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.451+0000 7f9243400640 1 --2- 172.21.15.60:0/2147232209 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f922806ba10 0x7f922806ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:53.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.451+0000 7f9243400640 1 -- 172.21.15.60:0/2147232209 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6255+0+0 (secure 0 0 0) 0x7f92440ad6f0 con 0x7f924c154cf0 2024-04-14T23:57:53.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.451+0000 7f924b400640 1 --2- 172.21.15.60:0/2147232209 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f922806ba10 0x7f922806ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:53.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.452+0000 7f924b400640 1 --2- 172.21.15.60:0/2147232209 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f922806ba10 0x7f922806ded0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f923c009560 tx=0x7f923c009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:57:53.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.456+0000 7f9243400640 1 -- 172.21.15.60:0/2147232209 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f924407bb90 con 0x7f924c154cf0 2024-04-14T23:57:53.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.700+0000 7f9250c00640 1 -- 172.21.15.60:0/2147232209 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f924c0cbb10 con 0x7f924c154cf0 2024-04-14T23:57:53.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.701+0000 7f9243400640 1 -- 172.21.15.60:0/2147232209 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+6007 (secure 0 0 0) 0x7f924407fc30 con 0x7f924c154cf0 2024-04-14T23:57:53.707 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:57:53.707 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:57:51.480229+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"a","rank":0,"incarnation":32,"state":"up:rejoin","state_seq":17,"addr":"172.21.15.60:6833/3281566884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3281566884},{"type":"v1","addr":"172.21.15.60:6833","nonce":3281566884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-14T23:57:53.707 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 34 2024-04-14T23:57:53.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.704+0000 7f9250c00640 1 -- 172.21.15.60:0/2147232209 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f922806ba10 msgr2=0x7f922806ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:53.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.704+0000 7f9250c00640 1 --2- 172.21.15.60:0/2147232209 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f922806ba10 0x7f922806ded0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f923c009560 tx=0x7f923c009190 comp rx=0 tx=0).stop 2024-04-14T23:57:53.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.704+0000 7f9250c00640 1 -- 172.21.15.60:0/2147232209 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f924c154cf0 msgr2=0x7f924c1f2ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:53.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.704+0000 7f9250c00640 1 --2- 172.21.15.60:0/2147232209 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f924c154cf0 0x7f924c1f2ea0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f924c0d13c0 tx=0x7f9244004a40 comp rx=0 tx=0).stop 2024-04-14T23:57:53.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.704+0000 7f9250c00640 1 -- 172.21.15.60:0/2147232209 shutdown_connections 2024-04-14T23:57:53.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.704+0000 7f9250c00640 1 --2- 172.21.15.60:0/2147232209 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f922806ba10 0x7f922806ded0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:53.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.704+0000 7f9250c00640 1 --2- 172.21.15.60:0/2147232209 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f924c154cf0 0x7f924c1f2ea0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:53.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.704+0000 7f9250c00640 1 --2- 172.21.15.60:0/2147232209 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f924c06c710 0x7f924c1f2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:53.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.704+0000 7f9250c00640 1 --2- 172.21.15.60:0/2147232209 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f924c06c310 0x7f924c150160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:53.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.704+0000 7f9250c00640 1 -- 172.21.15.60:0/2147232209 >> 172.21.15.60:0/2147232209 conn(0x7f924c0670f0 msgr2=0x7f924c0caf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:57:53.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.704+0000 7f9250c00640 1 -- 172.21.15.60:0/2147232209 shutdown_connections 2024-04-14T23:57:53.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:53.704+0000 7f9250c00640 1 -- 172.21.15.60:0/2147232209 wait complete. 2024-04-14T23:57:53.730 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 34, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-14T23:48:42.700188+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 34, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5286": { "addr": "172.21.15.60:6833/3281566884", "addrs": { "addrvec": [ { "addr": "172.21.15.60:6832", "nonce": 3281566884, "type": "v2" }, { "addr": "172.21.15.60:6833", "nonce": 3281566884, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5286, "incarnation": 32, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:rejoin", "state_seq": 17 } }, "last_failure": 0, "last_failure_osd_epoch": 45, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-14T23:57:51.480229+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5286 } } } ], "standbys": [ { "addr": "172.21.15.60:6837/1527815634", "addrs": { "addrvec": [ { "addr": "172.21.15.60:6836", "nonce": 1527815634, "type": "v2" }, { "addr": "172.21.15.60:6837", "nonce": 1527815634, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4197, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6838/3958319486", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6835", "nonce": 3958319486, "type": "v2" }, { "addr": "172.21.15.186:6838", "nonce": 3958319486, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4220, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6837/3910951185", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6834", "nonce": 3910951185, "type": "v2" }, { "addr": "172.21.15.186:6837", "nonce": 3910951185, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4264, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6839/2155020404", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6836", "nonce": 2155020404, "type": "v2" }, { "addr": "172.21.15.186:6839", "nonce": 2155020404, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4267, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-14T23:57:55.732 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:57:55.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:55.957+0000 7fa4a5400640 1 Processor -- start 2024-04-14T23:57:55.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:55.957+0000 7fa4a5400640 1 -- start start 2024-04-14T23:57:55.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:55.959+0000 7fa4a5400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa4a00b1e00 0x7fa4a00c6950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:55.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:55.959+0000 7fa4a5400640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa4a00c3d90 0x7fa4a00c41e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:55.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:55.959+0000 7fa4a5400640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4a0152fb0 con 0x7fa4a0073320 2024-04-14T23:57:55.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:55.959+0000 7fa4a5400640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fa4a00c6540 con 0x7fa4a00c3d90 2024-04-14T23:57:55.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:55.959+0000 7fa4a5400640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fa4a00b21e0 con 0x7fa4a00b1e00 2024-04-14T23:57:55.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:55.959+0000 7fa4a4a00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa4a00c3d90 0x7fa4a00c41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:55.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:55.959+0000 7fa497e00640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fa4a0073320 0x7fa4a00c7fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:44418/0 (socket says 172.21.15.60:44418) 2024-04-14T23:57:55.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:55.959+0000 7fa497e00640 1 -- 172.21.15.60:0/1001445499 learned_addr learned my addr 172.21.15.60:0/1001445499 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:57:55.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:55.960+0000 7fa49fe00640 1 --2- 172.21.15.60:0/1001445499 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa4a00b1e00 0x7fa4a00c6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:55.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:55.960+0000 7fa4a4a00640 1 -- 172.21.15.60:0/1001445499 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa4a00b1e00 msgr2=0x7fa4a00c6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:57:55.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:55.960+0000 7fa4a4a00640 1 --2- 172.21.15.60:0/1001445499 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa4a00b1e00 0x7fa4a00c6950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:55.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:55.960+0000 7fa49f400640 1 -- 172.21.15.60:0/1001445499 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3718715979 0 0) 0x7fa4a0152fb0 con 0x7fa4a0073320 2024-04-14T23:57:55.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:55.960+0000 7fa4a4a00640 1 -- 172.21.15.60:0/1001445499 >> v1:172.21.15.60:6789/0 conn(0x7fa4a0073320 legacy=0x7fa4a00c7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:55.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:55.960+0000 7fa4a4a00640 1 -- 172.21.15.60:0/1001445499 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4a0159310 con 0x7fa4a00c3d90 2024-04-14T23:57:55.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:55.961+0000 7fa4a4a00640 1 --2- 172.21.15.60:0/1001445499 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4a00c3d90 0x7fa4a00c41e0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fa498009d90 tx=0x7fa49802bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7977be3ce2f7f979 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:57:55.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:55.961+0000 7fa49f400640 1 -- 172.21.15.60:0/1001445499 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa498039070 con 0x7fa4a00c3d90 2024-04-14T23:57:55.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:55.962+0000 7fa49f400640 1 -- 172.21.15.60:0/1001445499 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa49802cce0 con 0x7fa4a00c3d90 2024-04-14T23:57:55.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:55.962+0000 7fa49f400640 1 -- 172.21.15.60:0/1001445499 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa498034660 con 0x7fa4a00c3d90 2024-04-14T23:57:55.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:55.962+0000 7fa4a5400640 1 -- 172.21.15.60:0/1001445499 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4a00c3d90 msgr2=0x7fa4a00c41e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:55.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:55.962+0000 7fa4a5400640 1 --2- 172.21.15.60:0/1001445499 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4a00c3d90 0x7fa4a00c41e0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fa498009d90 tx=0x7fa49802bf30 comp rx=0 tx=0).stop 2024-04-14T23:57:55.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:55.962+0000 7fa4a5400640 1 -- 172.21.15.60:0/1001445499 shutdown_connections 2024-04-14T23:57:55.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:55.962+0000 7fa4a5400640 1 --2- 172.21.15.60:0/1001445499 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4a00c3d90 0x7fa4a00c41e0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:55.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:55.962+0000 7fa4a5400640 1 --2- 172.21.15.60:0/1001445499 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa4a00b1e00 0x7fa4a00c6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:55.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:55.962+0000 7fa4a5400640 1 -- 172.21.15.60:0/1001445499 >> 172.21.15.60:0/1001445499 conn(0x7fa4a00c8b50 msgr2=0x7fa4a00c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:57:55.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:55.963+0000 7fa4a5400640 1 -- 172.21.15.60:0/1001445499 shutdown_connections 2024-04-14T23:57:55.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:55.963+0000 7fa4a5400640 1 -- 172.21.15.60:0/1001445499 wait complete. 2024-04-14T23:57:55.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:55.964+0000 7fa4a5400640 1 Processor -- start 2024-04-14T23:57:55.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:55.965+0000 7fa4a5400640 1 -- start start 2024-04-14T23:57:55.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:55.965+0000 7fa4a5400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa4a0073320 0x7fa4a015c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:55.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:55.965+0000 7fa4a5400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4a00b1e00 0x7fa4a00b0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:55.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:55.966+0000 7fa4a5400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa4a00c3d90 0x7fa4a01f37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:55.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:55.966+0000 7fa4a5400640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fa4a00c38b0 con 0x7fa4a00b1e00 2024-04-14T23:57:55.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:55.966+0000 7fa4a5400640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa4a0159890 con 0x7fa4a0073320 2024-04-14T23:57:55.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:55.966+0000 7fa4a5400640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fa4a01595b0 con 0x7fa4a00c3d90 2024-04-14T23:57:55.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:55.966+0000 7fa497e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4a00b1e00 0x7fa4a00b0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:55.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:55.966+0000 7fa497e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4a00b1e00 0x7fa4a00b0d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41622/0 (socket says 172.21.15.60:41622) 2024-04-14T23:57:55.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:55.966+0000 7fa497e00640 1 -- 172.21.15.60:0/1381741609 learned_addr learned my addr 172.21.15.60:0/1381741609 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:57:55.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:55.966+0000 7fa4a4a00640 1 --2- 172.21.15.60:0/1381741609 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa4a00c3d90 0x7fa4a01f37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:55.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:55.966+0000 7fa49fe00640 1 --2- 172.21.15.60:0/1381741609 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa4a0073320 0x7fa4a015c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:55.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:55.966+0000 7fa497e00640 1 -- 172.21.15.60:0/1381741609 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa4a00c3d90 msgr2=0x7fa4a01f37e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:55.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:55.966+0000 7fa497e00640 1 --2- 172.21.15.60:0/1381741609 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa4a00c3d90 0x7fa4a01f37e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:55.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:55.966+0000 7fa497e00640 1 -- 172.21.15.60:0/1381741609 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa4a0073320 msgr2=0x7fa4a015c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:55.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:55.966+0000 7fa497e00640 1 --2- 172.21.15.60:0/1381741609 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa4a0073320 0x7fa4a015c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:55.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:55.966+0000 7fa497e00640 1 -- 172.21.15.60:0/1381741609 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4a015ca50 con 0x7fa4a00b1e00 2024-04-14T23:57:55.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:55.966+0000 7fa49fe00640 1 --2- 172.21.15.60:0/1381741609 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa4a0073320 0x7fa4a015c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-14T23:57:55.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:55.967+0000 7fa497e00640 1 --2- 172.21.15.60:0/1381741609 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4a00b1e00 0x7fa4a00b0d10 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fa48800c460 tx=0x7fa48800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:57:55.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:55.967+0000 7fa49cc00640 1 -- 172.21.15.60:0/1381741609 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa488018070 con 0x7fa4a00b1e00 2024-04-14T23:57:55.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:55.967+0000 7fa49cc00640 1 -- 172.21.15.60:0/1381741609 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa488007500 con 0x7fa4a00b1e00 2024-04-14T23:57:55.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:55.967+0000 7fa49cc00640 1 -- 172.21.15.60:0/1381741609 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa488004a20 con 0x7fa4a00b1e00 2024-04-14T23:57:55.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:55.967+0000 7fa4a5400640 1 -- 172.21.15.60:0/1381741609 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4a01e5a30 con 0x7fa4a00b1e00 2024-04-14T23:57:55.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:55.967+0000 7fa4a5400640 1 -- 172.21.15.60:0/1381741609 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4a01e5ff0 con 0x7fa4a00b1e00 2024-04-14T23:57:55.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:55.968+0000 7fa4a5400640 1 -- 172.21.15.60:0/1381741609 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4a004f540 con 0x7fa4a00b1e00 2024-04-14T23:57:55.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:55.969+0000 7fa49cc00640 1 -- 172.21.15.60:0/1381741609 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7fa48801b070 con 0x7fa4a00b1e00 2024-04-14T23:57:55.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:55.970+0000 7fa49cc00640 1 --2- 172.21.15.60:0/1381741609 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fa47c06b990 0x7fa47c06de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:55.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:55.970+0000 7fa49cc00640 1 -- 172.21.15.60:0/1381741609 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6255+0+0 (secure 0 0 0) 0x7fa488089d70 con 0x7fa4a00b1e00 2024-04-14T23:57:55.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:55.971+0000 7fa49fe00640 1 --2- 172.21.15.60:0/1381741609 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fa47c06b990 0x7fa47c06de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:55.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:55.971+0000 7fa49fe00640 1 --2- 172.21.15.60:0/1381741609 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fa47c06b990 0x7fa47c06de50 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa490009b90 tx=0x7fa490008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:57:55.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:55.975+0000 7fa49cc00640 1 -- 172.21.15.60:0/1381741609 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa488057fe0 con 0x7fa4a00b1e00 2024-04-14T23:57:56.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:56.216+0000 7fa4a5400640 1 -- 172.21.15.60:0/1381741609 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa4a00c99e0 con 0x7fa4a00b1e00 2024-04-14T23:57:56.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:56.218+0000 7fa49cc00640 1 -- 172.21.15.60:0/1381741609 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+6007 (secure 0 0 0) 0x7fa48805c080 con 0x7fa4a00b1e00 2024-04-14T23:57:56.223 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:57:56.223 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:57:51.480229+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"a","rank":0,"incarnation":32,"state":"up:rejoin","state_seq":17,"addr":"172.21.15.60:6833/3281566884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3281566884},{"type":"v1","addr":"172.21.15.60:6833","nonce":3281566884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-14T23:57:56.224 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 34 2024-04-14T23:57:56.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:56.220+0000 7fa4a5400640 1 -- 172.21.15.60:0/1381741609 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fa47c06b990 msgr2=0x7fa47c06de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:56.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:56.220+0000 7fa4a5400640 1 --2- 172.21.15.60:0/1381741609 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fa47c06b990 0x7fa47c06de50 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa490009b90 tx=0x7fa490008040 comp rx=0 tx=0).stop 2024-04-14T23:57:56.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:56.220+0000 7fa4a5400640 1 -- 172.21.15.60:0/1381741609 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4a00b1e00 msgr2=0x7fa4a00b0d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:56.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:56.220+0000 7fa4a5400640 1 --2- 172.21.15.60:0/1381741609 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4a00b1e00 0x7fa4a00b0d10 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fa48800c460 tx=0x7fa48800c920 comp rx=0 tx=0).stop 2024-04-14T23:57:56.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:56.221+0000 7fa4a5400640 1 -- 172.21.15.60:0/1381741609 shutdown_connections 2024-04-14T23:57:56.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:56.221+0000 7fa4a5400640 1 --2- 172.21.15.60:0/1381741609 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fa47c06b990 0x7fa47c06de50 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:56.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:56.221+0000 7fa4a5400640 1 --2- 172.21.15.60:0/1381741609 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa4a00c3d90 0x7fa4a01f37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:56.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:56.221+0000 7fa4a5400640 1 --2- 172.21.15.60:0/1381741609 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa4a00b1e00 0x7fa4a00b0d10 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:56.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:56.221+0000 7fa4a5400640 1 --2- 172.21.15.60:0/1381741609 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa4a0073320 0x7fa4a015c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:56.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:56.221+0000 7fa4a5400640 1 -- 172.21.15.60:0/1381741609 >> 172.21.15.60:0/1381741609 conn(0x7fa4a00c8b50 msgr2=0x7fa4a00cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:57:56.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:56.221+0000 7fa4a5400640 1 -- 172.21.15.60:0/1381741609 shutdown_connections 2024-04-14T23:57:56.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:56.221+0000 7fa4a5400640 1 -- 172.21.15.60:0/1381741609 wait complete. 2024-04-14T23:57:56.247 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-14T23:57:58.248 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:57:58.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.461+0000 7fb669400640 1 Processor -- start 2024-04-14T23:57:58.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.462+0000 7fb669400640 1 -- start start 2024-04-14T23:57:58.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.463+0000 7fb669400640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb6640c6950 0x7fb66416a900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:58.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.463+0000 7fb669400640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb664071bf0 con 0x7fb66415c8a0 2024-04-14T23:57:58.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.463+0000 7fb669400640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb66415cc80 con 0x7fb6640c86d0 2024-04-14T23:57:58.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.463+0000 7fb669400640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fb6641632a0 con 0x7fb6640c6950 2024-04-14T23:57:58.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.464+0000 7fb65be00640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fb6640c86d0 0x7fb6641625c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.60:34816/0 (socket says 172.21.15.60:34816) 2024-04-14T23:57:58.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.464+0000 7fb65be00640 1 -- 172.21.15.60:0/3438313460 learned_addr learned my addr 172.21.15.60:0/3438313460 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:57:58.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.464+0000 7fb668a00640 1 --2- 172.21.15.60:0/3438313460 >> v2:172.21.15.60:3300/0 conn(0x7fb6640c6950 0x7fb66416a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:58.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.464+0000 7fb663400640 1 -- 172.21.15.60:0/3438313460 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1248830319 0 0) 0x7fb66415cc80 con 0x7fb6640c86d0 2024-04-14T23:57:58.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.464+0000 7fb663400640 1 -- 172.21.15.60:0/3438313460 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb648003520 con 0x7fb6640c86d0 2024-04-14T23:57:58.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.464+0000 7fb663400640 1 -- 172.21.15.60:0/3438313460 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3480359714 0 0) 0x7fb654001110 con 0x7fb66415c8a0 2024-04-14T23:57:58.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.464+0000 7fb663400640 1 -- 172.21.15.60:0/3438313460 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb66415cc80 con 0x7fb66415c8a0 2024-04-14T23:57:58.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.464+0000 7fb668a00640 1 -- 172.21.15.60:0/3438313460 >> v1:172.21.15.186:6789/0 conn(0x7fb6640c86d0 legacy=0x7fb6641625c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:58.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.465+0000 7fb668a00640 1 -- 172.21.15.60:0/3438313460 >> v1:172.21.15.60:6789/0 conn(0x7fb66415c8a0 legacy=0x7fb664072390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:58.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.465+0000 7fb668a00640 1 -- 172.21.15.60:0/3438313460 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6640c6fb0 con 0x7fb6640c6950 2024-04-14T23:57:58.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.466+0000 7fb668a00640 1 --2- 172.21.15.60:0/3438313460 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb6640c6950 0x7fb66416a900 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fb65c009d90 tx=0x7fb65c02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9c38ad24292a416 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:57:58.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.466+0000 7fb663400640 1 -- 172.21.15.60:0/3438313460 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb65c039070 con 0x7fb6640c6950 2024-04-14T23:57:58.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.466+0000 7fb663400640 1 -- 172.21.15.60:0/3438313460 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb65c02cd00 con 0x7fb6640c6950 2024-04-14T23:57:58.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.466+0000 7fb663400640 1 -- 172.21.15.60:0/3438313460 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb65c034660 con 0x7fb6640c6950 2024-04-14T23:57:58.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.466+0000 7fb669400640 1 -- 172.21.15.60:0/3438313460 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb6640c6950 msgr2=0x7fb66416a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:58.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.466+0000 7fb669400640 1 --2- 172.21.15.60:0/3438313460 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb6640c6950 0x7fb66416a900 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fb65c009d90 tx=0x7fb65c02bf50 comp rx=0 tx=0).stop 2024-04-14T23:57:58.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.467+0000 7fb669400640 1 -- 172.21.15.60:0/3438313460 shutdown_connections 2024-04-14T23:57:58.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.467+0000 7fb669400640 1 --2- 172.21.15.60:0/3438313460 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb6640c6950 0x7fb66416a900 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:58.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.467+0000 7fb669400640 1 -- 172.21.15.60:0/3438313460 >> 172.21.15.60:0/3438313460 conn(0x7fb6640c8b50 msgr2=0x7fb6640c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:57:58.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.467+0000 7fb669400640 1 -- 172.21.15.60:0/3438313460 shutdown_connections 2024-04-14T23:57:58.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.467+0000 7fb669400640 1 -- 172.21.15.60:0/3438313460 wait complete. 2024-04-14T23:57:58.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.468+0000 7fb669400640 1 Processor -- start 2024-04-14T23:57:58.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.469+0000 7fb669400640 1 -- start start 2024-04-14T23:57:58.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.470+0000 7fb669400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb6640c6950 0x7fb6641e94b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:58.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.470+0000 7fb669400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6640c86d0 0x7fb6640cae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:58.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.470+0000 7fb669400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb66415c8a0 0x7fb66406bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:58.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.470+0000 7fb669400640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb66406b1c0 con 0x7fb6640c6950 2024-04-14T23:57:58.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.470+0000 7fb669400640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb664071e70 con 0x7fb6640c86d0 2024-04-14T23:57:58.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.470+0000 7fb669400640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fb66406b340 con 0x7fb66415c8a0 2024-04-14T23:57:58.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.470+0000 7fb663e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb6640c6950 0x7fb6641e94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:58.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.470+0000 7fb663e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb6640c6950 0x7fb6641e94b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41644/0 (socket says 172.21.15.60:41644) 2024-04-14T23:57:58.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.470+0000 7fb663e00640 1 -- 172.21.15.60:0/943090964 learned_addr learned my addr 172.21.15.60:0/943090964 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:57:58.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.470+0000 7fb65be00640 1 --2- 172.21.15.60:0/943090964 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6640c86d0 0x7fb6640cae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:58.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.470+0000 7fb668a00640 1 --2- 172.21.15.60:0/943090964 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb66415c8a0 0x7fb66406bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:58.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.470+0000 7fb663e00640 1 -- 172.21.15.60:0/943090964 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb66415c8a0 msgr2=0x7fb66406bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:58.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.470+0000 7fb663e00640 1 --2- 172.21.15.60:0/943090964 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb66415c8a0 0x7fb66406bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:58.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.470+0000 7fb663e00640 1 -- 172.21.15.60:0/943090964 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6640c86d0 msgr2=0x7fb6640cae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:58.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.470+0000 7fb663e00640 1 --2- 172.21.15.60:0/943090964 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6640c86d0 0x7fb6640cae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:58.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.470+0000 7fb663e00640 1 -- 172.21.15.60:0/943090964 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6640cbdd0 con 0x7fb6640c6950 2024-04-14T23:57:58.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.471+0000 7fb663e00640 1 --2- 172.21.15.60:0/943090964 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb6640c6950 0x7fb6641e94b0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fb65400c730 tx=0x7fb65400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:57:58.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.471+0000 7fb660c00640 1 -- 172.21.15.60:0/943090964 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6540052d0 con 0x7fb6640c6950 2024-04-14T23:57:58.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.471+0000 7fb660c00640 1 -- 172.21.15.60:0/943090964 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb654005490 con 0x7fb6640c6950 2024-04-14T23:57:58.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.471+0000 7fb660c00640 1 -- 172.21.15.60:0/943090964 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb65400a880 con 0x7fb6640c6950 2024-04-14T23:57:58.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.471+0000 7fb669400640 1 -- 172.21.15.60:0/943090964 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb664159520 con 0x7fb6640c6950 2024-04-14T23:57:58.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.471+0000 7fb669400640 1 -- 172.21.15.60:0/943090964 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6640ca630 con 0x7fb6640c6950 2024-04-14T23:57:58.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.473+0000 7fb669400640 1 -- 172.21.15.60:0/943090964 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb664159a40 con 0x7fb6640c6950 2024-04-14T23:57:58.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.473+0000 7fb660c00640 1 -- 172.21.15.60:0/943090964 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7fb65400aa20 con 0x7fb6640c6950 2024-04-14T23:57:58.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.474+0000 7fb660c00640 1 --2- 172.21.15.60:0/943090964 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fb64006b990 0x7fb64006de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:58.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.474+0000 7fb660c00640 1 -- 172.21.15.60:0/943090964 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6255+0+0 (secure 0 0 0) 0x7fb65408a470 con 0x7fb6640c6950 2024-04-14T23:57:58.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.475+0000 7fb65be00640 1 --2- 172.21.15.60:0/943090964 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fb64006b990 0x7fb64006de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:58.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.476+0000 7fb65be00640 1 --2- 172.21.15.60:0/943090964 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fb64006b990 0x7fb64006de50 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb66415b6d0 tx=0x7fb64c007d30 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:57:58.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.479+0000 7fb660c00640 1 -- 172.21.15.60:0/943090964 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb664159a40 con 0x7fb6640c6950 2024-04-14T23:57:58.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.728+0000 7fb669400640 1 -- 172.21.15.60:0/943090964 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb66404f540 con 0x7fb6640c6950 2024-04-14T23:57:58.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.730+0000 7fb660c00640 1 -- 172.21.15.60:0/943090964 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+6014 (secure 0 0 0) 0x7fb664071bf0 con 0x7fb6640c6950 2024-04-14T23:57:58.735 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:57:58.735 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:57:57.529555+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":19,"addr":"172.21.15.60:6833/3281566884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3281566884},{"type":"v1","addr":"172.21.15.60:6833","nonce":3281566884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":1}]} 2024-04-14T23:57:58.735 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 35 2024-04-14T23:57:58.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.733+0000 7fb669400640 1 -- 172.21.15.60:0/943090964 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fb64006b990 msgr2=0x7fb64006de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:58.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.733+0000 7fb669400640 1 --2- 172.21.15.60:0/943090964 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fb64006b990 0x7fb64006de50 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb66415b6d0 tx=0x7fb64c007d30 comp rx=0 tx=0).stop 2024-04-14T23:57:58.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.733+0000 7fb669400640 1 -- 172.21.15.60:0/943090964 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb6640c6950 msgr2=0x7fb6641e94b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:58.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.733+0000 7fb669400640 1 --2- 172.21.15.60:0/943090964 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb6640c6950 0x7fb6641e94b0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fb65400c730 tx=0x7fb65400cbf0 comp rx=0 tx=0).stop 2024-04-14T23:57:58.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.733+0000 7fb669400640 1 -- 172.21.15.60:0/943090964 shutdown_connections 2024-04-14T23:57:58.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.733+0000 7fb669400640 1 --2- 172.21.15.60:0/943090964 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fb64006b990 0x7fb64006de50 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:58.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.733+0000 7fb669400640 1 --2- 172.21.15.60:0/943090964 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb66415c8a0 0x7fb66406bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:58.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.733+0000 7fb669400640 1 --2- 172.21.15.60:0/943090964 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6640c86d0 0x7fb6640cae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:58.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.733+0000 7fb669400640 1 --2- 172.21.15.60:0/943090964 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb6640c6950 0x7fb6641e94b0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:58.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.733+0000 7fb669400640 1 -- 172.21.15.60:0/943090964 >> 172.21.15.60:0/943090964 conn(0x7fb6640c8b50 msgr2=0x7fb66415a9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:57:58.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.733+0000 7fb669400640 1 -- 172.21.15.60:0/943090964 shutdown_connections 2024-04-14T23:57:58.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:58.733+0000 7fb669400640 1 -- 172.21.15.60:0/943090964 wait complete. 2024-04-14T23:57:58.758 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=5191 2024-04-14T23:57:58.759 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 1 secs before reviving mds.c 2024-04-14T23:57:59.760 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.c 2024-04-14T23:57:59.760 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-14T23:57:59.760 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-14T23:57:59.768 INFO:tasks.ceph.mds.c:Started 2024-04-14T23:57:59.769 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:57:59.943 INFO:tasks.ceph.mds.c.smithi060.stdout:starting mds.c at 2024-04-14T23:57:59.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.970+0000 7f89c0400640 1 Processor -- start 2024-04-14T23:57:59.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.970+0000 7f89c0400640 1 -- start start 2024-04-14T23:57:59.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.971+0000 7f89c0400640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f89b806f1f0 0x7f89b80c72a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:59.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.971+0000 7f89c0400640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f89b806f5d0 0x7f89b80c77e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:59.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.971+0000 7f89c0400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f89b80731e0 0x7f89b8154980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:59.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.971+0000 7f89c0400640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f89b80c6e10 con 0x7f89b806f1f0 2024-04-14T23:57:59.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.971+0000 7f89c0400640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f89b815b0d0 con 0x7f89b806f5d0 2024-04-14T23:57:59.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.971+0000 7f89c0400640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f89b815b370 con 0x7f89b80731e0 2024-04-14T23:57:59.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.971+0000 7f89bd000640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f89b806f5d0 0x7f89b80c77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:59.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.971+0000 7f89bda00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f89b806f1f0 0x7f89b80c72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:59.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.971+0000 7f89be400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f89b80731e0 0x7f89b8154980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:59.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.972+0000 7f89bd000640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f89b806f5d0 0x7f89b80c77e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:47926/0 (socket says 172.21.15.60:47926) 2024-04-14T23:57:59.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.972+0000 7f89bd000640 1 -- 172.21.15.60:0/3407450310 learned_addr learned my addr 172.21.15.60:0/3407450310 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:57:59.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.972+0000 7f89be400640 1 -- 172.21.15.60:0/3407450310 >> v2:172.21.15.186:3300/0 conn(0x7f89b806f5d0 msgr2=0x7f89b80c77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:57:59.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.972+0000 7f89be400640 1 --2- 172.21.15.60:0/3407450310 >> v2:172.21.15.186:3300/0 conn(0x7f89b806f5d0 0x7f89b80c77e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:59.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.972+0000 7f89be400640 1 -- 172.21.15.60:0/3407450310 >> v2:172.21.15.60:3300/0 conn(0x7f89b806f1f0 msgr2=0x7f89b80c72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:57:59.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.972+0000 7f89be400640 1 --2- 172.21.15.60:0/3407450310 >> v2:172.21.15.60:3300/0 conn(0x7f89b806f1f0 0x7f89b80c72a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:59.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.972+0000 7f89be400640 1 -- 172.21.15.60:0/3407450310 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89b815b650 con 0x7f89b80731e0 2024-04-14T23:57:59.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.972+0000 7f89bd000640 1 --2- 172.21.15.60:0/3407450310 >> v2:172.21.15.186:3300/0 conn(0x7f89b806f5d0 0x7f89b80c77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:57:59.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.972+0000 7f89bda00640 1 --2- 172.21.15.60:0/3407450310 >> v2:172.21.15.60:3300/0 conn(0x7f89b806f1f0 0x7f89b80c72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:57:59.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.973+0000 7f89be400640 1 --2- 172.21.15.60:0/3407450310 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f89b80731e0 0x7f89b8154980 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f89b80c3540 tx=0x7f89b002beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e330d6c9aa0aedec server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:57:59.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.973+0000 7f89b7e00640 1 -- 172.21.15.60:0/3407450310 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89b0039070 con 0x7f89b80731e0 2024-04-14T23:57:59.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.974+0000 7f89b7e00640 1 -- 172.21.15.60:0/3407450310 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89b002cc60 con 0x7f89b80731e0 2024-04-14T23:57:59.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.974+0000 7f89b7e00640 1 -- 172.21.15.60:0/3407450310 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89b0034660 con 0x7f89b80731e0 2024-04-14T23:57:59.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.974+0000 7f89c0400640 1 -- 172.21.15.60:0/3407450310 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f89b80731e0 msgr2=0x7f89b8154980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:59.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.974+0000 7f89c0400640 1 --2- 172.21.15.60:0/3407450310 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f89b80731e0 0x7f89b8154980 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f89b80c3540 tx=0x7f89b002beb0 comp rx=0 tx=0).stop 2024-04-14T23:57:59.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.974+0000 7f89c0400640 1 -- 172.21.15.60:0/3407450310 shutdown_connections 2024-04-14T23:57:59.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.974+0000 7f89c0400640 1 --2- 172.21.15.60:0/3407450310 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f89b80731e0 0x7f89b8154980 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:59.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.974+0000 7f89c0400640 1 --2- 172.21.15.60:0/3407450310 >> v2:172.21.15.186:3300/0 conn(0x7f89b806f5d0 0x7f89b80c77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:59.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.974+0000 7f89c0400640 1 --2- 172.21.15.60:0/3407450310 >> v2:172.21.15.60:3300/0 conn(0x7f89b806f1f0 0x7f89b80c72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:59.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.974+0000 7f89c0400640 1 -- 172.21.15.60:0/3407450310 >> 172.21.15.60:0/3407450310 conn(0x7f89b80cc170 msgr2=0x7f89b80cc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:57:59.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.974+0000 7f89c0400640 1 -- 172.21.15.60:0/3407450310 shutdown_connections 2024-04-14T23:57:59.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.974+0000 7f89c0400640 1 -- 172.21.15.60:0/3407450310 wait complete. 2024-04-14T23:57:59.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.976+0000 7f89c0400640 1 Processor -- start 2024-04-14T23:57:59.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.976+0000 7f89c0400640 1 -- start start 2024-04-14T23:57:59.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.977+0000 7f89c0400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f89b806f1f0 0x7f89b815e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:59.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.977+0000 7f89c0400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89b806f5d0 0x7f89b80c5a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:59.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.977+0000 7f89c0400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89b80731e0 0x7f89b81f0a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:59.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.977+0000 7f89c0400640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f89b80c84b0 con 0x7f89b80731e0 2024-04-14T23:57:59.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.977+0000 7f89c0400640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f89b8071180 con 0x7f89b806f5d0 2024-04-14T23:57:59.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.977+0000 7f89c0400640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f89b80c8630 con 0x7f89b806f1f0 2024-04-14T23:57:59.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.977+0000 7f89be400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89b80731e0 0x7f89b81f0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:59.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.977+0000 7f89be400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89b80731e0 0x7f89b81f0a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41678/0 (socket says 172.21.15.60:41678) 2024-04-14T23:57:59.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.977+0000 7f89bda00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f89b806f1f0 0x7f89b815e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:59.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.977+0000 7f89bd000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89b806f5d0 0x7f89b80c5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:59.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.977+0000 7f89be400640 1 -- 172.21.15.60:0/84733922 learned_addr learned my addr 172.21.15.60:0/84733922 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:57:59.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.978+0000 7f89be400640 1 -- 172.21.15.60:0/84733922 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f89b806f1f0 msgr2=0x7f89b815e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:59.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.978+0000 7f89be400640 1 --2- 172.21.15.60:0/84733922 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f89b806f1f0 0x7f89b815e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:59.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.978+0000 7f89be400640 1 -- 172.21.15.60:0/84733922 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89b806f5d0 msgr2=0x7f89b80c5a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:57:59.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.978+0000 7f89be400640 1 --2- 172.21.15.60:0/84733922 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89b806f5d0 0x7f89b80c5a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:57:59.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.978+0000 7f89be400640 1 -- 172.21.15.60:0/84733922 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89b81f39c0 con 0x7f89b80731e0 2024-04-14T23:57:59.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.978+0000 7f89bd000640 1 --2- 172.21.15.60:0/84733922 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89b806f5d0 0x7f89b80c5a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:57:59.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.978+0000 7f89be400640 1 --2- 172.21.15.60:0/84733922 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89b80731e0 0x7f89b81f0a60 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f89b0009e80 tx=0x7f89b0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:57:59.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.979+0000 7f89b5600640 1 -- 172.21.15.60:0/84733922 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89b0039050 con 0x7f89b80731e0 2024-04-14T23:57:59.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.979+0000 7f89b5600640 1 -- 172.21.15.60:0/84733922 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89b0045070 con 0x7f89b80731e0 2024-04-14T23:57:59.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.979+0000 7f89b5600640 1 -- 172.21.15.60:0/84733922 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89b0009450 con 0x7f89b80731e0 2024-04-14T23:57:59.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.979+0000 7f89c0400640 1 -- 172.21.15.60:0/84733922 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89b81f3c60 con 0x7f89b80731e0 2024-04-14T23:57:59.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.979+0000 7f89c0400640 1 -- 172.21.15.60:0/84733922 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89b81f4220 con 0x7f89b80731e0 2024-04-14T23:57:59.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.980+0000 7f89c0400640 1 -- 172.21.15.60:0/84733922 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89b8076970 con 0x7f89b80731e0 2024-04-14T23:57:59.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.981+0000 7f89b5600640 1 -- 172.21.15.60:0/84733922 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f89b003e850 con 0x7f89b80731e0 2024-04-14T23:57:59.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.982+0000 7f89b5600640 1 --2- 172.21.15.60:0/84733922 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f899806b9c0 0x7f899806de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:57:59.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.982+0000 7f89b5600640 1 -- 172.21.15.60:0/84733922 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6255+0+0 (secure 0 0 0) 0x7f89b00acf50 con 0x7f89b80731e0 2024-04-14T23:57:59.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.982+0000 7f89bda00640 1 --2- 172.21.15.60:0/84733922 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f899806b9c0 0x7f899806de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:57:59.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.983+0000 7f89bda00640 1 --2- 172.21.15.60:0/84733922 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f899806b9c0 0x7f899806de80 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f89ac009580 tx=0x7f89ac009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:57:59.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:57:59.987+0000 7f89b5600640 1 -- 172.21.15.60:0/84733922 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f89b004e020 con 0x7f89b80731e0 2024-04-14T23:58:00.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:00.232+0000 7f89c0400640 1 -- 172.21.15.60:0/84733922 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f89b80ccc20 con 0x7f89b80731e0 2024-04-14T23:58:00.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:00.233+0000 7f89b5600640 1 -- 172.21.15.60:0/84733922 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+6014 (secure 0 0 0) 0x7f89b007b250 con 0x7f89b80731e0 2024-04-14T23:58:00.238 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:58:00.239 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:57:57.529555+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":19,"addr":"172.21.15.60:6833/3281566884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3281566884},{"type":"v1","addr":"172.21.15.60:6833","nonce":3281566884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":1}]} 2024-04-14T23:58:00.239 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 35 2024-04-14T23:58:00.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:00.235+0000 7f89c0400640 1 -- 172.21.15.60:0/84733922 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f899806b9c0 msgr2=0x7f899806de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:00.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:00.235+0000 7f89c0400640 1 --2- 172.21.15.60:0/84733922 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f899806b9c0 0x7f899806de80 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f89ac009580 tx=0x7f89ac009190 comp rx=0 tx=0).stop 2024-04-14T23:58:00.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:00.235+0000 7f89c0400640 1 -- 172.21.15.60:0/84733922 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89b80731e0 msgr2=0x7f89b81f0a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:00.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:00.235+0000 7f89c0400640 1 --2- 172.21.15.60:0/84733922 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89b80731e0 0x7f89b81f0a60 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f89b0009e80 tx=0x7f89b0004a40 comp rx=0 tx=0).stop 2024-04-14T23:58:00.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:00.236+0000 7f89c0400640 1 -- 172.21.15.60:0/84733922 shutdown_connections 2024-04-14T23:58:00.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:00.236+0000 7f89c0400640 1 --2- 172.21.15.60:0/84733922 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f899806b9c0 0x7f899806de80 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:00.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:00.236+0000 7f89c0400640 1 --2- 172.21.15.60:0/84733922 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89b80731e0 0x7f89b81f0a60 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:00.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:00.236+0000 7f89c0400640 1 --2- 172.21.15.60:0/84733922 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89b806f5d0 0x7f89b80c5a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:00.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:00.236+0000 7f89c0400640 1 --2- 172.21.15.60:0/84733922 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f89b806f1f0 0x7f89b815e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:00.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:00.236+0000 7f89c0400640 1 -- 172.21.15.60:0/84733922 >> 172.21.15.60:0/84733922 conn(0x7f89b80cc170 msgr2=0x7f89b80c4310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:00.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:00.236+0000 7f89c0400640 1 -- 172.21.15.60:0/84733922 shutdown_connections 2024-04-14T23:58:00.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:00.236+0000 7f89c0400640 1 -- 172.21.15.60:0/84733922 wait complete. 2024-04-14T23:58:00.259 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2024-04-14T23:58:02.261 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:58:02.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.474+0000 7f8db3200640 1 Processor -- start 2024-04-14T23:58:02.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.474+0000 7f8db3200640 1 -- start start 2024-04-14T23:58:02.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.475+0000 7f8db3200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8db4158380 0x7f8db406f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:02.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.475+0000 7f8db3200640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8db415e8a0 0x7f8db41654c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:02.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.475+0000 7f8db3200640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8db406fdf0 con 0x7f8db40b0df0 2024-04-14T23:58:02.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.475+0000 7f8db3200640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f8db406de40 con 0x7f8db415e8a0 2024-04-14T23:58:02.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.475+0000 7f8db3200640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f8db406c360 con 0x7f8db4158380 2024-04-14T23:58:02.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.476+0000 7f8db2800640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8db415e8a0 0x7f8db41654c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:02.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.476+0000 7f8db1e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8db4158380 0x7f8db406f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:02.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.476+0000 7f8db2800640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8db415e8a0 0x7f8db41654c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39640/0 (socket says 172.21.15.60:39640) 2024-04-14T23:58:02.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.476+0000 7f8db2800640 1 -- 172.21.15.60:0/1826429318 learned_addr learned my addr 172.21.15.60:0/1826429318 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:58:02.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.476+0000 7f8db1e00640 1 -- 172.21.15.60:0/1826429318 >> v2:172.21.15.60:3300/0 conn(0x7f8db415e8a0 msgr2=0x7f8db41654c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:58:02.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.476+0000 7f8db1e00640 1 --2- 172.21.15.60:0/1826429318 >> v2:172.21.15.60:3300/0 conn(0x7f8db415e8a0 0x7f8db41654c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:02.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.476+0000 7f8db1e00640 1 -- 172.21.15.60:0/1826429318 >> v1:172.21.15.60:6789/0 conn(0x7f8db40b0df0 legacy=0x7f8db415e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:02.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.476+0000 7f8db1e00640 1 -- 172.21.15.60:0/1826429318 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8db40c93c0 con 0x7f8db4158380 2024-04-14T23:58:02.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.477+0000 7f8db1e00640 1 --2- 172.21.15.60:0/1826429318 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8db4158380 0x7f8db406f8b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f8da40097a0 tx=0x7f8da402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb01e5f0073a1afb server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:02.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.478+0000 7f8db1400640 1 -- 172.21.15.60:0/1826429318 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da4035020 con 0x7f8db4158380 2024-04-14T23:58:02.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.478+0000 7f8db1400640 1 -- 172.21.15.60:0/1826429318 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8da4031810 con 0x7f8db4158380 2024-04-14T23:58:02.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.478+0000 7f8db1400640 1 -- 172.21.15.60:0/1826429318 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da4031be0 con 0x7f8db4158380 2024-04-14T23:58:02.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.478+0000 7f8db3200640 1 -- 172.21.15.60:0/1826429318 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8db4158380 msgr2=0x7f8db406f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:02.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.478+0000 7f8db3200640 1 --2- 172.21.15.60:0/1826429318 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8db4158380 0x7f8db406f8b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f8da40097a0 tx=0x7f8da402b8b0 comp rx=0 tx=0).stop 2024-04-14T23:58:02.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.478+0000 7f8db3200640 1 -- 172.21.15.60:0/1826429318 shutdown_connections 2024-04-14T23:58:02.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.478+0000 7f8db3200640 1 --2- 172.21.15.60:0/1826429318 >> v2:172.21.15.60:3300/0 conn(0x7f8db415e8a0 0x7f8db41654c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:02.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.478+0000 7f8db3200640 1 --2- 172.21.15.60:0/1826429318 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8db4158380 0x7f8db406f8b0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:02.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.478+0000 7f8db3200640 1 -- 172.21.15.60:0/1826429318 >> 172.21.15.60:0/1826429318 conn(0x7f8db4066f10 msgr2=0x7f8db4067320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:02.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.479+0000 7f8db3200640 1 -- 172.21.15.60:0/1826429318 shutdown_connections 2024-04-14T23:58:02.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.479+0000 7f8db3200640 1 -- 172.21.15.60:0/1826429318 wait complete. 2024-04-14T23:58:02.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.480+0000 7f8db3200640 1 Processor -- start 2024-04-14T23:58:02.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.480+0000 7f8db3200640 1 -- start start 2024-04-14T23:58:02.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.481+0000 7f8db3200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8db40b0df0 0x7f8db41590f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:02.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.481+0000 7f8db3200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8db4158380 0x7f8db41f4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:02.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.481+0000 7f8db3200640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8db415e8a0 0x7f8db40b2630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:02.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.481+0000 7f8db3200640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f8db4071380 con 0x7f8db415e8a0 2024-04-14T23:58:02.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.481+0000 7f8db3200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8db406e070 con 0x7f8db40b0df0 2024-04-14T23:58:02.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.481+0000 7f8db3200640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f8db4071660 con 0x7f8db4158380 2024-04-14T23:58:02.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.482+0000 7f8db2800640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8db415e8a0 0x7f8db40b2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:02.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.482+0000 7f8db2800640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8db415e8a0 0x7f8db40b2630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39644/0 (socket says 172.21.15.60:39644) 2024-04-14T23:58:02.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.482+0000 7f8db2800640 1 -- 172.21.15.60:0/4256712581 learned_addr learned my addr 172.21.15.60:0/4256712581 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:58:02.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.482+0000 7f8db1e00640 1 --2- 172.21.15.60:0/4256712581 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8db40b0df0 0x7f8db41590f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:02.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.482+0000 7f8da9400640 1 --2- 172.21.15.60:0/4256712581 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8db4158380 0x7f8db41f4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:02.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.482+0000 7f8db2800640 1 -- 172.21.15.60:0/4256712581 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8db4158380 msgr2=0x7f8db41f4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:02.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.482+0000 7f8db2800640 1 --2- 172.21.15.60:0/4256712581 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8db4158380 0x7f8db41f4760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:02.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.482+0000 7f8db2800640 1 -- 172.21.15.60:0/4256712581 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8db40b0df0 msgr2=0x7f8db41590f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:02.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.482+0000 7f8db2800640 1 --2- 172.21.15.60:0/4256712581 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8db40b0df0 0x7f8db41590f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:02.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.482+0000 7f8db2800640 1 -- 172.21.15.60:0/4256712581 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8db41f3cf0 con 0x7f8db415e8a0 2024-04-14T23:58:02.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.482+0000 7f8da9400640 1 --2- 172.21.15.60:0/4256712581 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8db4158380 0x7f8db41f4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-14T23:58:02.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.482+0000 7f8db2800640 1 --2- 172.21.15.60:0/4256712581 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8db415e8a0 0x7f8db40b2630 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f8dac00e840 tx=0x7f8dac00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:02.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.483+0000 7f8daaa00640 1 -- 172.21.15.60:0/4256712581 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dac017070 con 0x7f8db415e8a0 2024-04-14T23:58:02.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.483+0000 7f8daaa00640 1 -- 172.21.15.60:0/4256712581 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8dac00c730 con 0x7f8db415e8a0 2024-04-14T23:58:02.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.483+0000 7f8db3200640 1 -- 172.21.15.60:0/4256712581 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8db40b2e70 con 0x7f8db415e8a0 2024-04-14T23:58:02.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.483+0000 7f8daaa00640 1 -- 172.21.15.60:0/4256712581 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dac00cb00 con 0x7f8db415e8a0 2024-04-14T23:58:02.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.483+0000 7f8db3200640 1 -- 172.21.15.60:0/4256712581 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8db41f3f60 con 0x7f8db415e8a0 2024-04-14T23:58:02.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.484+0000 7f8db3200640 1 -- 172.21.15.60:0/4256712581 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8db4157ee0 con 0x7f8db415e8a0 2024-04-14T23:58:02.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.485+0000 7f8daaa00640 1 -- 172.21.15.60:0/4256712581 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f8dac01a070 con 0x7f8db415e8a0 2024-04-14T23:58:02.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.486+0000 7f8daaa00640 1 --2- 172.21.15.60:0/4256712581 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f8d9006b9c0 0x7f8d9006de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:02.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.486+0000 7f8daaa00640 1 -- 172.21.15.60:0/4256712581 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6255+0+0 (secure 0 0 0) 0x7f8dac08a790 con 0x7f8db415e8a0 2024-04-14T23:58:02.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.487+0000 7f8db1e00640 1 --2- 172.21.15.60:0/4256712581 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f8d9006b9c0 0x7f8d9006de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:02.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.487+0000 7f8db1e00640 1 --2- 172.21.15.60:0/4256712581 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f8d9006b9c0 0x7f8d9006de80 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f8da4036000 tx=0x7f8da4008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:02.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.491+0000 7f8daaa00640 1 -- 172.21.15.60:0/4256712581 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8db4157ee0 con 0x7f8db415e8a0 2024-04-14T23:58:02.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.752+0000 7f8db3200640 1 -- 172.21.15.60:0/4256712581 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8db404f540 con 0x7f8db415e8a0 2024-04-14T23:58:02.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.753+0000 7f8daaa00640 1 -- 172.21.15.60:0/4256712581 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+6839 (secure 0 0 0) 0x7f8dac088e20 con 0x7f8db415e8a0 2024-04-14T23:58:02.758 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:58:02.758 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5409,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/1520534122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1520534122},{"type":"v1","addr":"172.21.15.60:6835","nonce":1520534122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:57:57.529555+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":19,"addr":"172.21.15.60:6833/3281566884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3281566884},{"type":"v1","addr":"172.21.15.60:6833","nonce":3281566884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":1}]} 2024-04-14T23:58:02.758 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 36 2024-04-14T23:58:02.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.755+0000 7f8db3200640 1 -- 172.21.15.60:0/4256712581 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f8d9006b9c0 msgr2=0x7f8d9006de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:02.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.755+0000 7f8db3200640 1 --2- 172.21.15.60:0/4256712581 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f8d9006b9c0 0x7f8d9006de80 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f8da4036000 tx=0x7f8da4008030 comp rx=0 tx=0).stop 2024-04-14T23:58:02.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.755+0000 7f8db3200640 1 -- 172.21.15.60:0/4256712581 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8db415e8a0 msgr2=0x7f8db40b2630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:02.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.755+0000 7f8db3200640 1 --2- 172.21.15.60:0/4256712581 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8db415e8a0 0x7f8db40b2630 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f8dac00e840 tx=0x7f8dac00ed00 comp rx=0 tx=0).stop 2024-04-14T23:58:02.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.755+0000 7f8db3200640 1 -- 172.21.15.60:0/4256712581 shutdown_connections 2024-04-14T23:58:02.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.755+0000 7f8db3200640 1 --2- 172.21.15.60:0/4256712581 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f8d9006b9c0 0x7f8d9006de80 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:02.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.755+0000 7f8db3200640 1 --2- 172.21.15.60:0/4256712581 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8db415e8a0 0x7f8db40b2630 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:02.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.755+0000 7f8db3200640 1 --2- 172.21.15.60:0/4256712581 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8db4158380 0x7f8db41f4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:02.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.755+0000 7f8db3200640 1 --2- 172.21.15.60:0/4256712581 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8db40b0df0 0x7f8db41590f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:02.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.755+0000 7f8db3200640 1 -- 172.21.15.60:0/4256712581 >> 172.21.15.60:0/4256712581 conn(0x7f8db4066f10 msgr2=0x7f8db4157270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:02.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.755+0000 7f8db3200640 1 -- 172.21.15.60:0/4256712581 shutdown_connections 2024-04-14T23:58:02.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:02.755+0000 7f8db3200640 1 -- 172.21.15.60:0/4256712581 wait complete. 2024-04-14T23:58:02.778 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported in up:standby state 2024-04-14T23:58:02.779 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 44 secs before thrashing 2024-04-14T23:58:03.071 INFO:tasks.workunit.client.0.smithi060.stderr:error: RPC failed; curl 56 OpenSSL SSL_read: error:0A000126:SSL routines::unexpected eof while reading, errno 0 2024-04-14T23:58:03.071 INFO:tasks.workunit.client.0.smithi060.stderr:fetch-pack: unexpected disconnect while reading sideband packet 2024-04-14T23:58:03.073 INFO:tasks.workunit.client.0.smithi060.stderr:fatal: early EOF 2024-04-14T23:58:03.079 INFO:tasks.workunit.client.0.smithi060.stderr:fatal: fetch-pack: invalid index-pack output 2024-04-14T23:58:03.926 DEBUG:teuthology.orchestra.run:got remote process result: 128 2024-04-14T23:58:03.926 INFO:tasks.workunit:Stopping ['fs/snaps'] on client.0... 2024-04-14T23:58:03.926 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-14T23:58:05.070 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_a9a752df26c63acad72e1b3569fd79a515ca0765/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph_a9a752df26c63acad72e1b3569fd79a515ca0765/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_a9a752df26c63acad72e1b3569fd79a515ca0765/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi060 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a9a752df26c63acad72e1b3569fd79a515ca0765 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh' 2024-04-14T23:58:05.307 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=85164b7883984711a8e74c5d09decf90 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_a9a752df26c63acad72e1b3569fd79a515ca0765/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph_a9a752df26c63acad72e1b3569fd79a515ca0765/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_a9a752df26c63acad72e1b3569fd79a515ca0765/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi060 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a9a752df26c63acad72e1b3569fd79a515ca0765 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh' 2024-04-14T23:58:05.310 DEBUG:teuthology.run_tasks:Unwinding manager mds_thrash 2024-04-14T23:58:05.391 INFO:tasks.mds_thrash:joining mds_thrasher 2024-04-14T23:58:05.393 INFO:tasks.mds_thrash.fs.[cephfs]:stat['max_mds'] = 0 2024-04-14T23:58:05.393 INFO:tasks.mds_thrash.fs.[cephfs]:stat['deactivate'] = 0 2024-04-14T23:58:05.393 INFO:tasks.mds_thrash.fs.[cephfs]:stat['kill'] = 5 2024-04-14T23:58:05.394 INFO:tasks.mds_thrash:done joining 2024-04-14T23:58:05.394 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-14T23:58:05.435 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-14T23:58:05.436 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-04-14T23:58:05.436 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-14T23:58:05.496 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-04-14T23:58:05.496 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-14T23:58:05.550 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-14T23:58:05.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.763+0000 7f6910400640 1 Processor -- start 2024-04-14T23:58:05.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.763+0000 7f6910400640 1 -- start start 2024-04-14T23:58:05.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.765+0000 7f6910400640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f69080c3910 0x7f69080c52b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:05.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.765+0000 7f6910400640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f690815ae00 0x7f690815b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:05.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.765+0000 7f6910400640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69080ccf80 con 0x7f6908151ae0 2024-04-14T23:58:05.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.765+0000 7f6910400640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f690806e920 con 0x7f69080c3910 2024-04-14T23:58:05.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.765+0000 7f6910400640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f69080c3da0 con 0x7f690815ae00 2024-04-14T23:58:05.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.765+0000 7f690d000640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f69080c3910 0x7f69080c52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:05.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.765+0000 7f690d000640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f69080c3910 0x7f69080c52b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39652/0 (socket says 172.21.15.60:39652) 2024-04-14T23:58:05.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.765+0000 7f690d000640 1 -- 172.21.15.60:0/657868482 learned_addr learned my addr 172.21.15.60:0/657868482 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:58:05.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.765+0000 7f690e400640 1 --2- 172.21.15.60:0/657868482 >> v2:172.21.15.186:3300/0 conn(0x7f690815ae00 0x7f690815b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:05.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.766+0000 7f690d000640 1 -- 172.21.15.60:0/657868482 >> v2:172.21.15.186:3300/0 conn(0x7f690815ae00 msgr2=0x7f690815b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:58:05.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.766+0000 7f690d000640 1 --2- 172.21.15.60:0/657868482 >> v2:172.21.15.186:3300/0 conn(0x7f690815ae00 0x7f690815b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:05.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.766+0000 7f690d000640 1 -- 172.21.15.60:0/657868482 >> v1:172.21.15.60:6789/0 conn(0x7f6908151ae0 legacy=0x7f690806e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:05.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.766+0000 7f690d000640 1 -- 172.21.15.60:0/657868482 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6908072dc0 con 0x7f69080c3910 2024-04-14T23:58:05.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.766+0000 7f6907e00640 1 -- 172.21.15.60:0/657868482 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 72999709 0 0) 0x7f69080ccf80 con 0x7f6908151ae0 2024-04-14T23:58:05.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.767+0000 7f690d000640 1 --2- 172.21.15.60:0/657868482 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69080c3910 0x7f69080c52b0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f69080739b0 tx=0x7f68f402ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c628ca73a766fba3 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:05.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.767+0000 7f6907e00640 1 -- 172.21.15.60:0/657868482 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68f402f850 con 0x7f69080c3910 2024-04-14T23:58:05.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.767+0000 7f6907e00640 1 -- 172.21.15.60:0/657868482 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68f402f9f0 con 0x7f69080c3910 2024-04-14T23:58:05.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.767+0000 7f6907e00640 1 -- 172.21.15.60:0/657868482 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68f402fdc0 con 0x7f69080c3910 2024-04-14T23:58:05.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.768+0000 7f6910400640 1 -- 172.21.15.60:0/657868482 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69080c3910 msgr2=0x7f69080c52b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:05.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.768+0000 7f6910400640 1 --2- 172.21.15.60:0/657868482 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69080c3910 0x7f69080c52b0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f69080739b0 tx=0x7f68f402ba30 comp rx=0 tx=0).stop 2024-04-14T23:58:05.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.768+0000 7f6910400640 1 -- 172.21.15.60:0/657868482 shutdown_connections 2024-04-14T23:58:05.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.768+0000 7f6910400640 1 --2- 172.21.15.60:0/657868482 >> v2:172.21.15.186:3300/0 conn(0x7f690815ae00 0x7f690815b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:05.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.768+0000 7f6910400640 1 --2- 172.21.15.60:0/657868482 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69080c3910 0x7f69080c52b0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:05.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.768+0000 7f6910400640 1 -- 172.21.15.60:0/657868482 >> 172.21.15.60:0/657868482 conn(0x7f69080cc170 msgr2=0x7f69080cc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:05.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.768+0000 7f6910400640 1 -- 172.21.15.60:0/657868482 shutdown_connections 2024-04-14T23:58:05.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.768+0000 7f6910400640 1 -- 172.21.15.60:0/657868482 wait complete. 2024-04-14T23:58:05.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.770+0000 7f6910400640 1 Processor -- start 2024-04-14T23:58:05.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.770+0000 7f6910400640 1 -- start start 2024-04-14T23:58:05.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.772+0000 7f6910400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69080c3910 0x7f6908150180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:05.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.772+0000 7f6910400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6908151ae0 0x7f690814d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:05.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.772+0000 7f6910400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f690815ae00 0x7f69081f05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:05.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.772+0000 7f6910400640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f69080726c0 con 0x7f69080c3910 2024-04-14T23:58:05.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.772+0000 7f6910400640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f69080723e0 con 0x7f6908151ae0 2024-04-14T23:58:05.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.772+0000 7f6910400640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f69080729a0 con 0x7f690815ae00 2024-04-14T23:58:05.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.772+0000 7f690da00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69080c3910 0x7f6908150180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:05.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.772+0000 7f690da00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69080c3910 0x7f6908150180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39660/0 (socket says 172.21.15.60:39660) 2024-04-14T23:58:05.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.772+0000 7f690da00640 1 -- 172.21.15.60:0/3179775243 learned_addr learned my addr 172.21.15.60:0/3179775243 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:58:05.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.772+0000 7f690e400640 1 --2- 172.21.15.60:0/3179775243 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f690815ae00 0x7f69081f05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:05.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.772+0000 7f690d000640 1 --2- 172.21.15.60:0/3179775243 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6908151ae0 0x7f690814d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:05.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.772+0000 7f690da00640 1 -- 172.21.15.60:0/3179775243 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f690815ae00 msgr2=0x7f69081f05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:05.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.772+0000 7f690da00640 1 --2- 172.21.15.60:0/3179775243 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f690815ae00 0x7f69081f05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:05.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.772+0000 7f690da00640 1 -- 172.21.15.60:0/3179775243 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6908151ae0 msgr2=0x7f690814d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:05.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.772+0000 7f690da00640 1 --2- 172.21.15.60:0/3179775243 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6908151ae0 0x7f690814d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:05.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.772+0000 7f690da00640 1 -- 172.21.15.60:0/3179775243 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6908150810 con 0x7f69080c3910 2024-04-14T23:58:05.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.772+0000 7f690e400640 1 --2- 172.21.15.60:0/3179775243 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f690815ae00 0x7f69081f05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-14T23:58:05.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.772+0000 7f690d000640 1 --2- 172.21.15.60:0/3179775243 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6908151ae0 0x7f690814d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-14T23:58:05.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.773+0000 7f690da00640 1 --2- 172.21.15.60:0/3179775243 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69080c3910 0x7f6908150180 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f68fc00c4b0 tx=0x7f68fc00c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:05.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.773+0000 7f6905600640 1 -- 172.21.15.60:0/3179775243 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68fc017070 con 0x7f69080c3910 2024-04-14T23:58:05.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.773+0000 7f6905600640 1 -- 172.21.15.60:0/3179775243 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68fc005480 con 0x7f69080c3910 2024-04-14T23:58:05.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.773+0000 7f6905600640 1 -- 172.21.15.60:0/3179775243 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68fc003830 con 0x7f69080c3910 2024-04-14T23:58:05.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.773+0000 7f6910400640 1 -- 172.21.15.60:0/3179775243 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6908150ad0 con 0x7f69080c3910 2024-04-14T23:58:05.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.773+0000 7f6910400640 1 -- 172.21.15.60:0/3179775243 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6908157ae0 con 0x7f69080c3910 2024-04-14T23:58:05.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.774+0000 7f6910400640 1 -- 172.21.15.60:0/3179775243 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6908157620 con 0x7f69080c3910 2024-04-14T23:58:05.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.776+0000 7f6905600640 1 -- 172.21.15.60:0/3179775243 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f68fc01a070 con 0x7f69080c3910 2024-04-14T23:58:05.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.777+0000 7f6905600640 1 --2- 172.21.15.60:0/3179775243 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f68e806b990 0x7f68e806de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:05.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.777+0000 7f6905600640 1 -- 172.21.15.60:0/3179775243 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6255+0+0 (secure 0 0 0) 0x7f68fc089e80 con 0x7f69080c3910 2024-04-14T23:58:05.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.777+0000 7f690d000640 1 --2- 172.21.15.60:0/3179775243 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f68e806b990 0x7f68e806de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:05.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.778+0000 7f690d000640 1 --2- 172.21.15.60:0/3179775243 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f68e806b990 0x7f68e806de50 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f68f4036000 tx=0x7f68f4008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:05.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.782+0000 7f6905600640 1 -- 172.21.15.60:0/3179775243 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6908157620 con 0x7f69080c3910 2024-04-14T23:58:05.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.985+0000 7f6910400640 1 -- 172.21.15.60:0/3179775243 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6908076970 con 0x7f69080c3910 2024-04-14T23:58:05.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.988+0000 7f6905600640 1 -- 172.21.15.60:0/3179775243 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) ==== 74+0+17524 (secure 0 0 0) 0x7f68fc0587d0 con 0x7f69080c3910 2024-04-14T23:58:05.994 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:58:05.994 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":45,"fsid":"5847db56-a76d-4486-b563-d83f1186af92","created":"2024-04-14T23:48:23.900003+0000","modified":"2024-04-14T23:57:39.001988+0000","last_up_change":"2024-04-14T23:48:34.203573+0000","last_in_change":"2024-04-14T23:48:26.345499+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-14T23:48:36.247372+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-14T23:48:40.274195+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-14T23:48:41.722183+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":505,"snap_epoch":41,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b2b2b755-7f42-4f9b-af9a-a178f7cc5a52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6817","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6818","nonce":2536528895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6822","nonce":2536528895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6827","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6828","nonce":2536528895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6825","nonce":2536528895}]},"public_addr":"172.21.15.60:6818/2536528895","cluster_addr":"172.21.15.60:6822/2536528895","heartbeat_back_addr":"172.21.15.60:6828/2536528895","heartbeat_front_addr":"172.21.15.60:6825/2536528895","state":["exists","up"]},{"osd":1,"uuid":"321418cd-5ab7-41a5-b9c8-b3fc12f68ebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6801","nonce":3115695676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6803","nonce":3115695676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6807","nonce":3115695676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6805","nonce":3115695676}]},"public_addr":"172.21.15.60:6801/3115695676","cluster_addr":"172.21.15.60:6803/3115695676","heartbeat_back_addr":"172.21.15.60:6807/3115695676","heartbeat_front_addr":"172.21.15.60:6805/3115695676","state":["exists","up"]},{"osd":2,"uuid":"07874e9f-3a38-41dc-be9f-2b75e84e53d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6819","nonce":3475168698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6824","nonce":3475168698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6831","nonce":3475168698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6829","nonce":3475168698}]},"public_addr":"172.21.15.60:6819/3475168698","cluster_addr":"172.21.15.60:6824/3475168698","heartbeat_back_addr":"172.21.15.60:6831/3475168698","heartbeat_front_addr":"172.21.15.60:6829/3475168698","state":["exists","up"]},{"osd":3,"uuid":"7e4ef6c9-b17a-47c6-a320-f5d16a6fe05e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6809","nonce":1730813674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6811","nonce":1730813674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6815","nonce":1730813674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6813","nonce":1730813674}]},"public_addr":"172.21.15.60:6809/1730813674","cluster_addr":"172.21.15.60:6811/1730813674","heartbeat_back_addr":"172.21.15.60:6815/1730813674","heartbeat_front_addr":"172.21.15.60:6813/1730813674","state":["exists","up"]},{"osd":4,"uuid":"759a3efc-dade-44c5-b57a-1623499aba50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6805","nonce":1464587038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6810","nonce":1464587038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6819","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6822","nonce":1464587038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6813","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6816","nonce":1464587038}]},"public_addr":"172.21.15.186:6805/1464587038","cluster_addr":"172.21.15.186:6810/1464587038","heartbeat_back_addr":"172.21.15.186:6822/1464587038","heartbeat_front_addr":"172.21.15.186:6816/1464587038","state":["exists","up"]},{"osd":5,"uuid":"24cd58c1-0cf1-4077-ab33-84c39c6907ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6801","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6803","nonce":1697098067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6809","nonce":1697098067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6820","nonce":1697098067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6815","nonce":1697098067}]},"public_addr":"172.21.15.186:6803/1697098067","cluster_addr":"172.21.15.186:6809/1697098067","heartbeat_back_addr":"172.21.15.186:6820/1697098067","heartbeat_front_addr":"172.21.15.186:6815/1697098067","state":["exists","up"]},{"osd":6,"uuid":"6add38f1-65e9-4b0d-927a-4ab77f5e2323","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6804","nonce":1837108559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6807","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6811","nonce":1837108559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6823","nonce":1837108559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6817","nonce":1837108559}]},"public_addr":"172.21.15.186:6804/1837108559","cluster_addr":"172.21.15.186:6811/1837108559","heartbeat_back_addr":"172.21.15.186:6823/1837108559","heartbeat_front_addr":"172.21.15.186:6817/1837108559","state":["exists","up"]},{"osd":7,"uuid":"a2326b6f-7601-4dda-a456-67c443ea5929","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6825","nonce":842537875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6827","nonce":842537875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6831","nonce":842537875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6829","nonce":842537875}]},"public_addr":"172.21.15.186:6825/842537875","cluster_addr":"172.21.15.186:6827/842537875","heartbeat_back_addr":"172.21.15.186:6831/842537875","heartbeat_front_addr":"172.21.15.186:6829/842537875","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.975583+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.976806+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.722725+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.763705+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.502140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.528972+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.517961+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.513350+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.23","mappings":[{"from":3,"to":2}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"3.3b","mappings":[{"from":7,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.60:6835/2356847578":"2024-04-15T23:57:39.001841+0000","172.21.15.60:6834/2356847578":"2024-04-15T23:57:39.001841+0000","172.21.15.60:6833/1971580590":"2024-04-15T23:56:28.972769+0000","172.21.15.60:6835/1714133698":"2024-04-15T23:54:48.942868+0000","172.21.15.60:6832/1971580590":"2024-04-15T23:56:28.972769+0000","172.21.15.60:6834/1714133698":"2024-04-15T23:54:48.942868+0000","172.21.15.60:6833/78303888":"2024-04-15T23:52:28.905933+0000","172.21.15.60:6832/78303888":"2024-04-15T23:52:28.905933+0000","172.21.15.60:6835/2677133122":"2024-04-15T23:50:58.881136+0000","172.21.15.60:6834/2677133122":"2024-04-15T23:50:58.881136+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-14T23:58:05.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.991+0000 7f6910400640 1 -- 172.21.15.60:0/3179775243 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f68e806b990 msgr2=0x7f68e806de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:05.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.991+0000 7f6910400640 1 --2- 172.21.15.60:0/3179775243 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f68e806b990 0x7f68e806de50 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f68f4036000 tx=0x7f68f4008030 comp rx=0 tx=0).stop 2024-04-14T23:58:05.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.991+0000 7f6910400640 1 -- 172.21.15.60:0/3179775243 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69080c3910 msgr2=0x7f6908150180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:05.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.991+0000 7f6910400640 1 --2- 172.21.15.60:0/3179775243 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69080c3910 0x7f6908150180 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f68fc00c4b0 tx=0x7f68fc00c970 comp rx=0 tx=0).stop 2024-04-14T23:58:05.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.991+0000 7f6910400640 1 -- 172.21.15.60:0/3179775243 shutdown_connections 2024-04-14T23:58:05.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.991+0000 7f6910400640 1 --2- 172.21.15.60:0/3179775243 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f68e806b990 0x7f68e806de50 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:05.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.991+0000 7f6910400640 1 --2- 172.21.15.60:0/3179775243 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f690815ae00 0x7f69081f05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:05.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.991+0000 7f6910400640 1 --2- 172.21.15.60:0/3179775243 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6908151ae0 0x7f690814d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:05.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.991+0000 7f6910400640 1 --2- 172.21.15.60:0/3179775243 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69080c3910 0x7f6908150180 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:05.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.991+0000 7f6910400640 1 -- 172.21.15.60:0/3179775243 >> 172.21.15.60:0/3179775243 conn(0x7f69080cc170 msgr2=0x7f690815b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:05.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.991+0000 7f6910400640 1 -- 172.21.15.60:0/3179775243 shutdown_connections 2024-04-14T23:58:05.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:05.991+0000 7f6910400640 1 -- 172.21.15.60:0/3179775243 wait complete. 2024-04-14T23:58:06.018 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-14T23:48:36.247372+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-14T23:48:40.274195+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-14T23:48:41.722183+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '41', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 505, 'snap_epoch': 41, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-14T23:58:06.019 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-14T23:58:06.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.224+0000 7fb32b200640 1 Processor -- start 2024-04-14T23:58:06.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.224+0000 7fb32b200640 1 -- start start 2024-04-14T23:58:06.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.225+0000 7fb32b200640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb32c158610 0x7fb32c0b1740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:06.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.225+0000 7fb32b200640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb32c0c6800 0x7fb32c0c6c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:06.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.225+0000 7fb32b200640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb32c0b1fd0 con 0x7fb32c0c3400 2024-04-14T23:58:06.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.225+0000 7fb32b200640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fb32c0c71d0 con 0x7fb32c0c6800 2024-04-14T23:58:06.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.225+0000 7fb32b200640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fb32c15b070 con 0x7fb32c158610 2024-04-14T23:58:06.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.226+0000 7fb321400640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fb32c0c3400 0x7fb32c06eb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:55588/0 (socket says 172.21.15.60:55588) 2024-04-14T23:58:06.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.226+0000 7fb321400640 1 -- 172.21.15.60:0/41986113 learned_addr learned my addr 172.21.15.60:0/41986113 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:58:06.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.226+0000 7fb32a800640 1 --2- 172.21.15.60:0/41986113 >> v2:172.21.15.60:3300/0 conn(0x7fb32c0c6800 0x7fb32c0c6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:06.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.226+0000 7fb329e00640 1 --2- 172.21.15.60:0/41986113 >> v2:172.21.15.186:3300/0 conn(0x7fb32c158610 0x7fb32c0b1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:06.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.226+0000 7fb329400640 1 -- 172.21.15.60:0/41986113 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3788503693 0 0) 0x7fb32c0b1fd0 con 0x7fb32c0c3400 2024-04-14T23:58:06.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.226+0000 7fb329400640 1 -- 172.21.15.60:0/41986113 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb310003520 con 0x7fb32c0c3400 2024-04-14T23:58:06.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.226+0000 7fb32a800640 1 -- 172.21.15.60:0/41986113 >> v2:172.21.15.186:3300/0 conn(0x7fb32c158610 msgr2=0x7fb32c0b1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:58:06.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.226+0000 7fb32a800640 1 --2- 172.21.15.60:0/41986113 >> v2:172.21.15.186:3300/0 conn(0x7fb32c158610 0x7fb32c0b1740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:06.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.226+0000 7fb32a800640 1 -- 172.21.15.60:0/41986113 >> v1:172.21.15.60:6789/0 conn(0x7fb32c0c3400 legacy=0x7fb32c06eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:06.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.226+0000 7fb32a800640 1 -- 172.21.15.60:0/41986113 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb32c15b350 con 0x7fb32c0c6800 2024-04-14T23:58:06.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.226+0000 7fb329e00640 1 --2- 172.21.15.60:0/41986113 >> v2:172.21.15.186:3300/0 conn(0x7fb32c158610 0x7fb32c0b1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:58:06.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.227+0000 7fb32a800640 1 --2- 172.21.15.60:0/41986113 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb32c0c6800 0x7fb32c0c6c90 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fb32c0ccfc0 tx=0x7fb32402bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ce34054e39ddfeb server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:06.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.228+0000 7fb329400640 1 -- 172.21.15.60:0/41986113 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb324039070 con 0x7fb32c0c6800 2024-04-14T23:58:06.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.228+0000 7fb329400640 1 -- 172.21.15.60:0/41986113 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb32402ccf0 con 0x7fb32c0c6800 2024-04-14T23:58:06.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.228+0000 7fb329400640 1 -- 172.21.15.60:0/41986113 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb324034660 con 0x7fb32c0c6800 2024-04-14T23:58:06.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.228+0000 7fb32b200640 1 -- 172.21.15.60:0/41986113 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb32c0c6800 msgr2=0x7fb32c0c6c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:06.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.228+0000 7fb32b200640 1 --2- 172.21.15.60:0/41986113 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb32c0c6800 0x7fb32c0c6c90 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fb32c0ccfc0 tx=0x7fb32402bf40 comp rx=0 tx=0).stop 2024-04-14T23:58:06.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.228+0000 7fb32b200640 1 -- 172.21.15.60:0/41986113 shutdown_connections 2024-04-14T23:58:06.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.228+0000 7fb32b200640 1 --2- 172.21.15.60:0/41986113 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb32c0c6800 0x7fb32c0c6c90 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:06.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.228+0000 7fb32b200640 1 --2- 172.21.15.60:0/41986113 >> v2:172.21.15.186:3300/0 conn(0x7fb32c158610 0x7fb32c0b1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:06.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.228+0000 7fb32b200640 1 -- 172.21.15.60:0/41986113 >> 172.21.15.60:0/41986113 conn(0x7fb32c0cc2f0 msgr2=0x7fb32c0cc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:06.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.229+0000 7fb32b200640 1 -- 172.21.15.60:0/41986113 shutdown_connections 2024-04-14T23:58:06.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.229+0000 7fb32b200640 1 -- 172.21.15.60:0/41986113 wait complete. 2024-04-14T23:58:06.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.230+0000 7fb32b200640 1 Processor -- start 2024-04-14T23:58:06.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.230+0000 7fb32b200640 1 -- start start 2024-04-14T23:58:06.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.231+0000 7fb32b200640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb32c0c3400 0x7fb32c14f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:06.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.231+0000 7fb32b200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb32c0c6800 0x7fb32c0c21c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:06.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.231+0000 7fb32b200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb32c158610 0x7fb32c1f0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:06.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.231+0000 7fb32b200640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb32c15bbf0 con 0x7fb32c0c3400 2024-04-14T23:58:06.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.231+0000 7fb32b200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb32c15b910 con 0x7fb32c158610 2024-04-14T23:58:06.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.231+0000 7fb32b200640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fb32c0b1c80 con 0x7fb32c0c6800 2024-04-14T23:58:06.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.231+0000 7fb329e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb32c0c3400 0x7fb32c14f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:06.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.232+0000 7fb329e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb32c0c3400 0x7fb32c14f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39684/0 (socket says 172.21.15.60:39684) 2024-04-14T23:58:06.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.232+0000 7fb329e00640 1 -- 172.21.15.60:0/1472050128 learned_addr learned my addr 172.21.15.60:0/1472050128 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:58:06.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.232+0000 7fb32a800640 1 --2- 172.21.15.60:0/1472050128 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb32c158610 0x7fb32c1f0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:06.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.232+0000 7fb321400640 1 --2- 172.21.15.60:0/1472050128 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb32c0c6800 0x7fb32c0c21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:06.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.232+0000 7fb329e00640 1 -- 172.21.15.60:0/1472050128 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb32c0c6800 msgr2=0x7fb32c0c21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:06.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.232+0000 7fb329e00640 1 --2- 172.21.15.60:0/1472050128 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb32c0c6800 0x7fb32c0c21c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:06.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.232+0000 7fb329e00640 1 -- 172.21.15.60:0/1472050128 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb32c158610 msgr2=0x7fb32c1f0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:06.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.232+0000 7fb329e00640 1 --2- 172.21.15.60:0/1472050128 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb32c158610 0x7fb32c1f0bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:06.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.232+0000 7fb329e00640 1 -- 172.21.15.60:0/1472050128 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb32c0cc770 con 0x7fb32c0c3400 2024-04-14T23:58:06.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.232+0000 7fb32a800640 1 --2- 172.21.15.60:0/1472050128 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb32c158610 0x7fb32c1f0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-14T23:58:06.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.232+0000 7fb329e00640 1 --2- 172.21.15.60:0/1472050128 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb32c0c3400 0x7fb32c14f4c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fb31c00ec10 tx=0x7fb31c00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:06.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.232+0000 7fb322a00640 1 -- 172.21.15.60:0/1472050128 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb31c017070 con 0x7fb32c0c3400 2024-04-14T23:58:06.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.232+0000 7fb322a00640 1 -- 172.21.15.60:0/1472050128 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb31c00cc30 con 0x7fb32c0c3400 2024-04-14T23:58:06.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.232+0000 7fb322a00640 1 -- 172.21.15.60:0/1472050128 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb31c013470 con 0x7fb32c0c3400 2024-04-14T23:58:06.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.233+0000 7fb32b200640 1 -- 172.21.15.60:0/1472050128 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb32c1f4650 con 0x7fb32c0c3400 2024-04-14T23:58:06.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.233+0000 7fb32b200640 1 -- 172.21.15.60:0/1472050128 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb32c0cb990 con 0x7fb32c0c3400 2024-04-14T23:58:06.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.234+0000 7fb32b200640 1 -- 172.21.15.60:0/1472050128 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb32c04f540 con 0x7fb32c0c3400 2024-04-14T23:58:06.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.235+0000 7fb322a00640 1 -- 172.21.15.60:0/1472050128 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7fb31c01a070 con 0x7fb32c0c3400 2024-04-14T23:58:06.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.235+0000 7fb322a00640 1 --2- 172.21.15.60:0/1472050128 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fb30806b990 0x7fb30806de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:06.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.236+0000 7fb322a00640 1 -- 172.21.15.60:0/1472050128 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6255+0+0 (secure 0 0 0) 0x7fb31c08abd0 con 0x7fb32c0c3400 2024-04-14T23:58:06.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.236+0000 7fb321400640 1 --2- 172.21.15.60:0/1472050128 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fb30806b990 0x7fb30806de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:06.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.237+0000 7fb321400640 1 --2- 172.21.15.60:0/1472050128 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fb30806b990 0x7fb30806de50 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fb32c158ee0 tx=0x7fb314009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:06.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.240+0000 7fb322a00640 1 -- 172.21.15.60:0/1472050128 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb31c059000 con 0x7fb32c0c3400 2024-04-14T23:58:06.432 INFO:tasks.ceph.mds.a.smithi060.stderr:2024-04-14T23:58:06.427+0000 7f3cf7400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-14T23:58:06.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.439+0000 7fb32b200640 1 -- 172.21.15.60:0/1472050128 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb32c070d10 con 0x7fb32c0c3400 2024-04-14T23:58:06.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.440+0000 7fb322a00640 1 -- 172.21.15.60:0/1472050128 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v45) ==== 93+0+10 (secure 0 0 0) 0x7fb31c05d0a0 con 0x7fb32c0c3400 2024-04-14T23:58:06.445 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2024-04-14T23:58:06.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.442+0000 7fb32b200640 1 -- 172.21.15.60:0/1472050128 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fb30806b990 msgr2=0x7fb30806de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:06.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.442+0000 7fb32b200640 1 --2- 172.21.15.60:0/1472050128 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fb30806b990 0x7fb30806de50 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fb32c158ee0 tx=0x7fb314009040 comp rx=0 tx=0).stop 2024-04-14T23:58:06.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.442+0000 7fb32b200640 1 -- 172.21.15.60:0/1472050128 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb32c0c3400 msgr2=0x7fb32c14f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:06.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.442+0000 7fb32b200640 1 --2- 172.21.15.60:0/1472050128 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb32c0c3400 0x7fb32c14f4c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fb31c00ec10 tx=0x7fb31c00c520 comp rx=0 tx=0).stop 2024-04-14T23:58:06.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.442+0000 7fb32b200640 1 -- 172.21.15.60:0/1472050128 shutdown_connections 2024-04-14T23:58:06.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.442+0000 7fb32b200640 1 --2- 172.21.15.60:0/1472050128 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fb30806b990 0x7fb30806de50 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:06.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.442+0000 7fb32b200640 1 --2- 172.21.15.60:0/1472050128 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb32c158610 0x7fb32c1f0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:06.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.443+0000 7fb32b200640 1 --2- 172.21.15.60:0/1472050128 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb32c0c6800 0x7fb32c0c21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:06.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.443+0000 7fb32b200640 1 --2- 172.21.15.60:0/1472050128 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb32c0c3400 0x7fb32c14f4c0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:06.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.443+0000 7fb32b200640 1 -- 172.21.15.60:0/1472050128 >> 172.21.15.60:0/1472050128 conn(0x7fb32c0cc2f0 msgr2=0x7fb32c072ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:06.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.443+0000 7fb32b200640 1 -- 172.21.15.60:0/1472050128 shutdown_connections 2024-04-14T23:58:06.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.443+0000 7fb32b200640 1 -- 172.21.15.60:0/1472050128 wait complete. 2024-04-14T23:58:06.468 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-14T23:58:06.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.687+0000 7f7262000640 1 Processor -- start 2024-04-14T23:58:06.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.687+0000 7f7262000640 1 -- start start 2024-04-14T23:58:06.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.689+0000 7f7262000640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f725c15c870 0x7f725c162580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:06.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.689+0000 7f7262000640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f725c0712e0 0x7f725c15fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:06.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.689+0000 7f7262000640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f725c15ff60 con 0x7f725c0c8560 2024-04-14T23:58:06.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.689+0000 7f7262000640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f725c1634d0 con 0x7f725c0712e0 2024-04-14T23:58:06.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.689+0000 7f7262000640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f725c162c30 con 0x7f725c15c870 2024-04-14T23:58:06.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.689+0000 7f725be00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f725c0712e0 0x7f725c15fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:06.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.689+0000 7f7260c00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f725c15c870 0x7f725c162580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:06.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.689+0000 7f725be00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f725c0712e0 0x7f725c15fa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39694/0 (socket says 172.21.15.60:39694) 2024-04-14T23:58:06.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.689+0000 7f725be00640 1 -- 172.21.15.60:0/2584282861 learned_addr learned my addr 172.21.15.60:0/2584282861 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:58:06.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.689+0000 7f7261600640 1 --1- 172.21.15.60:0/2584282861 >> v1:172.21.15.60:6789/0 conn(0x7f725c0c8560 0x7f725c06eb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:55592/0 (socket says 172.21.15.60:55592) 2024-04-14T23:58:06.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.690+0000 7f725be00640 1 -- 172.21.15.60:0/2584282861 >> v2:172.21.15.186:3300/0 conn(0x7f725c15c870 msgr2=0x7f725c162580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:58:06.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.690+0000 7f725be00640 1 --2- 172.21.15.60:0/2584282861 >> v2:172.21.15.186:3300/0 conn(0x7f725c15c870 0x7f725c162580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:06.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.690+0000 7f725be00640 1 -- 172.21.15.60:0/2584282861 >> v1:172.21.15.60:6789/0 conn(0x7f725c0c8560 legacy=0x7f725c06eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:06.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.690+0000 7f725be00640 1 -- 172.21.15.60:0/2584282861 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f725c162f10 con 0x7f725c0712e0 2024-04-14T23:58:06.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.690+0000 7f7260c00640 1 --2- 172.21.15.60:0/2584282861 >> v2:172.21.15.186:3300/0 conn(0x7f725c15c870 0x7f725c162580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:58:06.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.690+0000 7f725b400640 1 -- 172.21.15.60:0/2584282861 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2746512317 0 0) 0x7f725c15ff60 con 0x7f725c0c8560 2024-04-14T23:58:06.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.691+0000 7f725be00640 1 --2- 172.21.15.60:0/2584282861 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f725c0712e0 0x7f725c15fa20 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f725c06c780 tx=0x7f724402ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8774c8696a52417c server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:06.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.692+0000 7f725b400640 1 -- 172.21.15.60:0/2584282861 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f724402f850 con 0x7f725c0712e0 2024-04-14T23:58:06.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.692+0000 7f725b400640 1 -- 172.21.15.60:0/2584282861 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f724402f9f0 con 0x7f725c0712e0 2024-04-14T23:58:06.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.692+0000 7f725b400640 1 -- 172.21.15.60:0/2584282861 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f724402fdc0 con 0x7f725c0712e0 2024-04-14T23:58:06.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.692+0000 7f7262000640 1 -- 172.21.15.60:0/2584282861 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f725c0712e0 msgr2=0x7f725c15fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:06.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.692+0000 7f7262000640 1 --2- 172.21.15.60:0/2584282861 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f725c0712e0 0x7f725c15fa20 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f725c06c780 tx=0x7f724402ba30 comp rx=0 tx=0).stop 2024-04-14T23:58:06.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.692+0000 7f7262000640 1 -- 172.21.15.60:0/2584282861 shutdown_connections 2024-04-14T23:58:06.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.692+0000 7f7262000640 1 --2- 172.21.15.60:0/2584282861 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f725c0712e0 0x7f725c15fa20 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:06.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.692+0000 7f7262000640 1 --2- 172.21.15.60:0/2584282861 >> v2:172.21.15.186:3300/0 conn(0x7f725c15c870 0x7f725c162580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:06.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.692+0000 7f7262000640 1 -- 172.21.15.60:0/2584282861 >> 172.21.15.60:0/2584282861 conn(0x7f725c0cc2f0 msgr2=0x7f725c0cc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:06.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.693+0000 7f7262000640 1 -- 172.21.15.60:0/2584282861 shutdown_connections 2024-04-14T23:58:06.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.693+0000 7f7262000640 1 -- 172.21.15.60:0/2584282861 wait complete. 2024-04-14T23:58:06.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.694+0000 7f7262000640 1 Processor -- start 2024-04-14T23:58:06.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.695+0000 7f7262000640 1 -- start start 2024-04-14T23:58:06.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.695+0000 7f7262000640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f725c0712e0 0x7f725c1546f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:06.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.696+0000 7f7262000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f725c0c8560 0x7f725c158230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:06.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.696+0000 7f7262000640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f725c15c870 0x7f725c158770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:06.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.696+0000 7f7262000640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f725c15f500 con 0x7f725c15c870 2024-04-14T23:58:06.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.696+0000 7f7262000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f725c15f220 con 0x7f725c0c8560 2024-04-14T23:58:06.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.696+0000 7f7262000640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f725c06f270 con 0x7f725c0712e0 2024-04-14T23:58:06.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.696+0000 7f7261600640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f725c15c870 0x7f725c158770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:06.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.696+0000 7f7261600640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f725c15c870 0x7f725c158770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39700/0 (socket says 172.21.15.60:39700) 2024-04-14T23:58:06.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.696+0000 7f7261600640 1 -- 172.21.15.60:0/1110997931 learned_addr learned my addr 172.21.15.60:0/1110997931 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:58:06.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.696+0000 7f7260c00640 1 --2- 172.21.15.60:0/1110997931 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f725c0712e0 0x7f725c1546f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:06.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.696+0000 7f725be00640 1 --2- 172.21.15.60:0/1110997931 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f725c0c8560 0x7f725c158230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:06.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.696+0000 7f7261600640 1 -- 172.21.15.60:0/1110997931 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f725c0712e0 msgr2=0x7f725c1546f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:06.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.696+0000 7f7261600640 1 --2- 172.21.15.60:0/1110997931 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f725c0712e0 0x7f725c1546f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:06.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.696+0000 7f7261600640 1 -- 172.21.15.60:0/1110997931 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f725c0c8560 msgr2=0x7f725c158230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:06.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.696+0000 7f7261600640 1 --2- 172.21.15.60:0/1110997931 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f725c0c8560 0x7f725c158230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:06.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.696+0000 7f7261600640 1 -- 172.21.15.60:0/1110997931 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f725c1f5030 con 0x7f725c15c870 2024-04-14T23:58:06.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.697+0000 7f7261600640 1 --2- 172.21.15.60:0/1110997931 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f725c15c870 0x7f725c158770 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f725400c700 tx=0x7f725400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:06.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.697+0000 7f7258c00640 1 -- 172.21.15.60:0/1110997931 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7254018070 con 0x7f725c15c870 2024-04-14T23:58:06.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.697+0000 7f7258c00640 1 -- 172.21.15.60:0/1110997931 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f725400ce10 con 0x7f725c15c870 2024-04-14T23:58:06.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.697+0000 7f7262000640 1 -- 172.21.15.60:0/1110997931 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f725c1f52f0 con 0x7f725c15c870 2024-04-14T23:58:06.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.697+0000 7f7258c00640 1 -- 172.21.15.60:0/1110997931 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f725400a880 con 0x7f725c15c870 2024-04-14T23:58:06.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.697+0000 7f7262000640 1 -- 172.21.15.60:0/1110997931 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f725c1f58b0 con 0x7f725c15c870 2024-04-14T23:58:06.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.698+0000 7f7262000640 1 -- 172.21.15.60:0/1110997931 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f725c1f5af0 con 0x7f725c15c870 2024-04-14T23:58:06.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.699+0000 7f7258c00640 1 -- 172.21.15.60:0/1110997931 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f725401b070 con 0x7f725c15c870 2024-04-14T23:58:06.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.700+0000 7f7258c00640 1 --2- 172.21.15.60:0/1110997931 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f723806b990 0x7f723806de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:06.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.700+0000 7f7258c00640 1 -- 172.21.15.60:0/1110997931 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6255+0+0 (secure 0 0 0) 0x7f7254089f90 con 0x7f725c15c870 2024-04-14T23:58:06.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.701+0000 7f7260c00640 1 --2- 172.21.15.60:0/1110997931 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f723806b990 0x7f723806de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:06.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.701+0000 7f7260c00640 1 --2- 172.21.15.60:0/1110997931 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f723806b990 0x7f723806de50 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f724c009560 tx=0x7f724c009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:06.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.705+0000 7f7258c00640 1 -- 172.21.15.60:0/1110997931 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f725c1f5af0 con 0x7f725c15c870 2024-04-14T23:58:06.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.910+0000 7f7262000640 1 -- 172.21.15.60:0/1110997931 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f725c072ca0 con 0x7f725c15c870 2024-04-14T23:58:06.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.911+0000 7f7258c00640 1 -- 172.21.15.60:0/1110997931 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v45) ==== 104+0+11 (secure 0 0 0) 0x7f725c072ca0 con 0x7f725c15c870 2024-04-14T23:58:06.917 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2024-04-14T23:58:06.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.914+0000 7f7262000640 1 -- 172.21.15.60:0/1110997931 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f723806b990 msgr2=0x7f723806de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:06.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.914+0000 7f7262000640 1 --2- 172.21.15.60:0/1110997931 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f723806b990 0x7f723806de50 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f724c009560 tx=0x7f724c009190 comp rx=0 tx=0).stop 2024-04-14T23:58:06.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.914+0000 7f7262000640 1 -- 172.21.15.60:0/1110997931 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f725c15c870 msgr2=0x7f725c158770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:06.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.914+0000 7f7262000640 1 --2- 172.21.15.60:0/1110997931 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f725c15c870 0x7f725c158770 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f725400c700 tx=0x7f725400cbc0 comp rx=0 tx=0).stop 2024-04-14T23:58:06.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.914+0000 7f7262000640 1 -- 172.21.15.60:0/1110997931 shutdown_connections 2024-04-14T23:58:06.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.914+0000 7f7262000640 1 --2- 172.21.15.60:0/1110997931 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f723806b990 0x7f723806de50 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:06.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.914+0000 7f7262000640 1 --2- 172.21.15.60:0/1110997931 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f725c15c870 0x7f725c158770 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:06.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.914+0000 7f7262000640 1 --2- 172.21.15.60:0/1110997931 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f725c0c8560 0x7f725c158230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:06.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.914+0000 7f7262000640 1 --2- 172.21.15.60:0/1110997931 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f725c0712e0 0x7f725c1546f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:06.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.914+0000 7f7262000640 1 -- 172.21.15.60:0/1110997931 >> 172.21.15.60:0/1110997931 conn(0x7f725c0cc2f0 msgr2=0x7f725c0c9dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:06.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.915+0000 7f7262000640 1 -- 172.21.15.60:0/1110997931 shutdown_connections 2024-04-14T23:58:06.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:06.915+0000 7f7262000640 1 -- 172.21.15.60:0/1110997931 wait complete. 2024-04-14T23:58:06.939 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-14T23:58:07.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.163+0000 7fe3dbc00640 1 Processor -- start 2024-04-14T23:58:07.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.163+0000 7fe3dbc00640 1 -- start start 2024-04-14T23:58:07.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.165+0000 7fe3dbc00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe3d40c63c0 0x7fe3d40c6b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:07.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.165+0000 7fe3dbc00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3d41530d0 con 0x7fe3d415c9b0 2024-04-14T23:58:07.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.165+0000 7fe3dbc00640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3d40751f0 con 0x7fe3d4152cf0 2024-04-14T23:58:07.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.165+0000 7fe3dbc00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fe3d40c8370 con 0x7fe3d40c63c0 2024-04-14T23:58:07.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.166+0000 7fe3d9200640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fe3d4152cf0 0x7fe3d415c2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.60:45422/0 (socket says 172.21.15.60:45422) 2024-04-14T23:58:07.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.166+0000 7fe3d9200640 1 -- 172.21.15.60:0/65042260 learned_addr learned my addr 172.21.15.60:0/65042260 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:58:07.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.166+0000 7fe3d3e00640 1 --2- 172.21.15.60:0/65042260 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe3d40c63c0 0x7fe3d40c6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:07.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.166+0000 7fe3d3400640 1 -- 172.21.15.60:0/65042260 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3458976665 0 0) 0x7fe3d40751f0 con 0x7fe3d4152cf0 2024-04-14T23:58:07.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.166+0000 7fe3d3400640 1 -- 172.21.15.60:0/65042260 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe3bc003520 con 0x7fe3d4152cf0 2024-04-14T23:58:07.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.166+0000 7fe3d3400640 1 -- 172.21.15.60:0/65042260 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3234773189 0 0) 0x7fe3d41530d0 con 0x7fe3d415c9b0 2024-04-14T23:58:07.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.166+0000 7fe3d3400640 1 -- 172.21.15.60:0/65042260 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe3d40751f0 con 0x7fe3d415c9b0 2024-04-14T23:58:07.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.166+0000 7fe3d3e00640 1 -- 172.21.15.60:0/65042260 >> v1:172.21.15.186:6789/0 conn(0x7fe3d4152cf0 legacy=0x7fe3d415c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:07.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.166+0000 7fe3d3e00640 1 -- 172.21.15.60:0/65042260 >> v1:172.21.15.60:6789/0 conn(0x7fe3d415c9b0 legacy=0x7fe3d40c70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:07.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.166+0000 7fe3d3e00640 1 -- 172.21.15.60:0/65042260 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3d40c3810 con 0x7fe3d40c63c0 2024-04-14T23:58:07.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.167+0000 7fe3d3e00640 1 --2- 172.21.15.60:0/65042260 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe3d40c63c0 0x7fe3d40c6b70 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fe3c00097d0 tx=0x7fe3c002b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=acd6907741f565e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:07.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.168+0000 7fe3d3400640 1 -- 172.21.15.60:0/65042260 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3c0034070 con 0x7fe3d40c63c0 2024-04-14T23:58:07.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.168+0000 7fe3d3400640 1 -- 172.21.15.60:0/65042260 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3c0030880 con 0x7fe3d40c63c0 2024-04-14T23:58:07.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.168+0000 7fe3d3400640 1 -- 172.21.15.60:0/65042260 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3c0030c50 con 0x7fe3d40c63c0 2024-04-14T23:58:07.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.168+0000 7fe3dbc00640 1 -- 172.21.15.60:0/65042260 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe3d40c63c0 msgr2=0x7fe3d40c6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:07.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.168+0000 7fe3dbc00640 1 --2- 172.21.15.60:0/65042260 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe3d40c63c0 0x7fe3d40c6b70 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fe3c00097d0 tx=0x7fe3c002b9c0 comp rx=0 tx=0).stop 2024-04-14T23:58:07.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.168+0000 7fe3dbc00640 1 -- 172.21.15.60:0/65042260 shutdown_connections 2024-04-14T23:58:07.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.168+0000 7fe3dbc00640 1 --2- 172.21.15.60:0/65042260 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe3d40c63c0 0x7fe3d40c6b70 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:07.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.168+0000 7fe3dbc00640 1 -- 172.21.15.60:0/65042260 >> 172.21.15.60:0/65042260 conn(0x7fe3d40cc170 msgr2=0x7fe3d40cc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:07.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.168+0000 7fe3dbc00640 1 -- 172.21.15.60:0/65042260 shutdown_connections 2024-04-14T23:58:07.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.168+0000 7fe3dbc00640 1 -- 172.21.15.60:0/65042260 wait complete. 2024-04-14T23:58:07.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.170+0000 7fe3dbc00640 1 Processor -- start 2024-04-14T23:58:07.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.170+0000 7fe3dbc00640 1 -- start start 2024-04-14T23:58:07.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.171+0000 7fe3dbc00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe3d40c63c0 0x7fe3d414d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:07.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.171+0000 7fe3dbc00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe3d4152cf0 0x7fe3d415b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:07.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.171+0000 7fe3dbc00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3d415c9b0 0x7fe3d41e3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:07.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.171+0000 7fe3dbc00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fe3d40c88f0 con 0x7fe3d415c9b0 2024-04-14T23:58:07.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.171+0000 7fe3dbc00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe3d40c8610 con 0x7fe3d40c63c0 2024-04-14T23:58:07.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.171+0000 7fe3dbc00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fe3d40c77c0 con 0x7fe3d4152cf0 2024-04-14T23:58:07.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.171+0000 7fe3d9c00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3d415c9b0 0x7fe3d41e3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:07.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.172+0000 7fe3d9c00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3d415c9b0 0x7fe3d41e3770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39716/0 (socket says 172.21.15.60:39716) 2024-04-14T23:58:07.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.172+0000 7fe3d9c00640 1 -- 172.21.15.60:0/2964255227 learned_addr learned my addr 172.21.15.60:0/2964255227 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:58:07.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.172+0000 7fe3d9200640 1 --2- 172.21.15.60:0/2964255227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe3d40c63c0 0x7fe3d414d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:07.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.172+0000 7fe3d3e00640 1 --2- 172.21.15.60:0/2964255227 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe3d4152cf0 0x7fe3d415b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:07.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.172+0000 7fe3d9c00640 1 -- 172.21.15.60:0/2964255227 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe3d4152cf0 msgr2=0x7fe3d415b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:07.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.172+0000 7fe3d9c00640 1 --2- 172.21.15.60:0/2964255227 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe3d4152cf0 0x7fe3d415b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:07.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.172+0000 7fe3d9c00640 1 -- 172.21.15.60:0/2964255227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe3d40c63c0 msgr2=0x7fe3d414d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:07.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.172+0000 7fe3d9c00640 1 --2- 172.21.15.60:0/2964255227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe3d40c63c0 0x7fe3d414d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:07.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.172+0000 7fe3d9c00640 1 -- 172.21.15.60:0/2964255227 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3d40ca940 con 0x7fe3d415c9b0 2024-04-14T23:58:07.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.172+0000 7fe3d9c00640 1 --2- 172.21.15.60:0/2964255227 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3d415c9b0 0x7fe3d41e3770 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc00c7c0 tx=0x7fe3cc00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:07.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.173+0000 7fe3d0c00640 1 -- 172.21.15.60:0/2964255227 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3cc003b70 con 0x7fe3d415c9b0 2024-04-14T23:58:07.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.173+0000 7fe3d0c00640 1 -- 172.21.15.60:0/2964255227 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3cc003d10 con 0x7fe3d415c9b0 2024-04-14T23:58:07.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.173+0000 7fe3d0c00640 1 -- 172.21.15.60:0/2964255227 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3cc00aa50 con 0x7fe3d415c9b0 2024-04-14T23:58:07.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.173+0000 7fe3dbc00640 1 -- 172.21.15.60:0/2964255227 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3d414dab0 con 0x7fe3d415c9b0 2024-04-14T23:58:07.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.173+0000 7fe3dbc00640 1 -- 172.21.15.60:0/2964255227 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3d40caec0 con 0x7fe3d415c9b0 2024-04-14T23:58:07.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.174+0000 7fe3dbc00640 1 -- 172.21.15.60:0/2964255227 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3d4076970 con 0x7fe3d415c9b0 2024-04-14T23:58:07.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.175+0000 7fe3d0c00640 1 -- 172.21.15.60:0/2964255227 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7fe3cc00abf0 con 0x7fe3d415c9b0 2024-04-14T23:58:07.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.176+0000 7fe3d0c00640 1 --2- 172.21.15.60:0/2964255227 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fe3b406b9c0 0x7fe3b406de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:07.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.176+0000 7fe3d0c00640 1 -- 172.21.15.60:0/2964255227 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6255+0+0 (secure 0 0 0) 0x7fe3cc08a9d0 con 0x7fe3d415c9b0 2024-04-14T23:58:07.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.176+0000 7fe3d9200640 1 --2- 172.21.15.60:0/2964255227 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fe3b406b9c0 0x7fe3b406de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:07.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.177+0000 7fe3d9200640 1 --2- 172.21.15.60:0/2964255227 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fe3b406b9c0 0x7fe3b406de80 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe3c8000f60 tx=0x7fe3c8007d30 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:07.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.180+0000 7fe3d0c00640 1 -- 172.21.15.60:0/2964255227 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe3cc058c40 con 0x7fe3d415c9b0 2024-04-14T23:58:07.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.382+0000 7fe3dbc00640 1 -- 172.21.15.60:0/2964255227 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe3d414f650 con 0x7fe3d415c9b0 2024-04-14T23:58:07.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.384+0000 7fe3d0c00640 1 -- 172.21.15.60:0/2964255227 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v45) ==== 100+0+11 (secure 0 0 0) 0x7fe3cc05cce0 con 0x7fe3d415c9b0 2024-04-14T23:58:07.389 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2024-04-14T23:58:07.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.387+0000 7fe3dbc00640 1 -- 172.21.15.60:0/2964255227 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fe3b406b9c0 msgr2=0x7fe3b406de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:07.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.387+0000 7fe3dbc00640 1 --2- 172.21.15.60:0/2964255227 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fe3b406b9c0 0x7fe3b406de80 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe3c8000f60 tx=0x7fe3c8007d30 comp rx=0 tx=0).stop 2024-04-14T23:58:07.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.387+0000 7fe3dbc00640 1 -- 172.21.15.60:0/2964255227 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3d415c9b0 msgr2=0x7fe3d41e3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:07.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.387+0000 7fe3dbc00640 1 --2- 172.21.15.60:0/2964255227 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3d415c9b0 0x7fe3d41e3770 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc00c7c0 tx=0x7fe3cc00cc80 comp rx=0 tx=0).stop 2024-04-14T23:58:07.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.387+0000 7fe3dbc00640 1 -- 172.21.15.60:0/2964255227 shutdown_connections 2024-04-14T23:58:07.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.387+0000 7fe3dbc00640 1 --2- 172.21.15.60:0/2964255227 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fe3b406b9c0 0x7fe3b406de80 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:07.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.387+0000 7fe3dbc00640 1 --2- 172.21.15.60:0/2964255227 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3d415c9b0 0x7fe3d41e3770 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:07.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.387+0000 7fe3dbc00640 1 --2- 172.21.15.60:0/2964255227 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe3d4152cf0 0x7fe3d415b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:07.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.387+0000 7fe3dbc00640 1 --2- 172.21.15.60:0/2964255227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe3d40c63c0 0x7fe3d414d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:07.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.387+0000 7fe3dbc00640 1 -- 172.21.15.60:0/2964255227 >> 172.21.15.60:0/2964255227 conn(0x7fe3d40cc170 msgr2=0x7fe3d4156e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:07.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.387+0000 7fe3dbc00640 1 -- 172.21.15.60:0/2964255227 shutdown_connections 2024-04-14T23:58:07.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.387+0000 7fe3dbc00640 1 -- 172.21.15.60:0/2964255227 wait complete. 2024-04-14T23:58:07.414 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-14T23:58:07.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.637+0000 7f7159200640 1 Processor -- start 2024-04-14T23:58:07.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.637+0000 7f7159200640 1 -- start start 2024-04-14T23:58:07.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.639+0000 7f7159200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7154151cb0 0x7f71540cc340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:07.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.639+0000 7f7159200640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f71540672c0 0x7f71540c5b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:07.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.639+0000 7f7159200640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f71540c6070 0x7f715415b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:07.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.639+0000 7f7159200640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f7154152090 con 0x7f71540c6070 2024-04-14T23:58:07.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.639+0000 7f7159200640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f71540c3710 con 0x7f71540672c0 2024-04-14T23:58:07.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.639+0000 7f7159200640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f71540c39f0 con 0x7f7154151cb0 2024-04-14T23:58:07.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.639+0000 7f7152800640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f71540c6070 0x7f715415b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:07.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.639+0000 7f7152800640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f71540c6070 0x7f715415b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39726/0 (socket says 172.21.15.60:39726) 2024-04-14T23:58:07.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.639+0000 7f7152800640 1 -- 172.21.15.60:0/2975186306 learned_addr learned my addr 172.21.15.60:0/2975186306 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:58:07.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.639+0000 7f7151400640 1 --2- 172.21.15.60:0/2975186306 >> v2:172.21.15.186:3300/0 conn(0x7f71540672c0 0x7f71540c5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:07.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.639+0000 7f7151e00640 1 --2- 172.21.15.60:0/2975186306 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7154151cb0 0x7f71540cc340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:07.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.640+0000 7f7152800640 1 -- 172.21.15.60:0/2975186306 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7154151cb0 msgr2=0x7f71540cc340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:58:07.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.640+0000 7f7152800640 1 --2- 172.21.15.60:0/2975186306 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7154151cb0 0x7f71540cc340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:07.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.640+0000 7f7152800640 1 -- 172.21.15.60:0/2975186306 >> v2:172.21.15.186:3300/0 conn(0x7f71540672c0 msgr2=0x7f71540c5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:58:07.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.640+0000 7f7152800640 1 --2- 172.21.15.60:0/2975186306 >> v2:172.21.15.186:3300/0 conn(0x7f71540672c0 0x7f71540c5b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:07.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.640+0000 7f7152800640 1 -- 172.21.15.60:0/2975186306 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71540c3e50 con 0x7f71540c6070 2024-04-14T23:58:07.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.641+0000 7f7152800640 1 --2- 172.21.15.60:0/2975186306 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f71540c6070 0x7f715415b450 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f7154069060 tx=0x7f714c02dba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61d1970386de1e18 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:07.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.641+0000 7f7150a00640 1 -- 172.21.15.60:0/2975186306 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f714c03b070 con 0x7f71540c6070 2024-04-14T23:58:07.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.641+0000 7f7150a00640 1 -- 172.21.15.60:0/2975186306 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f714c02e950 con 0x7f71540c6070 2024-04-14T23:58:07.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.641+0000 7f7150a00640 1 -- 172.21.15.60:0/2975186306 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f714c02ed20 con 0x7f71540c6070 2024-04-14T23:58:07.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.642+0000 7f7159200640 1 -- 172.21.15.60:0/2975186306 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f71540c6070 msgr2=0x7f715415b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:07.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.642+0000 7f7159200640 1 --2- 172.21.15.60:0/2975186306 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f71540c6070 0x7f715415b450 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f7154069060 tx=0x7f714c02dba0 comp rx=0 tx=0).stop 2024-04-14T23:58:07.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.642+0000 7f7159200640 1 -- 172.21.15.60:0/2975186306 shutdown_connections 2024-04-14T23:58:07.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.642+0000 7f7159200640 1 --2- 172.21.15.60:0/2975186306 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f71540c6070 0x7f715415b450 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:07.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.642+0000 7f7159200640 1 --2- 172.21.15.60:0/2975186306 >> v2:172.21.15.186:3300/0 conn(0x7f71540672c0 0x7f71540c5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:07.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.642+0000 7f7159200640 1 --2- 172.21.15.60:0/2975186306 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7154151cb0 0x7f71540cc340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:07.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.642+0000 7f7159200640 1 -- 172.21.15.60:0/2975186306 >> 172.21.15.60:0/2975186306 conn(0x7f71540cc8b0 msgr2=0x7f71540cccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:07.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.642+0000 7f7159200640 1 -- 172.21.15.60:0/2975186306 shutdown_connections 2024-04-14T23:58:07.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.642+0000 7f7159200640 1 -- 172.21.15.60:0/2975186306 wait complete. 2024-04-14T23:58:07.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.644+0000 7f7159200640 1 Processor -- start 2024-04-14T23:58:07.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.644+0000 7f7159200640 1 -- start start 2024-04-14T23:58:07.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.645+0000 7f7159200640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f71540672c0 0x7f7154156770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:07.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.645+0000 7f7159200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f71540c6070 0x7f7154156cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:07.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.645+0000 7f7159200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7154151cb0 0x7f7154145ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:07.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.645+0000 7f7159200640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f715415aee0 con 0x7f71540672c0 2024-04-14T23:58:07.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.645+0000 7f7159200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f71540c4410 con 0x7f7154151cb0 2024-04-14T23:58:07.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.645+0000 7f7159200640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f71540c3340 con 0x7f71540c6070 2024-04-14T23:58:07.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.645+0000 7f7151e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f71540672c0 0x7f7154156770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:07.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.645+0000 7f7151e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f71540672c0 0x7f7154156770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39736/0 (socket says 172.21.15.60:39736) 2024-04-14T23:58:07.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.645+0000 7f7151e00640 1 -- 172.21.15.60:0/2294725346 learned_addr learned my addr 172.21.15.60:0/2294725346 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:58:07.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.645+0000 7f7151400640 1 --2- 172.21.15.60:0/2294725346 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f71540c6070 0x7f7154156cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:07.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.645+0000 7f7152800640 1 --2- 172.21.15.60:0/2294725346 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7154151cb0 0x7f7154145ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:07.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.646+0000 7f7151e00640 1 -- 172.21.15.60:0/2294725346 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f71540c6070 msgr2=0x7f7154156cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:07.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.646+0000 7f7151e00640 1 --2- 172.21.15.60:0/2294725346 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f71540c6070 0x7f7154156cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:07.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.646+0000 7f7151e00640 1 -- 172.21.15.60:0/2294725346 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7154151cb0 msgr2=0x7f7154145ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:07.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.646+0000 7f7151e00640 1 --2- 172.21.15.60:0/2294725346 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7154151cb0 0x7f7154145ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:07.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.646+0000 7f7151e00640 1 -- 172.21.15.60:0/2294725346 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7154158660 con 0x7f71540672c0 2024-04-14T23:58:07.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.646+0000 7f7152800640 1 --2- 172.21.15.60:0/2294725346 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7154151cb0 0x7f7154145ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:58:07.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.646+0000 7f7151e00640 1 --2- 172.21.15.60:0/2294725346 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f71540672c0 0x7f7154156770 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f714400e9e0 tx=0x7f714400eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:07.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.646+0000 7f714a000640 1 -- 172.21.15.60:0/2294725346 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7144017070 con 0x7f71540672c0 2024-04-14T23:58:07.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.646+0000 7f714a000640 1 -- 172.21.15.60:0/2294725346 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f714400c9d0 con 0x7f71540672c0 2024-04-14T23:58:07.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.646+0000 7f714a000640 1 -- 172.21.15.60:0/2294725346 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f714400cdc0 con 0x7f71540672c0 2024-04-14T23:58:07.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.646+0000 7f7159200640 1 -- 172.21.15.60:0/2294725346 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7154158940 con 0x7f71540672c0 2024-04-14T23:58:07.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.647+0000 7f7159200640 1 -- 172.21.15.60:0/2294725346 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7154155d00 con 0x7f71540672c0 2024-04-14T23:58:07.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.648+0000 7f7159200640 1 -- 172.21.15.60:0/2294725346 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7154155f00 con 0x7f71540672c0 2024-04-14T23:58:07.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.649+0000 7f714a000640 1 -- 172.21.15.60:0/2294725346 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f714401a070 con 0x7f71540672c0 2024-04-14T23:58:07.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.649+0000 7f714a000640 1 --2- 172.21.15.60:0/2294725346 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f713006b990 0x7f713006de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:07.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.650+0000 7f714a000640 1 -- 172.21.15.60:0/2294725346 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6255+0+0 (secure 0 0 0) 0x7f714408ae00 con 0x7f71540672c0 2024-04-14T23:58:07.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.650+0000 7f7151400640 1 --2- 172.21.15.60:0/2294725346 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f713006b990 0x7f713006de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:07.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.651+0000 7f7151400640 1 --2- 172.21.15.60:0/2294725346 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f713006b990 0x7f713006de50 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f71540c6950 tx=0x7f713c00b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:07.659 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.654+0000 7f714a000640 1 -- 172.21.15.60:0/2294725346 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7144059070 con 0x7f71540672c0 2024-04-14T23:58:07.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.883+0000 7f7159200640 1 -- 172.21.15.60:0/2294725346 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f7154068dd0 con 0x7f71540672c0 2024-04-14T23:58:07.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.884+0000 7f714a000640 1 -- 172.21.15.60:0/2294725346 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v45) ==== 81+0+606 (secure 0 0 0) 0x7f7154068dd0 con 0x7f71540672c0 2024-04-14T23:58:07.890 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.60:6835/2356847578 2024-04-15T23:57:39.001841+0000 2024-04-14T23:58:07.890 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.60:6834/2356847578 2024-04-15T23:57:39.001841+0000 2024-04-14T23:58:07.890 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.60:6833/1971580590 2024-04-15T23:56:28.972769+0000 2024-04-14T23:58:07.890 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.60:6835/1714133698 2024-04-15T23:54:48.942868+0000 2024-04-14T23:58:07.890 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.60:6832/1971580590 2024-04-15T23:56:28.972769+0000 2024-04-14T23:58:07.890 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.60:6834/1714133698 2024-04-15T23:54:48.942868+0000 2024-04-14T23:58:07.890 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.60:6833/78303888 2024-04-15T23:52:28.905933+0000 2024-04-14T23:58:07.890 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.60:6832/78303888 2024-04-15T23:52:28.905933+0000 2024-04-14T23:58:07.891 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.60:6835/2677133122 2024-04-15T23:50:58.881136+0000 2024-04-14T23:58:07.891 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.60:6834/2677133122 2024-04-15T23:50:58.881136+0000 2024-04-14T23:58:07.891 INFO:teuthology.orchestra.run.smithi060.stderr:listed 10 entries 2024-04-14T23:58:07.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.887+0000 7f7159200640 1 -- 172.21.15.60:0/2294725346 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f713006b990 msgr2=0x7f713006de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:07.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.887+0000 7f7159200640 1 --2- 172.21.15.60:0/2294725346 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f713006b990 0x7f713006de50 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f71540c6950 tx=0x7f713c00b040 comp rx=0 tx=0).stop 2024-04-14T23:58:07.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.887+0000 7f7159200640 1 -- 172.21.15.60:0/2294725346 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f71540672c0 msgr2=0x7f7154156770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:07.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.887+0000 7f7159200640 1 --2- 172.21.15.60:0/2294725346 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f71540672c0 0x7f7154156770 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f714400e9e0 tx=0x7f714400eea0 comp rx=0 tx=0).stop 2024-04-14T23:58:07.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.887+0000 7f7159200640 1 -- 172.21.15.60:0/2294725346 shutdown_connections 2024-04-14T23:58:07.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.887+0000 7f7159200640 1 --2- 172.21.15.60:0/2294725346 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f713006b990 0x7f713006de50 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:07.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.887+0000 7f7159200640 1 --2- 172.21.15.60:0/2294725346 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7154151cb0 0x7f7154145ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:07.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.887+0000 7f7159200640 1 --2- 172.21.15.60:0/2294725346 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f71540c6070 0x7f7154156cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:07.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.887+0000 7f7159200640 1 --2- 172.21.15.60:0/2294725346 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f71540672c0 0x7f7154156770 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:07.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.887+0000 7f7159200640 1 -- 172.21.15.60:0/2294725346 >> 172.21.15.60:0/2294725346 conn(0x7f71540cc8b0 msgr2=0x7f71540ca6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:07.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.887+0000 7f7159200640 1 -- 172.21.15.60:0/2294725346 shutdown_connections 2024-04-14T23:58:07.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:07.887+0000 7f7159200640 1 -- 172.21.15.60:0/2294725346 wait complete. 2024-04-14T23:58:07.913 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-14T23:58:07.913 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-14T23:58:07.913 DEBUG:teuthology.orchestra.run.smithi060:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-14T23:58:08.433 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi060.front.sepia.ceph.com 2024-04-14T23:58:08.433 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-14T23:58:08.434 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-14T23:58:08.491 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-14T23:58:08.492 DEBUG:teuthology.orchestra.run.smithi060:> ip netns list 2024-04-14T23:58:08.547 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-14T23:58:08.547 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-14T23:58:08.548 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-14T23:58:08.645 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-14T23:58:08.645 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link delete ceph-brx 2024-04-14T23:58:08.765 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-14T23:58:08.774 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-14T23:58:08.775 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-14T23:58:08.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:08.958+0000 7fc2abc00640 1 Processor -- start 2024-04-14T23:58:08.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:08.959+0000 7fc2abc00640 1 -- start start 2024-04-14T23:58:08.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:08.960+0000 7fc2abc00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc2a4152cf0 0x7fc2a415c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:08.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:08.960+0000 7fc2abc00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fc2a4159450 0x7fc2a415c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:08.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:08.960+0000 7fc2abc00640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2a41530d0 con 0x7fc2a40c63c0 2024-04-14T23:58:08.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:08.960+0000 7fc2abc00640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fc2a40751f0 con 0x7fc2a4152cf0 2024-04-14T23:58:08.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:08.960+0000 7fc2abc00640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fc2a40c8370 con 0x7fc2a4159450 2024-04-14T23:58:08.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:08.961+0000 7fc2a9200640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc2a4152cf0 0x7fc2a415c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:08.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:08.961+0000 7fc2a9200640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc2a4152cf0 0x7fc2a415c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39746/0 (socket says 172.21.15.60:39746) 2024-04-14T23:58:08.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:08.961+0000 7fc2a9200640 1 -- 172.21.15.60:0/709488764 learned_addr learned my addr 172.21.15.60:0/709488764 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:58:08.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:08.961+0000 7fc2a9c00640 1 --2- 172.21.15.60:0/709488764 >> v2:172.21.15.186:3300/0 conn(0x7fc2a4159450 0x7fc2a415c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:08.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:08.961+0000 7fc2a9200640 1 -- 172.21.15.60:0/709488764 >> v2:172.21.15.186:3300/0 conn(0x7fc2a4159450 msgr2=0x7fc2a415c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:58:08.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:08.961+0000 7fc2a9200640 1 --2- 172.21.15.60:0/709488764 >> v2:172.21.15.186:3300/0 conn(0x7fc2a4159450 0x7fc2a415c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:08.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:08.961+0000 7fc2a9200640 1 -- 172.21.15.60:0/709488764 >> v1:172.21.15.186:6789/0 conn(0x7fc2a40c63c0 legacy=0x7fc2a40c3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:08.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:08.961+0000 7fc2a9200640 1 -- 172.21.15.60:0/709488764 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2a40c3810 con 0x7fc2a4152cf0 2024-04-14T23:58:08.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:08.962+0000 7fc2a9200640 1 --2- 172.21.15.60:0/709488764 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2a4152cf0 0x7fc2a415c2a0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fc2a406c750 tx=0x7fc29802ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fde4f1888bcac32c server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:08.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:08.963+0000 7fc2a3400640 1 -- 172.21.15.60:0/709488764 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc29802f850 con 0x7fc2a4152cf0 2024-04-14T23:58:08.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:08.963+0000 7fc2a3400640 1 -- 172.21.15.60:0/709488764 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc29802f9f0 con 0x7fc2a4152cf0 2024-04-14T23:58:08.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:08.963+0000 7fc2a3400640 1 -- 172.21.15.60:0/709488764 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc29802fdc0 con 0x7fc2a4152cf0 2024-04-14T23:58:08.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:08.963+0000 7fc2abc00640 1 -- 172.21.15.60:0/709488764 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2a4152cf0 msgr2=0x7fc2a415c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:08.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:08.963+0000 7fc2abc00640 1 --2- 172.21.15.60:0/709488764 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2a4152cf0 0x7fc2a415c2a0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fc2a406c750 tx=0x7fc29802ba30 comp rx=0 tx=0).stop 2024-04-14T23:58:08.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:08.964+0000 7fc2abc00640 1 -- 172.21.15.60:0/709488764 shutdown_connections 2024-04-14T23:58:08.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:08.964+0000 7fc2abc00640 1 --2- 172.21.15.60:0/709488764 >> v2:172.21.15.186:3300/0 conn(0x7fc2a4159450 0x7fc2a415c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:08.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:08.964+0000 7fc2abc00640 1 --2- 172.21.15.60:0/709488764 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2a4152cf0 0x7fc2a415c2a0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:08.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:08.964+0000 7fc2abc00640 1 -- 172.21.15.60:0/709488764 >> 172.21.15.60:0/709488764 conn(0x7fc2a40cc170 msgr2=0x7fc2a40cc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:08.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:08.964+0000 7fc2abc00640 1 -- 172.21.15.60:0/709488764 shutdown_connections 2024-04-14T23:58:08.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:08.964+0000 7fc2abc00640 1 -- 172.21.15.60:0/709488764 wait complete. 2024-04-14T23:58:08.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:08.966+0000 7fc2abc00640 1 Processor -- start 2024-04-14T23:58:08.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:08.966+0000 7fc2abc00640 1 -- start start 2024-04-14T23:58:08.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:08.967+0000 7fc2abc00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc2a40c63c0 0x7fc2a41f40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:08.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:08.967+0000 7fc2abc00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2a4152cf0 0x7fc2a41f4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:08.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:08.967+0000 7fc2abc00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2a4159450 0x7fc2a415a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:08.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:08.967+0000 7fc2abc00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fc2a40c88f0 con 0x7fc2a4159450 2024-04-14T23:58:08.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:08.967+0000 7fc2abc00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc2a40c8610 con 0x7fc2a4152cf0 2024-04-14T23:58:08.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:08.967+0000 7fc2abc00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fc2a40c4690 con 0x7fc2a40c63c0 2024-04-14T23:58:08.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:08.967+0000 7fc2a9c00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2a4159450 0x7fc2a415a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:08.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:08.967+0000 7fc2a9200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc2a40c63c0 0x7fc2a41f40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:08.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:08.967+0000 7fc2a9c00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2a4159450 0x7fc2a415a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39750/0 (socket says 172.21.15.60:39750) 2024-04-14T23:58:08.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:08.967+0000 7fc2a9c00640 1 -- 172.21.15.60:0/3874742005 learned_addr learned my addr 172.21.15.60:0/3874742005 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:58:08.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:08.967+0000 7fc2a3e00640 1 --2- 172.21.15.60:0/3874742005 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2a4152cf0 0x7fc2a41f4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:08.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:08.968+0000 7fc2a9c00640 1 -- 172.21.15.60:0/3874742005 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc2a40c63c0 msgr2=0x7fc2a41f40e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:08.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:08.968+0000 7fc2a9c00640 1 --2- 172.21.15.60:0/3874742005 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc2a40c63c0 0x7fc2a41f40e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:08.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:08.968+0000 7fc2a9c00640 1 -- 172.21.15.60:0/3874742005 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2a4152cf0 msgr2=0x7fc2a41f4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:08.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:08.968+0000 7fc2a9c00640 1 --2- 172.21.15.60:0/3874742005 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2a4152cf0 0x7fc2a41f4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:08.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:08.968+0000 7fc2a9c00640 1 -- 172.21.15.60:0/3874742005 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2a41513b0 con 0x7fc2a4159450 2024-04-14T23:58:08.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:08.968+0000 7fc2a9c00640 1 --2- 172.21.15.60:0/3874742005 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2a4159450 0x7fc2a415a4c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fc29c009d30 tx=0x7fc29c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:08.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:08.968+0000 7fc2a0c00640 1 -- 172.21.15.60:0/3874742005 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc29c018020 con 0x7fc2a4159450 2024-04-14T23:58:08.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:08.968+0000 7fc2a0c00640 1 -- 172.21.15.60:0/3874742005 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc29c014480 con 0x7fc2a4159450 2024-04-14T23:58:08.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:08.968+0000 7fc2a0c00640 1 -- 172.21.15.60:0/3874742005 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc29c014870 con 0x7fc2a4159450 2024-04-14T23:58:08.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:08.968+0000 7fc2abc00640 1 -- 172.21.15.60:0/3874742005 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2a4151690 con 0x7fc2a4159450 2024-04-14T23:58:08.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:08.969+0000 7fc2abc00640 1 -- 172.21.15.60:0/3874742005 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2a41503c0 con 0x7fc2a4159450 2024-04-14T23:58:08.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:08.970+0000 7fc2abc00640 1 -- 172.21.15.60:0/3874742005 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2a4076970 con 0x7fc2a4159450 2024-04-14T23:58:08.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:08.971+0000 7fc2a0c00640 1 -- 172.21.15.60:0/3874742005 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7fc29c005d10 con 0x7fc2a4159450 2024-04-14T23:58:08.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:08.972+0000 7fc2a0c00640 1 --2- 172.21.15.60:0/3874742005 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fc28406b990 0x7fc28406de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:08.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:08.972+0000 7fc2a0c00640 1 -- 172.21.15.60:0/3874742005 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6255+0+0 (secure 0 0 0) 0x7fc29c08b730 con 0x7fc2a4159450 2024-04-14T23:58:08.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:08.972+0000 7fc2a9200640 1 --2- 172.21.15.60:0/3874742005 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fc28406b990 0x7fc28406de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:08.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:08.973+0000 7fc2a9200640 1 --2- 172.21.15.60:0/3874742005 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fc28406b990 0x7fc28406de50 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fc298036000 tx=0x7fc298008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:08.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:08.976+0000 7fc2a0c00640 1 -- 172.21.15.60:0/3874742005 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc29c022030 con 0x7fc2a4159450 2024-04-14T23:58:09.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.174+0000 7fc2abc00640 1 -- 172.21.15.60:0/3874742005 --> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc2a4153480 con 0x7fc28406b990 2024-04-14T23:58:09.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.181+0000 7fc2a0c00640 1 -- 172.21.15.60:0/3874742005 <== mgr.4100 v2:172.21.15.186:6832/714104835 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+255177 (secure 0 0 0) 0x7fc29802f820 con 0x7fc28406b990 2024-04-14T23:58:09.190 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:58:09.192 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-04-14T23:58:09.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.187+0000 7fc2abc00640 1 -- 172.21.15.60:0/3874742005 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fc28406b990 msgr2=0x7fc28406de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:09.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.188+0000 7fc2abc00640 1 --2- 172.21.15.60:0/3874742005 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fc28406b990 0x7fc28406de50 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fc298036000 tx=0x7fc298008030 comp rx=0 tx=0).stop 2024-04-14T23:58:09.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.188+0000 7fc2abc00640 1 -- 172.21.15.60:0/3874742005 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2a4159450 msgr2=0x7fc2a415a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:09.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.188+0000 7fc2abc00640 1 --2- 172.21.15.60:0/3874742005 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2a4159450 0x7fc2a415a4c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fc29c009d30 tx=0x7fc29c00c490 comp rx=0 tx=0).stop 2024-04-14T23:58:09.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.188+0000 7fc2abc00640 1 -- 172.21.15.60:0/3874742005 shutdown_connections 2024-04-14T23:58:09.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.188+0000 7fc2abc00640 1 --2- 172.21.15.60:0/3874742005 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fc28406b990 0x7fc28406de50 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:09.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.188+0000 7fc2abc00640 1 --2- 172.21.15.60:0/3874742005 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2a4159450 0x7fc2a415a4c0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:09.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.188+0000 7fc2abc00640 1 --2- 172.21.15.60:0/3874742005 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2a4152cf0 0x7fc2a41f4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:09.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.188+0000 7fc2abc00640 1 --2- 172.21.15.60:0/3874742005 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc2a40c63c0 0x7fc2a41f40e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:09.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.188+0000 7fc2abc00640 1 -- 172.21.15.60:0/3874742005 >> 172.21.15.60:0/3874742005 conn(0x7fc2a40cc170 msgr2=0x7fc2a40cc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:09.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.188+0000 7fc2abc00640 1 -- 172.21.15.60:0/3874742005 shutdown_connections 2024-04-14T23:58:09.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.188+0000 7fc2abc00640 1 -- 172.21.15.60:0/3874742005 wait complete. 2024-04-14T23:58:09.213 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":325,"stamp":"2024-04-14T23:58:08.314240+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":202752180,"num_objects":92,"num_object_clones":0,"num_object_copies":184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92,"num_whiteouts":0,"num_read":2892,"num_read_kb":545033,"num_write":14569,"num_write_kb":430090,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14479,"ondisk_log_size":14479,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":631756,"kb_used_data":165760,"kb_used_omap":12,"kb_used_meta":465907,"kb_avail":754342964,"statfs":{"total":773094113280,"available":772447195136,"internally_reserved":0,"allocated":169738240,"data_stored":407419304,"data_compressed":34457476,"data_compressed_allocated":44892160,"data_compressed_original":284942336,"omap_allocated":12752,"internal_metadata":477089328},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":567468,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":53,"num_read_kb":40,"num_write":60,"num_write_kb":587,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003437"},"pg_stats":[{"pgid":"3.3f","version":"36'158","reported_seq":2280,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.085799+0000","last_change":"2024-04-14T23:55:41.994501+0000","last_active":"2024-04-14T23:57:39.085799+0000","last_peered":"2024-04-14T23:57:39.085799+0000","last_clean":"2024-04-14T23:57:39.085799+0000","last_became_active":"2024-04-14T23:48:43.222278+0000","last_became_peered":"2024-04-14T23:48:43.222278+0000","last_unstale":"2024-04-14T23:57:39.085799+0000","last_undegraded":"2024-04-14T23:57:39.085799+0000","last_fullsized":"2024-04-14T23:57:39.085799+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T04:04:17.956339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025657449999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":29,"num_read_kb":15,"num_write":158,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"44'142","reported_seq":2269,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.088582+0000","last_change":"2024-04-14T23:55:42.310665+0000","last_active":"2024-04-14T23:57:39.088582+0000","last_peered":"2024-04-14T23:57:39.088582+0000","last_clean":"2024-04-14T23:57:39.088582+0000","last_became_active":"2024-04-14T23:48:43.226426+0000","last_became_peered":"2024-04-14T23:48:43.226426+0000","last_unstale":"2024-04-14T23:57:39.088582+0000","last_undegraded":"2024-04-14T23:57:39.088582+0000","last_fullsized":"2024-04-14T23:57:39.088582+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:58:52.815232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011741033999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":16,"num_write":142,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"37'171","reported_seq":2299,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.087926+0000","last_change":"2024-04-14T23:55:42.315743+0000","last_active":"2024-04-14T23:57:39.087926+0000","last_peered":"2024-04-14T23:57:39.087926+0000","last_clean":"2024-04-14T23:57:39.087926+0000","last_became_active":"2024-04-14T23:48:43.236616+0000","last_became_peered":"2024-04-14T23:48:43.236616+0000","last_unstale":"2024-04-14T23:57:39.087926+0000","last_undegraded":"2024-04-14T23:57:39.087926+0000","last_fullsized":"2024-04-14T23:57:39.087926+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:56:34.635393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01445958,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":38,"num_read_kb":8211,"num_write":164,"num_write_kb":8272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"44'140","reported_seq":2253,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.083881+0000","last_change":"2024-04-14T23:55:42.084275+0000","last_active":"2024-04-14T23:57:39.083881+0000","last_peered":"2024-04-14T23:57:39.083881+0000","last_clean":"2024-04-14T23:57:39.083881+0000","last_became_active":"2024-04-14T23:48:43.242000+0000","last_became_peered":"2024-04-14T23:48:43.242000+0000","last_unstale":"2024-04-14T23:57:39.083881+0000","last_undegraded":"2024-04-14T23:57:39.083881+0000","last_fullsized":"2024-04-14T23:57:39.083881+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:06:54.726051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020739690000000002,"stat_sum":{"num_bytes":193,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":9,"num_write":139,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"35'197","reported_seq":2310,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.086395+0000","last_change":"2024-04-14T23:55:42.308076+0000","last_active":"2024-04-14T23:57:39.086395+0000","last_peered":"2024-04-14T23:57:39.086395+0000","last_clean":"2024-04-14T23:57:39.086395+0000","last_became_active":"2024-04-14T23:49:29.448078+0000","last_became_peered":"2024-04-14T23:49:29.448078+0000","last_unstale":"2024-04-14T23:57:39.086395+0000","last_undegraded":"2024-04-14T23:57:39.086395+0000","last_fullsized":"2024-04-14T23:57:39.086395+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":197,"log_dups_size":0,"ondisk_log_size":197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:09:40.348011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011461469,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":27,"num_read_kb":18,"num_write":196,"num_write_kb":98,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3a","version":"35'170","reported_seq":2294,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.083401+0000","last_change":"2024-04-14T23:55:42.084342+0000","last_active":"2024-04-14T23:57:39.083401+0000","last_peered":"2024-04-14T23:57:39.083401+0000","last_clean":"2024-04-14T23:57:39.083401+0000","last_became_active":"2024-04-14T23:48:43.230405+0000","last_became_peered":"2024-04-14T23:48:43.230405+0000","last_unstale":"2024-04-14T23:57:39.083401+0000","last_undegraded":"2024-04-14T23:57:39.083401+0000","last_fullsized":"2024-04-14T23:57:39.083401+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:40:55.017190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002322771,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":32,"num_read_kb":19,"num_write":170,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"44'198","reported_seq":2327,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.084946+0000","last_change":"2024-04-14T23:55:41.992292+0000","last_active":"2024-04-14T23:57:39.084946+0000","last_peered":"2024-04-14T23:57:39.084946+0000","last_clean":"2024-04-14T23:57:39.084946+0000","last_became_active":"2024-04-14T23:48:43.230784+0000","last_became_peered":"2024-04-14T23:48:43.230784+0000","last_unstale":"2024-04-14T23:57:39.084946+0000","last_undegraded":"2024-04-14T23:57:39.084946+0000","last_fullsized":"2024-04-14T23:57:39.084946+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":198,"log_dups_size":0,"ondisk_log_size":198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T04:49:40.437771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023027080000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":37,"num_read_kb":16,"num_write":198,"num_write_kb":4194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"44'133","reported_seq":2251,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.091139+0000","last_change":"2024-04-14T23:55:42.317071+0000","last_active":"2024-04-14T23:57:39.091139+0000","last_peered":"2024-04-14T23:57:39.091139+0000","last_clean":"2024-04-14T23:57:39.091139+0000","last_became_active":"2024-04-14T23:48:43.235719+0000","last_became_peered":"2024-04-14T23:48:43.235719+0000","last_unstale":"2024-04-14T23:57:39.091139+0000","last_undegraded":"2024-04-14T23:57:39.091139+0000","last_fullsized":"2024-04-14T23:57:39.091139+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:40:21.290823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015616025,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":26,"num_read_kb":10,"num_write":132,"num_write_kb":8256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"44'157","reported_seq":2283,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.085798+0000","last_change":"2024-04-14T23:55:41.994669+0000","last_active":"2024-04-14T23:57:39.085798+0000","last_peered":"2024-04-14T23:57:39.085798+0000","last_clean":"2024-04-14T23:57:39.085798+0000","last_became_active":"2024-04-14T23:48:43.231751+0000","last_became_peered":"2024-04-14T23:48:43.231751+0000","last_unstale":"2024-04-14T23:57:39.085798+0000","last_undegraded":"2024-04-14T23:57:39.085798+0000","last_fullsized":"2024-04-14T23:57:39.085798+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:44:15.830503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025112200000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":4114,"num_write":154,"num_write_kb":8271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"35'164","reported_seq":2283,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.010548+0000","last_change":"2024-04-14T23:55:41.992813+0000","last_active":"2024-04-14T23:57:39.010548+0000","last_peered":"2024-04-14T23:57:39.010548+0000","last_clean":"2024-04-14T23:57:39.010548+0000","last_became_active":"2024-04-14T23:48:43.224592+0000","last_became_peered":"2024-04-14T23:48:43.224592+0000","last_unstale":"2024-04-14T23:57:39.010548+0000","last_undegraded":"2024-04-14T23:57:39.010548+0000","last_fullsized":"2024-04-14T23:57:39.010548+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:36:03.843739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014741629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":27,"num_read_kb":16,"num_write":162,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"44'171","reported_seq":2298,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.010380+0000","last_change":"2024-04-14T23:55:41.991958+0000","last_active":"2024-04-14T23:57:39.010380+0000","last_peered":"2024-04-14T23:57:39.010380+0000","last_clean":"2024-04-14T23:57:39.010380+0000","last_became_active":"2024-04-14T23:48:43.224989+0000","last_became_peered":"2024-04-14T23:48:43.224989+0000","last_unstale":"2024-04-14T23:57:39.010380+0000","last_undegraded":"2024-04-14T23:57:39.010380+0000","last_fullsized":"2024-04-14T23:57:39.010380+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:31:12.485950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001581301,"stat_sum":{"num_bytes":4197091,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":35,"num_read_kb":18,"num_write":170,"num_write_kb":4183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"44'146","reported_seq":2262,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.086409+0000","last_change":"2024-04-14T23:55:42.224637+0000","last_active":"2024-04-14T23:57:39.086409+0000","last_peered":"2024-04-14T23:57:39.086409+0000","last_clean":"2024-04-14T23:57:39.086409+0000","last_became_active":"2024-04-14T23:48:43.225892+0000","last_became_peered":"2024-04-14T23:48:43.225892+0000","last_unstale":"2024-04-14T23:57:39.086409+0000","last_undegraded":"2024-04-14T23:57:39.086409+0000","last_fullsized":"2024-04-14T23:57:39.086409+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":146,"log_dups_size":0,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:30:03.715201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011254640000000001,"stat_sum":{"num_bytes":73,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":24,"num_read_kb":10,"num_write":145,"num_write_kb":8263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"37'146","reported_seq":2267,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.092688+0000","last_change":"2024-04-14T23:55:42.317691+0000","last_active":"2024-04-14T23:57:39.092688+0000","last_peered":"2024-04-14T23:57:39.092688+0000","last_clean":"2024-04-14T23:57:39.092688+0000","last_became_active":"2024-04-14T23:48:43.219614+0000","last_became_peered":"2024-04-14T23:48:43.219614+0000","last_unstale":"2024-04-14T23:57:39.092688+0000","last_undegraded":"2024-04-14T23:57:39.092688+0000","last_fullsized":"2024-04-14T23:57:39.092688+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":146,"log_dups_size":0,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:11:13.830037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012872641000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":30,"num_read_kb":4115,"num_write":142,"num_write_kb":8261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"44'175","reported_seq":2308,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.090824+0000","last_change":"2024-04-14T23:55:42.319365+0000","last_active":"2024-04-14T23:57:39.090824+0000","last_peered":"2024-04-14T23:57:39.090824+0000","last_clean":"2024-04-14T23:57:39.090824+0000","last_became_active":"2024-04-14T23:48:43.239136+0000","last_became_peered":"2024-04-14T23:48:43.239136+0000","last_unstale":"2024-04-14T23:57:39.090824+0000","last_undegraded":"2024-04-14T23:57:39.090824+0000","last_fullsized":"2024-04-14T23:57:39.090824+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:47:21.467652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015761819,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":40,"num_read_kb":19,"num_write":174,"num_write_kb":12372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"44'159","reported_seq":2289,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.090236+0000","last_change":"2024-04-14T23:55:42.308265+0000","last_active":"2024-04-14T23:57:39.090236+0000","last_peered":"2024-04-14T23:57:39.090236+0000","last_clean":"2024-04-14T23:57:39.090236+0000","last_became_active":"2024-04-14T23:48:43.227898+0000","last_became_peered":"2024-04-14T23:48:43.227898+0000","last_unstale":"2024-04-14T23:57:39.090236+0000","last_undegraded":"2024-04-14T23:57:39.090236+0000","last_fullsized":"2024-04-14T23:57:39.090236+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:18:28.787829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011044834999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":37,"num_read_kb":19,"num_write":159,"num_write_kb":4178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"44'184","reported_seq":2317,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.083264+0000","last_change":"2024-04-14T23:55:42.083783+0000","last_active":"2024-04-14T23:57:39.083264+0000","last_peered":"2024-04-14T23:57:39.083264+0000","last_clean":"2024-04-14T23:57:39.083264+0000","last_became_active":"2024-04-14T23:48:43.238714+0000","last_became_peered":"2024-04-14T23:48:43.238714+0000","last_unstale":"2024-04-14T23:57:39.083264+0000","last_undegraded":"2024-04-14T23:57:39.083264+0000","last_fullsized":"2024-04-14T23:57:39.083264+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":184,"log_dups_size":0,"ondisk_log_size":184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:37:44.837525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019546920000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":40,"num_read_kb":19,"num_write":182,"num_write_kb":7459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"44'174","reported_seq":2306,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.083777+0000","last_change":"2024-04-14T23:55:42.085173+0000","last_active":"2024-04-14T23:57:39.083777+0000","last_peered":"2024-04-14T23:57:39.083777+0000","last_clean":"2024-04-14T23:57:39.083777+0000","last_became_active":"2024-04-14T23:48:43.228618+0000","last_became_peered":"2024-04-14T23:48:43.228618+0000","last_unstale":"2024-04-14T23:57:39.083777+0000","last_undegraded":"2024-04-14T23:57:39.083777+0000","last_fullsized":"2024-04-14T23:57:39.083777+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":174,"log_dups_size":0,"ondisk_log_size":174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T09:07:18.121366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002424158,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":40,"num_read_kb":4117,"num_write":170,"num_write_kb":4181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"37'153","reported_seq":2293,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.011857+0000","last_change":"2024-04-14T23:55:42.175749+0000","last_active":"2024-04-14T23:57:39.011857+0000","last_peered":"2024-04-14T23:57:39.011857+0000","last_clean":"2024-04-14T23:57:39.011857+0000","last_became_active":"2024-04-14T23:48:43.232971+0000","last_became_peered":"2024-04-14T23:48:43.232971+0000","last_unstale":"2024-04-14T23:57:39.011857+0000","last_undegraded":"2024-04-14T23:57:39.011857+0000","last_fullsized":"2024-04-14T23:57:39.011857+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:34:34.148700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030089610000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":47,"num_read_kb":26,"num_write":152,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"44'163","reported_seq":2290,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.086169+0000","last_change":"2024-04-14T23:55:41.994445+0000","last_active":"2024-04-14T23:57:39.086169+0000","last_peered":"2024-04-14T23:57:39.086169+0000","last_clean":"2024-04-14T23:57:39.086169+0000","last_became_active":"2024-04-14T23:48:43.232135+0000","last_became_peered":"2024-04-14T23:48:43.232135+0000","last_unstale":"2024-04-14T23:57:39.086169+0000","last_undegraded":"2024-04-14T23:57:39.086169+0000","last_fullsized":"2024-04-14T23:57:39.086169+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":163,"log_dups_size":0,"ondisk_log_size":163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:37:12.273966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029988570000000002,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":35,"num_read_kb":4112,"num_write":159,"num_write_kb":8270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"44'195","reported_seq":2329,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.010441+0000","last_change":"2024-04-14T23:55:41.992691+0000","last_active":"2024-04-14T23:57:39.010441+0000","last_peered":"2024-04-14T23:57:39.010441+0000","last_clean":"2024-04-14T23:57:39.010441+0000","last_became_active":"2024-04-14T23:48:43.226225+0000","last_became_peered":"2024-04-14T23:48:43.226225+0000","last_unstale":"2024-04-14T23:57:39.010441+0000","last_undegraded":"2024-04-14T23:57:39.010441+0000","last_fullsized":"2024-04-14T23:57:39.010441+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":195,"log_dups_size":0,"ondisk_log_size":195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:43:42.017755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019674760000000001,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":42,"num_read_kb":4116,"num_write":192,"num_write_kb":16477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"44'154","reported_seq":2291,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.090622+0000","last_change":"2024-04-14T23:55:42.316046+0000","last_active":"2024-04-14T23:57:39.090622+0000","last_peered":"2024-04-14T23:57:39.090622+0000","last_clean":"2024-04-14T23:57:39.090622+0000","last_became_active":"2024-04-14T23:48:43.230056+0000","last_became_peered":"2024-04-14T23:48:43.230056+0000","last_unstale":"2024-04-14T23:57:39.090622+0000","last_undegraded":"2024-04-14T23:57:39.090622+0000","last_fullsized":"2024-04-14T23:57:39.090622+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:05:52.381327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015336659000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":44,"num_read_kb":22,"num_write":154,"num_write_kb":77,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"44'175","reported_seq":2304,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.083498+0000","last_change":"2024-04-14T23:55:42.084499+0000","last_active":"2024-04-14T23:57:39.083498+0000","last_peered":"2024-04-14T23:57:39.083498+0000","last_clean":"2024-04-14T23:57:39.083498+0000","last_became_active":"2024-04-14T23:48:43.229110+0000","last_became_peered":"2024-04-14T23:48:43.229110+0000","last_unstale":"2024-04-14T23:57:39.083498+0000","last_undegraded":"2024-04-14T23:57:39.083498+0000","last_fullsized":"2024-04-14T23:57:39.083498+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T09:03:22.176338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022905320000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":36,"num_read_kb":17,"num_write":174,"num_write_kb":4183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"44'170","reported_seq":2301,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:45.295592+0000","last_change":"2024-04-14T23:55:42.314003+0000","last_active":"2024-04-14T23:57:45.295592+0000","last_peered":"2024-04-14T23:57:45.295592+0000","last_clean":"2024-04-14T23:57:45.295592+0000","last_became_active":"2024-04-14T23:48:43.233853+0000","last_became_peered":"2024-04-14T23:48:43.233853+0000","last_unstale":"2024-04-14T23:57:45.295592+0000","last_undegraded":"2024-04-14T23:57:45.295592+0000","last_fullsized":"2024-04-14T23:57:45.295592+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T09:33:21.684120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012858206,"stat_sum":{"num_bytes":1378,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":39,"num_read_kb":4117,"num_write":167,"num_write_kb":4180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"44'173","reported_seq":2295,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.011239+0000","last_change":"2024-04-14T23:55:42.175389+0000","last_active":"2024-04-14T23:57:39.011239+0000","last_peered":"2024-04-14T23:57:39.011239+0000","last_clean":"2024-04-14T23:57:39.011239+0000","last_became_active":"2024-04-14T23:49:29.512499+0000","last_became_peered":"2024-04-14T23:49:29.512499+0000","last_unstale":"2024-04-14T23:57:39.011239+0000","last_undegraded":"2024-04-14T23:57:39.011239+0000","last_fullsized":"2024-04-14T23:57:39.011239+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":173,"log_dups_size":0,"ondisk_log_size":173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:04:34.423852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037213820000000001,"stat_sum":{"num_bytes":23,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":36,"num_read_kb":16,"num_write":173,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"37'160","reported_seq":2287,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.093353+0000","last_change":"2024-04-14T23:55:42.319252+0000","last_active":"2024-04-14T23:57:39.093353+0000","last_peered":"2024-04-14T23:57:39.093353+0000","last_clean":"2024-04-14T23:57:39.093353+0000","last_became_active":"2024-04-14T23:48:43.233238+0000","last_became_peered":"2024-04-14T23:48:43.233238+0000","last_unstale":"2024-04-14T23:57:39.093353+0000","last_undegraded":"2024-04-14T23:57:39.093353+0000","last_fullsized":"2024-04-14T23:57:39.093353+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":160,"log_dups_size":0,"ondisk_log_size":160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T23:56:25.118737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012037971,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":18,"num_write":158,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"36'184","reported_seq":2312,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.010657+0000","last_change":"2024-04-14T23:55:41.993037+0000","last_active":"2024-04-14T23:57:39.010657+0000","last_peered":"2024-04-14T23:57:39.010657+0000","last_clean":"2024-04-14T23:57:39.010657+0000","last_became_active":"2024-04-14T23:48:43.222624+0000","last_became_peered":"2024-04-14T23:48:43.222624+0000","last_unstale":"2024-04-14T23:57:39.010657+0000","last_undegraded":"2024-04-14T23:57:39.010657+0000","last_fullsized":"2024-04-14T23:57:39.010657+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":184,"log_dups_size":0,"ondisk_log_size":184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:43:37.661814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001456707,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":19,"num_write":184,"num_write_kb":92,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"44'140","reported_seq":2257,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.086226+0000","last_change":"2024-04-14T23:55:42.309950+0000","last_active":"2024-04-14T23:57:39.086226+0000","last_peered":"2024-04-14T23:57:39.086226+0000","last_clean":"2024-04-14T23:57:39.086226+0000","last_became_active":"2024-04-14T23:48:43.232277+0000","last_became_peered":"2024-04-14T23:48:43.232277+0000","last_unstale":"2024-04-14T23:57:39.086226+0000","last_undegraded":"2024-04-14T23:57:39.086226+0000","last_fullsized":"2024-04-14T23:57:39.086226+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:17:15.623645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011200412,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":138,"num_write_kb":4167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"44'156","reported_seq":2279,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.094007+0000","last_change":"2024-04-14T23:55:42.319096+0000","last_active":"2024-04-14T23:57:39.094007+0000","last_peered":"2024-04-14T23:57:39.094007+0000","last_clean":"2024-04-14T23:57:39.094007+0000","last_became_active":"2024-04-14T23:48:43.229172+0000","last_became_peered":"2024-04-14T23:48:43.229172+0000","last_unstale":"2024-04-14T23:57:39.094007+0000","last_undegraded":"2024-04-14T23:57:39.094007+0000","last_fullsized":"2024-04-14T23:57:39.094007+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:05:34.205391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011889788,"stat_sum":{"num_bytes":482,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":31,"num_read_kb":3287,"num_write":153,"num_write_kb":3350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"36'184","reported_seq":2321,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.088053+0000","last_change":"2024-04-14T23:55:42.317181+0000","last_active":"2024-04-14T23:57:39.088053+0000","last_peered":"2024-04-14T23:57:39.088053+0000","last_clean":"2024-04-14T23:57:39.088053+0000","last_became_active":"2024-04-14T23:49:29.449548+0000","last_became_peered":"2024-04-14T23:49:29.449548+0000","last_unstale":"2024-04-14T23:57:39.088053+0000","last_undegraded":"2024-04-14T23:57:39.088053+0000","last_fullsized":"2024-04-14T23:57:39.088053+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":184,"log_dups_size":0,"ondisk_log_size":184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:07:19.034727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015790789,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":39,"num_read_kb":18,"num_write":184,"num_write_kb":92,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"35'188","reported_seq":2318,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.011493+0000","last_change":"2024-04-14T23:55:42.174354+0000","last_active":"2024-04-14T23:57:39.011493+0000","last_peered":"2024-04-14T23:57:39.011493+0000","last_clean":"2024-04-14T23:57:39.011493+0000","last_became_active":"2024-04-14T23:49:29.512523+0000","last_became_peered":"2024-04-14T23:49:29.512523+0000","last_unstale":"2024-04-14T23:57:39.011493+0000","last_undegraded":"2024-04-14T23:57:39.011493+0000","last_fullsized":"2024-04-14T23:57:39.011493+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":188,"log_dups_size":0,"ondisk_log_size":188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T09:13:33.216335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026059680000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":43,"num_read_kb":22,"num_write":188,"num_write_kb":94,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"44'205","reported_seq":2333,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.011575+0000","last_change":"2024-04-14T23:55:42.175131+0000","last_active":"2024-04-14T23:57:39.011575+0000","last_peered":"2024-04-14T23:57:39.011575+0000","last_clean":"2024-04-14T23:57:39.011575+0000","last_became_active":"2024-04-14T23:48:43.233567+0000","last_became_peered":"2024-04-14T23:48:43.233567+0000","last_unstale":"2024-04-14T23:57:39.011575+0000","last_undegraded":"2024-04-14T23:57:39.011575+0000","last_fullsized":"2024-04-14T23:57:39.011575+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":205,"log_dups_size":0,"ondisk_log_size":205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:43:07.999415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003348115,"stat_sum":{"num_bytes":4726,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":35,"num_read_kb":17,"num_write":205,"num_write_kb":14807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"44'199","reported_seq":2339,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.010040+0000","last_change":"2024-04-14T23:55:41.992153+0000","last_active":"2024-04-14T23:57:39.010040+0000","last_peered":"2024-04-14T23:57:39.010040+0000","last_clean":"2024-04-14T23:57:39.010040+0000","last_became_active":"2024-04-14T23:48:43.230626+0000","last_became_peered":"2024-04-14T23:48:43.230626+0000","last_unstale":"2024-04-14T23:57:39.010040+0000","last_undegraded":"2024-04-14T23:57:39.010040+0000","last_fullsized":"2024-04-14T23:57:39.010040+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":199,"log_dups_size":0,"ondisk_log_size":199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T23:53:31.323623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001517437,"stat_sum":{"num_bytes":4194851,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":47,"num_read_kb":27,"num_write":198,"num_write_kb":12385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"44'158","reported_seq":2278,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.084838+0000","last_change":"2024-04-14T23:55:42.224072+0000","last_active":"2024-04-14T23:57:39.084838+0000","last_peered":"2024-04-14T23:57:39.084838+0000","last_clean":"2024-04-14T23:57:39.084838+0000","last_became_active":"2024-04-14T23:48:43.226773+0000","last_became_peered":"2024-04-14T23:48:43.226773+0000","last_unstale":"2024-04-14T23:57:39.084838+0000","last_undegraded":"2024-04-14T23:57:39.084838+0000","last_fullsized":"2024-04-14T23:57:39.084838+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:04:13.345776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00101596,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":28,"num_read_kb":16,"num_write":158,"num_write_kb":4174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"34'20","reported_seq":90,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.094244+0000","last_change":"2024-04-14T23:48:42.239254+0000","last_active":"2024-04-14T23:57:39.094244+0000","last_peered":"2024-04-14T23:57:39.094244+0000","last_clean":"2024-04-14T23:57:39.094244+0000","last_became_active":"2024-04-14T23:48:42.238847+0000","last_became_peered":"2024-04-14T23:48:42.238847+0000","last_unstale":"2024-04-14T23:57:39.094244+0000","last_undegraded":"2024-04-14T23:57:39.094244+0000","last_fullsized":"2024-04-14T23:57:39.094244+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:46:16.301900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":16386,"num_write":22,"num_write_kb":4233,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"44'168","reported_seq":2296,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.091757+0000","last_change":"2024-04-14T23:55:42.317522+0000","last_active":"2024-04-14T23:57:39.091757+0000","last_peered":"2024-04-14T23:57:39.091757+0000","last_clean":"2024-04-14T23:57:39.091757+0000","last_became_active":"2024-04-14T23:48:43.237391+0000","last_became_peered":"2024-04-14T23:48:43.237391+0000","last_unstale":"2024-04-14T23:57:39.091757+0000","last_undegraded":"2024-04-14T23:57:39.091757+0000","last_fullsized":"2024-04-14T23:57:39.091757+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:01:14.668174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01542551,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":36,"num_read_kb":15,"num_write":168,"num_write_kb":4179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"34'164","reported_seq":240,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:45.069610+0000","last_change":"2024-04-14T23:48:42.219277+0000","last_active":"2024-04-14T23:57:45.069610+0000","last_peered":"2024-04-14T23:57:45.069610+0000","last_clean":"2024-04-14T23:57:45.069610+0000","last_became_active":"2024-04-14T23:48:42.218995+0000","last_became_peered":"2024-04-14T23:48:42.218995+0000","last_unstale":"2024-04-14T23:57:45.069610+0000","last_undegraded":"2024-04-14T23:57:45.069610+0000","last_fullsized":"2024-04-14T23:57:45.069610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T23:51:33.370857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":11,"num_read_kb":40960,"num_write":162,"num_write_kb":12391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"44'24","reported_seq":92,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.090540+0000","last_change":"2024-04-14T23:48:42.236803+0000","last_active":"2024-04-14T23:57:39.090540+0000","last_peered":"2024-04-14T23:57:39.090540+0000","last_clean":"2024-04-14T23:57:39.090540+0000","last_became_active":"2024-04-14T23:48:42.225937+0000","last_became_peered":"2024-04-14T23:48:42.225937+0000","last_unstale":"2024-04-14T23:57:39.090540+0000","last_undegraded":"2024-04-14T23:57:39.090540+0000","last_fullsized":"2024-04-14T23:57:39.090540+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:52:32.609126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":22,"num_write_kb":4105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"35'209","reported_seq":2340,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.091590+0000","last_change":"2024-04-14T23:55:42.316735+0000","last_active":"2024-04-14T23:57:39.091590+0000","last_peered":"2024-04-14T23:57:39.091590+0000","last_clean":"2024-04-14T23:57:39.091590+0000","last_became_active":"2024-04-14T23:48:43.227356+0000","last_became_peered":"2024-04-14T23:48:43.227356+0000","last_unstale":"2024-04-14T23:57:39.091590+0000","last_undegraded":"2024-04-14T23:57:39.091590+0000","last_fullsized":"2024-04-14T23:57:39.091590+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":209,"log_dups_size":0,"ondisk_log_size":209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:55:38.737429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014613774,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":40,"num_read_kb":4118,"num_write":206,"num_write_kb":8293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"45'189","reported_seq":260,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:44.808805+0000","last_change":"2024-04-14T23:48:42.210883+0000","last_active":"2024-04-14T23:57:44.808805+0000","last_peered":"2024-04-14T23:57:44.808805+0000","last_clean":"2024-04-14T23:57:44.808805+0000","last_became_active":"2024-04-14T23:48:42.210607+0000","last_became_peered":"2024-04-14T23:48:42.210607+0000","last_unstale":"2024-04-14T23:57:44.808805+0000","last_undegraded":"2024-04-14T23:57:44.808805+0000","last_fullsized":"2024-04-14T23:57:44.808805+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":189,"log_dups_size":0,"ondisk_log_size":189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:33:32.526223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":186,"num_write_kb":8305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"42'164","reported_seq":2296,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.085956+0000","last_change":"2024-04-14T23:55:42.310398+0000","last_active":"2024-04-14T23:57:39.085956+0000","last_peered":"2024-04-14T23:57:39.085956+0000","last_clean":"2024-04-14T23:57:39.085956+0000","last_became_active":"2024-04-14T23:48:43.217801+0000","last_became_peered":"2024-04-14T23:48:43.217801+0000","last_unstale":"2024-04-14T23:57:39.085956+0000","last_undegraded":"2024-04-14T23:57:39.085956+0000","last_fullsized":"2024-04-14T23:57:39.085956+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:39:59.476515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012539378,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":41,"num_read_kb":4114,"num_write":158,"num_write_kb":4174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"44'5","reported_seq":70,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.010369+0000","last_change":"2024-04-14T23:48:42.218652+0000","last_active":"2024-04-14T23:57:39.010369+0000","last_peered":"2024-04-14T23:57:39.010369+0000","last_clean":"2024-04-14T23:57:39.010369+0000","last_became_active":"2024-04-14T23:48:42.218321+0000","last_became_peered":"2024-04-14T23:48:42.218321+0000","last_unstale":"2024-04-14T23:57:39.010369+0000","last_undegraded":"2024-04-14T23:57:39.010369+0000","last_fullsized":"2024-04-14T23:57:39.010369+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T04:22:06.957747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"44'189","reported_seq":2326,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.083734+0000","last_change":"2024-04-14T23:55:42.084801+0000","last_active":"2024-04-14T23:57:39.083734+0000","last_peered":"2024-04-14T23:57:39.083734+0000","last_clean":"2024-04-14T23:57:39.083734+0000","last_became_active":"2024-04-14T23:48:43.228805+0000","last_became_peered":"2024-04-14T23:48:43.228805+0000","last_unstale":"2024-04-14T23:57:39.083734+0000","last_undegraded":"2024-04-14T23:57:39.083734+0000","last_fullsized":"2024-04-14T23:57:39.083734+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":189,"log_dups_size":0,"ondisk_log_size":189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:59:51.260017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022812650000000002,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":45,"num_read_kb":20,"num_write":189,"num_write_kb":4190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"44'27","reported_seq":95,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.083988+0000","last_change":"2024-04-14T23:48:42.221104+0000","last_active":"2024-04-14T23:57:39.083988+0000","last_peered":"2024-04-14T23:57:39.083988+0000","last_clean":"2024-04-14T23:57:39.083988+0000","last_became_active":"2024-04-14T23:48:42.220767+0000","last_became_peered":"2024-04-14T23:48:42.220767+0000","last_unstale":"2024-04-14T23:57:39.083988+0000","last_undegraded":"2024-04-14T23:57:39.083988+0000","last_fullsized":"2024-04-14T23:57:39.083988+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:23:57.213764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":24,"num_write_kb":4108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"44'162","reported_seq":2287,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.089328+0000","last_change":"2024-04-14T23:55:42.316479+0000","last_active":"2024-04-14T23:57:39.089328+0000","last_peered":"2024-04-14T23:57:39.089328+0000","last_clean":"2024-04-14T23:57:39.089328+0000","last_became_active":"2024-04-14T23:48:43.236168+0000","last_became_peered":"2024-04-14T23:48:43.236168+0000","last_unstale":"2024-04-14T23:57:39.089328+0000","last_undegraded":"2024-04-14T23:57:39.089328+0000","last_fullsized":"2024-04-14T23:57:39.089328+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:10:13.592872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015342639999999999,"stat_sum":{"num_bytes":420,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":33,"num_read_kb":18,"num_write":161,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"44'145","reported_seq":226,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:45.493309+0000","last_change":"2024-04-14T23:48:42.231329+0000","last_active":"2024-04-14T23:57:45.493309+0000","last_peered":"2024-04-14T23:57:45.493309+0000","last_clean":"2024-04-14T23:57:45.493309+0000","last_became_active":"2024-04-14T23:48:42.230969+0000","last_became_peered":"2024-04-14T23:48:42.230969+0000","last_unstale":"2024-04-14T23:57:45.493309+0000","last_undegraded":"2024-04-14T23:57:45.493309+0000","last_fullsized":"2024-04-14T23:57:45.493309+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:06:16.557589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194394,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":30,"num_read_kb":9177,"num_write":144,"num_write_kb":4199,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"44'151","reported_seq":2272,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.086423+0000","last_change":"2024-04-14T23:55:42.224786+0000","last_active":"2024-04-14T23:57:39.086423+0000","last_peered":"2024-04-14T23:57:39.086423+0000","last_clean":"2024-04-14T23:57:39.086423+0000","last_became_active":"2024-04-14T23:48:43.230253+0000","last_became_peered":"2024-04-14T23:48:43.230253+0000","last_unstale":"2024-04-14T23:57:39.086423+0000","last_undegraded":"2024-04-14T23:57:39.086423+0000","last_fullsized":"2024-04-14T23:57:39.086423+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":151,"log_dups_size":0,"ondisk_log_size":151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:04:12.081827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011901889999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":28,"num_read_kb":14,"num_write":150,"num_write_kb":4170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"45'705","reported_seq":801,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:02.669217+0000","last_change":"2024-04-14T23:48:42.237069+0000","last_active":"2024-04-14T23:58:02.669217+0000","last_peered":"2024-04-14T23:58:02.669217+0000","last_clean":"2024-04-14T23:58:02.669217+0000","last_became_active":"2024-04-14T23:48:42.236804+0000","last_became_peered":"2024-04-14T23:48:42.236804+0000","last_unstale":"2024-04-14T23:58:02.669217+0000","last_undegraded":"2024-04-14T23:58:02.669217+0000","last_fullsized":"2024-04-14T23:58:02.669217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":705,"log_dups_size":0,"ondisk_log_size":705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:25:37.318379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583584,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":41866,"num_write":697,"num_write_kb":12860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"44'154","reported_seq":2275,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.085057+0000","last_change":"2024-04-14T23:55:41.994228+0000","last_active":"2024-04-14T23:57:39.085057+0000","last_peered":"2024-04-14T23:57:39.085057+0000","last_clean":"2024-04-14T23:57:39.085057+0000","last_became_active":"2024-04-14T23:48:43.221462+0000","last_became_peered":"2024-04-14T23:48:43.221462+0000","last_unstale":"2024-04-14T23:57:39.085057+0000","last_undegraded":"2024-04-14T23:57:39.085057+0000","last_fullsized":"2024-04-14T23:57:39.085057+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:09:26.659126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033453850000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":154,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"44'99","reported_seq":183,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:02.545555+0000","last_change":"2024-04-14T23:48:42.230518+0000","last_active":"2024-04-14T23:58:02.545555+0000","last_peered":"2024-04-14T23:58:02.545555+0000","last_clean":"2024-04-14T23:58:02.545555+0000","last_became_active":"2024-04-14T23:48:42.229758+0000","last_became_peered":"2024-04-14T23:48:42.229758+0000","last_unstale":"2024-04-14T23:58:02.545555+0000","last_undegraded":"2024-04-14T23:58:02.545555+0000","last_fullsized":"2024-04-14T23:58:02.545555+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:01:41.332553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388642,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":28,"num_read_kb":32808,"num_write":95,"num_write_kb":8238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"44'160","reported_seq":2289,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:02.545168+0000","last_change":"2024-04-14T23:55:42.319479+0000","last_active":"2024-04-14T23:58:02.545168+0000","last_peered":"2024-04-14T23:58:02.545168+0000","last_clean":"2024-04-14T23:58:02.545168+0000","last_became_active":"2024-04-14T23:48:43.229476+0000","last_became_peered":"2024-04-14T23:48:43.229476+0000","last_unstale":"2024-04-14T23:58:02.545168+0000","last_undegraded":"2024-04-14T23:58:02.545168+0000","last_fullsized":"2024-04-14T23:58:02.545168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":160,"log_dups_size":0,"ondisk_log_size":160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:36:00.949626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015192783,"stat_sum":{"num_bytes":4198858,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":37,"num_read_kb":4115,"num_write":155,"num_write_kb":12367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":66,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.084193+0000","last_change":"2024-04-14T23:48:42.225404+0000","last_active":"2024-04-14T23:57:39.084193+0000","last_peered":"2024-04-14T23:57:39.084193+0000","last_clean":"2024-04-14T23:57:39.084193+0000","last_became_active":"2024-04-14T23:48:42.225211+0000","last_became_peered":"2024-04-14T23:48:42.225211+0000","last_unstale":"2024-04-14T23:57:39.084193+0000","last_undegraded":"2024-04-14T23:57:39.084193+0000","last_fullsized":"2024-04-14T23:57:39.084193+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:49:45.282259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"36'172","reported_seq":2308,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.085033+0000","last_change":"2024-04-14T23:55:41.993454+0000","last_active":"2024-04-14T23:57:39.085033+0000","last_peered":"2024-04-14T23:57:39.085033+0000","last_clean":"2024-04-14T23:57:39.085033+0000","last_became_active":"2024-04-14T23:48:43.229297+0000","last_became_peered":"2024-04-14T23:48:43.229297+0000","last_unstale":"2024-04-14T23:57:39.085033+0000","last_undegraded":"2024-04-14T23:57:39.085033+0000","last_fullsized":"2024-04-14T23:57:39.085033+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:21:50.218321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029516669999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":43,"num_read_kb":23,"num_write":172,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"34'2","reported_seq":68,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.010088+0000","last_change":"2024-04-14T23:48:42.220492+0000","last_active":"2024-04-14T23:57:39.010088+0000","last_peered":"2024-04-14T23:57:39.010088+0000","last_clean":"2024-04-14T23:57:39.010088+0000","last_became_active":"2024-04-14T23:48:42.220145+0000","last_became_peered":"2024-04-14T23:48:42.220145+0000","last_unstale":"2024-04-14T23:57:39.010088+0000","last_undegraded":"2024-04-14T23:57:39.010088+0000","last_fullsized":"2024-04-14T23:57:39.010088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:17:45.278993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"44'168","reported_seq":2296,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:02.542597+0000","last_change":"2024-04-14T23:55:41.992426+0000","last_active":"2024-04-14T23:58:02.542597+0000","last_peered":"2024-04-14T23:58:02.542597+0000","last_clean":"2024-04-14T23:58:02.542597+0000","last_became_active":"2024-04-14T23:48:43.235319+0000","last_became_peered":"2024-04-14T23:48:43.235319+0000","last_unstale":"2024-04-14T23:58:02.542597+0000","last_undegraded":"2024-04-14T23:58:02.542597+0000","last_fullsized":"2024-04-14T23:58:02.542597+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:38:08.635546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017199240000000001,"stat_sum":{"num_bytes":428,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":36,"num_read_kb":14,"num_write":163,"num_write_kb":82,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"35'101","reported_seq":199,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:02.579303+0000","last_change":"2024-04-14T23:48:42.231178+0000","last_active":"2024-04-14T23:58:02.579303+0000","last_peered":"2024-04-14T23:58:02.579303+0000","last_clean":"2024-04-14T23:58:02.579303+0000","last_became_active":"2024-04-14T23:48:42.230792+0000","last_became_peered":"2024-04-14T23:48:42.230792+0000","last_unstale":"2024-04-14T23:58:02.579303+0000","last_undegraded":"2024-04-14T23:58:02.579303+0000","last_fullsized":"2024-04-14T23:58:02.579303+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:22:40.744314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":84,"num_read_kb":20558,"num_write":102,"num_write_kb":8239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"44'181","reported_seq":2315,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:02.544562+0000","last_change":"2024-04-14T23:55:41.993119+0000","last_active":"2024-04-14T23:58:02.544562+0000","last_peered":"2024-04-14T23:58:02.544562+0000","last_clean":"2024-04-14T23:58:02.544562+0000","last_became_active":"2024-04-14T23:48:43.228004+0000","last_became_peered":"2024-04-14T23:48:43.228004+0000","last_unstale":"2024-04-14T23:58:02.544562+0000","last_undegraded":"2024-04-14T23:58:02.544562+0000","last_fullsized":"2024-04-14T23:58:02.544562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:09:06.909970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003032402,"stat_sum":{"num_bytes":1647,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":44,"num_read_kb":8209,"num_write":175,"num_write_kb":12374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"34'22","reported_seq":96,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:44.962330+0000","last_change":"2024-04-14T23:48:42.231936+0000","last_active":"2024-04-14T23:57:44.962330+0000","last_peered":"2024-04-14T23:57:44.962330+0000","last_clean":"2024-04-14T23:57:44.962330+0000","last_became_active":"2024-04-14T23:48:42.231612+0000","last_became_peered":"2024-04-14T23:48:42.231612+0000","last_unstale":"2024-04-14T23:57:44.962330+0000","last_undegraded":"2024-04-14T23:57:44.962330+0000","last_fullsized":"2024-04-14T23:57:44.962330+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:45:03.098024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194326,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":12293,"num_write":21,"num_write_kb":4101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"36'127","reported_seq":2246,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.086202+0000","last_change":"2024-04-14T23:55:41.993658+0000","last_active":"2024-04-14T23:57:39.086202+0000","last_peered":"2024-04-14T23:57:39.086202+0000","last_clean":"2024-04-14T23:57:39.086202+0000","last_became_active":"2024-04-14T23:48:43.228665+0000","last_became_peered":"2024-04-14T23:48:43.228665+0000","last_unstale":"2024-04-14T23:57:39.086202+0000","last_undegraded":"2024-04-14T23:57:39.086202+0000","last_fullsized":"2024-04-14T23:57:39.086202+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T23:49:58.331034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003047124,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":27,"num_read_kb":4107,"num_write":124,"num_write_kb":4157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"30'100","reported_seq":176,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:44.881117+0000","last_change":"2024-04-14T23:48:42.239794+0000","last_active":"2024-04-14T23:57:44.881117+0000","last_peered":"2024-04-14T23:57:44.881117+0000","last_clean":"2024-04-14T23:57:44.881117+0000","last_became_active":"2024-04-14T23:48:42.239505+0000","last_became_peered":"2024-04-14T23:48:42.239505+0000","last_unstale":"2024-04-14T23:57:44.881117+0000","last_undegraded":"2024-04-14T23:57:44.881117+0000","last_fullsized":"2024-04-14T23:57:44.881117+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T09:43:16.582708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":24,"num_read_kb":12309,"num_write":100,"num_write_kb":4158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"44'164","reported_seq":2293,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:45.262232+0000","last_change":"2024-04-14T23:55:41.994666+0000","last_active":"2024-04-14T23:57:45.262232+0000","last_peered":"2024-04-14T23:57:45.262232+0000","last_clean":"2024-04-14T23:57:45.262232+0000","last_became_active":"2024-04-14T23:48:43.229949+0000","last_became_peered":"2024-04-14T23:48:43.229949+0000","last_unstale":"2024-04-14T23:57:45.262232+0000","last_undegraded":"2024-04-14T23:57:45.262232+0000","last_fullsized":"2024-04-14T23:57:45.262232+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:26:59.738426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002291033,"stat_sum":{"num_bytes":240,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":37,"num_read_kb":4115,"num_write":159,"num_write_kb":4175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"34'18","reported_seq":93,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:44.346309+0000","last_change":"2024-04-14T23:48:42.227708+0000","last_active":"2024-04-14T23:57:44.346309+0000","last_peered":"2024-04-14T23:57:44.346309+0000","last_clean":"2024-04-14T23:57:44.346309+0000","last_became_active":"2024-04-14T23:48:42.226845+0000","last_became_peered":"2024-04-14T23:48:42.226845+0000","last_unstale":"2024-04-14T23:57:44.346309+0000","last_undegraded":"2024-04-14T23:57:44.346309+0000","last_fullsized":"2024-04-14T23:57:44.346309+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:02:54.468796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194374,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":9,"num_read_kb":16389,"num_write":18,"num_write_kb":4108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"43'183","reported_seq":2304,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.084349+0000","last_change":"2024-04-14T23:55:42.084740+0000","last_active":"2024-04-14T23:57:39.084349+0000","last_peered":"2024-04-14T23:57:39.084349+0000","last_clean":"2024-04-14T23:57:39.084349+0000","last_became_active":"2024-04-14T23:48:43.226717+0000","last_became_peered":"2024-04-14T23:48:43.226717+0000","last_unstale":"2024-04-14T23:57:39.084349+0000","last_undegraded":"2024-04-14T23:57:39.084349+0000","last_fullsized":"2024-04-14T23:57:39.084349+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:44:01.195501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022579950000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":28,"num_read_kb":14,"num_write":180,"num_write_kb":90,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"34'2","reported_seq":68,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.012051+0000","last_change":"2024-04-14T23:48:42.232164+0000","last_active":"2024-04-14T23:57:39.012051+0000","last_peered":"2024-04-14T23:57:39.012051+0000","last_clean":"2024-04-14T23:57:39.012051+0000","last_became_active":"2024-04-14T23:48:42.229785+0000","last_became_peered":"2024-04-14T23:48:42.229785+0000","last_unstale":"2024-04-14T23:57:39.012051+0000","last_undegraded":"2024-04-14T23:57:39.012051+0000","last_fullsized":"2024-04-14T23:57:39.012051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T04:21:37.538735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"36'172","reported_seq":2296,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.085217+0000","last_change":"2024-04-14T23:55:41.994270+0000","last_active":"2024-04-14T23:57:39.085217+0000","last_peered":"2024-04-14T23:57:39.085217+0000","last_clean":"2024-04-14T23:57:39.085217+0000","last_became_active":"2024-04-14T23:48:43.232612+0000","last_became_peered":"2024-04-14T23:48:43.232612+0000","last_unstale":"2024-04-14T23:57:39.085217+0000","last_undegraded":"2024-04-14T23:57:39.085217+0000","last_fullsized":"2024-04-14T23:57:39.085217+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T04:34:43.974518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031586029999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":31,"num_read_kb":17,"num_write":172,"num_write_kb":4181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":222,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.085334+0000","last_change":"2024-04-14T23:48:38.150200+0000","last_active":"2024-04-14T23:57:39.085334+0000","last_peered":"2024-04-14T23:57:39.085334+0000","last_clean":"2024-04-14T23:57:39.085334+0000","last_became_active":"2024-04-14T23:48:38.149301+0000","last_became_peered":"2024-04-14T23:48:38.149301+0000","last_unstale":"2024-04-14T23:57:39.085334+0000","last_undegraded":"2024-04-14T23:57:39.085334+0000","last_fullsized":"2024-04-14T23:57:39.085334+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:37.129663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:37.129663+0000","last_clean_scrub_stamp":"2024-04-14T23:48:37.129663+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T23:58:01.303851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"45'4","reported_seq":70,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.083463+0000","last_change":"2024-04-14T23:48:42.229238+0000","last_active":"2024-04-14T23:57:39.083463+0000","last_peered":"2024-04-14T23:57:39.083463+0000","last_clean":"2024-04-14T23:57:39.083463+0000","last_became_active":"2024-04-14T23:48:42.228994+0000","last_became_peered":"2024-04-14T23:48:42.228994+0000","last_unstale":"2024-04-14T23:57:39.083463+0000","last_undegraded":"2024-04-14T23:57:39.083463+0000","last_fullsized":"2024-04-14T23:57:39.083463+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:47:15.406228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"44'163","reported_seq":2289,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.092269+0000","last_change":"2024-04-14T23:55:42.318283+0000","last_active":"2024-04-14T23:57:39.092269+0000","last_peered":"2024-04-14T23:57:39.092269+0000","last_clean":"2024-04-14T23:57:39.092269+0000","last_became_active":"2024-04-14T23:48:43.223665+0000","last_became_peered":"2024-04-14T23:48:43.223665+0000","last_unstale":"2024-04-14T23:57:39.092269+0000","last_undegraded":"2024-04-14T23:57:39.092269+0000","last_fullsized":"2024-04-14T23:57:39.092269+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":163,"log_dups_size":0,"ondisk_log_size":163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:47:10.121417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012914624,"stat_sum":{"num_bytes":4902,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":33,"num_read_kb":14,"num_write":161,"num_write_kb":4180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"45'89","reported_seq":161,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:03.889202+0000","last_change":"2024-04-14T23:48:42.230404+0000","last_active":"2024-04-14T23:58:03.889202+0000","last_peered":"2024-04-14T23:58:03.889202+0000","last_clean":"2024-04-14T23:58:03.889202+0000","last_became_active":"2024-04-14T23:48:42.230173+0000","last_became_peered":"2024-04-14T23:48:42.230173+0000","last_unstale":"2024-04-14T23:58:03.889202+0000","last_undegraded":"2024-04-14T23:58:03.889202+0000","last_fullsized":"2024-04-14T23:58:03.889202+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:23:45.666680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8660106,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":80,"num_write_kb":8498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"42'152","reported_seq":2272,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.087977+0000","last_change":"2024-04-14T23:55:42.315208+0000","last_active":"2024-04-14T23:57:39.087977+0000","last_peered":"2024-04-14T23:57:39.087977+0000","last_clean":"2024-04-14T23:57:39.087977+0000","last_became_active":"2024-04-14T23:48:43.231025+0000","last_became_peered":"2024-04-14T23:48:43.231025+0000","last_unstale":"2024-04-14T23:57:39.087977+0000","last_undegraded":"2024-04-14T23:57:39.087977+0000","last_fullsized":"2024-04-14T23:57:39.087977+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:22:50.033217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014709888000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":27,"num_read_kb":10,"num_write":150,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'131","reported_seq":328,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.011644+0000","last_change":"2024-04-14T23:48:42.240651+0000","last_active":"2024-04-14T23:57:39.011644+0000","last_peered":"2024-04-14T23:57:39.011644+0000","last_clean":"2024-04-14T23:57:39.011644+0000","last_became_active":"2024-04-14T23:48:42.240351+0000","last_became_peered":"2024-04-14T23:48:42.240351+0000","last_unstale":"2024-04-14T23:57:39.011644+0000","last_undegraded":"2024-04-14T23:57:39.011644+0000","last_fullsized":"2024-04-14T23:57:39.011644+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:03:02.804444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":533490,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":131,"num_read_kb":605,"num_write":126,"num_write_kb":605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"44'152","reported_seq":2274,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.086207+0000","last_change":"2024-04-14T23:55:42.306590+0000","last_active":"2024-04-14T23:57:39.086207+0000","last_peered":"2024-04-14T23:57:39.086207+0000","last_clean":"2024-04-14T23:57:39.086207+0000","last_became_active":"2024-04-14T23:48:43.232120+0000","last_became_peered":"2024-04-14T23:48:43.232120+0000","last_unstale":"2024-04-14T23:57:39.086207+0000","last_undegraded":"2024-04-14T23:57:39.086207+0000","last_fullsized":"2024-04-14T23:57:39.086207+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:07:57.642700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0096232650000000006,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":29,"num_read_kb":17,"num_write":152,"num_write_kb":4171,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"34'156","reported_seq":228,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:44.101825+0000","last_change":"2024-04-14T23:48:42.233743+0000","last_active":"2024-04-14T23:57:44.101825+0000","last_peered":"2024-04-14T23:57:44.101825+0000","last_clean":"2024-04-14T23:57:44.101825+0000","last_became_active":"2024-04-14T23:48:42.233354+0000","last_became_peered":"2024-04-14T23:48:42.233354+0000","last_unstale":"2024-04-14T23:57:44.101825+0000","last_undegraded":"2024-04-14T23:57:44.101825+0000","last_fullsized":"2024-04-14T23:57:44.101825+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:49:23.045968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":154,"num_write_kb":8284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"44'183","reported_seq":2314,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.085863+0000","last_change":"2024-04-14T23:55:42.224478+0000","last_active":"2024-04-14T23:57:39.085863+0000","last_peered":"2024-04-14T23:57:39.085863+0000","last_clean":"2024-04-14T23:57:39.085863+0000","last_became_active":"2024-04-14T23:48:43.232119+0000","last_became_peered":"2024-04-14T23:48:43.232119+0000","last_unstale":"2024-04-14T23:57:39.085863+0000","last_undegraded":"2024-04-14T23:57:39.085863+0000","last_fullsized":"2024-04-14T23:57:39.085863+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:37:54.631325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012271420000000001,"stat_sum":{"num_bytes":192,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":38,"num_read_kb":16,"num_write":183,"num_write_kb":92,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"45'3","reported_seq":69,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.088128+0000","last_change":"2024-04-14T23:48:42.242249+0000","last_active":"2024-04-14T23:57:39.088128+0000","last_peered":"2024-04-14T23:57:39.088128+0000","last_clean":"2024-04-14T23:57:39.088128+0000","last_became_active":"2024-04-14T23:48:42.242039+0000","last_became_peered":"2024-04-14T23:48:42.242039+0000","last_unstale":"2024-04-14T23:57:39.088128+0000","last_undegraded":"2024-04-14T23:57:39.088128+0000","last_fullsized":"2024-04-14T23:57:39.088128+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:46:16.301900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"44'162","reported_seq":2291,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.084067+0000","last_change":"2024-04-14T23:55:42.085024+0000","last_active":"2024-04-14T23:57:39.084067+0000","last_peered":"2024-04-14T23:57:39.084067+0000","last_clean":"2024-04-14T23:57:39.084067+0000","last_became_active":"2024-04-14T23:48:43.241776+0000","last_became_peered":"2024-04-14T23:48:43.241776+0000","last_unstale":"2024-04-14T23:57:39.084067+0000","last_undegraded":"2024-04-14T23:57:39.084067+0000","last_fullsized":"2024-04-14T23:57:39.084067+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:05:29.929626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024520940000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":36,"num_read_kb":17,"num_write":162,"num_write_kb":8271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"45'29","reported_seq":98,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:45.314264+0000","last_change":"2024-04-14T23:48:42.232537+0000","last_active":"2024-04-14T23:57:45.314264+0000","last_peered":"2024-04-14T23:57:45.314264+0000","last_clean":"2024-04-14T23:57:45.314264+0000","last_became_active":"2024-04-14T23:48:42.232325+0000","last_became_peered":"2024-04-14T23:48:42.232325+0000","last_unstale":"2024-04-14T23:57:45.314264+0000","last_undegraded":"2024-04-14T23:57:45.314264+0000","last_fullsized":"2024-04-14T23:57:45.314264+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:34:37.448301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":25,"num_write_kb":4108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"44'166","reported_seq":2288,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.086381+0000","last_change":"2024-04-14T23:55:42.308941+0000","last_active":"2024-04-14T23:57:39.086381+0000","last_peered":"2024-04-14T23:57:39.086381+0000","last_clean":"2024-04-14T23:57:39.086381+0000","last_became_active":"2024-04-14T23:48:43.234150+0000","last_became_peered":"2024-04-14T23:48:43.234150+0000","last_unstale":"2024-04-14T23:57:39.086381+0000","last_undegraded":"2024-04-14T23:57:39.086381+0000","last_fullsized":"2024-04-14T23:57:39.086381+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":166,"log_dups_size":0,"ondisk_log_size":166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:55:59.231275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011050757,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":29,"num_read_kb":14,"num_write":166,"num_write_kb":4178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"45'182","reported_seq":254,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:02.552170+0000","last_change":"2024-04-14T23:48:42.230014+0000","last_active":"2024-04-14T23:58:02.552170+0000","last_peered":"2024-04-14T23:58:02.552170+0000","last_clean":"2024-04-14T23:58:02.552170+0000","last_became_active":"2024-04-14T23:48:42.229712+0000","last_became_peered":"2024-04-14T23:48:42.229712+0000","last_unstale":"2024-04-14T23:58:02.552170+0000","last_undegraded":"2024-04-14T23:58:02.552170+0000","last_fullsized":"2024-04-14T23:58:02.552170+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":182,"log_dups_size":0,"ondisk_log_size":182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:54:20.595391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":142,"num_write":414,"num_write_kb":617,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"44'166","reported_seq":2297,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.094045+0000","last_change":"2024-04-14T23:55:42.320181+0000","last_active":"2024-04-14T23:57:39.094045+0000","last_peered":"2024-04-14T23:57:39.094045+0000","last_clean":"2024-04-14T23:57:39.094045+0000","last_became_active":"2024-04-14T23:48:43.228739+0000","last_became_peered":"2024-04-14T23:48:43.228739+0000","last_unstale":"2024-04-14T23:57:39.094045+0000","last_undegraded":"2024-04-14T23:57:39.094045+0000","last_fullsized":"2024-04-14T23:57:39.094045+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":166,"log_dups_size":0,"ondisk_log_size":166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:27:33.331878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012444432,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":38,"num_read_kb":16,"num_write":166,"num_write_kb":4178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"45'208","reported_seq":285,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:02.646576+0000","last_change":"2024-04-14T23:48:42.220035+0000","last_active":"2024-04-14T23:58:02.646576+0000","last_peered":"2024-04-14T23:58:02.646576+0000","last_clean":"2024-04-14T23:58:02.646576+0000","last_became_active":"2024-04-14T23:48:42.219750+0000","last_became_peered":"2024-04-14T23:48:42.219750+0000","last_unstale":"2024-04-14T23:58:02.646576+0000","last_undegraded":"2024-04-14T23:58:02.646576+0000","last_fullsized":"2024-04-14T23:58:02.646576+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:00:58.928166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":25,"num_read_kb":6094,"num_write":202,"num_write_kb":4203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"44'176","reported_seq":2313,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.091734+0000","last_change":"2024-04-14T23:55:42.319020+0000","last_active":"2024-04-14T23:57:39.091734+0000","last_peered":"2024-04-14T23:57:39.091734+0000","last_clean":"2024-04-14T23:57:39.091734+0000","last_became_active":"2024-04-14T23:48:43.232730+0000","last_became_peered":"2024-04-14T23:48:43.232730+0000","last_unstale":"2024-04-14T23:57:39.091734+0000","last_undegraded":"2024-04-14T23:57:39.091734+0000","last_fullsized":"2024-04-14T23:57:39.091734+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":176,"log_dups_size":0,"ondisk_log_size":176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:38:48.078009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014562113999999999,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":45,"num_read_kb":26,"num_write":171,"num_write_kb":8277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'28","reported_seq":97,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.084866+0000","last_change":"2024-04-14T23:48:42.232087+0000","last_active":"2024-04-14T23:57:39.084866+0000","last_peered":"2024-04-14T23:57:39.084866+0000","last_clean":"2024-04-14T23:57:39.084866+0000","last_became_active":"2024-04-14T23:48:42.231796+0000","last_became_peered":"2024-04-14T23:48:42.231796+0000","last_unstale":"2024-04-14T23:57:39.084866+0000","last_undegraded":"2024-04-14T23:57:39.084866+0000","last_fullsized":"2024-04-14T23:57:39.084866+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:14:35.050093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":27,"num_write_kb":4121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"43'183","reported_seq":2313,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.085628+0000","last_change":"2024-04-14T23:55:42.224366+0000","last_active":"2024-04-14T23:57:39.085628+0000","last_peered":"2024-04-14T23:57:39.085628+0000","last_clean":"2024-04-14T23:57:39.085628+0000","last_became_active":"2024-04-14T23:48:43.232594+0000","last_became_peered":"2024-04-14T23:48:43.232594+0000","last_unstale":"2024-04-14T23:57:39.085628+0000","last_undegraded":"2024-04-14T23:57:39.085628+0000","last_fullsized":"2024-04-14T23:57:39.085628+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T09:41:55.211912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001144237,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":37,"num_read_kb":20,"num_write":182,"num_write_kb":91,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"44'240","reported_seq":308,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.087346+0000","last_change":"2024-04-14T23:48:42.231237+0000","last_active":"2024-04-14T23:57:39.087346+0000","last_peered":"2024-04-14T23:57:39.087346+0000","last_clean":"2024-04-14T23:57:39.087346+0000","last_became_active":"2024-04-14T23:48:42.230902+0000","last_became_peered":"2024-04-14T23:48:42.230902+0000","last_unstale":"2024-04-14T23:57:39.087346+0000","last_undegraded":"2024-04-14T23:57:39.087346+0000","last_fullsized":"2024-04-14T23:57:39.087346+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":240,"log_dups_size":0,"ondisk_log_size":240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:28:37.228039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":238,"num_write_kb":4263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"44'168","reported_seq":2294,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.092200+0000","last_change":"2024-04-14T23:55:42.316834+0000","last_active":"2024-04-14T23:57:39.092200+0000","last_peered":"2024-04-14T23:57:39.092200+0000","last_clean":"2024-04-14T23:57:39.092200+0000","last_became_active":"2024-04-14T23:48:43.232747+0000","last_became_peered":"2024-04-14T23:48:43.232747+0000","last_unstale":"2024-04-14T23:57:39.092200+0000","last_undegraded":"2024-04-14T23:57:39.092200+0000","last_fullsized":"2024-04-14T23:57:39.092200+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:13:25.640712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014090612000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":33,"num_read_kb":16,"num_write":168,"num_write_kb":8274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"34'24","reported_seq":94,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:45.019436+0000","last_change":"2024-04-14T23:48:42.240496+0000","last_active":"2024-04-14T23:57:45.019436+0000","last_peered":"2024-04-14T23:57:45.019436+0000","last_clean":"2024-04-14T23:57:45.019436+0000","last_became_active":"2024-04-14T23:48:42.238524+0000","last_became_peered":"2024-04-14T23:48:42.238524+0000","last_unstale":"2024-04-14T23:57:45.019436+0000","last_undegraded":"2024-04-14T23:57:45.019436+0000","last_fullsized":"2024-04-14T23:57:45.019436+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:29:35.218759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":23,"num_write_kb":4102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"44'163","reported_seq":2291,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:45.295617+0000","last_change":"2024-04-14T23:55:42.318594+0000","last_active":"2024-04-14T23:57:45.295617+0000","last_peered":"2024-04-14T23:57:45.295617+0000","last_clean":"2024-04-14T23:57:45.295617+0000","last_became_active":"2024-04-14T23:48:43.233651+0000","last_became_peered":"2024-04-14T23:48:43.233651+0000","last_unstale":"2024-04-14T23:57:45.295617+0000","last_undegraded":"2024-04-14T23:57:45.295617+0000","last_fullsized":"2024-04-14T23:57:45.295617+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":163,"log_dups_size":0,"ondisk_log_size":163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:17:15.414893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015058134000000001,"stat_sum":{"num_bytes":8390104,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":35,"num_read_kb":13,"num_write":163,"num_write_kb":8274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"44'70","reported_seq":148,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:45.105836+0000","last_change":"2024-04-14T23:48:42.218897+0000","last_active":"2024-04-14T23:57:45.105836+0000","last_peered":"2024-04-14T23:57:45.105836+0000","last_clean":"2024-04-14T23:57:45.105836+0000","last_became_active":"2024-04-14T23:48:42.218580+0000","last_became_peered":"2024-04-14T23:48:42.218580+0000","last_unstale":"2024-04-14T23:57:45.105836+0000","last_undegraded":"2024-04-14T23:57:45.105836+0000","last_fullsized":"2024-04-14T23:57:45.105836+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:26:44.477638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":28682,"num_write":62,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"35'166","reported_seq":2282,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.010030+0000","last_change":"2024-04-14T23:55:41.991867+0000","last_active":"2024-04-14T23:57:39.010030+0000","last_peered":"2024-04-14T23:57:39.010030+0000","last_clean":"2024-04-14T23:57:39.010030+0000","last_became_active":"2024-04-14T23:48:43.219601+0000","last_became_peered":"2024-04-14T23:48:43.219601+0000","last_unstale":"2024-04-14T23:57:39.010030+0000","last_undegraded":"2024-04-14T23:57:39.010030+0000","last_fullsized":"2024-04-14T23:57:39.010030+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":166,"log_dups_size":0,"ondisk_log_size":166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:23:08.643833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001540055,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":23,"num_read_kb":13,"num_write":166,"num_write_kb":83,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"44'151","reported_seq":2286,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.083755+0000","last_change":"2024-04-14T23:55:42.084976+0000","last_active":"2024-04-14T23:57:39.083755+0000","last_peered":"2024-04-14T23:57:39.083755+0000","last_clean":"2024-04-14T23:57:39.083755+0000","last_became_active":"2024-04-14T23:48:43.221842+0000","last_became_peered":"2024-04-14T23:48:43.221842+0000","last_unstale":"2024-04-14T23:57:39.083755+0000","last_undegraded":"2024-04-14T23:57:39.083755+0000","last_fullsized":"2024-04-14T23:57:39.083755+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":151,"log_dups_size":0,"ondisk_log_size":151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:58:33.633046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024436879999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":43,"num_read_kb":23,"num_write":150,"num_write_kb":4170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"33'62","reported_seq":137,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:44.931184+0000","last_change":"2024-04-14T23:48:42.221314+0000","last_active":"2024-04-14T23:57:44.931184+0000","last_peered":"2024-04-14T23:57:44.931184+0000","last_clean":"2024-04-14T23:57:44.931184+0000","last_became_active":"2024-04-14T23:48:42.221028+0000","last_became_peered":"2024-04-14T23:48:42.221028+0000","last_unstale":"2024-04-14T23:57:44.931184+0000","last_undegraded":"2024-04-14T23:57:44.931184+0000","last_fullsized":"2024-04-14T23:57:44.931184+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T09:58:18.195400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":24,"num_read_kb":12309,"num_write":62,"num_write_kb":4128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"44'165","reported_seq":2288,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.083812+0000","last_change":"2024-04-14T23:55:42.084580+0000","last_active":"2024-04-14T23:57:39.083812+0000","last_peered":"2024-04-14T23:57:39.083812+0000","last_clean":"2024-04-14T23:57:39.083812+0000","last_became_active":"2024-04-14T23:48:43.224312+0000","last_became_peered":"2024-04-14T23:48:43.224312+0000","last_unstale":"2024-04-14T23:57:39.083812+0000","last_undegraded":"2024-04-14T23:57:39.083812+0000","last_fullsized":"2024-04-14T23:57:39.083812+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:55:12.275566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00232957,"stat_sum":{"num_bytes":4674,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":31,"num_read_kb":13,"num_write":165,"num_write_kb":4182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"44'668","reported_seq":753,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:44.946209+0000","last_change":"2024-04-14T23:48:42.221000+0000","last_active":"2024-04-14T23:57:44.946209+0000","last_peered":"2024-04-14T23:57:44.946209+0000","last_clean":"2024-04-14T23:57:44.946209+0000","last_became_active":"2024-04-14T23:48:42.219813+0000","last_became_peered":"2024-04-14T23:48:42.219813+0000","last_unstale":"2024-04-14T23:57:44.946209+0000","last_undegraded":"2024-04-14T23:57:44.946209+0000","last_fullsized":"2024-04-14T23:57:44.946209+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":668,"log_dups_size":0,"ondisk_log_size":668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:00:38.107953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":44198,"num_write":659,"num_write_kb":12787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"37'165","reported_seq":2293,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.087948+0000","last_change":"2024-04-14T23:55:42.313777+0000","last_active":"2024-04-14T23:57:39.087948+0000","last_peered":"2024-04-14T23:57:39.087948+0000","last_clean":"2024-04-14T23:57:39.087948+0000","last_became_active":"2024-04-14T23:48:43.236595+0000","last_became_peered":"2024-04-14T23:48:43.236595+0000","last_unstale":"2024-04-14T23:57:39.087948+0000","last_undegraded":"2024-04-14T23:57:39.087948+0000","last_fullsized":"2024-04-14T23:57:39.087948+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:13:32.883032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013391734000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":19,"num_write":164,"num_write_kb":82,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"44'47","reported_seq":118,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.094119+0000","last_change":"2024-04-14T23:48:42.239255+0000","last_active":"2024-04-14T23:57:39.094119+0000","last_peered":"2024-04-14T23:57:39.094119+0000","last_clean":"2024-04-14T23:57:39.094119+0000","last_became_active":"2024-04-14T23:48:42.238892+0000","last_became_peered":"2024-04-14T23:48:42.238892+0000","last_unstale":"2024-04-14T23:57:39.094119+0000","last_undegraded":"2024-04-14T23:57:39.094119+0000","last_fullsized":"2024-04-14T23:57:39.094119+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:44:39.632155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":16384,"num_write":44,"num_write_kb":4126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"44'175","reported_seq":2298,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.092968+0000","last_change":"2024-04-14T23:55:42.319487+0000","last_active":"2024-04-14T23:57:39.092968+0000","last_peered":"2024-04-14T23:57:39.092968+0000","last_clean":"2024-04-14T23:57:39.092968+0000","last_became_active":"2024-04-14T23:48:43.222707+0000","last_became_peered":"2024-04-14T23:48:43.222707+0000","last_unstale":"2024-04-14T23:57:39.092968+0000","last_undegraded":"2024-04-14T23:57:39.092968+0000","last_fullsized":"2024-04-14T23:57:39.092968+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:47:16.501806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011942019,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":30,"num_read_kb":12,"num_write":175,"num_write_kb":4183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"45'76","reported_seq":157,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:46.454133+0000","last_change":"2024-04-14T23:48:42.226563+0000","last_active":"2024-04-14T23:57:46.454133+0000","last_peered":"2024-04-14T23:57:46.454133+0000","last_clean":"2024-04-14T23:57:46.454133+0000","last_became_active":"2024-04-14T23:48:42.226238+0000","last_became_peered":"2024-04-14T23:48:42.226238+0000","last_unstale":"2024-04-14T23:57:46.454133+0000","last_undegraded":"2024-04-14T23:57:46.454133+0000","last_fullsized":"2024-04-14T23:57:46.454133+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:01:36.516898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":25,"num_read_kb":20500,"num_write":71,"num_write_kb":4136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":50360410,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":2223,"num_read_kb":65817,"num_write":10624,"num_write_kb":276412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":100835328,"data_stored":100720820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10722,"ondisk_log_size":10722,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":151801402,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":553,"num_read_kb":479116,"num_write":3786,"num_write_kb":151262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":64675840,"data_stored":303602804,"data_compressed":34392348,"data_compressed_allocated":44802048,"data_compressed_original":283795456,"omap_allocated":0,"internal_metadata":0},"log_size":3640,"ondisk_log_size":3640,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65128,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574965,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":88644,"kb_used_data":25532,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94283196,"statfs":{"total":96636764160,"available":96545992704,"internally_reserved":0,"allocated":26144768,"data_stored":72411425,"data_compressed":6887616,"data_compressed_allocated":9027584,"data_compressed_original":55586816,"omap_allocated":1590,"internal_metadata":64616906},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574966,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77724,"kb_used_data":32660,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94294116,"statfs":{"total":96636764160,"available":96557174784,"internally_reserved":0,"allocated":33443840,"data_stored":71547043,"data_compressed":5492002,"data_compressed_allocated":7053312,"data_compressed_original":45449216,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574965,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81436,"kb_used_data":14164,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94290404,"statfs":{"total":96636764160,"available":96553373696,"internally_reserved":0,"allocated":14503936,"data_stored":38005896,"data_compressed":3110985,"data_compressed_allocated":4100096,"data_compressed_original":27897856,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574966,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84544,"kb_used_data":17268,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94287296,"statfs":{"total":96636764160,"available":96550191104,"internally_reserved":0,"allocated":17682432,"data_stored":54774132,"data_compressed":5258839,"data_compressed_allocated":6774784,"data_compressed_original":44175360,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574964,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":87932,"kb_used_data":20656,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94283908,"statfs":{"total":96636764160,"available":96546721792,"internally_reserved":0,"allocated":21151744,"data_stored":46919348,"data_compressed":3819253,"data_compressed_allocated":4960256,"data_compressed_original":31031296,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574965,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51228,"kb_used_data":7568,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94320612,"statfs":{"total":96636764160,"available":96584306688,"internally_reserved":0,"allocated":7749632,"data_stored":17554823,"data_compressed":1445694,"data_compressed_allocated":2031616,"data_compressed_original":12120064,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574965,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62352,"kb_used_data":17288,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94309488,"statfs":{"total":96636764160,"available":96572915712,"internally_reserved":0,"allocated":17702912,"data_stored":50842674,"data_compressed":4902040,"data_compressed_allocated":6463488,"data_compressed_original":39886848,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574964,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":97896,"kb_used_data":30624,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94273944,"statfs":{"total":96636764160,"available":96536518656,"internally_reserved":0,"allocated":31358976,"data_stored":55363963,"data_compressed":3541047,"data_compressed_allocated":4481024,"data_compressed_original":28794880,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32564,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32564,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5586944,"data_stored":29360766,"data_compressed":3508483,"data_compressed_allocated":4435968,"data_compressed_original":28221440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8802304,"data_stored":42214698,"data_compressed":4902040,"data_compressed_allocated":6463488,"data_compressed_original":39886848,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3031040,"data_stored":13116402,"data_compressed":1445694,"data_compressed_allocated":2031616,"data_compressed_original":12120064,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8028160,"data_stored":34088594,"data_compressed":3819253,"data_compressed_allocated":4960256,"data_compressed_original":31031296,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8749056,"data_stored":46137470,"data_compressed":5258839,"data_compressed_allocated":6774784,"data_compressed_original":44175360,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":5566464,"data_stored":29360218,"data_compressed":3110985,"data_compressed_allocated":4100096,"data_compressed_original":27897856,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":11943936,"data_stored":50332320,"data_compressed":5492002,"data_compressed_allocated":7053312,"data_compressed_original":45449216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12967936,"data_stored":58992336,"data_compressed":6855052,"data_compressed_allocated":8982528,"data_compressed_original":55013376,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":25194496,"data_stored":25173461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8388608,"data_stored":8388608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4206592,"data_stored":4199053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12611584,"data_stored":12591386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8421376,"data_stored":8397294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8425472,"data_stored":8406310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20987904,"data_stored":20975355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12599296,"data_stored":12589353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-14T23:58:09.218 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-14T23:58:09.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.471+0000 7fb1ebe00640 1 Processor -- start 2024-04-14T23:58:09.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.471+0000 7fb1ebe00640 1 -- start start 2024-04-14T23:58:09.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.472+0000 7fb1ebe00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb1ec15c8a0 0x7fb1ec072390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:09.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.473+0000 7fb1ebe00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb1ec0c6950 0x7fb1ec1698f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:09.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.473+0000 7fb1ebe00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1ec071bf0 con 0x7fb1ec0c86d0 2024-04-14T23:58:09.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.473+0000 7fb1ebe00640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fb1ec15cc80 con 0x7fb1ec0c6950 2024-04-14T23:58:09.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.473+0000 7fb1ebe00640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fb1ec1632a0 con 0x7fb1ec15c8a0 2024-04-14T23:58:09.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.473+0000 7fb1eb400640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb1ec0c6950 0x7fb1ec1698f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:09.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.473+0000 7fb1eb400640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb1ec0c6950 0x7fb1ec1698f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39762/0 (socket says 172.21.15.60:39762) 2024-04-14T23:58:09.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.473+0000 7fb1eb400640 1 -- 172.21.15.60:0/2785650279 learned_addr learned my addr 172.21.15.60:0/2785650279 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:58:09.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.473+0000 7fb1eaa00640 1 --2- 172.21.15.60:0/2785650279 >> v2:172.21.15.186:3300/0 conn(0x7fb1ec15c8a0 0x7fb1ec072390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:09.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.474+0000 7fb1eaa00640 1 -- 172.21.15.60:0/2785650279 >> v2:172.21.15.60:3300/0 conn(0x7fb1ec0c6950 msgr2=0x7fb1ec1698f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:58:09.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.474+0000 7fb1eaa00640 1 --2- 172.21.15.60:0/2785650279 >> v2:172.21.15.60:3300/0 conn(0x7fb1ec0c6950 0x7fb1ec1698f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:09.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.474+0000 7fb1eaa00640 1 -- 172.21.15.60:0/2785650279 >> v1:172.21.15.60:6789/0 conn(0x7fb1ec0c86d0 legacy=0x7fb1ec1625c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:09.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.474+0000 7fb1eaa00640 1 -- 172.21.15.60:0/2785650279 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1ec0c6fb0 con 0x7fb1ec15c8a0 2024-04-14T23:58:09.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.474+0000 7fb1eb400640 1 --2- 172.21.15.60:0/2785650279 >> v2:172.21.15.60:3300/0 conn(0x7fb1ec0c6950 0x7fb1ec1698f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:58:09.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.474+0000 7fb1e9600640 1 -- 172.21.15.60:0/2785650279 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2744210424 0 0) 0x7fb1d8001110 con 0x7fb1ec0c86d0 2024-04-14T23:58:09.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.475+0000 7fb1eaa00640 1 --2- 172.21.15.60:0/2785650279 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1ec15c8a0 0x7fb1ec072390 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fb1ec0c7630 tx=0x7fb1dc02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=574261aab12c1c89 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:09.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.475+0000 7fb1e9600640 1 -- 172.21.15.60:0/2785650279 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1dc035020 con 0x7fb1ec15c8a0 2024-04-14T23:58:09.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.475+0000 7fb1e9600640 1 -- 172.21.15.60:0/2785650279 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1dc031830 con 0x7fb1ec15c8a0 2024-04-14T23:58:09.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.476+0000 7fb1e9600640 1 -- 172.21.15.60:0/2785650279 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1dc031c00 con 0x7fb1ec15c8a0 2024-04-14T23:58:09.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.476+0000 7fb1ebe00640 1 -- 172.21.15.60:0/2785650279 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1ec15c8a0 msgr2=0x7fb1ec072390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:09.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.476+0000 7fb1ebe00640 1 --2- 172.21.15.60:0/2785650279 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1ec15c8a0 0x7fb1ec072390 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fb1ec0c7630 tx=0x7fb1dc02b8d0 comp rx=0 tx=0).stop 2024-04-14T23:58:09.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.476+0000 7fb1ebe00640 1 -- 172.21.15.60:0/2785650279 shutdown_connections 2024-04-14T23:58:09.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.476+0000 7fb1ebe00640 1 --2- 172.21.15.60:0/2785650279 >> v2:172.21.15.60:3300/0 conn(0x7fb1ec0c6950 0x7fb1ec1698f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:09.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.476+0000 7fb1ebe00640 1 --2- 172.21.15.60:0/2785650279 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1ec15c8a0 0x7fb1ec072390 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:09.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.476+0000 7fb1ebe00640 1 -- 172.21.15.60:0/2785650279 >> 172.21.15.60:0/2785650279 conn(0x7fb1ec0c8b50 msgr2=0x7fb1ec0c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:09.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.476+0000 7fb1ebe00640 1 -- 172.21.15.60:0/2785650279 shutdown_connections 2024-04-14T23:58:09.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.476+0000 7fb1ebe00640 1 -- 172.21.15.60:0/2785650279 wait complete. 2024-04-14T23:58:09.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.478+0000 7fb1ebe00640 1 Processor -- start 2024-04-14T23:58:09.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.478+0000 7fb1ebe00640 1 -- start start 2024-04-14T23:58:09.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.479+0000 7fb1ebe00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb1ec0c6950 0x7fb1ec1f5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:09.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.479+0000 7fb1ebe00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb1ec0c86d0 0x7fb1ec1e88c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:09.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.479+0000 7fb1ebe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1ec15c8a0 0x7fb1ec1e9d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:09.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.479+0000 7fb1ebe00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb1ec0728d0 con 0x7fb1ec0c86d0 2024-04-14T23:58:09.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.479+0000 7fb1ebe00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb1ec071e70 con 0x7fb1ec15c8a0 2024-04-14T23:58:09.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.479+0000 7fb1ebe00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fb1ec06b1c0 con 0x7fb1ec0c6950 2024-04-14T23:58:09.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.480+0000 7fb1eaa00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb1ec0c6950 0x7fb1ec1f5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:09.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.480+0000 7fb1ea000640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb1ec0c86d0 0x7fb1ec1e88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:09.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.480+0000 7fb1eaa00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb1ec0c6950 0x7fb1ec1f5650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.60:51136/0 (socket says 172.21.15.60:51136) 2024-04-14T23:58:09.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.480+0000 7fb1eaa00640 1 -- 172.21.15.60:0/454399071 learned_addr learned my addr 172.21.15.60:0/454399071 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:58:09.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.480+0000 7fb1eb400640 1 --2- 172.21.15.60:0/454399071 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1ec15c8a0 0x7fb1ec1e9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:09.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.480+0000 7fb1ea000640 1 -- 172.21.15.60:0/454399071 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb1ec0c6950 msgr2=0x7fb1ec1f5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:09.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.480+0000 7fb1ea000640 1 --2- 172.21.15.60:0/454399071 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb1ec0c6950 0x7fb1ec1f5650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:09.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.480+0000 7fb1ea000640 1 -- 172.21.15.60:0/454399071 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1ec15c8a0 msgr2=0x7fb1ec1e9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:09.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.480+0000 7fb1ea000640 1 --2- 172.21.15.60:0/454399071 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1ec15c8a0 0x7fb1ec1e9d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:09.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.480+0000 7fb1ea000640 1 -- 172.21.15.60:0/454399071 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1ec1618e0 con 0x7fb1ec0c86d0 2024-04-14T23:58:09.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.481+0000 7fb1ea000640 1 --2- 172.21.15.60:0/454399071 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb1ec0c86d0 0x7fb1ec1e88c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fb1d800c750 tx=0x7fb1d800cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:09.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.481+0000 7fb1e2a00640 1 -- 172.21.15.60:0/454399071 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1d8017070 con 0x7fb1ec0c86d0 2024-04-14T23:58:09.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.481+0000 7fb1e2a00640 1 -- 172.21.15.60:0/454399071 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1d8005070 con 0x7fb1ec0c86d0 2024-04-14T23:58:09.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.481+0000 7fb1ebe00640 1 -- 172.21.15.60:0/454399071 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1ec161bc0 con 0x7fb1ec0c86d0 2024-04-14T23:58:09.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.481+0000 7fb1e2a00640 1 -- 172.21.15.60:0/454399071 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1d8005460 con 0x7fb1ec0c86d0 2024-04-14T23:58:09.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.481+0000 7fb1ebe00640 1 -- 172.21.15.60:0/454399071 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1ec1e9550 con 0x7fb1ec0c86d0 2024-04-14T23:58:09.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.482+0000 7fb1ebe00640 1 -- 172.21.15.60:0/454399071 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1ec1620e0 con 0x7fb1ec0c86d0 2024-04-14T23:58:09.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.483+0000 7fb1e2a00640 1 -- 172.21.15.60:0/454399071 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7fb1d801a070 con 0x7fb1ec0c86d0 2024-04-14T23:58:09.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.484+0000 7fb1e2a00640 1 --2- 172.21.15.60:0/454399071 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fb1cc06b990 0x7fb1cc06de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:09.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.484+0000 7fb1e2a00640 1 -- 172.21.15.60:0/454399071 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6255+0+0 (secure 0 0 0) 0x7fb1d8089da0 con 0x7fb1ec0c86d0 2024-04-14T23:58:09.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.484+0000 7fb1eaa00640 1 --2- 172.21.15.60:0/454399071 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fb1cc06b990 0x7fb1cc06de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:09.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.485+0000 7fb1eaa00640 1 --2- 172.21.15.60:0/454399071 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fb1cc06b990 0x7fb1cc06de50 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc009560 tx=0x7fb1dc008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:09.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.489+0000 7fb1e2a00640 1 -- 172.21.15.60:0/454399071 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb1ec071bf0 con 0x7fb1ec0c86d0 2024-04-14T23:58:09.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.692+0000 7fb1ebe00640 1 -- 172.21.15.60:0/454399071 --> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb1ec1613c0 con 0x7fb1cc06b990 2024-04-14T23:58:09.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.700+0000 7fb1e2a00640 1 -- 172.21.15.60:0/454399071 <== mgr.4100 v2:172.21.15.186:6832/714104835 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+255177 (secure 0 0 0) 0x7fb1dc0884e0 con 0x7fb1cc06b990 2024-04-14T23:58:09.708 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:58:09.710 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-04-14T23:58:09.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.705+0000 7fb1ebe00640 1 -- 172.21.15.60:0/454399071 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fb1cc06b990 msgr2=0x7fb1cc06de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:09.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.705+0000 7fb1ebe00640 1 --2- 172.21.15.60:0/454399071 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fb1cc06b990 0x7fb1cc06de50 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc009560 tx=0x7fb1dc008030 comp rx=0 tx=0).stop 2024-04-14T23:58:09.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.705+0000 7fb1ebe00640 1 -- 172.21.15.60:0/454399071 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb1ec0c86d0 msgr2=0x7fb1ec1e88c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:09.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.705+0000 7fb1ebe00640 1 --2- 172.21.15.60:0/454399071 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb1ec0c86d0 0x7fb1ec1e88c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fb1d800c750 tx=0x7fb1d800cc10 comp rx=0 tx=0).stop 2024-04-14T23:58:09.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.705+0000 7fb1ebe00640 1 -- 172.21.15.60:0/454399071 shutdown_connections 2024-04-14T23:58:09.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.705+0000 7fb1ebe00640 1 --2- 172.21.15.60:0/454399071 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fb1cc06b990 0x7fb1cc06de50 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:09.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.705+0000 7fb1ebe00640 1 --2- 172.21.15.60:0/454399071 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1ec15c8a0 0x7fb1ec1e9d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:09.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.705+0000 7fb1ebe00640 1 --2- 172.21.15.60:0/454399071 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb1ec0c86d0 0x7fb1ec1e88c0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:09.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.705+0000 7fb1ebe00640 1 --2- 172.21.15.60:0/454399071 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb1ec0c6950 0x7fb1ec1f5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:09.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.705+0000 7fb1ebe00640 1 -- 172.21.15.60:0/454399071 >> 172.21.15.60:0/454399071 conn(0x7fb1ec0c8b50 msgr2=0x7fb1ec159f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:09.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.705+0000 7fb1ebe00640 1 -- 172.21.15.60:0/454399071 shutdown_connections 2024-04-14T23:58:09.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.705+0000 7fb1ebe00640 1 -- 172.21.15.60:0/454399071 wait complete. 2024-04-14T23:58:09.728 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":325,"stamp":"2024-04-14T23:58:08.314240+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":202752180,"num_objects":92,"num_object_clones":0,"num_object_copies":184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92,"num_whiteouts":0,"num_read":2892,"num_read_kb":545033,"num_write":14569,"num_write_kb":430090,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14479,"ondisk_log_size":14479,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":631756,"kb_used_data":165760,"kb_used_omap":12,"kb_used_meta":465907,"kb_avail":754342964,"statfs":{"total":773094113280,"available":772447195136,"internally_reserved":0,"allocated":169738240,"data_stored":407419304,"data_compressed":34457476,"data_compressed_allocated":44892160,"data_compressed_original":284942336,"omap_allocated":12752,"internal_metadata":477089328},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":567468,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":53,"num_read_kb":40,"num_write":60,"num_write_kb":587,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003437"},"pg_stats":[{"pgid":"3.3f","version":"36'158","reported_seq":2280,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.085799+0000","last_change":"2024-04-14T23:55:41.994501+0000","last_active":"2024-04-14T23:57:39.085799+0000","last_peered":"2024-04-14T23:57:39.085799+0000","last_clean":"2024-04-14T23:57:39.085799+0000","last_became_active":"2024-04-14T23:48:43.222278+0000","last_became_peered":"2024-04-14T23:48:43.222278+0000","last_unstale":"2024-04-14T23:57:39.085799+0000","last_undegraded":"2024-04-14T23:57:39.085799+0000","last_fullsized":"2024-04-14T23:57:39.085799+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T04:04:17.956339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025657449999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":29,"num_read_kb":15,"num_write":158,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"44'142","reported_seq":2269,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.088582+0000","last_change":"2024-04-14T23:55:42.310665+0000","last_active":"2024-04-14T23:57:39.088582+0000","last_peered":"2024-04-14T23:57:39.088582+0000","last_clean":"2024-04-14T23:57:39.088582+0000","last_became_active":"2024-04-14T23:48:43.226426+0000","last_became_peered":"2024-04-14T23:48:43.226426+0000","last_unstale":"2024-04-14T23:57:39.088582+0000","last_undegraded":"2024-04-14T23:57:39.088582+0000","last_fullsized":"2024-04-14T23:57:39.088582+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:58:52.815232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011741033999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":16,"num_write":142,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"37'171","reported_seq":2299,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.087926+0000","last_change":"2024-04-14T23:55:42.315743+0000","last_active":"2024-04-14T23:57:39.087926+0000","last_peered":"2024-04-14T23:57:39.087926+0000","last_clean":"2024-04-14T23:57:39.087926+0000","last_became_active":"2024-04-14T23:48:43.236616+0000","last_became_peered":"2024-04-14T23:48:43.236616+0000","last_unstale":"2024-04-14T23:57:39.087926+0000","last_undegraded":"2024-04-14T23:57:39.087926+0000","last_fullsized":"2024-04-14T23:57:39.087926+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:56:34.635393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01445958,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":38,"num_read_kb":8211,"num_write":164,"num_write_kb":8272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"44'140","reported_seq":2253,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.083881+0000","last_change":"2024-04-14T23:55:42.084275+0000","last_active":"2024-04-14T23:57:39.083881+0000","last_peered":"2024-04-14T23:57:39.083881+0000","last_clean":"2024-04-14T23:57:39.083881+0000","last_became_active":"2024-04-14T23:48:43.242000+0000","last_became_peered":"2024-04-14T23:48:43.242000+0000","last_unstale":"2024-04-14T23:57:39.083881+0000","last_undegraded":"2024-04-14T23:57:39.083881+0000","last_fullsized":"2024-04-14T23:57:39.083881+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:06:54.726051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020739690000000002,"stat_sum":{"num_bytes":193,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":9,"num_write":139,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"35'197","reported_seq":2310,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.086395+0000","last_change":"2024-04-14T23:55:42.308076+0000","last_active":"2024-04-14T23:57:39.086395+0000","last_peered":"2024-04-14T23:57:39.086395+0000","last_clean":"2024-04-14T23:57:39.086395+0000","last_became_active":"2024-04-14T23:49:29.448078+0000","last_became_peered":"2024-04-14T23:49:29.448078+0000","last_unstale":"2024-04-14T23:57:39.086395+0000","last_undegraded":"2024-04-14T23:57:39.086395+0000","last_fullsized":"2024-04-14T23:57:39.086395+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":197,"log_dups_size":0,"ondisk_log_size":197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:09:40.348011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011461469,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":27,"num_read_kb":18,"num_write":196,"num_write_kb":98,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3a","version":"35'170","reported_seq":2294,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.083401+0000","last_change":"2024-04-14T23:55:42.084342+0000","last_active":"2024-04-14T23:57:39.083401+0000","last_peered":"2024-04-14T23:57:39.083401+0000","last_clean":"2024-04-14T23:57:39.083401+0000","last_became_active":"2024-04-14T23:48:43.230405+0000","last_became_peered":"2024-04-14T23:48:43.230405+0000","last_unstale":"2024-04-14T23:57:39.083401+0000","last_undegraded":"2024-04-14T23:57:39.083401+0000","last_fullsized":"2024-04-14T23:57:39.083401+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:40:55.017190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002322771,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":32,"num_read_kb":19,"num_write":170,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"44'198","reported_seq":2327,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.084946+0000","last_change":"2024-04-14T23:55:41.992292+0000","last_active":"2024-04-14T23:57:39.084946+0000","last_peered":"2024-04-14T23:57:39.084946+0000","last_clean":"2024-04-14T23:57:39.084946+0000","last_became_active":"2024-04-14T23:48:43.230784+0000","last_became_peered":"2024-04-14T23:48:43.230784+0000","last_unstale":"2024-04-14T23:57:39.084946+0000","last_undegraded":"2024-04-14T23:57:39.084946+0000","last_fullsized":"2024-04-14T23:57:39.084946+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":198,"log_dups_size":0,"ondisk_log_size":198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T04:49:40.437771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023027080000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":37,"num_read_kb":16,"num_write":198,"num_write_kb":4194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"44'133","reported_seq":2251,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.091139+0000","last_change":"2024-04-14T23:55:42.317071+0000","last_active":"2024-04-14T23:57:39.091139+0000","last_peered":"2024-04-14T23:57:39.091139+0000","last_clean":"2024-04-14T23:57:39.091139+0000","last_became_active":"2024-04-14T23:48:43.235719+0000","last_became_peered":"2024-04-14T23:48:43.235719+0000","last_unstale":"2024-04-14T23:57:39.091139+0000","last_undegraded":"2024-04-14T23:57:39.091139+0000","last_fullsized":"2024-04-14T23:57:39.091139+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:40:21.290823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015616025,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":26,"num_read_kb":10,"num_write":132,"num_write_kb":8256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"44'157","reported_seq":2283,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.085798+0000","last_change":"2024-04-14T23:55:41.994669+0000","last_active":"2024-04-14T23:57:39.085798+0000","last_peered":"2024-04-14T23:57:39.085798+0000","last_clean":"2024-04-14T23:57:39.085798+0000","last_became_active":"2024-04-14T23:48:43.231751+0000","last_became_peered":"2024-04-14T23:48:43.231751+0000","last_unstale":"2024-04-14T23:57:39.085798+0000","last_undegraded":"2024-04-14T23:57:39.085798+0000","last_fullsized":"2024-04-14T23:57:39.085798+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:44:15.830503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025112200000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":4114,"num_write":154,"num_write_kb":8271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"35'164","reported_seq":2283,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.010548+0000","last_change":"2024-04-14T23:55:41.992813+0000","last_active":"2024-04-14T23:57:39.010548+0000","last_peered":"2024-04-14T23:57:39.010548+0000","last_clean":"2024-04-14T23:57:39.010548+0000","last_became_active":"2024-04-14T23:48:43.224592+0000","last_became_peered":"2024-04-14T23:48:43.224592+0000","last_unstale":"2024-04-14T23:57:39.010548+0000","last_undegraded":"2024-04-14T23:57:39.010548+0000","last_fullsized":"2024-04-14T23:57:39.010548+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:36:03.843739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014741629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":27,"num_read_kb":16,"num_write":162,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"44'171","reported_seq":2298,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.010380+0000","last_change":"2024-04-14T23:55:41.991958+0000","last_active":"2024-04-14T23:57:39.010380+0000","last_peered":"2024-04-14T23:57:39.010380+0000","last_clean":"2024-04-14T23:57:39.010380+0000","last_became_active":"2024-04-14T23:48:43.224989+0000","last_became_peered":"2024-04-14T23:48:43.224989+0000","last_unstale":"2024-04-14T23:57:39.010380+0000","last_undegraded":"2024-04-14T23:57:39.010380+0000","last_fullsized":"2024-04-14T23:57:39.010380+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:31:12.485950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001581301,"stat_sum":{"num_bytes":4197091,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":35,"num_read_kb":18,"num_write":170,"num_write_kb":4183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"44'146","reported_seq":2262,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.086409+0000","last_change":"2024-04-14T23:55:42.224637+0000","last_active":"2024-04-14T23:57:39.086409+0000","last_peered":"2024-04-14T23:57:39.086409+0000","last_clean":"2024-04-14T23:57:39.086409+0000","last_became_active":"2024-04-14T23:48:43.225892+0000","last_became_peered":"2024-04-14T23:48:43.225892+0000","last_unstale":"2024-04-14T23:57:39.086409+0000","last_undegraded":"2024-04-14T23:57:39.086409+0000","last_fullsized":"2024-04-14T23:57:39.086409+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":146,"log_dups_size":0,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:30:03.715201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011254640000000001,"stat_sum":{"num_bytes":73,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":24,"num_read_kb":10,"num_write":145,"num_write_kb":8263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"37'146","reported_seq":2267,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.092688+0000","last_change":"2024-04-14T23:55:42.317691+0000","last_active":"2024-04-14T23:57:39.092688+0000","last_peered":"2024-04-14T23:57:39.092688+0000","last_clean":"2024-04-14T23:57:39.092688+0000","last_became_active":"2024-04-14T23:48:43.219614+0000","last_became_peered":"2024-04-14T23:48:43.219614+0000","last_unstale":"2024-04-14T23:57:39.092688+0000","last_undegraded":"2024-04-14T23:57:39.092688+0000","last_fullsized":"2024-04-14T23:57:39.092688+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":146,"log_dups_size":0,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:11:13.830037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012872641000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":30,"num_read_kb":4115,"num_write":142,"num_write_kb":8261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"44'175","reported_seq":2308,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.090824+0000","last_change":"2024-04-14T23:55:42.319365+0000","last_active":"2024-04-14T23:57:39.090824+0000","last_peered":"2024-04-14T23:57:39.090824+0000","last_clean":"2024-04-14T23:57:39.090824+0000","last_became_active":"2024-04-14T23:48:43.239136+0000","last_became_peered":"2024-04-14T23:48:43.239136+0000","last_unstale":"2024-04-14T23:57:39.090824+0000","last_undegraded":"2024-04-14T23:57:39.090824+0000","last_fullsized":"2024-04-14T23:57:39.090824+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:47:21.467652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015761819,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":40,"num_read_kb":19,"num_write":174,"num_write_kb":12372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"44'159","reported_seq":2289,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.090236+0000","last_change":"2024-04-14T23:55:42.308265+0000","last_active":"2024-04-14T23:57:39.090236+0000","last_peered":"2024-04-14T23:57:39.090236+0000","last_clean":"2024-04-14T23:57:39.090236+0000","last_became_active":"2024-04-14T23:48:43.227898+0000","last_became_peered":"2024-04-14T23:48:43.227898+0000","last_unstale":"2024-04-14T23:57:39.090236+0000","last_undegraded":"2024-04-14T23:57:39.090236+0000","last_fullsized":"2024-04-14T23:57:39.090236+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:18:28.787829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011044834999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":37,"num_read_kb":19,"num_write":159,"num_write_kb":4178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"44'184","reported_seq":2317,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.083264+0000","last_change":"2024-04-14T23:55:42.083783+0000","last_active":"2024-04-14T23:57:39.083264+0000","last_peered":"2024-04-14T23:57:39.083264+0000","last_clean":"2024-04-14T23:57:39.083264+0000","last_became_active":"2024-04-14T23:48:43.238714+0000","last_became_peered":"2024-04-14T23:48:43.238714+0000","last_unstale":"2024-04-14T23:57:39.083264+0000","last_undegraded":"2024-04-14T23:57:39.083264+0000","last_fullsized":"2024-04-14T23:57:39.083264+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":184,"log_dups_size":0,"ondisk_log_size":184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:37:44.837525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019546920000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":40,"num_read_kb":19,"num_write":182,"num_write_kb":7459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"44'174","reported_seq":2306,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.083777+0000","last_change":"2024-04-14T23:55:42.085173+0000","last_active":"2024-04-14T23:57:39.083777+0000","last_peered":"2024-04-14T23:57:39.083777+0000","last_clean":"2024-04-14T23:57:39.083777+0000","last_became_active":"2024-04-14T23:48:43.228618+0000","last_became_peered":"2024-04-14T23:48:43.228618+0000","last_unstale":"2024-04-14T23:57:39.083777+0000","last_undegraded":"2024-04-14T23:57:39.083777+0000","last_fullsized":"2024-04-14T23:57:39.083777+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":174,"log_dups_size":0,"ondisk_log_size":174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T09:07:18.121366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002424158,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":40,"num_read_kb":4117,"num_write":170,"num_write_kb":4181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"37'153","reported_seq":2293,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.011857+0000","last_change":"2024-04-14T23:55:42.175749+0000","last_active":"2024-04-14T23:57:39.011857+0000","last_peered":"2024-04-14T23:57:39.011857+0000","last_clean":"2024-04-14T23:57:39.011857+0000","last_became_active":"2024-04-14T23:48:43.232971+0000","last_became_peered":"2024-04-14T23:48:43.232971+0000","last_unstale":"2024-04-14T23:57:39.011857+0000","last_undegraded":"2024-04-14T23:57:39.011857+0000","last_fullsized":"2024-04-14T23:57:39.011857+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:34:34.148700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030089610000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":47,"num_read_kb":26,"num_write":152,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"44'163","reported_seq":2290,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.086169+0000","last_change":"2024-04-14T23:55:41.994445+0000","last_active":"2024-04-14T23:57:39.086169+0000","last_peered":"2024-04-14T23:57:39.086169+0000","last_clean":"2024-04-14T23:57:39.086169+0000","last_became_active":"2024-04-14T23:48:43.232135+0000","last_became_peered":"2024-04-14T23:48:43.232135+0000","last_unstale":"2024-04-14T23:57:39.086169+0000","last_undegraded":"2024-04-14T23:57:39.086169+0000","last_fullsized":"2024-04-14T23:57:39.086169+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":163,"log_dups_size":0,"ondisk_log_size":163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:37:12.273966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029988570000000002,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":35,"num_read_kb":4112,"num_write":159,"num_write_kb":8270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"44'195","reported_seq":2329,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.010441+0000","last_change":"2024-04-14T23:55:41.992691+0000","last_active":"2024-04-14T23:57:39.010441+0000","last_peered":"2024-04-14T23:57:39.010441+0000","last_clean":"2024-04-14T23:57:39.010441+0000","last_became_active":"2024-04-14T23:48:43.226225+0000","last_became_peered":"2024-04-14T23:48:43.226225+0000","last_unstale":"2024-04-14T23:57:39.010441+0000","last_undegraded":"2024-04-14T23:57:39.010441+0000","last_fullsized":"2024-04-14T23:57:39.010441+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":195,"log_dups_size":0,"ondisk_log_size":195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:43:42.017755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019674760000000001,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":42,"num_read_kb":4116,"num_write":192,"num_write_kb":16477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"44'154","reported_seq":2291,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.090622+0000","last_change":"2024-04-14T23:55:42.316046+0000","last_active":"2024-04-14T23:57:39.090622+0000","last_peered":"2024-04-14T23:57:39.090622+0000","last_clean":"2024-04-14T23:57:39.090622+0000","last_became_active":"2024-04-14T23:48:43.230056+0000","last_became_peered":"2024-04-14T23:48:43.230056+0000","last_unstale":"2024-04-14T23:57:39.090622+0000","last_undegraded":"2024-04-14T23:57:39.090622+0000","last_fullsized":"2024-04-14T23:57:39.090622+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:05:52.381327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015336659000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":44,"num_read_kb":22,"num_write":154,"num_write_kb":77,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"44'175","reported_seq":2304,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.083498+0000","last_change":"2024-04-14T23:55:42.084499+0000","last_active":"2024-04-14T23:57:39.083498+0000","last_peered":"2024-04-14T23:57:39.083498+0000","last_clean":"2024-04-14T23:57:39.083498+0000","last_became_active":"2024-04-14T23:48:43.229110+0000","last_became_peered":"2024-04-14T23:48:43.229110+0000","last_unstale":"2024-04-14T23:57:39.083498+0000","last_undegraded":"2024-04-14T23:57:39.083498+0000","last_fullsized":"2024-04-14T23:57:39.083498+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T09:03:22.176338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022905320000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":36,"num_read_kb":17,"num_write":174,"num_write_kb":4183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"44'170","reported_seq":2301,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:45.295592+0000","last_change":"2024-04-14T23:55:42.314003+0000","last_active":"2024-04-14T23:57:45.295592+0000","last_peered":"2024-04-14T23:57:45.295592+0000","last_clean":"2024-04-14T23:57:45.295592+0000","last_became_active":"2024-04-14T23:48:43.233853+0000","last_became_peered":"2024-04-14T23:48:43.233853+0000","last_unstale":"2024-04-14T23:57:45.295592+0000","last_undegraded":"2024-04-14T23:57:45.295592+0000","last_fullsized":"2024-04-14T23:57:45.295592+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T09:33:21.684120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012858206,"stat_sum":{"num_bytes":1378,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":39,"num_read_kb":4117,"num_write":167,"num_write_kb":4180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"44'173","reported_seq":2295,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.011239+0000","last_change":"2024-04-14T23:55:42.175389+0000","last_active":"2024-04-14T23:57:39.011239+0000","last_peered":"2024-04-14T23:57:39.011239+0000","last_clean":"2024-04-14T23:57:39.011239+0000","last_became_active":"2024-04-14T23:49:29.512499+0000","last_became_peered":"2024-04-14T23:49:29.512499+0000","last_unstale":"2024-04-14T23:57:39.011239+0000","last_undegraded":"2024-04-14T23:57:39.011239+0000","last_fullsized":"2024-04-14T23:57:39.011239+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":173,"log_dups_size":0,"ondisk_log_size":173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:04:34.423852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037213820000000001,"stat_sum":{"num_bytes":23,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":36,"num_read_kb":16,"num_write":173,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"37'160","reported_seq":2287,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.093353+0000","last_change":"2024-04-14T23:55:42.319252+0000","last_active":"2024-04-14T23:57:39.093353+0000","last_peered":"2024-04-14T23:57:39.093353+0000","last_clean":"2024-04-14T23:57:39.093353+0000","last_became_active":"2024-04-14T23:48:43.233238+0000","last_became_peered":"2024-04-14T23:48:43.233238+0000","last_unstale":"2024-04-14T23:57:39.093353+0000","last_undegraded":"2024-04-14T23:57:39.093353+0000","last_fullsized":"2024-04-14T23:57:39.093353+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":160,"log_dups_size":0,"ondisk_log_size":160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T23:56:25.118737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012037971,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":18,"num_write":158,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"36'184","reported_seq":2312,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.010657+0000","last_change":"2024-04-14T23:55:41.993037+0000","last_active":"2024-04-14T23:57:39.010657+0000","last_peered":"2024-04-14T23:57:39.010657+0000","last_clean":"2024-04-14T23:57:39.010657+0000","last_became_active":"2024-04-14T23:48:43.222624+0000","last_became_peered":"2024-04-14T23:48:43.222624+0000","last_unstale":"2024-04-14T23:57:39.010657+0000","last_undegraded":"2024-04-14T23:57:39.010657+0000","last_fullsized":"2024-04-14T23:57:39.010657+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":184,"log_dups_size":0,"ondisk_log_size":184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:43:37.661814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001456707,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":19,"num_write":184,"num_write_kb":92,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"44'140","reported_seq":2257,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.086226+0000","last_change":"2024-04-14T23:55:42.309950+0000","last_active":"2024-04-14T23:57:39.086226+0000","last_peered":"2024-04-14T23:57:39.086226+0000","last_clean":"2024-04-14T23:57:39.086226+0000","last_became_active":"2024-04-14T23:48:43.232277+0000","last_became_peered":"2024-04-14T23:48:43.232277+0000","last_unstale":"2024-04-14T23:57:39.086226+0000","last_undegraded":"2024-04-14T23:57:39.086226+0000","last_fullsized":"2024-04-14T23:57:39.086226+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:17:15.623645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011200412,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":138,"num_write_kb":4167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"44'156","reported_seq":2279,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.094007+0000","last_change":"2024-04-14T23:55:42.319096+0000","last_active":"2024-04-14T23:57:39.094007+0000","last_peered":"2024-04-14T23:57:39.094007+0000","last_clean":"2024-04-14T23:57:39.094007+0000","last_became_active":"2024-04-14T23:48:43.229172+0000","last_became_peered":"2024-04-14T23:48:43.229172+0000","last_unstale":"2024-04-14T23:57:39.094007+0000","last_undegraded":"2024-04-14T23:57:39.094007+0000","last_fullsized":"2024-04-14T23:57:39.094007+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:05:34.205391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011889788,"stat_sum":{"num_bytes":482,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":31,"num_read_kb":3287,"num_write":153,"num_write_kb":3350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"36'184","reported_seq":2321,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.088053+0000","last_change":"2024-04-14T23:55:42.317181+0000","last_active":"2024-04-14T23:57:39.088053+0000","last_peered":"2024-04-14T23:57:39.088053+0000","last_clean":"2024-04-14T23:57:39.088053+0000","last_became_active":"2024-04-14T23:49:29.449548+0000","last_became_peered":"2024-04-14T23:49:29.449548+0000","last_unstale":"2024-04-14T23:57:39.088053+0000","last_undegraded":"2024-04-14T23:57:39.088053+0000","last_fullsized":"2024-04-14T23:57:39.088053+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":184,"log_dups_size":0,"ondisk_log_size":184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:07:19.034727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015790789,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":39,"num_read_kb":18,"num_write":184,"num_write_kb":92,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"35'188","reported_seq":2318,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.011493+0000","last_change":"2024-04-14T23:55:42.174354+0000","last_active":"2024-04-14T23:57:39.011493+0000","last_peered":"2024-04-14T23:57:39.011493+0000","last_clean":"2024-04-14T23:57:39.011493+0000","last_became_active":"2024-04-14T23:49:29.512523+0000","last_became_peered":"2024-04-14T23:49:29.512523+0000","last_unstale":"2024-04-14T23:57:39.011493+0000","last_undegraded":"2024-04-14T23:57:39.011493+0000","last_fullsized":"2024-04-14T23:57:39.011493+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":188,"log_dups_size":0,"ondisk_log_size":188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T09:13:33.216335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026059680000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":43,"num_read_kb":22,"num_write":188,"num_write_kb":94,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"44'205","reported_seq":2333,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.011575+0000","last_change":"2024-04-14T23:55:42.175131+0000","last_active":"2024-04-14T23:57:39.011575+0000","last_peered":"2024-04-14T23:57:39.011575+0000","last_clean":"2024-04-14T23:57:39.011575+0000","last_became_active":"2024-04-14T23:48:43.233567+0000","last_became_peered":"2024-04-14T23:48:43.233567+0000","last_unstale":"2024-04-14T23:57:39.011575+0000","last_undegraded":"2024-04-14T23:57:39.011575+0000","last_fullsized":"2024-04-14T23:57:39.011575+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":205,"log_dups_size":0,"ondisk_log_size":205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:43:07.999415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003348115,"stat_sum":{"num_bytes":4726,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":35,"num_read_kb":17,"num_write":205,"num_write_kb":14807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"44'199","reported_seq":2339,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.010040+0000","last_change":"2024-04-14T23:55:41.992153+0000","last_active":"2024-04-14T23:57:39.010040+0000","last_peered":"2024-04-14T23:57:39.010040+0000","last_clean":"2024-04-14T23:57:39.010040+0000","last_became_active":"2024-04-14T23:48:43.230626+0000","last_became_peered":"2024-04-14T23:48:43.230626+0000","last_unstale":"2024-04-14T23:57:39.010040+0000","last_undegraded":"2024-04-14T23:57:39.010040+0000","last_fullsized":"2024-04-14T23:57:39.010040+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":199,"log_dups_size":0,"ondisk_log_size":199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T23:53:31.323623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001517437,"stat_sum":{"num_bytes":4194851,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":47,"num_read_kb":27,"num_write":198,"num_write_kb":12385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"44'158","reported_seq":2278,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.084838+0000","last_change":"2024-04-14T23:55:42.224072+0000","last_active":"2024-04-14T23:57:39.084838+0000","last_peered":"2024-04-14T23:57:39.084838+0000","last_clean":"2024-04-14T23:57:39.084838+0000","last_became_active":"2024-04-14T23:48:43.226773+0000","last_became_peered":"2024-04-14T23:48:43.226773+0000","last_unstale":"2024-04-14T23:57:39.084838+0000","last_undegraded":"2024-04-14T23:57:39.084838+0000","last_fullsized":"2024-04-14T23:57:39.084838+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:04:13.345776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00101596,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":28,"num_read_kb":16,"num_write":158,"num_write_kb":4174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"34'20","reported_seq":90,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.094244+0000","last_change":"2024-04-14T23:48:42.239254+0000","last_active":"2024-04-14T23:57:39.094244+0000","last_peered":"2024-04-14T23:57:39.094244+0000","last_clean":"2024-04-14T23:57:39.094244+0000","last_became_active":"2024-04-14T23:48:42.238847+0000","last_became_peered":"2024-04-14T23:48:42.238847+0000","last_unstale":"2024-04-14T23:57:39.094244+0000","last_undegraded":"2024-04-14T23:57:39.094244+0000","last_fullsized":"2024-04-14T23:57:39.094244+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:46:16.301900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":16386,"num_write":22,"num_write_kb":4233,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"44'168","reported_seq":2296,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.091757+0000","last_change":"2024-04-14T23:55:42.317522+0000","last_active":"2024-04-14T23:57:39.091757+0000","last_peered":"2024-04-14T23:57:39.091757+0000","last_clean":"2024-04-14T23:57:39.091757+0000","last_became_active":"2024-04-14T23:48:43.237391+0000","last_became_peered":"2024-04-14T23:48:43.237391+0000","last_unstale":"2024-04-14T23:57:39.091757+0000","last_undegraded":"2024-04-14T23:57:39.091757+0000","last_fullsized":"2024-04-14T23:57:39.091757+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:01:14.668174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01542551,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":36,"num_read_kb":15,"num_write":168,"num_write_kb":4179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"34'164","reported_seq":240,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:45.069610+0000","last_change":"2024-04-14T23:48:42.219277+0000","last_active":"2024-04-14T23:57:45.069610+0000","last_peered":"2024-04-14T23:57:45.069610+0000","last_clean":"2024-04-14T23:57:45.069610+0000","last_became_active":"2024-04-14T23:48:42.218995+0000","last_became_peered":"2024-04-14T23:48:42.218995+0000","last_unstale":"2024-04-14T23:57:45.069610+0000","last_undegraded":"2024-04-14T23:57:45.069610+0000","last_fullsized":"2024-04-14T23:57:45.069610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T23:51:33.370857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":11,"num_read_kb":40960,"num_write":162,"num_write_kb":12391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"44'24","reported_seq":92,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.090540+0000","last_change":"2024-04-14T23:48:42.236803+0000","last_active":"2024-04-14T23:57:39.090540+0000","last_peered":"2024-04-14T23:57:39.090540+0000","last_clean":"2024-04-14T23:57:39.090540+0000","last_became_active":"2024-04-14T23:48:42.225937+0000","last_became_peered":"2024-04-14T23:48:42.225937+0000","last_unstale":"2024-04-14T23:57:39.090540+0000","last_undegraded":"2024-04-14T23:57:39.090540+0000","last_fullsized":"2024-04-14T23:57:39.090540+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:52:32.609126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":22,"num_write_kb":4105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"35'209","reported_seq":2340,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.091590+0000","last_change":"2024-04-14T23:55:42.316735+0000","last_active":"2024-04-14T23:57:39.091590+0000","last_peered":"2024-04-14T23:57:39.091590+0000","last_clean":"2024-04-14T23:57:39.091590+0000","last_became_active":"2024-04-14T23:48:43.227356+0000","last_became_peered":"2024-04-14T23:48:43.227356+0000","last_unstale":"2024-04-14T23:57:39.091590+0000","last_undegraded":"2024-04-14T23:57:39.091590+0000","last_fullsized":"2024-04-14T23:57:39.091590+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":209,"log_dups_size":0,"ondisk_log_size":209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:55:38.737429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014613774,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":40,"num_read_kb":4118,"num_write":206,"num_write_kb":8293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"45'189","reported_seq":260,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:44.808805+0000","last_change":"2024-04-14T23:48:42.210883+0000","last_active":"2024-04-14T23:57:44.808805+0000","last_peered":"2024-04-14T23:57:44.808805+0000","last_clean":"2024-04-14T23:57:44.808805+0000","last_became_active":"2024-04-14T23:48:42.210607+0000","last_became_peered":"2024-04-14T23:48:42.210607+0000","last_unstale":"2024-04-14T23:57:44.808805+0000","last_undegraded":"2024-04-14T23:57:44.808805+0000","last_fullsized":"2024-04-14T23:57:44.808805+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":189,"log_dups_size":0,"ondisk_log_size":189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:33:32.526223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":186,"num_write_kb":8305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"42'164","reported_seq":2296,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.085956+0000","last_change":"2024-04-14T23:55:42.310398+0000","last_active":"2024-04-14T23:57:39.085956+0000","last_peered":"2024-04-14T23:57:39.085956+0000","last_clean":"2024-04-14T23:57:39.085956+0000","last_became_active":"2024-04-14T23:48:43.217801+0000","last_became_peered":"2024-04-14T23:48:43.217801+0000","last_unstale":"2024-04-14T23:57:39.085956+0000","last_undegraded":"2024-04-14T23:57:39.085956+0000","last_fullsized":"2024-04-14T23:57:39.085956+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:39:59.476515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012539378,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":41,"num_read_kb":4114,"num_write":158,"num_write_kb":4174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"44'5","reported_seq":70,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.010369+0000","last_change":"2024-04-14T23:48:42.218652+0000","last_active":"2024-04-14T23:57:39.010369+0000","last_peered":"2024-04-14T23:57:39.010369+0000","last_clean":"2024-04-14T23:57:39.010369+0000","last_became_active":"2024-04-14T23:48:42.218321+0000","last_became_peered":"2024-04-14T23:48:42.218321+0000","last_unstale":"2024-04-14T23:57:39.010369+0000","last_undegraded":"2024-04-14T23:57:39.010369+0000","last_fullsized":"2024-04-14T23:57:39.010369+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T04:22:06.957747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"44'189","reported_seq":2326,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.083734+0000","last_change":"2024-04-14T23:55:42.084801+0000","last_active":"2024-04-14T23:57:39.083734+0000","last_peered":"2024-04-14T23:57:39.083734+0000","last_clean":"2024-04-14T23:57:39.083734+0000","last_became_active":"2024-04-14T23:48:43.228805+0000","last_became_peered":"2024-04-14T23:48:43.228805+0000","last_unstale":"2024-04-14T23:57:39.083734+0000","last_undegraded":"2024-04-14T23:57:39.083734+0000","last_fullsized":"2024-04-14T23:57:39.083734+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":189,"log_dups_size":0,"ondisk_log_size":189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:59:51.260017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022812650000000002,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":45,"num_read_kb":20,"num_write":189,"num_write_kb":4190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"44'27","reported_seq":95,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.083988+0000","last_change":"2024-04-14T23:48:42.221104+0000","last_active":"2024-04-14T23:57:39.083988+0000","last_peered":"2024-04-14T23:57:39.083988+0000","last_clean":"2024-04-14T23:57:39.083988+0000","last_became_active":"2024-04-14T23:48:42.220767+0000","last_became_peered":"2024-04-14T23:48:42.220767+0000","last_unstale":"2024-04-14T23:57:39.083988+0000","last_undegraded":"2024-04-14T23:57:39.083988+0000","last_fullsized":"2024-04-14T23:57:39.083988+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:23:57.213764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":24,"num_write_kb":4108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"44'162","reported_seq":2287,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.089328+0000","last_change":"2024-04-14T23:55:42.316479+0000","last_active":"2024-04-14T23:57:39.089328+0000","last_peered":"2024-04-14T23:57:39.089328+0000","last_clean":"2024-04-14T23:57:39.089328+0000","last_became_active":"2024-04-14T23:48:43.236168+0000","last_became_peered":"2024-04-14T23:48:43.236168+0000","last_unstale":"2024-04-14T23:57:39.089328+0000","last_undegraded":"2024-04-14T23:57:39.089328+0000","last_fullsized":"2024-04-14T23:57:39.089328+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:10:13.592872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015342639999999999,"stat_sum":{"num_bytes":420,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":33,"num_read_kb":18,"num_write":161,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"44'145","reported_seq":226,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:45.493309+0000","last_change":"2024-04-14T23:48:42.231329+0000","last_active":"2024-04-14T23:57:45.493309+0000","last_peered":"2024-04-14T23:57:45.493309+0000","last_clean":"2024-04-14T23:57:45.493309+0000","last_became_active":"2024-04-14T23:48:42.230969+0000","last_became_peered":"2024-04-14T23:48:42.230969+0000","last_unstale":"2024-04-14T23:57:45.493309+0000","last_undegraded":"2024-04-14T23:57:45.493309+0000","last_fullsized":"2024-04-14T23:57:45.493309+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:06:16.557589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194394,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":30,"num_read_kb":9177,"num_write":144,"num_write_kb":4199,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"44'151","reported_seq":2272,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.086423+0000","last_change":"2024-04-14T23:55:42.224786+0000","last_active":"2024-04-14T23:57:39.086423+0000","last_peered":"2024-04-14T23:57:39.086423+0000","last_clean":"2024-04-14T23:57:39.086423+0000","last_became_active":"2024-04-14T23:48:43.230253+0000","last_became_peered":"2024-04-14T23:48:43.230253+0000","last_unstale":"2024-04-14T23:57:39.086423+0000","last_undegraded":"2024-04-14T23:57:39.086423+0000","last_fullsized":"2024-04-14T23:57:39.086423+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":151,"log_dups_size":0,"ondisk_log_size":151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:04:12.081827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011901889999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":28,"num_read_kb":14,"num_write":150,"num_write_kb":4170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"45'705","reported_seq":801,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:02.669217+0000","last_change":"2024-04-14T23:48:42.237069+0000","last_active":"2024-04-14T23:58:02.669217+0000","last_peered":"2024-04-14T23:58:02.669217+0000","last_clean":"2024-04-14T23:58:02.669217+0000","last_became_active":"2024-04-14T23:48:42.236804+0000","last_became_peered":"2024-04-14T23:48:42.236804+0000","last_unstale":"2024-04-14T23:58:02.669217+0000","last_undegraded":"2024-04-14T23:58:02.669217+0000","last_fullsized":"2024-04-14T23:58:02.669217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":705,"log_dups_size":0,"ondisk_log_size":705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:25:37.318379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583584,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":41866,"num_write":697,"num_write_kb":12860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"44'154","reported_seq":2275,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.085057+0000","last_change":"2024-04-14T23:55:41.994228+0000","last_active":"2024-04-14T23:57:39.085057+0000","last_peered":"2024-04-14T23:57:39.085057+0000","last_clean":"2024-04-14T23:57:39.085057+0000","last_became_active":"2024-04-14T23:48:43.221462+0000","last_became_peered":"2024-04-14T23:48:43.221462+0000","last_unstale":"2024-04-14T23:57:39.085057+0000","last_undegraded":"2024-04-14T23:57:39.085057+0000","last_fullsized":"2024-04-14T23:57:39.085057+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:09:26.659126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033453850000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":154,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"44'99","reported_seq":183,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:02.545555+0000","last_change":"2024-04-14T23:48:42.230518+0000","last_active":"2024-04-14T23:58:02.545555+0000","last_peered":"2024-04-14T23:58:02.545555+0000","last_clean":"2024-04-14T23:58:02.545555+0000","last_became_active":"2024-04-14T23:48:42.229758+0000","last_became_peered":"2024-04-14T23:48:42.229758+0000","last_unstale":"2024-04-14T23:58:02.545555+0000","last_undegraded":"2024-04-14T23:58:02.545555+0000","last_fullsized":"2024-04-14T23:58:02.545555+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:01:41.332553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388642,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":28,"num_read_kb":32808,"num_write":95,"num_write_kb":8238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"44'160","reported_seq":2289,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:02.545168+0000","last_change":"2024-04-14T23:55:42.319479+0000","last_active":"2024-04-14T23:58:02.545168+0000","last_peered":"2024-04-14T23:58:02.545168+0000","last_clean":"2024-04-14T23:58:02.545168+0000","last_became_active":"2024-04-14T23:48:43.229476+0000","last_became_peered":"2024-04-14T23:48:43.229476+0000","last_unstale":"2024-04-14T23:58:02.545168+0000","last_undegraded":"2024-04-14T23:58:02.545168+0000","last_fullsized":"2024-04-14T23:58:02.545168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":160,"log_dups_size":0,"ondisk_log_size":160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:36:00.949626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015192783,"stat_sum":{"num_bytes":4198858,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":37,"num_read_kb":4115,"num_write":155,"num_write_kb":12367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":66,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.084193+0000","last_change":"2024-04-14T23:48:42.225404+0000","last_active":"2024-04-14T23:57:39.084193+0000","last_peered":"2024-04-14T23:57:39.084193+0000","last_clean":"2024-04-14T23:57:39.084193+0000","last_became_active":"2024-04-14T23:48:42.225211+0000","last_became_peered":"2024-04-14T23:48:42.225211+0000","last_unstale":"2024-04-14T23:57:39.084193+0000","last_undegraded":"2024-04-14T23:57:39.084193+0000","last_fullsized":"2024-04-14T23:57:39.084193+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:49:45.282259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"36'172","reported_seq":2308,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.085033+0000","last_change":"2024-04-14T23:55:41.993454+0000","last_active":"2024-04-14T23:57:39.085033+0000","last_peered":"2024-04-14T23:57:39.085033+0000","last_clean":"2024-04-14T23:57:39.085033+0000","last_became_active":"2024-04-14T23:48:43.229297+0000","last_became_peered":"2024-04-14T23:48:43.229297+0000","last_unstale":"2024-04-14T23:57:39.085033+0000","last_undegraded":"2024-04-14T23:57:39.085033+0000","last_fullsized":"2024-04-14T23:57:39.085033+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:21:50.218321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029516669999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":43,"num_read_kb":23,"num_write":172,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"34'2","reported_seq":68,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.010088+0000","last_change":"2024-04-14T23:48:42.220492+0000","last_active":"2024-04-14T23:57:39.010088+0000","last_peered":"2024-04-14T23:57:39.010088+0000","last_clean":"2024-04-14T23:57:39.010088+0000","last_became_active":"2024-04-14T23:48:42.220145+0000","last_became_peered":"2024-04-14T23:48:42.220145+0000","last_unstale":"2024-04-14T23:57:39.010088+0000","last_undegraded":"2024-04-14T23:57:39.010088+0000","last_fullsized":"2024-04-14T23:57:39.010088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:17:45.278993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"44'168","reported_seq":2296,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:02.542597+0000","last_change":"2024-04-14T23:55:41.992426+0000","last_active":"2024-04-14T23:58:02.542597+0000","last_peered":"2024-04-14T23:58:02.542597+0000","last_clean":"2024-04-14T23:58:02.542597+0000","last_became_active":"2024-04-14T23:48:43.235319+0000","last_became_peered":"2024-04-14T23:48:43.235319+0000","last_unstale":"2024-04-14T23:58:02.542597+0000","last_undegraded":"2024-04-14T23:58:02.542597+0000","last_fullsized":"2024-04-14T23:58:02.542597+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:38:08.635546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017199240000000001,"stat_sum":{"num_bytes":428,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":36,"num_read_kb":14,"num_write":163,"num_write_kb":82,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"35'101","reported_seq":199,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:02.579303+0000","last_change":"2024-04-14T23:48:42.231178+0000","last_active":"2024-04-14T23:58:02.579303+0000","last_peered":"2024-04-14T23:58:02.579303+0000","last_clean":"2024-04-14T23:58:02.579303+0000","last_became_active":"2024-04-14T23:48:42.230792+0000","last_became_peered":"2024-04-14T23:48:42.230792+0000","last_unstale":"2024-04-14T23:58:02.579303+0000","last_undegraded":"2024-04-14T23:58:02.579303+0000","last_fullsized":"2024-04-14T23:58:02.579303+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:22:40.744314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":84,"num_read_kb":20558,"num_write":102,"num_write_kb":8239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"44'181","reported_seq":2315,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:02.544562+0000","last_change":"2024-04-14T23:55:41.993119+0000","last_active":"2024-04-14T23:58:02.544562+0000","last_peered":"2024-04-14T23:58:02.544562+0000","last_clean":"2024-04-14T23:58:02.544562+0000","last_became_active":"2024-04-14T23:48:43.228004+0000","last_became_peered":"2024-04-14T23:48:43.228004+0000","last_unstale":"2024-04-14T23:58:02.544562+0000","last_undegraded":"2024-04-14T23:58:02.544562+0000","last_fullsized":"2024-04-14T23:58:02.544562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:09:06.909970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003032402,"stat_sum":{"num_bytes":1647,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":44,"num_read_kb":8209,"num_write":175,"num_write_kb":12374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"34'22","reported_seq":96,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:44.962330+0000","last_change":"2024-04-14T23:48:42.231936+0000","last_active":"2024-04-14T23:57:44.962330+0000","last_peered":"2024-04-14T23:57:44.962330+0000","last_clean":"2024-04-14T23:57:44.962330+0000","last_became_active":"2024-04-14T23:48:42.231612+0000","last_became_peered":"2024-04-14T23:48:42.231612+0000","last_unstale":"2024-04-14T23:57:44.962330+0000","last_undegraded":"2024-04-14T23:57:44.962330+0000","last_fullsized":"2024-04-14T23:57:44.962330+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:45:03.098024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194326,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":12293,"num_write":21,"num_write_kb":4101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"36'127","reported_seq":2246,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.086202+0000","last_change":"2024-04-14T23:55:41.993658+0000","last_active":"2024-04-14T23:57:39.086202+0000","last_peered":"2024-04-14T23:57:39.086202+0000","last_clean":"2024-04-14T23:57:39.086202+0000","last_became_active":"2024-04-14T23:48:43.228665+0000","last_became_peered":"2024-04-14T23:48:43.228665+0000","last_unstale":"2024-04-14T23:57:39.086202+0000","last_undegraded":"2024-04-14T23:57:39.086202+0000","last_fullsized":"2024-04-14T23:57:39.086202+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T23:49:58.331034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003047124,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":27,"num_read_kb":4107,"num_write":124,"num_write_kb":4157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"30'100","reported_seq":176,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:44.881117+0000","last_change":"2024-04-14T23:48:42.239794+0000","last_active":"2024-04-14T23:57:44.881117+0000","last_peered":"2024-04-14T23:57:44.881117+0000","last_clean":"2024-04-14T23:57:44.881117+0000","last_became_active":"2024-04-14T23:48:42.239505+0000","last_became_peered":"2024-04-14T23:48:42.239505+0000","last_unstale":"2024-04-14T23:57:44.881117+0000","last_undegraded":"2024-04-14T23:57:44.881117+0000","last_fullsized":"2024-04-14T23:57:44.881117+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T09:43:16.582708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":24,"num_read_kb":12309,"num_write":100,"num_write_kb":4158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"44'164","reported_seq":2293,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:45.262232+0000","last_change":"2024-04-14T23:55:41.994666+0000","last_active":"2024-04-14T23:57:45.262232+0000","last_peered":"2024-04-14T23:57:45.262232+0000","last_clean":"2024-04-14T23:57:45.262232+0000","last_became_active":"2024-04-14T23:48:43.229949+0000","last_became_peered":"2024-04-14T23:48:43.229949+0000","last_unstale":"2024-04-14T23:57:45.262232+0000","last_undegraded":"2024-04-14T23:57:45.262232+0000","last_fullsized":"2024-04-14T23:57:45.262232+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:26:59.738426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002291033,"stat_sum":{"num_bytes":240,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":37,"num_read_kb":4115,"num_write":159,"num_write_kb":4175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"34'18","reported_seq":93,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:44.346309+0000","last_change":"2024-04-14T23:48:42.227708+0000","last_active":"2024-04-14T23:57:44.346309+0000","last_peered":"2024-04-14T23:57:44.346309+0000","last_clean":"2024-04-14T23:57:44.346309+0000","last_became_active":"2024-04-14T23:48:42.226845+0000","last_became_peered":"2024-04-14T23:48:42.226845+0000","last_unstale":"2024-04-14T23:57:44.346309+0000","last_undegraded":"2024-04-14T23:57:44.346309+0000","last_fullsized":"2024-04-14T23:57:44.346309+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:02:54.468796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194374,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":9,"num_read_kb":16389,"num_write":18,"num_write_kb":4108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"43'183","reported_seq":2304,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.084349+0000","last_change":"2024-04-14T23:55:42.084740+0000","last_active":"2024-04-14T23:57:39.084349+0000","last_peered":"2024-04-14T23:57:39.084349+0000","last_clean":"2024-04-14T23:57:39.084349+0000","last_became_active":"2024-04-14T23:48:43.226717+0000","last_became_peered":"2024-04-14T23:48:43.226717+0000","last_unstale":"2024-04-14T23:57:39.084349+0000","last_undegraded":"2024-04-14T23:57:39.084349+0000","last_fullsized":"2024-04-14T23:57:39.084349+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:44:01.195501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022579950000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":28,"num_read_kb":14,"num_write":180,"num_write_kb":90,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"34'2","reported_seq":68,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.012051+0000","last_change":"2024-04-14T23:48:42.232164+0000","last_active":"2024-04-14T23:57:39.012051+0000","last_peered":"2024-04-14T23:57:39.012051+0000","last_clean":"2024-04-14T23:57:39.012051+0000","last_became_active":"2024-04-14T23:48:42.229785+0000","last_became_peered":"2024-04-14T23:48:42.229785+0000","last_unstale":"2024-04-14T23:57:39.012051+0000","last_undegraded":"2024-04-14T23:57:39.012051+0000","last_fullsized":"2024-04-14T23:57:39.012051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T04:21:37.538735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"36'172","reported_seq":2296,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.085217+0000","last_change":"2024-04-14T23:55:41.994270+0000","last_active":"2024-04-14T23:57:39.085217+0000","last_peered":"2024-04-14T23:57:39.085217+0000","last_clean":"2024-04-14T23:57:39.085217+0000","last_became_active":"2024-04-14T23:48:43.232612+0000","last_became_peered":"2024-04-14T23:48:43.232612+0000","last_unstale":"2024-04-14T23:57:39.085217+0000","last_undegraded":"2024-04-14T23:57:39.085217+0000","last_fullsized":"2024-04-14T23:57:39.085217+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T04:34:43.974518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031586029999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":31,"num_read_kb":17,"num_write":172,"num_write_kb":4181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":222,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.085334+0000","last_change":"2024-04-14T23:48:38.150200+0000","last_active":"2024-04-14T23:57:39.085334+0000","last_peered":"2024-04-14T23:57:39.085334+0000","last_clean":"2024-04-14T23:57:39.085334+0000","last_became_active":"2024-04-14T23:48:38.149301+0000","last_became_peered":"2024-04-14T23:48:38.149301+0000","last_unstale":"2024-04-14T23:57:39.085334+0000","last_undegraded":"2024-04-14T23:57:39.085334+0000","last_fullsized":"2024-04-14T23:57:39.085334+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:37.129663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:37.129663+0000","last_clean_scrub_stamp":"2024-04-14T23:48:37.129663+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T23:58:01.303851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"45'4","reported_seq":70,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.083463+0000","last_change":"2024-04-14T23:48:42.229238+0000","last_active":"2024-04-14T23:57:39.083463+0000","last_peered":"2024-04-14T23:57:39.083463+0000","last_clean":"2024-04-14T23:57:39.083463+0000","last_became_active":"2024-04-14T23:48:42.228994+0000","last_became_peered":"2024-04-14T23:48:42.228994+0000","last_unstale":"2024-04-14T23:57:39.083463+0000","last_undegraded":"2024-04-14T23:57:39.083463+0000","last_fullsized":"2024-04-14T23:57:39.083463+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:47:15.406228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"44'163","reported_seq":2289,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.092269+0000","last_change":"2024-04-14T23:55:42.318283+0000","last_active":"2024-04-14T23:57:39.092269+0000","last_peered":"2024-04-14T23:57:39.092269+0000","last_clean":"2024-04-14T23:57:39.092269+0000","last_became_active":"2024-04-14T23:48:43.223665+0000","last_became_peered":"2024-04-14T23:48:43.223665+0000","last_unstale":"2024-04-14T23:57:39.092269+0000","last_undegraded":"2024-04-14T23:57:39.092269+0000","last_fullsized":"2024-04-14T23:57:39.092269+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":163,"log_dups_size":0,"ondisk_log_size":163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:47:10.121417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012914624,"stat_sum":{"num_bytes":4902,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":33,"num_read_kb":14,"num_write":161,"num_write_kb":4180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"45'89","reported_seq":161,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:03.889202+0000","last_change":"2024-04-14T23:48:42.230404+0000","last_active":"2024-04-14T23:58:03.889202+0000","last_peered":"2024-04-14T23:58:03.889202+0000","last_clean":"2024-04-14T23:58:03.889202+0000","last_became_active":"2024-04-14T23:48:42.230173+0000","last_became_peered":"2024-04-14T23:48:42.230173+0000","last_unstale":"2024-04-14T23:58:03.889202+0000","last_undegraded":"2024-04-14T23:58:03.889202+0000","last_fullsized":"2024-04-14T23:58:03.889202+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:23:45.666680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8660106,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":80,"num_write_kb":8498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"42'152","reported_seq":2272,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.087977+0000","last_change":"2024-04-14T23:55:42.315208+0000","last_active":"2024-04-14T23:57:39.087977+0000","last_peered":"2024-04-14T23:57:39.087977+0000","last_clean":"2024-04-14T23:57:39.087977+0000","last_became_active":"2024-04-14T23:48:43.231025+0000","last_became_peered":"2024-04-14T23:48:43.231025+0000","last_unstale":"2024-04-14T23:57:39.087977+0000","last_undegraded":"2024-04-14T23:57:39.087977+0000","last_fullsized":"2024-04-14T23:57:39.087977+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:22:50.033217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014709888000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":27,"num_read_kb":10,"num_write":150,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'131","reported_seq":328,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.011644+0000","last_change":"2024-04-14T23:48:42.240651+0000","last_active":"2024-04-14T23:57:39.011644+0000","last_peered":"2024-04-14T23:57:39.011644+0000","last_clean":"2024-04-14T23:57:39.011644+0000","last_became_active":"2024-04-14T23:48:42.240351+0000","last_became_peered":"2024-04-14T23:48:42.240351+0000","last_unstale":"2024-04-14T23:57:39.011644+0000","last_undegraded":"2024-04-14T23:57:39.011644+0000","last_fullsized":"2024-04-14T23:57:39.011644+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:03:02.804444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":533490,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":131,"num_read_kb":605,"num_write":126,"num_write_kb":605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"44'152","reported_seq":2274,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.086207+0000","last_change":"2024-04-14T23:55:42.306590+0000","last_active":"2024-04-14T23:57:39.086207+0000","last_peered":"2024-04-14T23:57:39.086207+0000","last_clean":"2024-04-14T23:57:39.086207+0000","last_became_active":"2024-04-14T23:48:43.232120+0000","last_became_peered":"2024-04-14T23:48:43.232120+0000","last_unstale":"2024-04-14T23:57:39.086207+0000","last_undegraded":"2024-04-14T23:57:39.086207+0000","last_fullsized":"2024-04-14T23:57:39.086207+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:07:57.642700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0096232650000000006,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":29,"num_read_kb":17,"num_write":152,"num_write_kb":4171,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"34'156","reported_seq":228,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:44.101825+0000","last_change":"2024-04-14T23:48:42.233743+0000","last_active":"2024-04-14T23:57:44.101825+0000","last_peered":"2024-04-14T23:57:44.101825+0000","last_clean":"2024-04-14T23:57:44.101825+0000","last_became_active":"2024-04-14T23:48:42.233354+0000","last_became_peered":"2024-04-14T23:48:42.233354+0000","last_unstale":"2024-04-14T23:57:44.101825+0000","last_undegraded":"2024-04-14T23:57:44.101825+0000","last_fullsized":"2024-04-14T23:57:44.101825+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:49:23.045968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":154,"num_write_kb":8284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"44'183","reported_seq":2314,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.085863+0000","last_change":"2024-04-14T23:55:42.224478+0000","last_active":"2024-04-14T23:57:39.085863+0000","last_peered":"2024-04-14T23:57:39.085863+0000","last_clean":"2024-04-14T23:57:39.085863+0000","last_became_active":"2024-04-14T23:48:43.232119+0000","last_became_peered":"2024-04-14T23:48:43.232119+0000","last_unstale":"2024-04-14T23:57:39.085863+0000","last_undegraded":"2024-04-14T23:57:39.085863+0000","last_fullsized":"2024-04-14T23:57:39.085863+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:37:54.631325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012271420000000001,"stat_sum":{"num_bytes":192,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":38,"num_read_kb":16,"num_write":183,"num_write_kb":92,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"45'3","reported_seq":69,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.088128+0000","last_change":"2024-04-14T23:48:42.242249+0000","last_active":"2024-04-14T23:57:39.088128+0000","last_peered":"2024-04-14T23:57:39.088128+0000","last_clean":"2024-04-14T23:57:39.088128+0000","last_became_active":"2024-04-14T23:48:42.242039+0000","last_became_peered":"2024-04-14T23:48:42.242039+0000","last_unstale":"2024-04-14T23:57:39.088128+0000","last_undegraded":"2024-04-14T23:57:39.088128+0000","last_fullsized":"2024-04-14T23:57:39.088128+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:46:16.301900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"44'162","reported_seq":2291,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.084067+0000","last_change":"2024-04-14T23:55:42.085024+0000","last_active":"2024-04-14T23:57:39.084067+0000","last_peered":"2024-04-14T23:57:39.084067+0000","last_clean":"2024-04-14T23:57:39.084067+0000","last_became_active":"2024-04-14T23:48:43.241776+0000","last_became_peered":"2024-04-14T23:48:43.241776+0000","last_unstale":"2024-04-14T23:57:39.084067+0000","last_undegraded":"2024-04-14T23:57:39.084067+0000","last_fullsized":"2024-04-14T23:57:39.084067+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:05:29.929626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024520940000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":36,"num_read_kb":17,"num_write":162,"num_write_kb":8271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"45'29","reported_seq":98,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:45.314264+0000","last_change":"2024-04-14T23:48:42.232537+0000","last_active":"2024-04-14T23:57:45.314264+0000","last_peered":"2024-04-14T23:57:45.314264+0000","last_clean":"2024-04-14T23:57:45.314264+0000","last_became_active":"2024-04-14T23:48:42.232325+0000","last_became_peered":"2024-04-14T23:48:42.232325+0000","last_unstale":"2024-04-14T23:57:45.314264+0000","last_undegraded":"2024-04-14T23:57:45.314264+0000","last_fullsized":"2024-04-14T23:57:45.314264+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:34:37.448301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":25,"num_write_kb":4108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"44'166","reported_seq":2288,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.086381+0000","last_change":"2024-04-14T23:55:42.308941+0000","last_active":"2024-04-14T23:57:39.086381+0000","last_peered":"2024-04-14T23:57:39.086381+0000","last_clean":"2024-04-14T23:57:39.086381+0000","last_became_active":"2024-04-14T23:48:43.234150+0000","last_became_peered":"2024-04-14T23:48:43.234150+0000","last_unstale":"2024-04-14T23:57:39.086381+0000","last_undegraded":"2024-04-14T23:57:39.086381+0000","last_fullsized":"2024-04-14T23:57:39.086381+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":166,"log_dups_size":0,"ondisk_log_size":166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:55:59.231275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011050757,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":29,"num_read_kb":14,"num_write":166,"num_write_kb":4178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"45'182","reported_seq":254,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:02.552170+0000","last_change":"2024-04-14T23:48:42.230014+0000","last_active":"2024-04-14T23:58:02.552170+0000","last_peered":"2024-04-14T23:58:02.552170+0000","last_clean":"2024-04-14T23:58:02.552170+0000","last_became_active":"2024-04-14T23:48:42.229712+0000","last_became_peered":"2024-04-14T23:48:42.229712+0000","last_unstale":"2024-04-14T23:58:02.552170+0000","last_undegraded":"2024-04-14T23:58:02.552170+0000","last_fullsized":"2024-04-14T23:58:02.552170+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":182,"log_dups_size":0,"ondisk_log_size":182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:54:20.595391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":142,"num_write":414,"num_write_kb":617,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"44'166","reported_seq":2297,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.094045+0000","last_change":"2024-04-14T23:55:42.320181+0000","last_active":"2024-04-14T23:57:39.094045+0000","last_peered":"2024-04-14T23:57:39.094045+0000","last_clean":"2024-04-14T23:57:39.094045+0000","last_became_active":"2024-04-14T23:48:43.228739+0000","last_became_peered":"2024-04-14T23:48:43.228739+0000","last_unstale":"2024-04-14T23:57:39.094045+0000","last_undegraded":"2024-04-14T23:57:39.094045+0000","last_fullsized":"2024-04-14T23:57:39.094045+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":166,"log_dups_size":0,"ondisk_log_size":166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:27:33.331878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012444432,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":38,"num_read_kb":16,"num_write":166,"num_write_kb":4178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"45'208","reported_seq":285,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:02.646576+0000","last_change":"2024-04-14T23:48:42.220035+0000","last_active":"2024-04-14T23:58:02.646576+0000","last_peered":"2024-04-14T23:58:02.646576+0000","last_clean":"2024-04-14T23:58:02.646576+0000","last_became_active":"2024-04-14T23:48:42.219750+0000","last_became_peered":"2024-04-14T23:48:42.219750+0000","last_unstale":"2024-04-14T23:58:02.646576+0000","last_undegraded":"2024-04-14T23:58:02.646576+0000","last_fullsized":"2024-04-14T23:58:02.646576+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:00:58.928166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":25,"num_read_kb":6094,"num_write":202,"num_write_kb":4203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"44'176","reported_seq":2313,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.091734+0000","last_change":"2024-04-14T23:55:42.319020+0000","last_active":"2024-04-14T23:57:39.091734+0000","last_peered":"2024-04-14T23:57:39.091734+0000","last_clean":"2024-04-14T23:57:39.091734+0000","last_became_active":"2024-04-14T23:48:43.232730+0000","last_became_peered":"2024-04-14T23:48:43.232730+0000","last_unstale":"2024-04-14T23:57:39.091734+0000","last_undegraded":"2024-04-14T23:57:39.091734+0000","last_fullsized":"2024-04-14T23:57:39.091734+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":176,"log_dups_size":0,"ondisk_log_size":176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:38:48.078009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014562113999999999,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":45,"num_read_kb":26,"num_write":171,"num_write_kb":8277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'28","reported_seq":97,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.084866+0000","last_change":"2024-04-14T23:48:42.232087+0000","last_active":"2024-04-14T23:57:39.084866+0000","last_peered":"2024-04-14T23:57:39.084866+0000","last_clean":"2024-04-14T23:57:39.084866+0000","last_became_active":"2024-04-14T23:48:42.231796+0000","last_became_peered":"2024-04-14T23:48:42.231796+0000","last_unstale":"2024-04-14T23:57:39.084866+0000","last_undegraded":"2024-04-14T23:57:39.084866+0000","last_fullsized":"2024-04-14T23:57:39.084866+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:14:35.050093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":27,"num_write_kb":4121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"43'183","reported_seq":2313,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.085628+0000","last_change":"2024-04-14T23:55:42.224366+0000","last_active":"2024-04-14T23:57:39.085628+0000","last_peered":"2024-04-14T23:57:39.085628+0000","last_clean":"2024-04-14T23:57:39.085628+0000","last_became_active":"2024-04-14T23:48:43.232594+0000","last_became_peered":"2024-04-14T23:48:43.232594+0000","last_unstale":"2024-04-14T23:57:39.085628+0000","last_undegraded":"2024-04-14T23:57:39.085628+0000","last_fullsized":"2024-04-14T23:57:39.085628+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T09:41:55.211912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001144237,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":37,"num_read_kb":20,"num_write":182,"num_write_kb":91,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"44'240","reported_seq":308,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.087346+0000","last_change":"2024-04-14T23:48:42.231237+0000","last_active":"2024-04-14T23:57:39.087346+0000","last_peered":"2024-04-14T23:57:39.087346+0000","last_clean":"2024-04-14T23:57:39.087346+0000","last_became_active":"2024-04-14T23:48:42.230902+0000","last_became_peered":"2024-04-14T23:48:42.230902+0000","last_unstale":"2024-04-14T23:57:39.087346+0000","last_undegraded":"2024-04-14T23:57:39.087346+0000","last_fullsized":"2024-04-14T23:57:39.087346+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":240,"log_dups_size":0,"ondisk_log_size":240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:28:37.228039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":238,"num_write_kb":4263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"44'168","reported_seq":2294,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.092200+0000","last_change":"2024-04-14T23:55:42.316834+0000","last_active":"2024-04-14T23:57:39.092200+0000","last_peered":"2024-04-14T23:57:39.092200+0000","last_clean":"2024-04-14T23:57:39.092200+0000","last_became_active":"2024-04-14T23:48:43.232747+0000","last_became_peered":"2024-04-14T23:48:43.232747+0000","last_unstale":"2024-04-14T23:57:39.092200+0000","last_undegraded":"2024-04-14T23:57:39.092200+0000","last_fullsized":"2024-04-14T23:57:39.092200+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:13:25.640712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014090612000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":33,"num_read_kb":16,"num_write":168,"num_write_kb":8274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"34'24","reported_seq":94,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:45.019436+0000","last_change":"2024-04-14T23:48:42.240496+0000","last_active":"2024-04-14T23:57:45.019436+0000","last_peered":"2024-04-14T23:57:45.019436+0000","last_clean":"2024-04-14T23:57:45.019436+0000","last_became_active":"2024-04-14T23:48:42.238524+0000","last_became_peered":"2024-04-14T23:48:42.238524+0000","last_unstale":"2024-04-14T23:57:45.019436+0000","last_undegraded":"2024-04-14T23:57:45.019436+0000","last_fullsized":"2024-04-14T23:57:45.019436+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:29:35.218759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":23,"num_write_kb":4102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"44'163","reported_seq":2291,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:45.295617+0000","last_change":"2024-04-14T23:55:42.318594+0000","last_active":"2024-04-14T23:57:45.295617+0000","last_peered":"2024-04-14T23:57:45.295617+0000","last_clean":"2024-04-14T23:57:45.295617+0000","last_became_active":"2024-04-14T23:48:43.233651+0000","last_became_peered":"2024-04-14T23:48:43.233651+0000","last_unstale":"2024-04-14T23:57:45.295617+0000","last_undegraded":"2024-04-14T23:57:45.295617+0000","last_fullsized":"2024-04-14T23:57:45.295617+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":163,"log_dups_size":0,"ondisk_log_size":163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:17:15.414893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015058134000000001,"stat_sum":{"num_bytes":8390104,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":35,"num_read_kb":13,"num_write":163,"num_write_kb":8274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"44'70","reported_seq":148,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:45.105836+0000","last_change":"2024-04-14T23:48:42.218897+0000","last_active":"2024-04-14T23:57:45.105836+0000","last_peered":"2024-04-14T23:57:45.105836+0000","last_clean":"2024-04-14T23:57:45.105836+0000","last_became_active":"2024-04-14T23:48:42.218580+0000","last_became_peered":"2024-04-14T23:48:42.218580+0000","last_unstale":"2024-04-14T23:57:45.105836+0000","last_undegraded":"2024-04-14T23:57:45.105836+0000","last_fullsized":"2024-04-14T23:57:45.105836+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:26:44.477638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":28682,"num_write":62,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"35'166","reported_seq":2282,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.010030+0000","last_change":"2024-04-14T23:55:41.991867+0000","last_active":"2024-04-14T23:57:39.010030+0000","last_peered":"2024-04-14T23:57:39.010030+0000","last_clean":"2024-04-14T23:57:39.010030+0000","last_became_active":"2024-04-14T23:48:43.219601+0000","last_became_peered":"2024-04-14T23:48:43.219601+0000","last_unstale":"2024-04-14T23:57:39.010030+0000","last_undegraded":"2024-04-14T23:57:39.010030+0000","last_fullsized":"2024-04-14T23:57:39.010030+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":166,"log_dups_size":0,"ondisk_log_size":166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:23:08.643833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001540055,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":23,"num_read_kb":13,"num_write":166,"num_write_kb":83,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"44'151","reported_seq":2286,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.083755+0000","last_change":"2024-04-14T23:55:42.084976+0000","last_active":"2024-04-14T23:57:39.083755+0000","last_peered":"2024-04-14T23:57:39.083755+0000","last_clean":"2024-04-14T23:57:39.083755+0000","last_became_active":"2024-04-14T23:48:43.221842+0000","last_became_peered":"2024-04-14T23:48:43.221842+0000","last_unstale":"2024-04-14T23:57:39.083755+0000","last_undegraded":"2024-04-14T23:57:39.083755+0000","last_fullsized":"2024-04-14T23:57:39.083755+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":151,"log_dups_size":0,"ondisk_log_size":151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:58:33.633046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024436879999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":43,"num_read_kb":23,"num_write":150,"num_write_kb":4170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"33'62","reported_seq":137,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:44.931184+0000","last_change":"2024-04-14T23:48:42.221314+0000","last_active":"2024-04-14T23:57:44.931184+0000","last_peered":"2024-04-14T23:57:44.931184+0000","last_clean":"2024-04-14T23:57:44.931184+0000","last_became_active":"2024-04-14T23:48:42.221028+0000","last_became_peered":"2024-04-14T23:48:42.221028+0000","last_unstale":"2024-04-14T23:57:44.931184+0000","last_undegraded":"2024-04-14T23:57:44.931184+0000","last_fullsized":"2024-04-14T23:57:44.931184+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T09:58:18.195400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":24,"num_read_kb":12309,"num_write":62,"num_write_kb":4128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"44'165","reported_seq":2288,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.083812+0000","last_change":"2024-04-14T23:55:42.084580+0000","last_active":"2024-04-14T23:57:39.083812+0000","last_peered":"2024-04-14T23:57:39.083812+0000","last_clean":"2024-04-14T23:57:39.083812+0000","last_became_active":"2024-04-14T23:48:43.224312+0000","last_became_peered":"2024-04-14T23:48:43.224312+0000","last_unstale":"2024-04-14T23:57:39.083812+0000","last_undegraded":"2024-04-14T23:57:39.083812+0000","last_fullsized":"2024-04-14T23:57:39.083812+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:55:12.275566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00232957,"stat_sum":{"num_bytes":4674,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":31,"num_read_kb":13,"num_write":165,"num_write_kb":4182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"44'668","reported_seq":753,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:44.946209+0000","last_change":"2024-04-14T23:48:42.221000+0000","last_active":"2024-04-14T23:57:44.946209+0000","last_peered":"2024-04-14T23:57:44.946209+0000","last_clean":"2024-04-14T23:57:44.946209+0000","last_became_active":"2024-04-14T23:48:42.219813+0000","last_became_peered":"2024-04-14T23:48:42.219813+0000","last_unstale":"2024-04-14T23:57:44.946209+0000","last_undegraded":"2024-04-14T23:57:44.946209+0000","last_fullsized":"2024-04-14T23:57:44.946209+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":668,"log_dups_size":0,"ondisk_log_size":668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:00:38.107953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":44198,"num_write":659,"num_write_kb":12787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"37'165","reported_seq":2293,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.087948+0000","last_change":"2024-04-14T23:55:42.313777+0000","last_active":"2024-04-14T23:57:39.087948+0000","last_peered":"2024-04-14T23:57:39.087948+0000","last_clean":"2024-04-14T23:57:39.087948+0000","last_became_active":"2024-04-14T23:48:43.236595+0000","last_became_peered":"2024-04-14T23:48:43.236595+0000","last_unstale":"2024-04-14T23:57:39.087948+0000","last_undegraded":"2024-04-14T23:57:39.087948+0000","last_fullsized":"2024-04-14T23:57:39.087948+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:13:32.883032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013391734000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":19,"num_write":164,"num_write_kb":82,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"44'47","reported_seq":118,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.094119+0000","last_change":"2024-04-14T23:48:42.239255+0000","last_active":"2024-04-14T23:57:39.094119+0000","last_peered":"2024-04-14T23:57:39.094119+0000","last_clean":"2024-04-14T23:57:39.094119+0000","last_became_active":"2024-04-14T23:48:42.238892+0000","last_became_peered":"2024-04-14T23:48:42.238892+0000","last_unstale":"2024-04-14T23:57:39.094119+0000","last_undegraded":"2024-04-14T23:57:39.094119+0000","last_fullsized":"2024-04-14T23:57:39.094119+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:44:39.632155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":16384,"num_write":44,"num_write_kb":4126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"44'175","reported_seq":2298,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.092968+0000","last_change":"2024-04-14T23:55:42.319487+0000","last_active":"2024-04-14T23:57:39.092968+0000","last_peered":"2024-04-14T23:57:39.092968+0000","last_clean":"2024-04-14T23:57:39.092968+0000","last_became_active":"2024-04-14T23:48:43.222707+0000","last_became_peered":"2024-04-14T23:48:43.222707+0000","last_unstale":"2024-04-14T23:57:39.092968+0000","last_undegraded":"2024-04-14T23:57:39.092968+0000","last_fullsized":"2024-04-14T23:57:39.092968+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:47:16.501806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011942019,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":30,"num_read_kb":12,"num_write":175,"num_write_kb":4183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"45'76","reported_seq":157,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:46.454133+0000","last_change":"2024-04-14T23:48:42.226563+0000","last_active":"2024-04-14T23:57:46.454133+0000","last_peered":"2024-04-14T23:57:46.454133+0000","last_clean":"2024-04-14T23:57:46.454133+0000","last_became_active":"2024-04-14T23:48:42.226238+0000","last_became_peered":"2024-04-14T23:48:42.226238+0000","last_unstale":"2024-04-14T23:57:46.454133+0000","last_undegraded":"2024-04-14T23:57:46.454133+0000","last_fullsized":"2024-04-14T23:57:46.454133+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:01:36.516898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":25,"num_read_kb":20500,"num_write":71,"num_write_kb":4136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":50360410,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":2223,"num_read_kb":65817,"num_write":10624,"num_write_kb":276412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":100835328,"data_stored":100720820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10722,"ondisk_log_size":10722,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":151801402,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":553,"num_read_kb":479116,"num_write":3786,"num_write_kb":151262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":64675840,"data_stored":303602804,"data_compressed":34392348,"data_compressed_allocated":44802048,"data_compressed_original":283795456,"omap_allocated":0,"internal_metadata":0},"log_size":3640,"ondisk_log_size":3640,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65128,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574965,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":88644,"kb_used_data":25532,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94283196,"statfs":{"total":96636764160,"available":96545992704,"internally_reserved":0,"allocated":26144768,"data_stored":72411425,"data_compressed":6887616,"data_compressed_allocated":9027584,"data_compressed_original":55586816,"omap_allocated":1590,"internal_metadata":64616906},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574966,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77724,"kb_used_data":32660,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94294116,"statfs":{"total":96636764160,"available":96557174784,"internally_reserved":0,"allocated":33443840,"data_stored":71547043,"data_compressed":5492002,"data_compressed_allocated":7053312,"data_compressed_original":45449216,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574965,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81436,"kb_used_data":14164,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94290404,"statfs":{"total":96636764160,"available":96553373696,"internally_reserved":0,"allocated":14503936,"data_stored":38005896,"data_compressed":3110985,"data_compressed_allocated":4100096,"data_compressed_original":27897856,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574966,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84544,"kb_used_data":17268,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94287296,"statfs":{"total":96636764160,"available":96550191104,"internally_reserved":0,"allocated":17682432,"data_stored":54774132,"data_compressed":5258839,"data_compressed_allocated":6774784,"data_compressed_original":44175360,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574964,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":87932,"kb_used_data":20656,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94283908,"statfs":{"total":96636764160,"available":96546721792,"internally_reserved":0,"allocated":21151744,"data_stored":46919348,"data_compressed":3819253,"data_compressed_allocated":4960256,"data_compressed_original":31031296,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574965,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51228,"kb_used_data":7568,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94320612,"statfs":{"total":96636764160,"available":96584306688,"internally_reserved":0,"allocated":7749632,"data_stored":17554823,"data_compressed":1445694,"data_compressed_allocated":2031616,"data_compressed_original":12120064,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574965,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62352,"kb_used_data":17288,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94309488,"statfs":{"total":96636764160,"available":96572915712,"internally_reserved":0,"allocated":17702912,"data_stored":50842674,"data_compressed":4902040,"data_compressed_allocated":6463488,"data_compressed_original":39886848,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574964,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":97896,"kb_used_data":30624,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94273944,"statfs":{"total":96636764160,"available":96536518656,"internally_reserved":0,"allocated":31358976,"data_stored":55363963,"data_compressed":3541047,"data_compressed_allocated":4481024,"data_compressed_original":28794880,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32564,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32564,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5586944,"data_stored":29360766,"data_compressed":3508483,"data_compressed_allocated":4435968,"data_compressed_original":28221440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8802304,"data_stored":42214698,"data_compressed":4902040,"data_compressed_allocated":6463488,"data_compressed_original":39886848,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3031040,"data_stored":13116402,"data_compressed":1445694,"data_compressed_allocated":2031616,"data_compressed_original":12120064,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8028160,"data_stored":34088594,"data_compressed":3819253,"data_compressed_allocated":4960256,"data_compressed_original":31031296,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8749056,"data_stored":46137470,"data_compressed":5258839,"data_compressed_allocated":6774784,"data_compressed_original":44175360,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":5566464,"data_stored":29360218,"data_compressed":3110985,"data_compressed_allocated":4100096,"data_compressed_original":27897856,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":11943936,"data_stored":50332320,"data_compressed":5492002,"data_compressed_allocated":7053312,"data_compressed_original":45449216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12967936,"data_stored":58992336,"data_compressed":6855052,"data_compressed_allocated":8982528,"data_compressed_original":55013376,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":25194496,"data_stored":25173461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8388608,"data_stored":8388608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4206592,"data_stored":4199053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12611584,"data_stored":12591386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8421376,"data_stored":8397294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8425472,"data_stored":8406310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20987904,"data_stored":20975355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12599296,"data_stored":12589353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-14T23:58:09.733 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-14T23:58:09.733 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-14T23:58:09.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.994+0000 7f2c9f200640 1 Processor -- start 2024-04-14T23:58:10.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.994+0000 7f2c9f200640 1 -- start start 2024-04-14T23:58:10.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.996+0000 7f2c9f200640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2ca00c5380 0x7f2ca015a980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:10.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.996+0000 7f2c9f200640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ca006b4b0 con 0x7f2ca0070d50 2024-04-14T23:58:10.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.996+0000 7f2c9f200640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ca006b790 con 0x7f2ca00c4fa0 2024-04-14T23:58:10.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.996+0000 7f2c9f200640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f2ca00c69f0 con 0x7f2ca00c5380 2024-04-14T23:58:10.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.996+0000 7f2c9e800640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2ca00c5380 0x7f2ca015a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:10.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.996+0000 7f2c97e00640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f2ca0070d50 0x7f2ca015a270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:55612/0 (socket says 172.21.15.60:55612) 2024-04-14T23:58:10.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.996+0000 7f2c97e00640 1 -- 172.21.15.60:0/1075848637 learned_addr learned my addr 172.21.15.60:0/1075848637 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:58:10.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.997+0000 7f2c9e800640 1 -- 172.21.15.60:0/1075848637 >> v1:172.21.15.186:6789/0 conn(0x7f2ca00c4fa0 legacy=0x7f2ca00c77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:10.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.997+0000 7f2c9e800640 1 -- 172.21.15.60:0/1075848637 >> v1:172.21.15.60:6789/0 conn(0x7f2ca0070d50 legacy=0x7f2ca015a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:10.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.997+0000 7f2c9e800640 1 -- 172.21.15.60:0/1075848637 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ca00c6cd0 con 0x7f2ca00c5380 2024-04-14T23:58:10.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.997+0000 7f2c9d400640 1 -- 172.21.15.60:0/1075848637 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1652420738 0 0) 0x7f2ca006b4b0 con 0x7f2ca0070d50 2024-04-14T23:58:10.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.998+0000 7f2c9e800640 1 --2- 172.21.15.60:0/1075848637 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ca00c5380 0x7f2ca015a980 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f2c98009d90 tx=0x7f2c9802e160 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e663a925987afbf server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:10.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.998+0000 7f2c9d400640 1 -- 172.21.15.60:0/1075848637 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c9803b070 con 0x7f2ca00c5380 2024-04-14T23:58:10.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.998+0000 7f2c9d400640 1 -- 172.21.15.60:0/1075848637 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c98036440 con 0x7f2ca00c5380 2024-04-14T23:58:10.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.998+0000 7f2c9d400640 1 -- 172.21.15.60:0/1075848637 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c98036810 con 0x7f2ca00c5380 2024-04-14T23:58:10.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.999+0000 7f2c9f200640 1 -- 172.21.15.60:0/1075848637 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ca00c5380 msgr2=0x7f2ca015a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:10.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.999+0000 7f2c9f200640 1 --2- 172.21.15.60:0/1075848637 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ca00c5380 0x7f2ca015a980 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f2c98009d90 tx=0x7f2c9802e160 comp rx=0 tx=0).stop 2024-04-14T23:58:10.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.999+0000 7f2c9f200640 1 -- 172.21.15.60:0/1075848637 shutdown_connections 2024-04-14T23:58:10.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.999+0000 7f2c9f200640 1 --2- 172.21.15.60:0/1075848637 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ca00c5380 0x7f2ca015a980 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:10.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.999+0000 7f2c9f200640 1 -- 172.21.15.60:0/1075848637 >> 172.21.15.60:0/1075848637 conn(0x7f2ca00c8b50 msgr2=0x7f2ca00c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:10.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.999+0000 7f2c9f200640 1 -- 172.21.15.60:0/1075848637 shutdown_connections 2024-04-14T23:58:10.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:09.999+0000 7f2c9f200640 1 -- 172.21.15.60:0/1075848637 wait complete. 2024-04-14T23:58:10.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.001+0000 7f2c9f200640 1 Processor -- start 2024-04-14T23:58:10.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.001+0000 7f2c9f200640 1 -- start start 2024-04-14T23:58:10.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.002+0000 7f2c9f200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ca0070d50 0x7f2ca014f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:10.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.002+0000 7f2c9f200640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ca00c4fa0 0x7f2ca0151ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:10.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.002+0000 7f2c9f200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ca00c5380 0x7f2ca01529d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:10.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.002+0000 7f2c9f200640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f2ca006b1c0 con 0x7f2ca00c4fa0 2024-04-14T23:58:10.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.002+0000 7f2c9f200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2ca00c7290 con 0x7f2ca0070d50 2024-04-14T23:58:10.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.002+0000 7f2c9f200640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f2ca00c99d0 con 0x7f2ca00c5380 2024-04-14T23:58:10.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.002+0000 7f2c9e800640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ca00c5380 0x7f2ca01529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:10.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.002+0000 7f2c97e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ca00c4fa0 0x7f2ca0151ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:10.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.003+0000 7f2c9e800640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ca00c5380 0x7f2ca01529d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.60:51150/0 (socket says 172.21.15.60:51150) 2024-04-14T23:58:10.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.003+0000 7f2c9e800640 1 -- 172.21.15.60:0/1560482697 learned_addr learned my addr 172.21.15.60:0/1560482697 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:58:10.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.003+0000 7f2c9de00640 1 --2- 172.21.15.60:0/1560482697 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ca0070d50 0x7f2ca014f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:10.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.003+0000 7f2c9e800640 1 -- 172.21.15.60:0/1560482697 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ca0070d50 msgr2=0x7f2ca014f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:10.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.003+0000 7f2c9e800640 1 --2- 172.21.15.60:0/1560482697 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ca0070d50 0x7f2ca014f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:10.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.003+0000 7f2c9e800640 1 -- 172.21.15.60:0/1560482697 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ca00c4fa0 msgr2=0x7f2ca0151ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:10.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.003+0000 7f2c9e800640 1 --2- 172.21.15.60:0/1560482697 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ca00c4fa0 0x7f2ca0151ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:10.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.003+0000 7f2c9e800640 1 -- 172.21.15.60:0/1560482697 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ca00c1a80 con 0x7f2ca00c5380 2024-04-14T23:58:10.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.003+0000 7f2c97e00640 1 --2- 172.21.15.60:0/1560482697 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ca00c4fa0 0x7f2ca0151ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:58:10.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.003+0000 7f2c9e800640 1 --2- 172.21.15.60:0/1560482697 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ca00c5380 0x7f2ca01529d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2c98009d90 tx=0x7f2c98004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:10.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.004+0000 7f2c96000640 1 -- 172.21.15.60:0/1560482697 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c98046070 con 0x7f2ca00c5380 2024-04-14T23:58:10.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.004+0000 7f2c96000640 1 -- 172.21.15.60:0/1560482697 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c98036e20 con 0x7f2ca00c5380 2024-04-14T23:58:10.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.004+0000 7f2c96000640 1 -- 172.21.15.60:0/1560482697 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c9803b040 con 0x7f2ca00c5380 2024-04-14T23:58:10.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.004+0000 7f2c9f200640 1 -- 172.21.15.60:0/1560482697 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ca01507a0 con 0x7f2ca00c5380 2024-04-14T23:58:10.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.004+0000 7f2c9f200640 1 -- 172.21.15.60:0/1560482697 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ca0150d60 con 0x7f2ca00c5380 2024-04-14T23:58:10.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.005+0000 7f2c9f200640 1 -- 172.21.15.60:0/1560482697 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ca004f540 con 0x7f2ca00c5380 2024-04-14T23:58:10.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.006+0000 7f2c96000640 1 -- 172.21.15.60:0/1560482697 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f2c98049070 con 0x7f2ca00c5380 2024-04-14T23:58:10.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.007+0000 7f2c96000640 1 --2- 172.21.15.60:0/1560482697 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f2c7c06b990 0x7f2c7c06de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:10.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.007+0000 7f2c96000640 1 -- 172.21.15.60:0/1560482697 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(45..45 src has 1..45) ==== 6255+0+0 (secure 0 0 0) 0x7f2c980acb20 con 0x7f2ca00c5380 2024-04-14T23:58:10.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.007+0000 7f2c9de00640 1 --2- 172.21.15.60:0/1560482697 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f2c7c06b990 0x7f2c7c06de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:10.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.008+0000 7f2c9de00640 1 --2- 172.21.15.60:0/1560482697 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f2c7c06b990 0x7f2c7c06de50 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f2c90000f60 tx=0x7f2c90009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:10.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.012+0000 7f2c96000640 1 -- 172.21.15.60:0/1560482697 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2c9807af70 con 0x7f2ca00c5380 2024-04-14T23:58:10.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.209+0000 7f2c9f200640 1 -- 172.21.15.60:0/1560482697 --> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2ca00c5850 con 0x7f2c7c06b990 2024-04-14T23:58:10.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.216+0000 7f2c96000640 1 -- 172.21.15.60:0/1560482697 <== mgr.4100 v2:172.21.15.186:6832/714104835 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+255177 (secure 0 0 0) 0x7f2ca00c5850 con 0x7f2c7c06b990 2024-04-14T23:58:10.224 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:58:10.226 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-04-14T23:58:10.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.221+0000 7f2c9f200640 1 -- 172.21.15.60:0/1560482697 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f2c7c06b990 msgr2=0x7f2c7c06de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:10.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.221+0000 7f2c9f200640 1 --2- 172.21.15.60:0/1560482697 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f2c7c06b990 0x7f2c7c06de50 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f2c90000f60 tx=0x7f2c90009040 comp rx=0 tx=0).stop 2024-04-14T23:58:10.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.221+0000 7f2c9f200640 1 -- 172.21.15.60:0/1560482697 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ca00c5380 msgr2=0x7f2ca01529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:10.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.221+0000 7f2c9f200640 1 --2- 172.21.15.60:0/1560482697 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ca00c5380 0x7f2ca01529d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2c98009d90 tx=0x7f2c98004a40 comp rx=0 tx=0).stop 2024-04-14T23:58:10.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.221+0000 7f2c9f200640 1 -- 172.21.15.60:0/1560482697 shutdown_connections 2024-04-14T23:58:10.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.221+0000 7f2c9f200640 1 --2- 172.21.15.60:0/1560482697 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f2c7c06b990 0x7f2c7c06de50 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:10.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.221+0000 7f2c9f200640 1 --2- 172.21.15.60:0/1560482697 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ca00c5380 0x7f2ca01529d0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:10.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.221+0000 7f2c9f200640 1 --2- 172.21.15.60:0/1560482697 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ca00c4fa0 0x7f2ca0151ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:10.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.221+0000 7f2c9f200640 1 --2- 172.21.15.60:0/1560482697 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ca0070d50 0x7f2ca014f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:10.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.221+0000 7f2c9f200640 1 -- 172.21.15.60:0/1560482697 >> 172.21.15.60:0/1560482697 conn(0x7f2ca00c8b50 msgr2=0x7f2ca015a040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:10.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.222+0000 7f2c9f200640 1 -- 172.21.15.60:0/1560482697 shutdown_connections 2024-04-14T23:58:10.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.222+0000 7f2c9f200640 1 -- 172.21.15.60:0/1560482697 wait complete. 2024-04-14T23:58:10.247 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":325,"stamp":"2024-04-14T23:58:08.314240+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":202752180,"num_objects":92,"num_object_clones":0,"num_object_copies":184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92,"num_whiteouts":0,"num_read":2892,"num_read_kb":545033,"num_write":14569,"num_write_kb":430090,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14479,"ondisk_log_size":14479,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":631756,"kb_used_data":165760,"kb_used_omap":12,"kb_used_meta":465907,"kb_avail":754342964,"statfs":{"total":773094113280,"available":772447195136,"internally_reserved":0,"allocated":169738240,"data_stored":407419304,"data_compressed":34457476,"data_compressed_allocated":44892160,"data_compressed_original":284942336,"omap_allocated":12752,"internal_metadata":477089328},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":567468,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":53,"num_read_kb":40,"num_write":60,"num_write_kb":587,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003437"},"pg_stats":[{"pgid":"3.3f","version":"36'158","reported_seq":2280,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.085799+0000","last_change":"2024-04-14T23:55:41.994501+0000","last_active":"2024-04-14T23:57:39.085799+0000","last_peered":"2024-04-14T23:57:39.085799+0000","last_clean":"2024-04-14T23:57:39.085799+0000","last_became_active":"2024-04-14T23:48:43.222278+0000","last_became_peered":"2024-04-14T23:48:43.222278+0000","last_unstale":"2024-04-14T23:57:39.085799+0000","last_undegraded":"2024-04-14T23:57:39.085799+0000","last_fullsized":"2024-04-14T23:57:39.085799+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T04:04:17.956339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025657449999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":29,"num_read_kb":15,"num_write":158,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"44'142","reported_seq":2269,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.088582+0000","last_change":"2024-04-14T23:55:42.310665+0000","last_active":"2024-04-14T23:57:39.088582+0000","last_peered":"2024-04-14T23:57:39.088582+0000","last_clean":"2024-04-14T23:57:39.088582+0000","last_became_active":"2024-04-14T23:48:43.226426+0000","last_became_peered":"2024-04-14T23:48:43.226426+0000","last_unstale":"2024-04-14T23:57:39.088582+0000","last_undegraded":"2024-04-14T23:57:39.088582+0000","last_fullsized":"2024-04-14T23:57:39.088582+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:58:52.815232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011741033999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":16,"num_write":142,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"37'171","reported_seq":2299,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.087926+0000","last_change":"2024-04-14T23:55:42.315743+0000","last_active":"2024-04-14T23:57:39.087926+0000","last_peered":"2024-04-14T23:57:39.087926+0000","last_clean":"2024-04-14T23:57:39.087926+0000","last_became_active":"2024-04-14T23:48:43.236616+0000","last_became_peered":"2024-04-14T23:48:43.236616+0000","last_unstale":"2024-04-14T23:57:39.087926+0000","last_undegraded":"2024-04-14T23:57:39.087926+0000","last_fullsized":"2024-04-14T23:57:39.087926+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:56:34.635393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01445958,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":38,"num_read_kb":8211,"num_write":164,"num_write_kb":8272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"44'140","reported_seq":2253,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.083881+0000","last_change":"2024-04-14T23:55:42.084275+0000","last_active":"2024-04-14T23:57:39.083881+0000","last_peered":"2024-04-14T23:57:39.083881+0000","last_clean":"2024-04-14T23:57:39.083881+0000","last_became_active":"2024-04-14T23:48:43.242000+0000","last_became_peered":"2024-04-14T23:48:43.242000+0000","last_unstale":"2024-04-14T23:57:39.083881+0000","last_undegraded":"2024-04-14T23:57:39.083881+0000","last_fullsized":"2024-04-14T23:57:39.083881+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:06:54.726051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020739690000000002,"stat_sum":{"num_bytes":193,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":9,"num_write":139,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"35'197","reported_seq":2310,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.086395+0000","last_change":"2024-04-14T23:55:42.308076+0000","last_active":"2024-04-14T23:57:39.086395+0000","last_peered":"2024-04-14T23:57:39.086395+0000","last_clean":"2024-04-14T23:57:39.086395+0000","last_became_active":"2024-04-14T23:49:29.448078+0000","last_became_peered":"2024-04-14T23:49:29.448078+0000","last_unstale":"2024-04-14T23:57:39.086395+0000","last_undegraded":"2024-04-14T23:57:39.086395+0000","last_fullsized":"2024-04-14T23:57:39.086395+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":197,"log_dups_size":0,"ondisk_log_size":197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:09:40.348011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011461469,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":27,"num_read_kb":18,"num_write":196,"num_write_kb":98,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3a","version":"35'170","reported_seq":2294,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.083401+0000","last_change":"2024-04-14T23:55:42.084342+0000","last_active":"2024-04-14T23:57:39.083401+0000","last_peered":"2024-04-14T23:57:39.083401+0000","last_clean":"2024-04-14T23:57:39.083401+0000","last_became_active":"2024-04-14T23:48:43.230405+0000","last_became_peered":"2024-04-14T23:48:43.230405+0000","last_unstale":"2024-04-14T23:57:39.083401+0000","last_undegraded":"2024-04-14T23:57:39.083401+0000","last_fullsized":"2024-04-14T23:57:39.083401+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:40:55.017190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002322771,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":32,"num_read_kb":19,"num_write":170,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"44'198","reported_seq":2327,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.084946+0000","last_change":"2024-04-14T23:55:41.992292+0000","last_active":"2024-04-14T23:57:39.084946+0000","last_peered":"2024-04-14T23:57:39.084946+0000","last_clean":"2024-04-14T23:57:39.084946+0000","last_became_active":"2024-04-14T23:48:43.230784+0000","last_became_peered":"2024-04-14T23:48:43.230784+0000","last_unstale":"2024-04-14T23:57:39.084946+0000","last_undegraded":"2024-04-14T23:57:39.084946+0000","last_fullsized":"2024-04-14T23:57:39.084946+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":198,"log_dups_size":0,"ondisk_log_size":198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T04:49:40.437771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023027080000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":37,"num_read_kb":16,"num_write":198,"num_write_kb":4194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"44'133","reported_seq":2251,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.091139+0000","last_change":"2024-04-14T23:55:42.317071+0000","last_active":"2024-04-14T23:57:39.091139+0000","last_peered":"2024-04-14T23:57:39.091139+0000","last_clean":"2024-04-14T23:57:39.091139+0000","last_became_active":"2024-04-14T23:48:43.235719+0000","last_became_peered":"2024-04-14T23:48:43.235719+0000","last_unstale":"2024-04-14T23:57:39.091139+0000","last_undegraded":"2024-04-14T23:57:39.091139+0000","last_fullsized":"2024-04-14T23:57:39.091139+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:40:21.290823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015616025,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":26,"num_read_kb":10,"num_write":132,"num_write_kb":8256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"44'157","reported_seq":2283,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.085798+0000","last_change":"2024-04-14T23:55:41.994669+0000","last_active":"2024-04-14T23:57:39.085798+0000","last_peered":"2024-04-14T23:57:39.085798+0000","last_clean":"2024-04-14T23:57:39.085798+0000","last_became_active":"2024-04-14T23:48:43.231751+0000","last_became_peered":"2024-04-14T23:48:43.231751+0000","last_unstale":"2024-04-14T23:57:39.085798+0000","last_undegraded":"2024-04-14T23:57:39.085798+0000","last_fullsized":"2024-04-14T23:57:39.085798+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:44:15.830503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025112200000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":4114,"num_write":154,"num_write_kb":8271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"35'164","reported_seq":2283,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.010548+0000","last_change":"2024-04-14T23:55:41.992813+0000","last_active":"2024-04-14T23:57:39.010548+0000","last_peered":"2024-04-14T23:57:39.010548+0000","last_clean":"2024-04-14T23:57:39.010548+0000","last_became_active":"2024-04-14T23:48:43.224592+0000","last_became_peered":"2024-04-14T23:48:43.224592+0000","last_unstale":"2024-04-14T23:57:39.010548+0000","last_undegraded":"2024-04-14T23:57:39.010548+0000","last_fullsized":"2024-04-14T23:57:39.010548+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:36:03.843739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014741629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":27,"num_read_kb":16,"num_write":162,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"44'171","reported_seq":2298,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.010380+0000","last_change":"2024-04-14T23:55:41.991958+0000","last_active":"2024-04-14T23:57:39.010380+0000","last_peered":"2024-04-14T23:57:39.010380+0000","last_clean":"2024-04-14T23:57:39.010380+0000","last_became_active":"2024-04-14T23:48:43.224989+0000","last_became_peered":"2024-04-14T23:48:43.224989+0000","last_unstale":"2024-04-14T23:57:39.010380+0000","last_undegraded":"2024-04-14T23:57:39.010380+0000","last_fullsized":"2024-04-14T23:57:39.010380+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:31:12.485950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001581301,"stat_sum":{"num_bytes":4197091,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":35,"num_read_kb":18,"num_write":170,"num_write_kb":4183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"44'146","reported_seq":2262,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.086409+0000","last_change":"2024-04-14T23:55:42.224637+0000","last_active":"2024-04-14T23:57:39.086409+0000","last_peered":"2024-04-14T23:57:39.086409+0000","last_clean":"2024-04-14T23:57:39.086409+0000","last_became_active":"2024-04-14T23:48:43.225892+0000","last_became_peered":"2024-04-14T23:48:43.225892+0000","last_unstale":"2024-04-14T23:57:39.086409+0000","last_undegraded":"2024-04-14T23:57:39.086409+0000","last_fullsized":"2024-04-14T23:57:39.086409+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":146,"log_dups_size":0,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:30:03.715201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011254640000000001,"stat_sum":{"num_bytes":73,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":24,"num_read_kb":10,"num_write":145,"num_write_kb":8263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"37'146","reported_seq":2267,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.092688+0000","last_change":"2024-04-14T23:55:42.317691+0000","last_active":"2024-04-14T23:57:39.092688+0000","last_peered":"2024-04-14T23:57:39.092688+0000","last_clean":"2024-04-14T23:57:39.092688+0000","last_became_active":"2024-04-14T23:48:43.219614+0000","last_became_peered":"2024-04-14T23:48:43.219614+0000","last_unstale":"2024-04-14T23:57:39.092688+0000","last_undegraded":"2024-04-14T23:57:39.092688+0000","last_fullsized":"2024-04-14T23:57:39.092688+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":146,"log_dups_size":0,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:11:13.830037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012872641000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":30,"num_read_kb":4115,"num_write":142,"num_write_kb":8261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"44'175","reported_seq":2308,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.090824+0000","last_change":"2024-04-14T23:55:42.319365+0000","last_active":"2024-04-14T23:57:39.090824+0000","last_peered":"2024-04-14T23:57:39.090824+0000","last_clean":"2024-04-14T23:57:39.090824+0000","last_became_active":"2024-04-14T23:48:43.239136+0000","last_became_peered":"2024-04-14T23:48:43.239136+0000","last_unstale":"2024-04-14T23:57:39.090824+0000","last_undegraded":"2024-04-14T23:57:39.090824+0000","last_fullsized":"2024-04-14T23:57:39.090824+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:47:21.467652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015761819,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":40,"num_read_kb":19,"num_write":174,"num_write_kb":12372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"44'159","reported_seq":2289,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.090236+0000","last_change":"2024-04-14T23:55:42.308265+0000","last_active":"2024-04-14T23:57:39.090236+0000","last_peered":"2024-04-14T23:57:39.090236+0000","last_clean":"2024-04-14T23:57:39.090236+0000","last_became_active":"2024-04-14T23:48:43.227898+0000","last_became_peered":"2024-04-14T23:48:43.227898+0000","last_unstale":"2024-04-14T23:57:39.090236+0000","last_undegraded":"2024-04-14T23:57:39.090236+0000","last_fullsized":"2024-04-14T23:57:39.090236+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:18:28.787829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011044834999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":37,"num_read_kb":19,"num_write":159,"num_write_kb":4178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"44'184","reported_seq":2317,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.083264+0000","last_change":"2024-04-14T23:55:42.083783+0000","last_active":"2024-04-14T23:57:39.083264+0000","last_peered":"2024-04-14T23:57:39.083264+0000","last_clean":"2024-04-14T23:57:39.083264+0000","last_became_active":"2024-04-14T23:48:43.238714+0000","last_became_peered":"2024-04-14T23:48:43.238714+0000","last_unstale":"2024-04-14T23:57:39.083264+0000","last_undegraded":"2024-04-14T23:57:39.083264+0000","last_fullsized":"2024-04-14T23:57:39.083264+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":184,"log_dups_size":0,"ondisk_log_size":184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:37:44.837525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019546920000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":40,"num_read_kb":19,"num_write":182,"num_write_kb":7459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"44'174","reported_seq":2306,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.083777+0000","last_change":"2024-04-14T23:55:42.085173+0000","last_active":"2024-04-14T23:57:39.083777+0000","last_peered":"2024-04-14T23:57:39.083777+0000","last_clean":"2024-04-14T23:57:39.083777+0000","last_became_active":"2024-04-14T23:48:43.228618+0000","last_became_peered":"2024-04-14T23:48:43.228618+0000","last_unstale":"2024-04-14T23:57:39.083777+0000","last_undegraded":"2024-04-14T23:57:39.083777+0000","last_fullsized":"2024-04-14T23:57:39.083777+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":174,"log_dups_size":0,"ondisk_log_size":174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T09:07:18.121366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002424158,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":40,"num_read_kb":4117,"num_write":170,"num_write_kb":4181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"37'153","reported_seq":2293,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.011857+0000","last_change":"2024-04-14T23:55:42.175749+0000","last_active":"2024-04-14T23:57:39.011857+0000","last_peered":"2024-04-14T23:57:39.011857+0000","last_clean":"2024-04-14T23:57:39.011857+0000","last_became_active":"2024-04-14T23:48:43.232971+0000","last_became_peered":"2024-04-14T23:48:43.232971+0000","last_unstale":"2024-04-14T23:57:39.011857+0000","last_undegraded":"2024-04-14T23:57:39.011857+0000","last_fullsized":"2024-04-14T23:57:39.011857+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:34:34.148700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030089610000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":47,"num_read_kb":26,"num_write":152,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"44'163","reported_seq":2290,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.086169+0000","last_change":"2024-04-14T23:55:41.994445+0000","last_active":"2024-04-14T23:57:39.086169+0000","last_peered":"2024-04-14T23:57:39.086169+0000","last_clean":"2024-04-14T23:57:39.086169+0000","last_became_active":"2024-04-14T23:48:43.232135+0000","last_became_peered":"2024-04-14T23:48:43.232135+0000","last_unstale":"2024-04-14T23:57:39.086169+0000","last_undegraded":"2024-04-14T23:57:39.086169+0000","last_fullsized":"2024-04-14T23:57:39.086169+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":163,"log_dups_size":0,"ondisk_log_size":163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:37:12.273966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029988570000000002,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":35,"num_read_kb":4112,"num_write":159,"num_write_kb":8270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"44'195","reported_seq":2329,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.010441+0000","last_change":"2024-04-14T23:55:41.992691+0000","last_active":"2024-04-14T23:57:39.010441+0000","last_peered":"2024-04-14T23:57:39.010441+0000","last_clean":"2024-04-14T23:57:39.010441+0000","last_became_active":"2024-04-14T23:48:43.226225+0000","last_became_peered":"2024-04-14T23:48:43.226225+0000","last_unstale":"2024-04-14T23:57:39.010441+0000","last_undegraded":"2024-04-14T23:57:39.010441+0000","last_fullsized":"2024-04-14T23:57:39.010441+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":195,"log_dups_size":0,"ondisk_log_size":195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:43:42.017755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019674760000000001,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":42,"num_read_kb":4116,"num_write":192,"num_write_kb":16477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"44'154","reported_seq":2291,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.090622+0000","last_change":"2024-04-14T23:55:42.316046+0000","last_active":"2024-04-14T23:57:39.090622+0000","last_peered":"2024-04-14T23:57:39.090622+0000","last_clean":"2024-04-14T23:57:39.090622+0000","last_became_active":"2024-04-14T23:48:43.230056+0000","last_became_peered":"2024-04-14T23:48:43.230056+0000","last_unstale":"2024-04-14T23:57:39.090622+0000","last_undegraded":"2024-04-14T23:57:39.090622+0000","last_fullsized":"2024-04-14T23:57:39.090622+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:05:52.381327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015336659000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":44,"num_read_kb":22,"num_write":154,"num_write_kb":77,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"44'175","reported_seq":2304,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.083498+0000","last_change":"2024-04-14T23:55:42.084499+0000","last_active":"2024-04-14T23:57:39.083498+0000","last_peered":"2024-04-14T23:57:39.083498+0000","last_clean":"2024-04-14T23:57:39.083498+0000","last_became_active":"2024-04-14T23:48:43.229110+0000","last_became_peered":"2024-04-14T23:48:43.229110+0000","last_unstale":"2024-04-14T23:57:39.083498+0000","last_undegraded":"2024-04-14T23:57:39.083498+0000","last_fullsized":"2024-04-14T23:57:39.083498+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T09:03:22.176338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022905320000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":36,"num_read_kb":17,"num_write":174,"num_write_kb":4183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"44'170","reported_seq":2301,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:45.295592+0000","last_change":"2024-04-14T23:55:42.314003+0000","last_active":"2024-04-14T23:57:45.295592+0000","last_peered":"2024-04-14T23:57:45.295592+0000","last_clean":"2024-04-14T23:57:45.295592+0000","last_became_active":"2024-04-14T23:48:43.233853+0000","last_became_peered":"2024-04-14T23:48:43.233853+0000","last_unstale":"2024-04-14T23:57:45.295592+0000","last_undegraded":"2024-04-14T23:57:45.295592+0000","last_fullsized":"2024-04-14T23:57:45.295592+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T09:33:21.684120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012858206,"stat_sum":{"num_bytes":1378,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":39,"num_read_kb":4117,"num_write":167,"num_write_kb":4180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"44'173","reported_seq":2295,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.011239+0000","last_change":"2024-04-14T23:55:42.175389+0000","last_active":"2024-04-14T23:57:39.011239+0000","last_peered":"2024-04-14T23:57:39.011239+0000","last_clean":"2024-04-14T23:57:39.011239+0000","last_became_active":"2024-04-14T23:49:29.512499+0000","last_became_peered":"2024-04-14T23:49:29.512499+0000","last_unstale":"2024-04-14T23:57:39.011239+0000","last_undegraded":"2024-04-14T23:57:39.011239+0000","last_fullsized":"2024-04-14T23:57:39.011239+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":173,"log_dups_size":0,"ondisk_log_size":173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:04:34.423852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037213820000000001,"stat_sum":{"num_bytes":23,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":36,"num_read_kb":16,"num_write":173,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"37'160","reported_seq":2287,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.093353+0000","last_change":"2024-04-14T23:55:42.319252+0000","last_active":"2024-04-14T23:57:39.093353+0000","last_peered":"2024-04-14T23:57:39.093353+0000","last_clean":"2024-04-14T23:57:39.093353+0000","last_became_active":"2024-04-14T23:48:43.233238+0000","last_became_peered":"2024-04-14T23:48:43.233238+0000","last_unstale":"2024-04-14T23:57:39.093353+0000","last_undegraded":"2024-04-14T23:57:39.093353+0000","last_fullsized":"2024-04-14T23:57:39.093353+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":160,"log_dups_size":0,"ondisk_log_size":160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T23:56:25.118737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012037971,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":18,"num_write":158,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"36'184","reported_seq":2312,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.010657+0000","last_change":"2024-04-14T23:55:41.993037+0000","last_active":"2024-04-14T23:57:39.010657+0000","last_peered":"2024-04-14T23:57:39.010657+0000","last_clean":"2024-04-14T23:57:39.010657+0000","last_became_active":"2024-04-14T23:48:43.222624+0000","last_became_peered":"2024-04-14T23:48:43.222624+0000","last_unstale":"2024-04-14T23:57:39.010657+0000","last_undegraded":"2024-04-14T23:57:39.010657+0000","last_fullsized":"2024-04-14T23:57:39.010657+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":184,"log_dups_size":0,"ondisk_log_size":184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:43:37.661814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001456707,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":19,"num_write":184,"num_write_kb":92,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"44'140","reported_seq":2257,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.086226+0000","last_change":"2024-04-14T23:55:42.309950+0000","last_active":"2024-04-14T23:57:39.086226+0000","last_peered":"2024-04-14T23:57:39.086226+0000","last_clean":"2024-04-14T23:57:39.086226+0000","last_became_active":"2024-04-14T23:48:43.232277+0000","last_became_peered":"2024-04-14T23:48:43.232277+0000","last_unstale":"2024-04-14T23:57:39.086226+0000","last_undegraded":"2024-04-14T23:57:39.086226+0000","last_fullsized":"2024-04-14T23:57:39.086226+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:17:15.623645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011200412,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":138,"num_write_kb":4167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"44'156","reported_seq":2279,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.094007+0000","last_change":"2024-04-14T23:55:42.319096+0000","last_active":"2024-04-14T23:57:39.094007+0000","last_peered":"2024-04-14T23:57:39.094007+0000","last_clean":"2024-04-14T23:57:39.094007+0000","last_became_active":"2024-04-14T23:48:43.229172+0000","last_became_peered":"2024-04-14T23:48:43.229172+0000","last_unstale":"2024-04-14T23:57:39.094007+0000","last_undegraded":"2024-04-14T23:57:39.094007+0000","last_fullsized":"2024-04-14T23:57:39.094007+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:05:34.205391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011889788,"stat_sum":{"num_bytes":482,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":31,"num_read_kb":3287,"num_write":153,"num_write_kb":3350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"36'184","reported_seq":2321,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.088053+0000","last_change":"2024-04-14T23:55:42.317181+0000","last_active":"2024-04-14T23:57:39.088053+0000","last_peered":"2024-04-14T23:57:39.088053+0000","last_clean":"2024-04-14T23:57:39.088053+0000","last_became_active":"2024-04-14T23:49:29.449548+0000","last_became_peered":"2024-04-14T23:49:29.449548+0000","last_unstale":"2024-04-14T23:57:39.088053+0000","last_undegraded":"2024-04-14T23:57:39.088053+0000","last_fullsized":"2024-04-14T23:57:39.088053+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":184,"log_dups_size":0,"ondisk_log_size":184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:07:19.034727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015790789,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":39,"num_read_kb":18,"num_write":184,"num_write_kb":92,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"35'188","reported_seq":2318,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.011493+0000","last_change":"2024-04-14T23:55:42.174354+0000","last_active":"2024-04-14T23:57:39.011493+0000","last_peered":"2024-04-14T23:57:39.011493+0000","last_clean":"2024-04-14T23:57:39.011493+0000","last_became_active":"2024-04-14T23:49:29.512523+0000","last_became_peered":"2024-04-14T23:49:29.512523+0000","last_unstale":"2024-04-14T23:57:39.011493+0000","last_undegraded":"2024-04-14T23:57:39.011493+0000","last_fullsized":"2024-04-14T23:57:39.011493+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":188,"log_dups_size":0,"ondisk_log_size":188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T09:13:33.216335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026059680000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":43,"num_read_kb":22,"num_write":188,"num_write_kb":94,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"44'205","reported_seq":2333,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.011575+0000","last_change":"2024-04-14T23:55:42.175131+0000","last_active":"2024-04-14T23:57:39.011575+0000","last_peered":"2024-04-14T23:57:39.011575+0000","last_clean":"2024-04-14T23:57:39.011575+0000","last_became_active":"2024-04-14T23:48:43.233567+0000","last_became_peered":"2024-04-14T23:48:43.233567+0000","last_unstale":"2024-04-14T23:57:39.011575+0000","last_undegraded":"2024-04-14T23:57:39.011575+0000","last_fullsized":"2024-04-14T23:57:39.011575+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":205,"log_dups_size":0,"ondisk_log_size":205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:43:07.999415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003348115,"stat_sum":{"num_bytes":4726,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":35,"num_read_kb":17,"num_write":205,"num_write_kb":14807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"44'199","reported_seq":2339,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.010040+0000","last_change":"2024-04-14T23:55:41.992153+0000","last_active":"2024-04-14T23:57:39.010040+0000","last_peered":"2024-04-14T23:57:39.010040+0000","last_clean":"2024-04-14T23:57:39.010040+0000","last_became_active":"2024-04-14T23:48:43.230626+0000","last_became_peered":"2024-04-14T23:48:43.230626+0000","last_unstale":"2024-04-14T23:57:39.010040+0000","last_undegraded":"2024-04-14T23:57:39.010040+0000","last_fullsized":"2024-04-14T23:57:39.010040+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":199,"log_dups_size":0,"ondisk_log_size":199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T23:53:31.323623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001517437,"stat_sum":{"num_bytes":4194851,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":47,"num_read_kb":27,"num_write":198,"num_write_kb":12385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"44'158","reported_seq":2278,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.084838+0000","last_change":"2024-04-14T23:55:42.224072+0000","last_active":"2024-04-14T23:57:39.084838+0000","last_peered":"2024-04-14T23:57:39.084838+0000","last_clean":"2024-04-14T23:57:39.084838+0000","last_became_active":"2024-04-14T23:48:43.226773+0000","last_became_peered":"2024-04-14T23:48:43.226773+0000","last_unstale":"2024-04-14T23:57:39.084838+0000","last_undegraded":"2024-04-14T23:57:39.084838+0000","last_fullsized":"2024-04-14T23:57:39.084838+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:04:13.345776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00101596,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":28,"num_read_kb":16,"num_write":158,"num_write_kb":4174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"34'20","reported_seq":90,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.094244+0000","last_change":"2024-04-14T23:48:42.239254+0000","last_active":"2024-04-14T23:57:39.094244+0000","last_peered":"2024-04-14T23:57:39.094244+0000","last_clean":"2024-04-14T23:57:39.094244+0000","last_became_active":"2024-04-14T23:48:42.238847+0000","last_became_peered":"2024-04-14T23:48:42.238847+0000","last_unstale":"2024-04-14T23:57:39.094244+0000","last_undegraded":"2024-04-14T23:57:39.094244+0000","last_fullsized":"2024-04-14T23:57:39.094244+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:46:16.301900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":16386,"num_write":22,"num_write_kb":4233,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"44'168","reported_seq":2296,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.091757+0000","last_change":"2024-04-14T23:55:42.317522+0000","last_active":"2024-04-14T23:57:39.091757+0000","last_peered":"2024-04-14T23:57:39.091757+0000","last_clean":"2024-04-14T23:57:39.091757+0000","last_became_active":"2024-04-14T23:48:43.237391+0000","last_became_peered":"2024-04-14T23:48:43.237391+0000","last_unstale":"2024-04-14T23:57:39.091757+0000","last_undegraded":"2024-04-14T23:57:39.091757+0000","last_fullsized":"2024-04-14T23:57:39.091757+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:01:14.668174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01542551,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":36,"num_read_kb":15,"num_write":168,"num_write_kb":4179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"34'164","reported_seq":240,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:45.069610+0000","last_change":"2024-04-14T23:48:42.219277+0000","last_active":"2024-04-14T23:57:45.069610+0000","last_peered":"2024-04-14T23:57:45.069610+0000","last_clean":"2024-04-14T23:57:45.069610+0000","last_became_active":"2024-04-14T23:48:42.218995+0000","last_became_peered":"2024-04-14T23:48:42.218995+0000","last_unstale":"2024-04-14T23:57:45.069610+0000","last_undegraded":"2024-04-14T23:57:45.069610+0000","last_fullsized":"2024-04-14T23:57:45.069610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T23:51:33.370857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":11,"num_read_kb":40960,"num_write":162,"num_write_kb":12391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"44'24","reported_seq":92,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.090540+0000","last_change":"2024-04-14T23:48:42.236803+0000","last_active":"2024-04-14T23:57:39.090540+0000","last_peered":"2024-04-14T23:57:39.090540+0000","last_clean":"2024-04-14T23:57:39.090540+0000","last_became_active":"2024-04-14T23:48:42.225937+0000","last_became_peered":"2024-04-14T23:48:42.225937+0000","last_unstale":"2024-04-14T23:57:39.090540+0000","last_undegraded":"2024-04-14T23:57:39.090540+0000","last_fullsized":"2024-04-14T23:57:39.090540+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:52:32.609126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":22,"num_write_kb":4105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"35'209","reported_seq":2340,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.091590+0000","last_change":"2024-04-14T23:55:42.316735+0000","last_active":"2024-04-14T23:57:39.091590+0000","last_peered":"2024-04-14T23:57:39.091590+0000","last_clean":"2024-04-14T23:57:39.091590+0000","last_became_active":"2024-04-14T23:48:43.227356+0000","last_became_peered":"2024-04-14T23:48:43.227356+0000","last_unstale":"2024-04-14T23:57:39.091590+0000","last_undegraded":"2024-04-14T23:57:39.091590+0000","last_fullsized":"2024-04-14T23:57:39.091590+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":209,"log_dups_size":0,"ondisk_log_size":209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:55:38.737429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014613774,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":40,"num_read_kb":4118,"num_write":206,"num_write_kb":8293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"45'189","reported_seq":260,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:44.808805+0000","last_change":"2024-04-14T23:48:42.210883+0000","last_active":"2024-04-14T23:57:44.808805+0000","last_peered":"2024-04-14T23:57:44.808805+0000","last_clean":"2024-04-14T23:57:44.808805+0000","last_became_active":"2024-04-14T23:48:42.210607+0000","last_became_peered":"2024-04-14T23:48:42.210607+0000","last_unstale":"2024-04-14T23:57:44.808805+0000","last_undegraded":"2024-04-14T23:57:44.808805+0000","last_fullsized":"2024-04-14T23:57:44.808805+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":189,"log_dups_size":0,"ondisk_log_size":189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:33:32.526223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":186,"num_write_kb":8305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"42'164","reported_seq":2296,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.085956+0000","last_change":"2024-04-14T23:55:42.310398+0000","last_active":"2024-04-14T23:57:39.085956+0000","last_peered":"2024-04-14T23:57:39.085956+0000","last_clean":"2024-04-14T23:57:39.085956+0000","last_became_active":"2024-04-14T23:48:43.217801+0000","last_became_peered":"2024-04-14T23:48:43.217801+0000","last_unstale":"2024-04-14T23:57:39.085956+0000","last_undegraded":"2024-04-14T23:57:39.085956+0000","last_fullsized":"2024-04-14T23:57:39.085956+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:39:59.476515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012539378,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":41,"num_read_kb":4114,"num_write":158,"num_write_kb":4174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"44'5","reported_seq":70,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.010369+0000","last_change":"2024-04-14T23:48:42.218652+0000","last_active":"2024-04-14T23:57:39.010369+0000","last_peered":"2024-04-14T23:57:39.010369+0000","last_clean":"2024-04-14T23:57:39.010369+0000","last_became_active":"2024-04-14T23:48:42.218321+0000","last_became_peered":"2024-04-14T23:48:42.218321+0000","last_unstale":"2024-04-14T23:57:39.010369+0000","last_undegraded":"2024-04-14T23:57:39.010369+0000","last_fullsized":"2024-04-14T23:57:39.010369+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T04:22:06.957747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"44'189","reported_seq":2326,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.083734+0000","last_change":"2024-04-14T23:55:42.084801+0000","last_active":"2024-04-14T23:57:39.083734+0000","last_peered":"2024-04-14T23:57:39.083734+0000","last_clean":"2024-04-14T23:57:39.083734+0000","last_became_active":"2024-04-14T23:48:43.228805+0000","last_became_peered":"2024-04-14T23:48:43.228805+0000","last_unstale":"2024-04-14T23:57:39.083734+0000","last_undegraded":"2024-04-14T23:57:39.083734+0000","last_fullsized":"2024-04-14T23:57:39.083734+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":189,"log_dups_size":0,"ondisk_log_size":189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:59:51.260017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022812650000000002,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":45,"num_read_kb":20,"num_write":189,"num_write_kb":4190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"44'27","reported_seq":95,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.083988+0000","last_change":"2024-04-14T23:48:42.221104+0000","last_active":"2024-04-14T23:57:39.083988+0000","last_peered":"2024-04-14T23:57:39.083988+0000","last_clean":"2024-04-14T23:57:39.083988+0000","last_became_active":"2024-04-14T23:48:42.220767+0000","last_became_peered":"2024-04-14T23:48:42.220767+0000","last_unstale":"2024-04-14T23:57:39.083988+0000","last_undegraded":"2024-04-14T23:57:39.083988+0000","last_fullsized":"2024-04-14T23:57:39.083988+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:23:57.213764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":24,"num_write_kb":4108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"44'162","reported_seq":2287,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.089328+0000","last_change":"2024-04-14T23:55:42.316479+0000","last_active":"2024-04-14T23:57:39.089328+0000","last_peered":"2024-04-14T23:57:39.089328+0000","last_clean":"2024-04-14T23:57:39.089328+0000","last_became_active":"2024-04-14T23:48:43.236168+0000","last_became_peered":"2024-04-14T23:48:43.236168+0000","last_unstale":"2024-04-14T23:57:39.089328+0000","last_undegraded":"2024-04-14T23:57:39.089328+0000","last_fullsized":"2024-04-14T23:57:39.089328+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:10:13.592872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015342639999999999,"stat_sum":{"num_bytes":420,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":33,"num_read_kb":18,"num_write":161,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"44'145","reported_seq":226,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:45.493309+0000","last_change":"2024-04-14T23:48:42.231329+0000","last_active":"2024-04-14T23:57:45.493309+0000","last_peered":"2024-04-14T23:57:45.493309+0000","last_clean":"2024-04-14T23:57:45.493309+0000","last_became_active":"2024-04-14T23:48:42.230969+0000","last_became_peered":"2024-04-14T23:48:42.230969+0000","last_unstale":"2024-04-14T23:57:45.493309+0000","last_undegraded":"2024-04-14T23:57:45.493309+0000","last_fullsized":"2024-04-14T23:57:45.493309+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:06:16.557589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194394,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":30,"num_read_kb":9177,"num_write":144,"num_write_kb":4199,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"44'151","reported_seq":2272,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.086423+0000","last_change":"2024-04-14T23:55:42.224786+0000","last_active":"2024-04-14T23:57:39.086423+0000","last_peered":"2024-04-14T23:57:39.086423+0000","last_clean":"2024-04-14T23:57:39.086423+0000","last_became_active":"2024-04-14T23:48:43.230253+0000","last_became_peered":"2024-04-14T23:48:43.230253+0000","last_unstale":"2024-04-14T23:57:39.086423+0000","last_undegraded":"2024-04-14T23:57:39.086423+0000","last_fullsized":"2024-04-14T23:57:39.086423+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":151,"log_dups_size":0,"ondisk_log_size":151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:04:12.081827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011901889999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":28,"num_read_kb":14,"num_write":150,"num_write_kb":4170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"45'705","reported_seq":801,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:02.669217+0000","last_change":"2024-04-14T23:48:42.237069+0000","last_active":"2024-04-14T23:58:02.669217+0000","last_peered":"2024-04-14T23:58:02.669217+0000","last_clean":"2024-04-14T23:58:02.669217+0000","last_became_active":"2024-04-14T23:48:42.236804+0000","last_became_peered":"2024-04-14T23:48:42.236804+0000","last_unstale":"2024-04-14T23:58:02.669217+0000","last_undegraded":"2024-04-14T23:58:02.669217+0000","last_fullsized":"2024-04-14T23:58:02.669217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":705,"log_dups_size":0,"ondisk_log_size":705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:25:37.318379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583584,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":41866,"num_write":697,"num_write_kb":12860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"44'154","reported_seq":2275,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.085057+0000","last_change":"2024-04-14T23:55:41.994228+0000","last_active":"2024-04-14T23:57:39.085057+0000","last_peered":"2024-04-14T23:57:39.085057+0000","last_clean":"2024-04-14T23:57:39.085057+0000","last_became_active":"2024-04-14T23:48:43.221462+0000","last_became_peered":"2024-04-14T23:48:43.221462+0000","last_unstale":"2024-04-14T23:57:39.085057+0000","last_undegraded":"2024-04-14T23:57:39.085057+0000","last_fullsized":"2024-04-14T23:57:39.085057+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:09:26.659126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033453850000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":154,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"44'99","reported_seq":183,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:02.545555+0000","last_change":"2024-04-14T23:48:42.230518+0000","last_active":"2024-04-14T23:58:02.545555+0000","last_peered":"2024-04-14T23:58:02.545555+0000","last_clean":"2024-04-14T23:58:02.545555+0000","last_became_active":"2024-04-14T23:48:42.229758+0000","last_became_peered":"2024-04-14T23:48:42.229758+0000","last_unstale":"2024-04-14T23:58:02.545555+0000","last_undegraded":"2024-04-14T23:58:02.545555+0000","last_fullsized":"2024-04-14T23:58:02.545555+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:01:41.332553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388642,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":28,"num_read_kb":32808,"num_write":95,"num_write_kb":8238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"44'160","reported_seq":2289,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:02.545168+0000","last_change":"2024-04-14T23:55:42.319479+0000","last_active":"2024-04-14T23:58:02.545168+0000","last_peered":"2024-04-14T23:58:02.545168+0000","last_clean":"2024-04-14T23:58:02.545168+0000","last_became_active":"2024-04-14T23:48:43.229476+0000","last_became_peered":"2024-04-14T23:48:43.229476+0000","last_unstale":"2024-04-14T23:58:02.545168+0000","last_undegraded":"2024-04-14T23:58:02.545168+0000","last_fullsized":"2024-04-14T23:58:02.545168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":160,"log_dups_size":0,"ondisk_log_size":160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:36:00.949626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015192783,"stat_sum":{"num_bytes":4198858,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":37,"num_read_kb":4115,"num_write":155,"num_write_kb":12367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":66,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.084193+0000","last_change":"2024-04-14T23:48:42.225404+0000","last_active":"2024-04-14T23:57:39.084193+0000","last_peered":"2024-04-14T23:57:39.084193+0000","last_clean":"2024-04-14T23:57:39.084193+0000","last_became_active":"2024-04-14T23:48:42.225211+0000","last_became_peered":"2024-04-14T23:48:42.225211+0000","last_unstale":"2024-04-14T23:57:39.084193+0000","last_undegraded":"2024-04-14T23:57:39.084193+0000","last_fullsized":"2024-04-14T23:57:39.084193+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:49:45.282259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"36'172","reported_seq":2308,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.085033+0000","last_change":"2024-04-14T23:55:41.993454+0000","last_active":"2024-04-14T23:57:39.085033+0000","last_peered":"2024-04-14T23:57:39.085033+0000","last_clean":"2024-04-14T23:57:39.085033+0000","last_became_active":"2024-04-14T23:48:43.229297+0000","last_became_peered":"2024-04-14T23:48:43.229297+0000","last_unstale":"2024-04-14T23:57:39.085033+0000","last_undegraded":"2024-04-14T23:57:39.085033+0000","last_fullsized":"2024-04-14T23:57:39.085033+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:21:50.218321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029516669999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":43,"num_read_kb":23,"num_write":172,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"34'2","reported_seq":68,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.010088+0000","last_change":"2024-04-14T23:48:42.220492+0000","last_active":"2024-04-14T23:57:39.010088+0000","last_peered":"2024-04-14T23:57:39.010088+0000","last_clean":"2024-04-14T23:57:39.010088+0000","last_became_active":"2024-04-14T23:48:42.220145+0000","last_became_peered":"2024-04-14T23:48:42.220145+0000","last_unstale":"2024-04-14T23:57:39.010088+0000","last_undegraded":"2024-04-14T23:57:39.010088+0000","last_fullsized":"2024-04-14T23:57:39.010088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:17:45.278993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"44'168","reported_seq":2296,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:02.542597+0000","last_change":"2024-04-14T23:55:41.992426+0000","last_active":"2024-04-14T23:58:02.542597+0000","last_peered":"2024-04-14T23:58:02.542597+0000","last_clean":"2024-04-14T23:58:02.542597+0000","last_became_active":"2024-04-14T23:48:43.235319+0000","last_became_peered":"2024-04-14T23:48:43.235319+0000","last_unstale":"2024-04-14T23:58:02.542597+0000","last_undegraded":"2024-04-14T23:58:02.542597+0000","last_fullsized":"2024-04-14T23:58:02.542597+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:38:08.635546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017199240000000001,"stat_sum":{"num_bytes":428,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":36,"num_read_kb":14,"num_write":163,"num_write_kb":82,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"35'101","reported_seq":199,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:02.579303+0000","last_change":"2024-04-14T23:48:42.231178+0000","last_active":"2024-04-14T23:58:02.579303+0000","last_peered":"2024-04-14T23:58:02.579303+0000","last_clean":"2024-04-14T23:58:02.579303+0000","last_became_active":"2024-04-14T23:48:42.230792+0000","last_became_peered":"2024-04-14T23:48:42.230792+0000","last_unstale":"2024-04-14T23:58:02.579303+0000","last_undegraded":"2024-04-14T23:58:02.579303+0000","last_fullsized":"2024-04-14T23:58:02.579303+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:22:40.744314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":84,"num_read_kb":20558,"num_write":102,"num_write_kb":8239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"44'181","reported_seq":2315,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:02.544562+0000","last_change":"2024-04-14T23:55:41.993119+0000","last_active":"2024-04-14T23:58:02.544562+0000","last_peered":"2024-04-14T23:58:02.544562+0000","last_clean":"2024-04-14T23:58:02.544562+0000","last_became_active":"2024-04-14T23:48:43.228004+0000","last_became_peered":"2024-04-14T23:48:43.228004+0000","last_unstale":"2024-04-14T23:58:02.544562+0000","last_undegraded":"2024-04-14T23:58:02.544562+0000","last_fullsized":"2024-04-14T23:58:02.544562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:09:06.909970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003032402,"stat_sum":{"num_bytes":1647,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":44,"num_read_kb":8209,"num_write":175,"num_write_kb":12374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"34'22","reported_seq":96,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:44.962330+0000","last_change":"2024-04-14T23:48:42.231936+0000","last_active":"2024-04-14T23:57:44.962330+0000","last_peered":"2024-04-14T23:57:44.962330+0000","last_clean":"2024-04-14T23:57:44.962330+0000","last_became_active":"2024-04-14T23:48:42.231612+0000","last_became_peered":"2024-04-14T23:48:42.231612+0000","last_unstale":"2024-04-14T23:57:44.962330+0000","last_undegraded":"2024-04-14T23:57:44.962330+0000","last_fullsized":"2024-04-14T23:57:44.962330+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:45:03.098024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194326,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":12293,"num_write":21,"num_write_kb":4101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"36'127","reported_seq":2246,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.086202+0000","last_change":"2024-04-14T23:55:41.993658+0000","last_active":"2024-04-14T23:57:39.086202+0000","last_peered":"2024-04-14T23:57:39.086202+0000","last_clean":"2024-04-14T23:57:39.086202+0000","last_became_active":"2024-04-14T23:48:43.228665+0000","last_became_peered":"2024-04-14T23:48:43.228665+0000","last_unstale":"2024-04-14T23:57:39.086202+0000","last_undegraded":"2024-04-14T23:57:39.086202+0000","last_fullsized":"2024-04-14T23:57:39.086202+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T23:49:58.331034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003047124,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":27,"num_read_kb":4107,"num_write":124,"num_write_kb":4157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"30'100","reported_seq":176,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:44.881117+0000","last_change":"2024-04-14T23:48:42.239794+0000","last_active":"2024-04-14T23:57:44.881117+0000","last_peered":"2024-04-14T23:57:44.881117+0000","last_clean":"2024-04-14T23:57:44.881117+0000","last_became_active":"2024-04-14T23:48:42.239505+0000","last_became_peered":"2024-04-14T23:48:42.239505+0000","last_unstale":"2024-04-14T23:57:44.881117+0000","last_undegraded":"2024-04-14T23:57:44.881117+0000","last_fullsized":"2024-04-14T23:57:44.881117+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T09:43:16.582708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":24,"num_read_kb":12309,"num_write":100,"num_write_kb":4158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"44'164","reported_seq":2293,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:45.262232+0000","last_change":"2024-04-14T23:55:41.994666+0000","last_active":"2024-04-14T23:57:45.262232+0000","last_peered":"2024-04-14T23:57:45.262232+0000","last_clean":"2024-04-14T23:57:45.262232+0000","last_became_active":"2024-04-14T23:48:43.229949+0000","last_became_peered":"2024-04-14T23:48:43.229949+0000","last_unstale":"2024-04-14T23:57:45.262232+0000","last_undegraded":"2024-04-14T23:57:45.262232+0000","last_fullsized":"2024-04-14T23:57:45.262232+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:26:59.738426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002291033,"stat_sum":{"num_bytes":240,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":37,"num_read_kb":4115,"num_write":159,"num_write_kb":4175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"34'18","reported_seq":93,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:44.346309+0000","last_change":"2024-04-14T23:48:42.227708+0000","last_active":"2024-04-14T23:57:44.346309+0000","last_peered":"2024-04-14T23:57:44.346309+0000","last_clean":"2024-04-14T23:57:44.346309+0000","last_became_active":"2024-04-14T23:48:42.226845+0000","last_became_peered":"2024-04-14T23:48:42.226845+0000","last_unstale":"2024-04-14T23:57:44.346309+0000","last_undegraded":"2024-04-14T23:57:44.346309+0000","last_fullsized":"2024-04-14T23:57:44.346309+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:02:54.468796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194374,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":9,"num_read_kb":16389,"num_write":18,"num_write_kb":4108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"43'183","reported_seq":2304,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.084349+0000","last_change":"2024-04-14T23:55:42.084740+0000","last_active":"2024-04-14T23:57:39.084349+0000","last_peered":"2024-04-14T23:57:39.084349+0000","last_clean":"2024-04-14T23:57:39.084349+0000","last_became_active":"2024-04-14T23:48:43.226717+0000","last_became_peered":"2024-04-14T23:48:43.226717+0000","last_unstale":"2024-04-14T23:57:39.084349+0000","last_undegraded":"2024-04-14T23:57:39.084349+0000","last_fullsized":"2024-04-14T23:57:39.084349+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:44:01.195501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022579950000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":28,"num_read_kb":14,"num_write":180,"num_write_kb":90,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"34'2","reported_seq":68,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.012051+0000","last_change":"2024-04-14T23:48:42.232164+0000","last_active":"2024-04-14T23:57:39.012051+0000","last_peered":"2024-04-14T23:57:39.012051+0000","last_clean":"2024-04-14T23:57:39.012051+0000","last_became_active":"2024-04-14T23:48:42.229785+0000","last_became_peered":"2024-04-14T23:48:42.229785+0000","last_unstale":"2024-04-14T23:57:39.012051+0000","last_undegraded":"2024-04-14T23:57:39.012051+0000","last_fullsized":"2024-04-14T23:57:39.012051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T04:21:37.538735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"36'172","reported_seq":2296,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.085217+0000","last_change":"2024-04-14T23:55:41.994270+0000","last_active":"2024-04-14T23:57:39.085217+0000","last_peered":"2024-04-14T23:57:39.085217+0000","last_clean":"2024-04-14T23:57:39.085217+0000","last_became_active":"2024-04-14T23:48:43.232612+0000","last_became_peered":"2024-04-14T23:48:43.232612+0000","last_unstale":"2024-04-14T23:57:39.085217+0000","last_undegraded":"2024-04-14T23:57:39.085217+0000","last_fullsized":"2024-04-14T23:57:39.085217+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T04:34:43.974518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031586029999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":31,"num_read_kb":17,"num_write":172,"num_write_kb":4181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":222,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.085334+0000","last_change":"2024-04-14T23:48:38.150200+0000","last_active":"2024-04-14T23:57:39.085334+0000","last_peered":"2024-04-14T23:57:39.085334+0000","last_clean":"2024-04-14T23:57:39.085334+0000","last_became_active":"2024-04-14T23:48:38.149301+0000","last_became_peered":"2024-04-14T23:48:38.149301+0000","last_unstale":"2024-04-14T23:57:39.085334+0000","last_undegraded":"2024-04-14T23:57:39.085334+0000","last_fullsized":"2024-04-14T23:57:39.085334+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:37.129663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:37.129663+0000","last_clean_scrub_stamp":"2024-04-14T23:48:37.129663+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T23:58:01.303851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"45'4","reported_seq":70,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.083463+0000","last_change":"2024-04-14T23:48:42.229238+0000","last_active":"2024-04-14T23:57:39.083463+0000","last_peered":"2024-04-14T23:57:39.083463+0000","last_clean":"2024-04-14T23:57:39.083463+0000","last_became_active":"2024-04-14T23:48:42.228994+0000","last_became_peered":"2024-04-14T23:48:42.228994+0000","last_unstale":"2024-04-14T23:57:39.083463+0000","last_undegraded":"2024-04-14T23:57:39.083463+0000","last_fullsized":"2024-04-14T23:57:39.083463+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:47:15.406228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"44'163","reported_seq":2289,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.092269+0000","last_change":"2024-04-14T23:55:42.318283+0000","last_active":"2024-04-14T23:57:39.092269+0000","last_peered":"2024-04-14T23:57:39.092269+0000","last_clean":"2024-04-14T23:57:39.092269+0000","last_became_active":"2024-04-14T23:48:43.223665+0000","last_became_peered":"2024-04-14T23:48:43.223665+0000","last_unstale":"2024-04-14T23:57:39.092269+0000","last_undegraded":"2024-04-14T23:57:39.092269+0000","last_fullsized":"2024-04-14T23:57:39.092269+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":163,"log_dups_size":0,"ondisk_log_size":163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:47:10.121417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012914624,"stat_sum":{"num_bytes":4902,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":33,"num_read_kb":14,"num_write":161,"num_write_kb":4180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"45'89","reported_seq":161,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:03.889202+0000","last_change":"2024-04-14T23:48:42.230404+0000","last_active":"2024-04-14T23:58:03.889202+0000","last_peered":"2024-04-14T23:58:03.889202+0000","last_clean":"2024-04-14T23:58:03.889202+0000","last_became_active":"2024-04-14T23:48:42.230173+0000","last_became_peered":"2024-04-14T23:48:42.230173+0000","last_unstale":"2024-04-14T23:58:03.889202+0000","last_undegraded":"2024-04-14T23:58:03.889202+0000","last_fullsized":"2024-04-14T23:58:03.889202+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:23:45.666680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8660106,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":80,"num_write_kb":8498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"42'152","reported_seq":2272,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.087977+0000","last_change":"2024-04-14T23:55:42.315208+0000","last_active":"2024-04-14T23:57:39.087977+0000","last_peered":"2024-04-14T23:57:39.087977+0000","last_clean":"2024-04-14T23:57:39.087977+0000","last_became_active":"2024-04-14T23:48:43.231025+0000","last_became_peered":"2024-04-14T23:48:43.231025+0000","last_unstale":"2024-04-14T23:57:39.087977+0000","last_undegraded":"2024-04-14T23:57:39.087977+0000","last_fullsized":"2024-04-14T23:57:39.087977+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:22:50.033217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014709888000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":27,"num_read_kb":10,"num_write":150,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'131","reported_seq":328,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.011644+0000","last_change":"2024-04-14T23:48:42.240651+0000","last_active":"2024-04-14T23:57:39.011644+0000","last_peered":"2024-04-14T23:57:39.011644+0000","last_clean":"2024-04-14T23:57:39.011644+0000","last_became_active":"2024-04-14T23:48:42.240351+0000","last_became_peered":"2024-04-14T23:48:42.240351+0000","last_unstale":"2024-04-14T23:57:39.011644+0000","last_undegraded":"2024-04-14T23:57:39.011644+0000","last_fullsized":"2024-04-14T23:57:39.011644+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:03:02.804444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":533490,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":131,"num_read_kb":605,"num_write":126,"num_write_kb":605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"44'152","reported_seq":2274,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.086207+0000","last_change":"2024-04-14T23:55:42.306590+0000","last_active":"2024-04-14T23:57:39.086207+0000","last_peered":"2024-04-14T23:57:39.086207+0000","last_clean":"2024-04-14T23:57:39.086207+0000","last_became_active":"2024-04-14T23:48:43.232120+0000","last_became_peered":"2024-04-14T23:48:43.232120+0000","last_unstale":"2024-04-14T23:57:39.086207+0000","last_undegraded":"2024-04-14T23:57:39.086207+0000","last_fullsized":"2024-04-14T23:57:39.086207+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:07:57.642700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0096232650000000006,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":29,"num_read_kb":17,"num_write":152,"num_write_kb":4171,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"34'156","reported_seq":228,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:44.101825+0000","last_change":"2024-04-14T23:48:42.233743+0000","last_active":"2024-04-14T23:57:44.101825+0000","last_peered":"2024-04-14T23:57:44.101825+0000","last_clean":"2024-04-14T23:57:44.101825+0000","last_became_active":"2024-04-14T23:48:42.233354+0000","last_became_peered":"2024-04-14T23:48:42.233354+0000","last_unstale":"2024-04-14T23:57:44.101825+0000","last_undegraded":"2024-04-14T23:57:44.101825+0000","last_fullsized":"2024-04-14T23:57:44.101825+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:49:23.045968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":154,"num_write_kb":8284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"44'183","reported_seq":2314,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.085863+0000","last_change":"2024-04-14T23:55:42.224478+0000","last_active":"2024-04-14T23:57:39.085863+0000","last_peered":"2024-04-14T23:57:39.085863+0000","last_clean":"2024-04-14T23:57:39.085863+0000","last_became_active":"2024-04-14T23:48:43.232119+0000","last_became_peered":"2024-04-14T23:48:43.232119+0000","last_unstale":"2024-04-14T23:57:39.085863+0000","last_undegraded":"2024-04-14T23:57:39.085863+0000","last_fullsized":"2024-04-14T23:57:39.085863+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:37:54.631325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012271420000000001,"stat_sum":{"num_bytes":192,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":38,"num_read_kb":16,"num_write":183,"num_write_kb":92,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"45'3","reported_seq":69,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.088128+0000","last_change":"2024-04-14T23:48:42.242249+0000","last_active":"2024-04-14T23:57:39.088128+0000","last_peered":"2024-04-14T23:57:39.088128+0000","last_clean":"2024-04-14T23:57:39.088128+0000","last_became_active":"2024-04-14T23:48:42.242039+0000","last_became_peered":"2024-04-14T23:48:42.242039+0000","last_unstale":"2024-04-14T23:57:39.088128+0000","last_undegraded":"2024-04-14T23:57:39.088128+0000","last_fullsized":"2024-04-14T23:57:39.088128+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:46:16.301900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"44'162","reported_seq":2291,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.084067+0000","last_change":"2024-04-14T23:55:42.085024+0000","last_active":"2024-04-14T23:57:39.084067+0000","last_peered":"2024-04-14T23:57:39.084067+0000","last_clean":"2024-04-14T23:57:39.084067+0000","last_became_active":"2024-04-14T23:48:43.241776+0000","last_became_peered":"2024-04-14T23:48:43.241776+0000","last_unstale":"2024-04-14T23:57:39.084067+0000","last_undegraded":"2024-04-14T23:57:39.084067+0000","last_fullsized":"2024-04-14T23:57:39.084067+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:05:29.929626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024520940000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":36,"num_read_kb":17,"num_write":162,"num_write_kb":8271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"45'29","reported_seq":98,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:45.314264+0000","last_change":"2024-04-14T23:48:42.232537+0000","last_active":"2024-04-14T23:57:45.314264+0000","last_peered":"2024-04-14T23:57:45.314264+0000","last_clean":"2024-04-14T23:57:45.314264+0000","last_became_active":"2024-04-14T23:48:42.232325+0000","last_became_peered":"2024-04-14T23:48:42.232325+0000","last_unstale":"2024-04-14T23:57:45.314264+0000","last_undegraded":"2024-04-14T23:57:45.314264+0000","last_fullsized":"2024-04-14T23:57:45.314264+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:34:37.448301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":25,"num_write_kb":4108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"44'166","reported_seq":2288,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.086381+0000","last_change":"2024-04-14T23:55:42.308941+0000","last_active":"2024-04-14T23:57:39.086381+0000","last_peered":"2024-04-14T23:57:39.086381+0000","last_clean":"2024-04-14T23:57:39.086381+0000","last_became_active":"2024-04-14T23:48:43.234150+0000","last_became_peered":"2024-04-14T23:48:43.234150+0000","last_unstale":"2024-04-14T23:57:39.086381+0000","last_undegraded":"2024-04-14T23:57:39.086381+0000","last_fullsized":"2024-04-14T23:57:39.086381+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":166,"log_dups_size":0,"ondisk_log_size":166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:55:59.231275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011050757,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":29,"num_read_kb":14,"num_write":166,"num_write_kb":4178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"45'182","reported_seq":254,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:02.552170+0000","last_change":"2024-04-14T23:48:42.230014+0000","last_active":"2024-04-14T23:58:02.552170+0000","last_peered":"2024-04-14T23:58:02.552170+0000","last_clean":"2024-04-14T23:58:02.552170+0000","last_became_active":"2024-04-14T23:48:42.229712+0000","last_became_peered":"2024-04-14T23:48:42.229712+0000","last_unstale":"2024-04-14T23:58:02.552170+0000","last_undegraded":"2024-04-14T23:58:02.552170+0000","last_fullsized":"2024-04-14T23:58:02.552170+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":182,"log_dups_size":0,"ondisk_log_size":182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:54:20.595391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":142,"num_write":414,"num_write_kb":617,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"44'166","reported_seq":2297,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.094045+0000","last_change":"2024-04-14T23:55:42.320181+0000","last_active":"2024-04-14T23:57:39.094045+0000","last_peered":"2024-04-14T23:57:39.094045+0000","last_clean":"2024-04-14T23:57:39.094045+0000","last_became_active":"2024-04-14T23:48:43.228739+0000","last_became_peered":"2024-04-14T23:48:43.228739+0000","last_unstale":"2024-04-14T23:57:39.094045+0000","last_undegraded":"2024-04-14T23:57:39.094045+0000","last_fullsized":"2024-04-14T23:57:39.094045+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":166,"log_dups_size":0,"ondisk_log_size":166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:27:33.331878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012444432,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":38,"num_read_kb":16,"num_write":166,"num_write_kb":4178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"45'208","reported_seq":285,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:02.646576+0000","last_change":"2024-04-14T23:48:42.220035+0000","last_active":"2024-04-14T23:58:02.646576+0000","last_peered":"2024-04-14T23:58:02.646576+0000","last_clean":"2024-04-14T23:58:02.646576+0000","last_became_active":"2024-04-14T23:48:42.219750+0000","last_became_peered":"2024-04-14T23:48:42.219750+0000","last_unstale":"2024-04-14T23:58:02.646576+0000","last_undegraded":"2024-04-14T23:58:02.646576+0000","last_fullsized":"2024-04-14T23:58:02.646576+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:00:58.928166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":25,"num_read_kb":6094,"num_write":202,"num_write_kb":4203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"44'176","reported_seq":2313,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.091734+0000","last_change":"2024-04-14T23:55:42.319020+0000","last_active":"2024-04-14T23:57:39.091734+0000","last_peered":"2024-04-14T23:57:39.091734+0000","last_clean":"2024-04-14T23:57:39.091734+0000","last_became_active":"2024-04-14T23:48:43.232730+0000","last_became_peered":"2024-04-14T23:48:43.232730+0000","last_unstale":"2024-04-14T23:57:39.091734+0000","last_undegraded":"2024-04-14T23:57:39.091734+0000","last_fullsized":"2024-04-14T23:57:39.091734+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":176,"log_dups_size":0,"ondisk_log_size":176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:38:48.078009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014562113999999999,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":45,"num_read_kb":26,"num_write":171,"num_write_kb":8277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'28","reported_seq":97,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.084866+0000","last_change":"2024-04-14T23:48:42.232087+0000","last_active":"2024-04-14T23:57:39.084866+0000","last_peered":"2024-04-14T23:57:39.084866+0000","last_clean":"2024-04-14T23:57:39.084866+0000","last_became_active":"2024-04-14T23:48:42.231796+0000","last_became_peered":"2024-04-14T23:48:42.231796+0000","last_unstale":"2024-04-14T23:57:39.084866+0000","last_undegraded":"2024-04-14T23:57:39.084866+0000","last_fullsized":"2024-04-14T23:57:39.084866+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:14:35.050093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":27,"num_write_kb":4121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"43'183","reported_seq":2313,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.085628+0000","last_change":"2024-04-14T23:55:42.224366+0000","last_active":"2024-04-14T23:57:39.085628+0000","last_peered":"2024-04-14T23:57:39.085628+0000","last_clean":"2024-04-14T23:57:39.085628+0000","last_became_active":"2024-04-14T23:48:43.232594+0000","last_became_peered":"2024-04-14T23:48:43.232594+0000","last_unstale":"2024-04-14T23:57:39.085628+0000","last_undegraded":"2024-04-14T23:57:39.085628+0000","last_fullsized":"2024-04-14T23:57:39.085628+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T09:41:55.211912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001144237,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":37,"num_read_kb":20,"num_write":182,"num_write_kb":91,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"44'240","reported_seq":308,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.087346+0000","last_change":"2024-04-14T23:48:42.231237+0000","last_active":"2024-04-14T23:57:39.087346+0000","last_peered":"2024-04-14T23:57:39.087346+0000","last_clean":"2024-04-14T23:57:39.087346+0000","last_became_active":"2024-04-14T23:48:42.230902+0000","last_became_peered":"2024-04-14T23:48:42.230902+0000","last_unstale":"2024-04-14T23:57:39.087346+0000","last_undegraded":"2024-04-14T23:57:39.087346+0000","last_fullsized":"2024-04-14T23:57:39.087346+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":240,"log_dups_size":0,"ondisk_log_size":240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:28:37.228039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":238,"num_write_kb":4263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"44'168","reported_seq":2294,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.092200+0000","last_change":"2024-04-14T23:55:42.316834+0000","last_active":"2024-04-14T23:57:39.092200+0000","last_peered":"2024-04-14T23:57:39.092200+0000","last_clean":"2024-04-14T23:57:39.092200+0000","last_became_active":"2024-04-14T23:48:43.232747+0000","last_became_peered":"2024-04-14T23:48:43.232747+0000","last_unstale":"2024-04-14T23:57:39.092200+0000","last_undegraded":"2024-04-14T23:57:39.092200+0000","last_fullsized":"2024-04-14T23:57:39.092200+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:13:25.640712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014090612000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":33,"num_read_kb":16,"num_write":168,"num_write_kb":8274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"34'24","reported_seq":94,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:45.019436+0000","last_change":"2024-04-14T23:48:42.240496+0000","last_active":"2024-04-14T23:57:45.019436+0000","last_peered":"2024-04-14T23:57:45.019436+0000","last_clean":"2024-04-14T23:57:45.019436+0000","last_became_active":"2024-04-14T23:48:42.238524+0000","last_became_peered":"2024-04-14T23:48:42.238524+0000","last_unstale":"2024-04-14T23:57:45.019436+0000","last_undegraded":"2024-04-14T23:57:45.019436+0000","last_fullsized":"2024-04-14T23:57:45.019436+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:29:35.218759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":23,"num_write_kb":4102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"44'163","reported_seq":2291,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:45.295617+0000","last_change":"2024-04-14T23:55:42.318594+0000","last_active":"2024-04-14T23:57:45.295617+0000","last_peered":"2024-04-14T23:57:45.295617+0000","last_clean":"2024-04-14T23:57:45.295617+0000","last_became_active":"2024-04-14T23:48:43.233651+0000","last_became_peered":"2024-04-14T23:48:43.233651+0000","last_unstale":"2024-04-14T23:57:45.295617+0000","last_undegraded":"2024-04-14T23:57:45.295617+0000","last_fullsized":"2024-04-14T23:57:45.295617+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":163,"log_dups_size":0,"ondisk_log_size":163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:17:15.414893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015058134000000001,"stat_sum":{"num_bytes":8390104,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":35,"num_read_kb":13,"num_write":163,"num_write_kb":8274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"44'70","reported_seq":148,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:45.105836+0000","last_change":"2024-04-14T23:48:42.218897+0000","last_active":"2024-04-14T23:57:45.105836+0000","last_peered":"2024-04-14T23:57:45.105836+0000","last_clean":"2024-04-14T23:57:45.105836+0000","last_became_active":"2024-04-14T23:48:42.218580+0000","last_became_peered":"2024-04-14T23:48:42.218580+0000","last_unstale":"2024-04-14T23:57:45.105836+0000","last_undegraded":"2024-04-14T23:57:45.105836+0000","last_fullsized":"2024-04-14T23:57:45.105836+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:26:44.477638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":28682,"num_write":62,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"35'166","reported_seq":2282,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.010030+0000","last_change":"2024-04-14T23:55:41.991867+0000","last_active":"2024-04-14T23:57:39.010030+0000","last_peered":"2024-04-14T23:57:39.010030+0000","last_clean":"2024-04-14T23:57:39.010030+0000","last_became_active":"2024-04-14T23:48:43.219601+0000","last_became_peered":"2024-04-14T23:48:43.219601+0000","last_unstale":"2024-04-14T23:57:39.010030+0000","last_undegraded":"2024-04-14T23:57:39.010030+0000","last_fullsized":"2024-04-14T23:57:39.010030+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":166,"log_dups_size":0,"ondisk_log_size":166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:23:08.643833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001540055,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":23,"num_read_kb":13,"num_write":166,"num_write_kb":83,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"44'151","reported_seq":2286,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.083755+0000","last_change":"2024-04-14T23:55:42.084976+0000","last_active":"2024-04-14T23:57:39.083755+0000","last_peered":"2024-04-14T23:57:39.083755+0000","last_clean":"2024-04-14T23:57:39.083755+0000","last_became_active":"2024-04-14T23:48:43.221842+0000","last_became_peered":"2024-04-14T23:48:43.221842+0000","last_unstale":"2024-04-14T23:57:39.083755+0000","last_undegraded":"2024-04-14T23:57:39.083755+0000","last_fullsized":"2024-04-14T23:57:39.083755+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":151,"log_dups_size":0,"ondisk_log_size":151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:58:33.633046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024436879999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":43,"num_read_kb":23,"num_write":150,"num_write_kb":4170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"33'62","reported_seq":137,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:44.931184+0000","last_change":"2024-04-14T23:48:42.221314+0000","last_active":"2024-04-14T23:57:44.931184+0000","last_peered":"2024-04-14T23:57:44.931184+0000","last_clean":"2024-04-14T23:57:44.931184+0000","last_became_active":"2024-04-14T23:48:42.221028+0000","last_became_peered":"2024-04-14T23:48:42.221028+0000","last_unstale":"2024-04-14T23:57:44.931184+0000","last_undegraded":"2024-04-14T23:57:44.931184+0000","last_fullsized":"2024-04-14T23:57:44.931184+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T09:58:18.195400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":24,"num_read_kb":12309,"num_write":62,"num_write_kb":4128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"44'165","reported_seq":2288,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.083812+0000","last_change":"2024-04-14T23:55:42.084580+0000","last_active":"2024-04-14T23:57:39.083812+0000","last_peered":"2024-04-14T23:57:39.083812+0000","last_clean":"2024-04-14T23:57:39.083812+0000","last_became_active":"2024-04-14T23:48:43.224312+0000","last_became_peered":"2024-04-14T23:48:43.224312+0000","last_unstale":"2024-04-14T23:57:39.083812+0000","last_undegraded":"2024-04-14T23:57:39.083812+0000","last_fullsized":"2024-04-14T23:57:39.083812+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:55:12.275566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00232957,"stat_sum":{"num_bytes":4674,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":31,"num_read_kb":13,"num_write":165,"num_write_kb":4182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"44'668","reported_seq":753,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:44.946209+0000","last_change":"2024-04-14T23:48:42.221000+0000","last_active":"2024-04-14T23:57:44.946209+0000","last_peered":"2024-04-14T23:57:44.946209+0000","last_clean":"2024-04-14T23:57:44.946209+0000","last_became_active":"2024-04-14T23:48:42.219813+0000","last_became_peered":"2024-04-14T23:48:42.219813+0000","last_unstale":"2024-04-14T23:57:44.946209+0000","last_undegraded":"2024-04-14T23:57:44.946209+0000","last_fullsized":"2024-04-14T23:57:44.946209+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":668,"log_dups_size":0,"ondisk_log_size":668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:00:38.107953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":44198,"num_write":659,"num_write_kb":12787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"37'165","reported_seq":2293,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.087948+0000","last_change":"2024-04-14T23:55:42.313777+0000","last_active":"2024-04-14T23:57:39.087948+0000","last_peered":"2024-04-14T23:57:39.087948+0000","last_clean":"2024-04-14T23:57:39.087948+0000","last_became_active":"2024-04-14T23:48:43.236595+0000","last_became_peered":"2024-04-14T23:48:43.236595+0000","last_unstale":"2024-04-14T23:57:39.087948+0000","last_undegraded":"2024-04-14T23:57:39.087948+0000","last_fullsized":"2024-04-14T23:57:39.087948+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:13:32.883032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013391734000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":19,"num_write":164,"num_write_kb":82,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"44'47","reported_seq":118,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.094119+0000","last_change":"2024-04-14T23:48:42.239255+0000","last_active":"2024-04-14T23:57:39.094119+0000","last_peered":"2024-04-14T23:57:39.094119+0000","last_clean":"2024-04-14T23:57:39.094119+0000","last_became_active":"2024-04-14T23:48:42.238892+0000","last_became_peered":"2024-04-14T23:48:42.238892+0000","last_unstale":"2024-04-14T23:57:39.094119+0000","last_undegraded":"2024-04-14T23:57:39.094119+0000","last_fullsized":"2024-04-14T23:57:39.094119+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:44:39.632155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":16384,"num_write":44,"num_write_kb":4126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"44'175","reported_seq":2298,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.092968+0000","last_change":"2024-04-14T23:55:42.319487+0000","last_active":"2024-04-14T23:57:39.092968+0000","last_peered":"2024-04-14T23:57:39.092968+0000","last_clean":"2024-04-14T23:57:39.092968+0000","last_became_active":"2024-04-14T23:48:43.222707+0000","last_became_peered":"2024-04-14T23:48:43.222707+0000","last_unstale":"2024-04-14T23:57:39.092968+0000","last_undegraded":"2024-04-14T23:57:39.092968+0000","last_fullsized":"2024-04-14T23:57:39.092968+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:47:16.501806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011942019,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":30,"num_read_kb":12,"num_write":175,"num_write_kb":4183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"45'76","reported_seq":157,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:46.454133+0000","last_change":"2024-04-14T23:48:42.226563+0000","last_active":"2024-04-14T23:57:46.454133+0000","last_peered":"2024-04-14T23:57:46.454133+0000","last_clean":"2024-04-14T23:57:46.454133+0000","last_became_active":"2024-04-14T23:48:42.226238+0000","last_became_peered":"2024-04-14T23:48:42.226238+0000","last_unstale":"2024-04-14T23:57:46.454133+0000","last_undegraded":"2024-04-14T23:57:46.454133+0000","last_fullsized":"2024-04-14T23:57:46.454133+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:01:36.516898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":25,"num_read_kb":20500,"num_write":71,"num_write_kb":4136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":50360410,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":2223,"num_read_kb":65817,"num_write":10624,"num_write_kb":276412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":100835328,"data_stored":100720820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10722,"ondisk_log_size":10722,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":151801402,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":553,"num_read_kb":479116,"num_write":3786,"num_write_kb":151262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":64675840,"data_stored":303602804,"data_compressed":34392348,"data_compressed_allocated":44802048,"data_compressed_original":283795456,"omap_allocated":0,"internal_metadata":0},"log_size":3640,"ondisk_log_size":3640,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65128,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574965,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":88644,"kb_used_data":25532,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94283196,"statfs":{"total":96636764160,"available":96545992704,"internally_reserved":0,"allocated":26144768,"data_stored":72411425,"data_compressed":6887616,"data_compressed_allocated":9027584,"data_compressed_original":55586816,"omap_allocated":1590,"internal_metadata":64616906},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574966,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77724,"kb_used_data":32660,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94294116,"statfs":{"total":96636764160,"available":96557174784,"internally_reserved":0,"allocated":33443840,"data_stored":71547043,"data_compressed":5492002,"data_compressed_allocated":7053312,"data_compressed_original":45449216,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574965,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81436,"kb_used_data":14164,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94290404,"statfs":{"total":96636764160,"available":96553373696,"internally_reserved":0,"allocated":14503936,"data_stored":38005896,"data_compressed":3110985,"data_compressed_allocated":4100096,"data_compressed_original":27897856,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574966,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84544,"kb_used_data":17268,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94287296,"statfs":{"total":96636764160,"available":96550191104,"internally_reserved":0,"allocated":17682432,"data_stored":54774132,"data_compressed":5258839,"data_compressed_allocated":6774784,"data_compressed_original":44175360,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574964,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":87932,"kb_used_data":20656,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94283908,"statfs":{"total":96636764160,"available":96546721792,"internally_reserved":0,"allocated":21151744,"data_stored":46919348,"data_compressed":3819253,"data_compressed_allocated":4960256,"data_compressed_original":31031296,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574965,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51228,"kb_used_data":7568,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94320612,"statfs":{"total":96636764160,"available":96584306688,"internally_reserved":0,"allocated":7749632,"data_stored":17554823,"data_compressed":1445694,"data_compressed_allocated":2031616,"data_compressed_original":12120064,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574965,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62352,"kb_used_data":17288,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94309488,"statfs":{"total":96636764160,"available":96572915712,"internally_reserved":0,"allocated":17702912,"data_stored":50842674,"data_compressed":4902040,"data_compressed_allocated":6463488,"data_compressed_original":39886848,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574964,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":97896,"kb_used_data":30624,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94273944,"statfs":{"total":96636764160,"available":96536518656,"internally_reserved":0,"allocated":31358976,"data_stored":55363963,"data_compressed":3541047,"data_compressed_allocated":4481024,"data_compressed_original":28794880,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32564,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32564,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5586944,"data_stored":29360766,"data_compressed":3508483,"data_compressed_allocated":4435968,"data_compressed_original":28221440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8802304,"data_stored":42214698,"data_compressed":4902040,"data_compressed_allocated":6463488,"data_compressed_original":39886848,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3031040,"data_stored":13116402,"data_compressed":1445694,"data_compressed_allocated":2031616,"data_compressed_original":12120064,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8028160,"data_stored":34088594,"data_compressed":3819253,"data_compressed_allocated":4960256,"data_compressed_original":31031296,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8749056,"data_stored":46137470,"data_compressed":5258839,"data_compressed_allocated":6774784,"data_compressed_original":44175360,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":5566464,"data_stored":29360218,"data_compressed":3110985,"data_compressed_allocated":4100096,"data_compressed_original":27897856,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":11943936,"data_stored":50332320,"data_compressed":5492002,"data_compressed_allocated":7053312,"data_compressed_original":45449216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12967936,"data_stored":58992336,"data_compressed":6855052,"data_compressed_allocated":8982528,"data_compressed_original":55013376,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":25194496,"data_stored":25173461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8388608,"data_stored":8388608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4206592,"data_stored":4199053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12611584,"data_stored":12591386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8421376,"data_stored":8397294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8425472,"data_stored":8406310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20987904,"data_stored":20975355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12599296,"data_stored":12589353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-14T23:58:10.250 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-14T23:58:10.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.500+0000 7f1ae2e00640 1 Processor -- start 2024-04-14T23:58:10.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.501+0000 7f1ae2e00640 1 -- start start 2024-04-14T23:58:10.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.502+0000 7f1ae2e00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f1adc071e60 0x7f1adc159560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:10.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.502+0000 7f1ae2e00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1adc158ac0 con 0x7f1adc06d0d0 2024-04-14T23:58:10.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.502+0000 7f1ae2e00640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1adc072de0 con 0x7f1adc06e010 2024-04-14T23:58:10.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.502+0000 7f1ae2e00640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f1adc06d540 con 0x7f1adc071e60 2024-04-14T23:58:10.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.503+0000 7f1ae1a00640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f1adc06e010 0x7f1adc0726d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.60:45442/0 (socket says 172.21.15.60:45442) 2024-04-14T23:58:10.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.503+0000 7f1ae1a00640 1 -- 172.21.15.60:0/1286702569 learned_addr learned my addr 172.21.15.60:0/1286702569 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:58:10.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.503+0000 7f1ae1000640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f1adc071e60 0x7f1adc159560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:10.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.503+0000 7f1adbe00640 1 -- 172.21.15.60:0/1286702569 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 989584987 0 0) 0x7f1adc072de0 con 0x7f1adc06e010 2024-04-14T23:58:10.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.503+0000 7f1adbe00640 1 -- 172.21.15.60:0/1286702569 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1ac4003520 con 0x7f1adc06e010 2024-04-14T23:58:10.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.503+0000 7f1ae1000640 1 -- 172.21.15.60:0/1286702569 >> v1:172.21.15.186:6789/0 conn(0x7f1adc06e010 legacy=0x7f1adc0726d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:10.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.503+0000 7f1ae1000640 1 -- 172.21.15.60:0/1286702569 >> v1:172.21.15.60:6789/0 conn(0x7f1adc06d0d0 legacy=0x7f1adc0c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:10.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.503+0000 7f1ae1000640 1 -- 172.21.15.60:0/1286702569 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1adc073130 con 0x7f1adc071e60 2024-04-14T23:58:10.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.504+0000 7f1ae1000640 1 --2- 172.21.15.60:0/1286702569 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1adc071e60 0x7f1adc159560 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f1ac80097a0 tx=0x7f1ac802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79887f43fa971f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:10.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.505+0000 7f1adbe00640 1 -- 172.21.15.60:0/1286702569 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac8034070 con 0x7f1adc071e60 2024-04-14T23:58:10.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.505+0000 7f1adbe00640 1 -- 172.21.15.60:0/1286702569 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ac8030610 con 0x7f1adc071e60 2024-04-14T23:58:10.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.505+0000 7f1adbe00640 1 -- 172.21.15.60:0/1286702569 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac80309e0 con 0x7f1adc071e60 2024-04-14T23:58:10.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.505+0000 7f1ae2e00640 1 -- 172.21.15.60:0/1286702569 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1adc071e60 msgr2=0x7f1adc159560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:10.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.505+0000 7f1ae2e00640 1 --2- 172.21.15.60:0/1286702569 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1adc071e60 0x7f1adc159560 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f1ac80097a0 tx=0x7f1ac802b960 comp rx=0 tx=0).stop 2024-04-14T23:58:10.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.505+0000 7f1ae2e00640 1 -- 172.21.15.60:0/1286702569 shutdown_connections 2024-04-14T23:58:10.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.505+0000 7f1ae2e00640 1 --2- 172.21.15.60:0/1286702569 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1adc071e60 0x7f1adc159560 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:10.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.505+0000 7f1ae2e00640 1 -- 172.21.15.60:0/1286702569 >> 172.21.15.60:0/1286702569 conn(0x7f1adc06fdc0 msgr2=0x7f1adc0701d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:10.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.505+0000 7f1ae2e00640 1 -- 172.21.15.60:0/1286702569 shutdown_connections 2024-04-14T23:58:10.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.505+0000 7f1ae2e00640 1 -- 172.21.15.60:0/1286702569 wait complete. 2024-04-14T23:58:10.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.507+0000 7f1ae2e00640 1 Processor -- start 2024-04-14T23:58:10.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.507+0000 7f1ae2e00640 1 -- start start 2024-04-14T23:58:10.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.508+0000 7f1ae2e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1adc06d0d0 0x7f1adc15a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:10.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.508+0000 7f1ae2e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1adc06e010 0x7f1adc0cc1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:10.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.508+0000 7f1ae2e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1adc071e60 0x7f1adc0cc720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:10.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.508+0000 7f1ae2e00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f1adc1590d0 con 0x7f1adc06e010 2024-04-14T23:58:10.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.508+0000 7f1ae2e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1adc153170 con 0x7f1adc06d0d0 2024-04-14T23:58:10.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.508+0000 7f1ae2e00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f1adc158d60 con 0x7f1adc071e60 2024-04-14T23:58:10.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.508+0000 7f1ae1000640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1adc06e010 0x7f1adc0cc1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:10.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.508+0000 7f1ae1000640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1adc06e010 0x7f1adc0cc1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:39786/0 (socket says 172.21.15.60:39786) 2024-04-14T23:58:10.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.508+0000 7f1ae1000640 1 -- 172.21.15.60:0/2996926797 learned_addr learned my addr 172.21.15.60:0/2996926797 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:58:10.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.509+0000 7f1ae2400640 1 --2- 172.21.15.60:0/2996926797 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1adc071e60 0x7f1adc0cc720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:10.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.509+0000 7f1ae1a00640 1 --2- 172.21.15.60:0/2996926797 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1adc06d0d0 0x7f1adc15a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:10.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.509+0000 7f1ae1000640 1 -- 172.21.15.60:0/2996926797 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1adc071e60 msgr2=0x7f1adc0cc720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:10.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.509+0000 7f1ae1000640 1 --2- 172.21.15.60:0/2996926797 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1adc071e60 0x7f1adc0cc720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:10.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.509+0000 7f1ae1000640 1 -- 172.21.15.60:0/2996926797 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1adc06d0d0 msgr2=0x7f1adc15a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:10.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.509+0000 7f1ae1000640 1 --2- 172.21.15.60:0/2996926797 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1adc06d0d0 0x7f1adc15a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:10.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.509+0000 7f1ae1000640 1 -- 172.21.15.60:0/2996926797 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1adc14e0e0 con 0x7f1adc06e010 2024-04-14T23:58:10.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.509+0000 7f1ae1000640 1 --2- 172.21.15.60:0/2996926797 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1adc06e010 0x7f1adc0cc1e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f1ac8000c00 tx=0x7f1ac80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:10.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.510+0000 7f1ad9600640 1 -- 172.21.15.60:0/2996926797 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac8034040 con 0x7f1adc06e010 2024-04-14T23:58:10.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.510+0000 7f1ad9600640 1 -- 172.21.15.60:0/2996926797 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ac803aaa0 con 0x7f1adc06e010 2024-04-14T23:58:10.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.510+0000 7f1ad9600640 1 -- 172.21.15.60:0/2996926797 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac80443f0 con 0x7f1adc06e010 2024-04-14T23:58:10.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.510+0000 7f1ae2e00640 1 -- 172.21.15.60:0/2996926797 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1adc14e3a0 con 0x7f1adc06e010 2024-04-14T23:58:10.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.510+0000 7f1ae2e00640 1 -- 172.21.15.60:0/2996926797 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1adc14f220 con 0x7f1adc06e010 2024-04-14T23:58:10.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.511+0000 7f1ae2e00640 1 -- 172.21.15.60:0/2996926797 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1adc068bc0 con 0x7f1adc06e010 2024-04-14T23:58:10.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.512+0000 7f1ad9600640 1 -- 172.21.15.60:0/2996926797 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f1ac804b060 con 0x7f1adc06e010 2024-04-14T23:58:10.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.513+0000 7f1ad9600640 1 --2- 172.21.15.60:0/2996926797 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f1abc06b9c0 0x7f1abc06de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:10.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.513+0000 7f1ad9600640 1 -- 172.21.15.60:0/2996926797 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6255+0+0 (secure 0 0 0) 0x7f1ac80ac650 con 0x7f1adc06e010 2024-04-14T23:58:10.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.513+0000 7f1ae1a00640 1 --2- 172.21.15.60:0/2996926797 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f1abc06b9c0 0x7f1abc06de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:10.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.514+0000 7f1ae1a00640 1 --2- 172.21.15.60:0/2996926797 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f1abc06b9c0 0x7f1abc06de80 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f1ad0001560 tx=0x7f1ad0001a90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:10.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.517+0000 7f1ad9600640 1 -- 172.21.15.60:0/2996926797 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1adc068bc0 con 0x7f1adc06e010 2024-04-14T23:58:10.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.721+0000 7f1ae2e00640 1 -- 172.21.15.60:0/2996926797 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1adc04f540 con 0x7f1adc06e010 2024-04-14T23:58:10.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.724+0000 7f1ad9600640 1 -- 172.21.15.60:0/2996926797 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) ==== 74+0+17524 (secure 0 0 0) 0x7f1ac803f070 con 0x7f1adc06e010 2024-04-14T23:58:10.729 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:58:10.729 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":45,"fsid":"5847db56-a76d-4486-b563-d83f1186af92","created":"2024-04-14T23:48:23.900003+0000","modified":"2024-04-14T23:57:39.001988+0000","last_up_change":"2024-04-14T23:48:34.203573+0000","last_in_change":"2024-04-14T23:48:26.345499+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-14T23:48:36.247372+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-14T23:48:40.274195+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-14T23:48:41.722183+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":505,"snap_epoch":41,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b2b2b755-7f42-4f9b-af9a-a178f7cc5a52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6817","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6818","nonce":2536528895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6822","nonce":2536528895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6827","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6828","nonce":2536528895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6825","nonce":2536528895}]},"public_addr":"172.21.15.60:6818/2536528895","cluster_addr":"172.21.15.60:6822/2536528895","heartbeat_back_addr":"172.21.15.60:6828/2536528895","heartbeat_front_addr":"172.21.15.60:6825/2536528895","state":["exists","up"]},{"osd":1,"uuid":"321418cd-5ab7-41a5-b9c8-b3fc12f68ebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6801","nonce":3115695676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6803","nonce":3115695676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6807","nonce":3115695676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6805","nonce":3115695676}]},"public_addr":"172.21.15.60:6801/3115695676","cluster_addr":"172.21.15.60:6803/3115695676","heartbeat_back_addr":"172.21.15.60:6807/3115695676","heartbeat_front_addr":"172.21.15.60:6805/3115695676","state":["exists","up"]},{"osd":2,"uuid":"07874e9f-3a38-41dc-be9f-2b75e84e53d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6819","nonce":3475168698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6824","nonce":3475168698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6831","nonce":3475168698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6829","nonce":3475168698}]},"public_addr":"172.21.15.60:6819/3475168698","cluster_addr":"172.21.15.60:6824/3475168698","heartbeat_back_addr":"172.21.15.60:6831/3475168698","heartbeat_front_addr":"172.21.15.60:6829/3475168698","state":["exists","up"]},{"osd":3,"uuid":"7e4ef6c9-b17a-47c6-a320-f5d16a6fe05e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6809","nonce":1730813674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6811","nonce":1730813674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6815","nonce":1730813674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6813","nonce":1730813674}]},"public_addr":"172.21.15.60:6809/1730813674","cluster_addr":"172.21.15.60:6811/1730813674","heartbeat_back_addr":"172.21.15.60:6815/1730813674","heartbeat_front_addr":"172.21.15.60:6813/1730813674","state":["exists","up"]},{"osd":4,"uuid":"759a3efc-dade-44c5-b57a-1623499aba50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6805","nonce":1464587038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6810","nonce":1464587038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6819","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6822","nonce":1464587038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6813","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6816","nonce":1464587038}]},"public_addr":"172.21.15.186:6805/1464587038","cluster_addr":"172.21.15.186:6810/1464587038","heartbeat_back_addr":"172.21.15.186:6822/1464587038","heartbeat_front_addr":"172.21.15.186:6816/1464587038","state":["exists","up"]},{"osd":5,"uuid":"24cd58c1-0cf1-4077-ab33-84c39c6907ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6801","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6803","nonce":1697098067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6809","nonce":1697098067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6820","nonce":1697098067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6815","nonce":1697098067}]},"public_addr":"172.21.15.186:6803/1697098067","cluster_addr":"172.21.15.186:6809/1697098067","heartbeat_back_addr":"172.21.15.186:6820/1697098067","heartbeat_front_addr":"172.21.15.186:6815/1697098067","state":["exists","up"]},{"osd":6,"uuid":"6add38f1-65e9-4b0d-927a-4ab77f5e2323","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6804","nonce":1837108559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6807","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6811","nonce":1837108559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6823","nonce":1837108559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6817","nonce":1837108559}]},"public_addr":"172.21.15.186:6804/1837108559","cluster_addr":"172.21.15.186:6811/1837108559","heartbeat_back_addr":"172.21.15.186:6823/1837108559","heartbeat_front_addr":"172.21.15.186:6817/1837108559","state":["exists","up"]},{"osd":7,"uuid":"a2326b6f-7601-4dda-a456-67c443ea5929","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6825","nonce":842537875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6827","nonce":842537875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6831","nonce":842537875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6829","nonce":842537875}]},"public_addr":"172.21.15.186:6825/842537875","cluster_addr":"172.21.15.186:6827/842537875","heartbeat_back_addr":"172.21.15.186:6831/842537875","heartbeat_front_addr":"172.21.15.186:6829/842537875","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.975583+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.976806+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.722725+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.763705+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.502140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.528972+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.517961+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.513350+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.23","mappings":[{"from":3,"to":2}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"3.3b","mappings":[{"from":7,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.60:6835/2356847578":"2024-04-15T23:57:39.001841+0000","172.21.15.60:6834/2356847578":"2024-04-15T23:57:39.001841+0000","172.21.15.60:6833/1971580590":"2024-04-15T23:56:28.972769+0000","172.21.15.60:6835/1714133698":"2024-04-15T23:54:48.942868+0000","172.21.15.60:6832/1971580590":"2024-04-15T23:56:28.972769+0000","172.21.15.60:6834/1714133698":"2024-04-15T23:54:48.942868+0000","172.21.15.60:6833/78303888":"2024-04-15T23:52:28.905933+0000","172.21.15.60:6832/78303888":"2024-04-15T23:52:28.905933+0000","172.21.15.60:6835/2677133122":"2024-04-15T23:50:58.881136+0000","172.21.15.60:6834/2677133122":"2024-04-15T23:50:58.881136+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-14T23:58:10.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.726+0000 7f1ae2e00640 1 -- 172.21.15.60:0/2996926797 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f1abc06b9c0 msgr2=0x7f1abc06de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:10.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.726+0000 7f1ae2e00640 1 --2- 172.21.15.60:0/2996926797 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f1abc06b9c0 0x7f1abc06de80 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f1ad0001560 tx=0x7f1ad0001a90 comp rx=0 tx=0).stop 2024-04-14T23:58:10.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.727+0000 7f1ae2e00640 1 -- 172.21.15.60:0/2996926797 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1adc06e010 msgr2=0x7f1adc0cc1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:10.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.727+0000 7f1ae2e00640 1 --2- 172.21.15.60:0/2996926797 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1adc06e010 0x7f1adc0cc1e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f1ac8000c00 tx=0x7f1ac80046c0 comp rx=0 tx=0).stop 2024-04-14T23:58:10.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.727+0000 7f1ae2e00640 1 -- 172.21.15.60:0/2996926797 shutdown_connections 2024-04-14T23:58:10.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.727+0000 7f1ae2e00640 1 --2- 172.21.15.60:0/2996926797 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f1abc06b9c0 0x7f1abc06de80 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:10.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.727+0000 7f1ae2e00640 1 --2- 172.21.15.60:0/2996926797 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1adc071e60 0x7f1adc0cc720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:10.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.727+0000 7f1ae2e00640 1 --2- 172.21.15.60:0/2996926797 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1adc06e010 0x7f1adc0cc1e0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:10.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.727+0000 7f1ae2e00640 1 --2- 172.21.15.60:0/2996926797 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1adc06d0d0 0x7f1adc15a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:10.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.727+0000 7f1ae2e00640 1 -- 172.21.15.60:0/2996926797 >> 172.21.15.60:0/2996926797 conn(0x7f1adc06fdc0 msgr2=0x7f1adc0701a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:10.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.727+0000 7f1ae2e00640 1 -- 172.21.15.60:0/2996926797 shutdown_connections 2024-04-14T23:58:10.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:10.728+0000 7f1ae2e00640 1 -- 172.21.15.60:0/2996926797 wait complete. 2024-04-14T23:58:11.753 INFO:tasks.ceph:Scrubbing osd.0 2024-04-14T23:58:11.753 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-14T23:58:11.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:11.989+0000 7f22d6000640 1 Processor -- start 2024-04-14T23:58:11.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:11.989+0000 7f22d6000640 1 -- start start 2024-04-14T23:58:11.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:11.991+0000 7f22d6000640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f22d0152a40 0x7f22d0159080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:11.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:11.991+0000 7f22d6000640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f22d0152e20 0x7f22d006e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:11.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:11.991+0000 7f22d6000640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22d015c380 con 0x7f22d01595c0 2024-04-14T23:58:11.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:11.991+0000 7f22d6000640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f22d015cf00 con 0x7f22d0152a40 2024-04-14T23:58:11.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:11.991+0000 7f22d6000640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f22d015c660 con 0x7f22d0152e20 2024-04-14T23:58:11.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:11.991+0000 7f22cf400640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f22d0152a40 0x7f22d0159080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:11.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:11.991+0000 7f22cf400640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f22d0152a40 0x7f22d0159080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59428/0 (socket says 172.21.15.60:59428) 2024-04-14T23:58:11.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:11.991+0000 7f22cf400640 1 -- 172.21.15.60:0/3805447697 learned_addr learned my addr 172.21.15.60:0/3805447697 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:58:11.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:11.991+0000 7f22cea00640 1 --2- 172.21.15.60:0/3805447697 >> v2:172.21.15.186:3300/0 conn(0x7f22d0152e20 0x7f22d006e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:11.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:11.992+0000 7f22ce000640 1 -- 172.21.15.60:0/3805447697 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1391830882 0 0) 0x7f22d015c380 con 0x7f22d01595c0 2024-04-14T23:58:11.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:11.992+0000 7f22cf400640 1 -- 172.21.15.60:0/3805447697 >> v2:172.21.15.186:3300/0 conn(0x7f22d0152e20 msgr2=0x7f22d006e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:58:11.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:11.992+0000 7f22cf400640 1 --2- 172.21.15.60:0/3805447697 >> v2:172.21.15.186:3300/0 conn(0x7f22d0152e20 0x7f22d006e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:11.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:11.992+0000 7f22cf400640 1 -- 172.21.15.60:0/3805447697 >> v1:172.21.15.186:6789/0 conn(0x7f22d01595c0 legacy=0x7f22d006ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:11.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:11.992+0000 7f22cf400640 1 -- 172.21.15.60:0/3805447697 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22d015c940 con 0x7f22d0152a40 2024-04-14T23:58:11.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:11.993+0000 7f22cf400640 1 --2- 172.21.15.60:0/3805447697 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f22d0152a40 0x7f22d0159080 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f22d00c68c0 tx=0x7f22c002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=703af1ab9ab65970 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:11.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:11.993+0000 7f22ce000640 1 -- 172.21.15.60:0/3805447697 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22c0034070 con 0x7f22d0152a40 2024-04-14T23:58:11.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:11.993+0000 7f22ce000640 1 -- 172.21.15.60:0/3805447697 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22c0030860 con 0x7f22d0152a40 2024-04-14T23:58:11.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:11.993+0000 7f22ce000640 1 -- 172.21.15.60:0/3805447697 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22c0030c30 con 0x7f22d0152a40 2024-04-14T23:58:12.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:11.994+0000 7f22d6000640 1 -- 172.21.15.60:0/3805447697 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f22d0152a40 msgr2=0x7f22d0159080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:12.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:11.994+0000 7f22d6000640 1 --2- 172.21.15.60:0/3805447697 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f22d0152a40 0x7f22d0159080 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f22d00c68c0 tx=0x7f22c002b9a0 comp rx=0 tx=0).stop 2024-04-14T23:58:12.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:11.994+0000 7f22d6000640 1 -- 172.21.15.60:0/3805447697 shutdown_connections 2024-04-14T23:58:12.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:11.994+0000 7f22d6000640 1 --2- 172.21.15.60:0/3805447697 >> v2:172.21.15.186:3300/0 conn(0x7f22d0152e20 0x7f22d006e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:12.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:11.994+0000 7f22d6000640 1 --2- 172.21.15.60:0/3805447697 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f22d0152a40 0x7f22d0159080 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:12.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:11.994+0000 7f22d6000640 1 -- 172.21.15.60:0/3805447697 >> 172.21.15.60:0/3805447697 conn(0x7f22d0067540 msgr2=0x7f22d006b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:12.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:11.994+0000 7f22d6000640 1 -- 172.21.15.60:0/3805447697 shutdown_connections 2024-04-14T23:58:12.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:11.994+0000 7f22d6000640 1 -- 172.21.15.60:0/3805447697 wait complete. 2024-04-14T23:58:12.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:11.996+0000 7f22d6000640 1 Processor -- start 2024-04-14T23:58:12.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:11.996+0000 7f22d6000640 1 -- start start 2024-04-14T23:58:12.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:11.997+0000 7f22d6000640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f22d0152a40 0x7f22d013e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:12.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:11.997+0000 7f22d6000640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f22d0152e20 0x7f22d01ee3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:12.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:11.997+0000 7f22d6000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22d01595c0 0x7f22d00c8930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:12.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:11.997+0000 7f22d6000640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f22d00c65c0 con 0x7f22d0152a40 2024-04-14T23:58:12.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:11.997+0000 7f22d6000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f22d0158b60 con 0x7f22d01595c0 2024-04-14T23:58:12.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:11.997+0000 7f22d6000640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f22d015bf40 con 0x7f22d0152e20 2024-04-14T23:58:12.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:11.997+0000 7f22cf400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f22d0152a40 0x7f22d013e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:12.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:11.997+0000 7f22cf400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f22d0152a40 0x7f22d013e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59432/0 (socket says 172.21.15.60:59432) 2024-04-14T23:58:12.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:11.997+0000 7f22cf400640 1 -- 172.21.15.60:0/1881998053 learned_addr learned my addr 172.21.15.60:0/1881998053 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:58:12.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:11.998+0000 7f22cea00640 1 --2- 172.21.15.60:0/1881998053 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f22d0152e20 0x7f22d01ee3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:12.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:11.998+0000 7f22cfe00640 1 --2- 172.21.15.60:0/1881998053 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22d01595c0 0x7f22d00c8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:12.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:11.998+0000 7f22cf400640 1 -- 172.21.15.60:0/1881998053 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f22d0152e20 msgr2=0x7f22d01ee3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:12.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:11.998+0000 7f22cf400640 1 --2- 172.21.15.60:0/1881998053 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f22d0152e20 0x7f22d01ee3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:12.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:11.998+0000 7f22cf400640 1 -- 172.21.15.60:0/1881998053 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22d01595c0 msgr2=0x7f22d00c8930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:12.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:11.998+0000 7f22cf400640 1 --2- 172.21.15.60:0/1881998053 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22d01595c0 0x7f22d00c8930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:12.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:11.998+0000 7f22cf400640 1 -- 172.21.15.60:0/1881998053 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22d00c94c0 con 0x7f22d0152a40 2024-04-14T23:58:12.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:11.998+0000 7f22cf400640 1 --2- 172.21.15.60:0/1881998053 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f22d0152a40 0x7f22d013e510 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f22c002b9e0 tx=0x7f22c0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:12.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:11.999+0000 7f22c7400640 1 -- 172.21.15.60:0/1881998053 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22c0048070 con 0x7f22d0152a40 2024-04-14T23:58:12.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:11.999+0000 7f22c7400640 1 -- 172.21.15.60:0/1881998053 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22c0030660 con 0x7f22d0152a40 2024-04-14T23:58:12.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:11.999+0000 7f22c7400640 1 -- 172.21.15.60:0/1881998053 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22c0034040 con 0x7f22d0152a40 2024-04-14T23:58:12.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:11.999+0000 7f22d6000640 1 -- 172.21.15.60:0/1881998053 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22d0142c40 con 0x7f22d0152a40 2024-04-14T23:58:12.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:11.999+0000 7f22d6000640 1 -- 172.21.15.60:0/1881998053 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22d0156d70 con 0x7f22d0152a40 2024-04-14T23:58:12.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.000+0000 7f22d6000640 1 -- 172.21.15.60:0/1881998053 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f22d015c380 con 0x7f22d0152a40 2024-04-14T23:58:12.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.001+0000 7f22c7400640 1 -- 172.21.15.60:0/1881998053 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f22c0037070 con 0x7f22d0152a40 2024-04-14T23:58:12.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.002+0000 7f22c7400640 1 --2- 172.21.15.60:0/1881998053 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f22b006b990 0x7f22b006de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:12.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.002+0000 7f22c7400640 1 -- 172.21.15.60:0/1881998053 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6255+0+0 (secure 0 0 0) 0x7f22c00ac9c0 con 0x7f22d0152a40 2024-04-14T23:58:12.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.002+0000 7f22cea00640 1 --2- 172.21.15.60:0/1881998053 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f22b006b990 0x7f22b006de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:12.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.002+0000 7f22c7400640 1 --2- 172.21.15.60:0/1881998053 >> [v2:172.21.15.60:6817/2536528895,v1:172.21.15.60:6818/2536528895] conn(0x7f22b0074040 0x7f22b00764c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:12.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.002+0000 7f22c7400640 1 -- 172.21.15.60:0/1881998053 --> [v2:172.21.15.60:6817/2536528895,v1:172.21.15.60:6818/2536528895] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f22b0076b90 con 0x7f22b0074040 2024-04-14T23:58:12.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.002+0000 7f22c7400640 1 -- 172.21.15.60:0/1881998053 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f22c00b12a0 con 0x7f22d0152a40 2024-04-14T23:58:12.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.003+0000 7f22cfe00640 1 --2- 172.21.15.60:0/1881998053 >> [v2:172.21.15.60:6817/2536528895,v1:172.21.15.60:6818/2536528895] conn(0x7f22b0074040 0x7f22b00764c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:12.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.003+0000 7f22cea00640 1 --2- 172.21.15.60:0/1881998053 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f22b006b990 0x7f22b006de50 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f22d01ef4d0 tx=0x7f22bc009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:12.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.003+0000 7f22cfe00640 1 --2- 172.21.15.60:0/1881998053 >> [v2:172.21.15.60:6817/2536528895,v1:172.21.15.60:6818/2536528895] conn(0x7f22b0074040 0x7f22b00764c0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:12.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.004+0000 7f22c7400640 1 -- 172.21.15.60:0/1881998053 <== osd.0 v2:172.21.15.60:6817/2536528895 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f22b0076b90 con 0x7f22b0074040 2024-04-14T23:58:12.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.019+0000 7f22d6000640 1 -- 172.21.15.60:0/1881998053 --> [v2:172.21.15.60:6817/2536528895,v1:172.21.15.60:6818/2536528895] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f22d0076970 con 0x7f22b0074040 2024-04-14T23:58:12.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.019+0000 7f22c7400640 1 -- 172.21.15.60:0/1881998053 <== osd.0 v2:172.21.15.60:6817/2536528895 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f22d0076970 con 0x7f22b0074040 2024-04-14T23:58:12.025 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-14T23:58:12.025 INFO:teuthology.orchestra.run.smithi060.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-14T23:58:12.025 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-14T23:58:12.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.020+0000 7f22d6000640 1 -- 172.21.15.60:0/1881998053 >> [v2:172.21.15.60:6817/2536528895,v1:172.21.15.60:6818/2536528895] conn(0x7f22b0074040 msgr2=0x7f22b00764c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:12.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.020+0000 7f22d6000640 1 --2- 172.21.15.60:0/1881998053 >> [v2:172.21.15.60:6817/2536528895,v1:172.21.15.60:6818/2536528895] conn(0x7f22b0074040 0x7f22b00764c0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:12.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.020+0000 7f22d6000640 1 -- 172.21.15.60:0/1881998053 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f22b006b990 msgr2=0x7f22b006de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:12.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.020+0000 7f22d6000640 1 --2- 172.21.15.60:0/1881998053 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f22b006b990 0x7f22b006de50 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f22d01ef4d0 tx=0x7f22bc009190 comp rx=0 tx=0).stop 2024-04-14T23:58:12.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.020+0000 7f22d6000640 1 -- 172.21.15.60:0/1881998053 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f22d0152a40 msgr2=0x7f22d013e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:12.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.020+0000 7f22d6000640 1 --2- 172.21.15.60:0/1881998053 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f22d0152a40 0x7f22d013e510 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f22c002b9e0 tx=0x7f22c0004730 comp rx=0 tx=0).stop 2024-04-14T23:58:12.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.020+0000 7f22cfe00640 1 -- 172.21.15.60:0/1881998053 reap_dead start 2024-04-14T23:58:12.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.020+0000 7f22d6000640 1 -- 172.21.15.60:0/1881998053 shutdown_connections 2024-04-14T23:58:12.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.020+0000 7f22d6000640 1 -- 172.21.15.60:0/1881998053 >> 172.21.15.60:0/1881998053 conn(0x7f22d0067540 msgr2=0x7f22d00c3700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:12.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.020+0000 7f22d6000640 1 -- 172.21.15.60:0/1881998053 shutdown_connections 2024-04-14T23:58:12.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.021+0000 7f22d6000640 1 -- 172.21.15.60:0/1881998053 wait complete. 2024-04-14T23:58:12.045 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-04-14T23:58:12.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.253+0000 7fdda0a00640 1 Processor -- start 2024-04-14T23:58:12.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.253+0000 7fdda0a00640 1 -- start start 2024-04-14T23:58:12.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.255+0000 7fdda0a00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd9c15b740 0x7fdd9c06eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:12.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.255+0000 7fdda0a00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdd9c0c7480 0x7fdd9c0711a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:12.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.255+0000 7fdda0a00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd9c15bb20 con 0x7fdd9c06f0a0 2024-04-14T23:58:12.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.255+0000 7fdda0a00640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fdd9c070d10 con 0x7fdd9c0c7480 2024-04-14T23:58:12.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.255+0000 7fdda0a00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fdd9c15e310 con 0x7fdd9c15b740 2024-04-14T23:58:12.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.255+0000 7fdd9aa00640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fdd9c06f0a0 0x7fdd9c0c6d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53234/0 (socket says 172.21.15.60:53234) 2024-04-14T23:58:12.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.255+0000 7fdd9aa00640 1 -- 172.21.15.60:0/14930395 learned_addr learned my addr 172.21.15.60:0/14930395 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:58:12.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.255+0000 7fdd9b400640 1 --2- 172.21.15.60:0/14930395 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd9c15b740 0x7fdd9c06eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:12.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.255+0000 7fdd9be00640 1 --2- 172.21.15.60:0/14930395 >> v2:172.21.15.60:3300/0 conn(0x7fdd9c0c7480 0x7fdd9c0711a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:12.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.256+0000 7fdd9a000640 1 -- 172.21.15.60:0/14930395 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2670482266 0 0) 0x7fdd9c15bb20 con 0x7fdd9c06f0a0 2024-04-14T23:58:12.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.256+0000 7fdd9a000640 1 -- 172.21.15.60:0/14930395 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd80003520 con 0x7fdd9c06f0a0 2024-04-14T23:58:12.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.256+0000 7fdd9b400640 1 -- 172.21.15.60:0/14930395 >> v2:172.21.15.60:3300/0 conn(0x7fdd9c0c7480 msgr2=0x7fdd9c0711a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:58:12.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.256+0000 7fdd9b400640 1 --2- 172.21.15.60:0/14930395 >> v2:172.21.15.60:3300/0 conn(0x7fdd9c0c7480 0x7fdd9c0711a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:12.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.256+0000 7fdd9b400640 1 -- 172.21.15.60:0/14930395 >> v1:172.21.15.60:6789/0 conn(0x7fdd9c06f0a0 legacy=0x7fdd9c0c6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:12.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.256+0000 7fdd9b400640 1 -- 172.21.15.60:0/14930395 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd9c15e5f0 con 0x7fdd9c15b740 2024-04-14T23:58:12.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.256+0000 7fdd9be00640 1 --2- 172.21.15.60:0/14930395 >> v2:172.21.15.60:3300/0 conn(0x7fdd9c0c7480 0x7fdd9c0711a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:58:12.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.257+0000 7fdd9b400640 1 --2- 172.21.15.60:0/14930395 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd9c15b740 0x7fdd9c06eb60 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c0097a0 tx=0x7fdd8c02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc1ee7d6d197784 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:12.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.258+0000 7fdd9a000640 1 -- 172.21.15.60:0/14930395 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd8c035020 con 0x7fdd9c15b740 2024-04-14T23:58:12.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.258+0000 7fdd9a000640 1 -- 172.21.15.60:0/14930395 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd8c031810 con 0x7fdd9c15b740 2024-04-14T23:58:12.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.258+0000 7fdd9a000640 1 -- 172.21.15.60:0/14930395 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd8c031be0 con 0x7fdd9c15b740 2024-04-14T23:58:12.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.259+0000 7fdda0a00640 1 -- 172.21.15.60:0/14930395 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd9c15b740 msgr2=0x7fdd9c06eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:12.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.259+0000 7fdda0a00640 1 --2- 172.21.15.60:0/14930395 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd9c15b740 0x7fdd9c06eb60 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c0097a0 tx=0x7fdd8c02b8b0 comp rx=0 tx=0).stop 2024-04-14T23:58:12.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.259+0000 7fdda0a00640 1 -- 172.21.15.60:0/14930395 shutdown_connections 2024-04-14T23:58:12.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.259+0000 7fdda0a00640 1 --2- 172.21.15.60:0/14930395 >> v2:172.21.15.60:3300/0 conn(0x7fdd9c0c7480 0x7fdd9c0711a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:12.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.259+0000 7fdda0a00640 1 --2- 172.21.15.60:0/14930395 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd9c15b740 0x7fdd9c06eb60 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:12.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.259+0000 7fdda0a00640 1 -- 172.21.15.60:0/14930395 >> 172.21.15.60:0/14930395 conn(0x7fdd9c0cc2f0 msgr2=0x7fdd9c0cc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:12.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.259+0000 7fdda0a00640 1 -- 172.21.15.60:0/14930395 shutdown_connections 2024-04-14T23:58:12.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.259+0000 7fdda0a00640 1 -- 172.21.15.60:0/14930395 wait complete. 2024-04-14T23:58:12.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.261+0000 7fdda0a00640 1 Processor -- start 2024-04-14T23:58:12.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.261+0000 7fdda0a00640 1 -- start start 2024-04-14T23:58:12.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.262+0000 7fdda0a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd9c06f0a0 0x7fdd9c1f4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:12.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.262+0000 7fdda0a00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd9c0c7480 0x7fdd9c157b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:12.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.262+0000 7fdda0a00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd9c15b740 0x7fdd9c0c4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:12.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.262+0000 7fdda0a00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fdd9c15ee90 con 0x7fdd9c15b740 2024-04-14T23:58:12.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.262+0000 7fdda0a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdd9c15ebb0 con 0x7fdd9c06f0a0 2024-04-14T23:58:12.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.262+0000 7fdda0a00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fdd9c0c86b0 con 0x7fdd9c0c7480 2024-04-14T23:58:12.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.263+0000 7fdd9be00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd9c15b740 0x7fdd9c0c4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:12.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.263+0000 7fdd9aa00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd9c0c7480 0x7fdd9c157b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:12.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.263+0000 7fdd9b400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd9c06f0a0 0x7fdd9c1f4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:12.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.263+0000 7fdd9be00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd9c15b740 0x7fdd9c0c4580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59462/0 (socket says 172.21.15.60:59462) 2024-04-14T23:58:12.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.263+0000 7fdd9be00640 1 -- 172.21.15.60:0/3508529316 learned_addr learned my addr 172.21.15.60:0/3508529316 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:58:12.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.263+0000 7fdd9be00640 1 -- 172.21.15.60:0/3508529316 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd9c0c7480 msgr2=0x7fdd9c157b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:12.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.263+0000 7fdd9be00640 1 --2- 172.21.15.60:0/3508529316 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd9c0c7480 0x7fdd9c157b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:12.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.263+0000 7fdd9be00640 1 -- 172.21.15.60:0/3508529316 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd9c06f0a0 msgr2=0x7fdd9c1f4f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:12.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.263+0000 7fdd9be00640 1 --2- 172.21.15.60:0/3508529316 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd9c06f0a0 0x7fdd9c1f4f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:12.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.263+0000 7fdd9be00640 1 -- 172.21.15.60:0/3508529316 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd9c1f5490 con 0x7fdd9c15b740 2024-04-14T23:58:12.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.263+0000 7fdd9b400640 1 --2- 172.21.15.60:0/3508529316 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd9c06f0a0 0x7fdd9c1f4f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:58:12.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.264+0000 7fdd9be00640 1 --2- 172.21.15.60:0/3508529316 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd9c15b740 0x7fdd9c0c4580 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fdd94009f00 tx=0x7fdd9400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:12.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.264+0000 7fdd93400640 1 -- 172.21.15.60:0/3508529316 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd94018030 con 0x7fdd9c15b740 2024-04-14T23:58:12.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.264+0000 7fdd93400640 1 -- 172.21.15.60:0/3508529316 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd9400ce50 con 0x7fdd9c15b740 2024-04-14T23:58:12.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.264+0000 7fdda0a00640 1 -- 172.21.15.60:0/3508529316 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd9c0c5310 con 0x7fdd9c15b740 2024-04-14T23:58:12.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.264+0000 7fdd93400640 1 -- 172.21.15.60:0/3508529316 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd94003bb0 con 0x7fdd9c15b740 2024-04-14T23:58:12.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.264+0000 7fdda0a00640 1 -- 172.21.15.60:0/3508529316 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd9c0c58d0 con 0x7fdd9c15b740 2024-04-14T23:58:12.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.266+0000 7fdda0a00640 1 -- 172.21.15.60:0/3508529316 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd9c0720f0 con 0x7fdd9c15b740 2024-04-14T23:58:12.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.267+0000 7fdd93400640 1 -- 172.21.15.60:0/3508529316 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7fdd94005d10 con 0x7fdd9c15b740 2024-04-14T23:58:12.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.267+0000 7fdd93400640 1 --2- 172.21.15.60:0/3508529316 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fdd7c06b9c0 0x7fdd7c06de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:12.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.267+0000 7fdd93400640 1 -- 172.21.15.60:0/3508529316 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6255+0+0 (secure 0 0 0) 0x7fdd9408b3a0 con 0x7fdd9c15b740 2024-04-14T23:58:12.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.268+0000 7fdd9b400640 1 --2- 172.21.15.60:0/3508529316 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fdd7c06b9c0 0x7fdd7c06de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:12.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.269+0000 7fdd9b400640 1 --2- 172.21.15.60:0/3508529316 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fdd7c06b9c0 0x7fdd7c06de80 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c036000 tx=0x7fdd8c03c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:12.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.274+0000 7fdd93400640 1 -- 172.21.15.60:0/3508529316 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fdd9c0720f0 con 0x7fdd9c15b740 2024-04-14T23:58:12.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.475+0000 7fdda0a00640 1 -- 172.21.15.60:0/3508529316 --> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7fdd9c04f540 con 0x7fdd7c06b9c0 2024-04-14T23:58:12.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.476+0000 7fdd93400640 1 -- 172.21.15.60:0/3508529316 <== mgr.4100 v2:172.21.15.186:6832/714104835 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fdd9c04f540 con 0x7fdd7c06b9c0 2024-04-14T23:58:12.481 INFO:teuthology.orchestra.run.smithi060.stderr:instructed osd(s) 0 to deep-scrub 2024-04-14T23:58:12.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.479+0000 7fdda0a00640 1 -- 172.21.15.60:0/3508529316 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fdd7c06b9c0 msgr2=0x7fdd7c06de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:12.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.479+0000 7fdda0a00640 1 --2- 172.21.15.60:0/3508529316 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fdd7c06b9c0 0x7fdd7c06de80 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c036000 tx=0x7fdd8c03c040 comp rx=0 tx=0).stop 2024-04-14T23:58:12.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.479+0000 7fdda0a00640 1 -- 172.21.15.60:0/3508529316 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd9c15b740 msgr2=0x7fdd9c0c4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:12.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.479+0000 7fdda0a00640 1 --2- 172.21.15.60:0/3508529316 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd9c15b740 0x7fdd9c0c4580 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fdd94009f00 tx=0x7fdd9400c490 comp rx=0 tx=0).stop 2024-04-14T23:58:12.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.479+0000 7fdda0a00640 1 -- 172.21.15.60:0/3508529316 shutdown_connections 2024-04-14T23:58:12.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.479+0000 7fdda0a00640 1 --2- 172.21.15.60:0/3508529316 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fdd7c06b9c0 0x7fdd7c06de80 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:12.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.479+0000 7fdda0a00640 1 --2- 172.21.15.60:0/3508529316 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd9c15b740 0x7fdd9c0c4580 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:12.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.479+0000 7fdda0a00640 1 --2- 172.21.15.60:0/3508529316 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd9c0c7480 0x7fdd9c157b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:12.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.479+0000 7fdda0a00640 1 --2- 172.21.15.60:0/3508529316 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd9c06f0a0 0x7fdd9c1f4f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:12.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.479+0000 7fdda0a00640 1 -- 172.21.15.60:0/3508529316 >> 172.21.15.60:0/3508529316 conn(0x7fdd9c0cc2f0 msgr2=0x7fdd9c0cc6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:12.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.479+0000 7fdda0a00640 1 -- 172.21.15.60:0/3508529316 shutdown_connections 2024-04-14T23:58:12.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.479+0000 7fdda0a00640 1 -- 172.21.15.60:0/3508529316 wait complete. 2024-04-14T23:58:12.504 INFO:tasks.ceph:Scrubbing osd.1 2024-04-14T23:58:12.504 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-14T23:58:12.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.733+0000 7f3e02800640 1 Processor -- start 2024-04-14T23:58:12.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.733+0000 7f3e02800640 1 -- start start 2024-04-14T23:58:12.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.735+0000 7f3e02800640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3dfc151b70 0x7f3dfc072ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:12.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.735+0000 7f3e02800640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f3dfc1581d0 0x7f3dfc0c6440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:12.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.735+0000 7f3e02800640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3dfc073100 con 0x7f3dfc151f50 2024-04-14T23:58:12.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.735+0000 7f3e02800640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f3dfc157d40 con 0x7f3dfc1581d0 2024-04-14T23:58:12.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.735+0000 7f3e02800640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f3dfc1579f0 con 0x7f3dfc151b70 2024-04-14T23:58:12.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.736+0000 7f3dfaa00640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f3dfc151f50 0x7f3dfc0722f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53250/0 (socket says 172.21.15.60:53250) 2024-04-14T23:58:12.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.736+0000 7f3dfaa00640 1 -- 172.21.15.60:0/344229246 learned_addr learned my addr 172.21.15.60:0/344229246 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:58:12.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.736+0000 7f3dfbe00640 1 --2- 172.21.15.60:0/344229246 >> v2:172.21.15.186:3300/0 conn(0x7f3dfc1581d0 0x7f3dfc0c6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:12.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.736+0000 7f3dfb400640 1 --2- 172.21.15.60:0/344229246 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3dfc151b70 0x7f3dfc072ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:12.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.736+0000 7f3dfa000640 1 -- 172.21.15.60:0/344229246 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 10621426 0 0) 0x7f3dfc073100 con 0x7f3dfc151f50 2024-04-14T23:58:12.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.736+0000 7f3dfbe00640 1 -- 172.21.15.60:0/344229246 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3dfc151b70 msgr2=0x7f3dfc072ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:58:12.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.736+0000 7f3dfbe00640 1 --2- 172.21.15.60:0/344229246 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3dfc151b70 0x7f3dfc072ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:12.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.736+0000 7f3dfbe00640 1 -- 172.21.15.60:0/344229246 >> v1:172.21.15.60:6789/0 conn(0x7f3dfc151f50 legacy=0x7f3dfc0722f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:12.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.736+0000 7f3dfbe00640 1 -- 172.21.15.60:0/344229246 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dfc0c4710 con 0x7f3dfc1581d0 2024-04-14T23:58:12.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.737+0000 7f3dfbe00640 1 --2- 172.21.15.60:0/344229246 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3dfc1581d0 0x7f3dfc0c6440 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f3df4009d90 tx=0x7f3df402bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d62f78dc349a941 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:12.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.738+0000 7f3dfa000640 1 -- 172.21.15.60:0/344229246 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3df4039070 con 0x7f3dfc1581d0 2024-04-14T23:58:12.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.738+0000 7f3dfa000640 1 -- 172.21.15.60:0/344229246 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3df402cce0 con 0x7f3dfc1581d0 2024-04-14T23:58:12.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.738+0000 7f3dfa000640 1 -- 172.21.15.60:0/344229246 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3df4034660 con 0x7f3dfc1581d0 2024-04-14T23:58:12.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.738+0000 7f3e02800640 1 -- 172.21.15.60:0/344229246 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3dfc1581d0 msgr2=0x7f3dfc0c6440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:12.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.738+0000 7f3e02800640 1 --2- 172.21.15.60:0/344229246 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3dfc1581d0 0x7f3dfc0c6440 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f3df4009d90 tx=0x7f3df402bf30 comp rx=0 tx=0).stop 2024-04-14T23:58:12.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.738+0000 7f3e02800640 1 -- 172.21.15.60:0/344229246 shutdown_connections 2024-04-14T23:58:12.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.738+0000 7f3e02800640 1 --2- 172.21.15.60:0/344229246 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3dfc1581d0 0x7f3dfc0c6440 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:12.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.738+0000 7f3e02800640 1 --2- 172.21.15.60:0/344229246 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3dfc151b70 0x7f3dfc072ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:12.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.738+0000 7f3e02800640 1 -- 172.21.15.60:0/344229246 >> 172.21.15.60:0/344229246 conn(0x7f3dfc0cd210 msgr2=0x7f3dfc0cd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:12.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.739+0000 7f3e02800640 1 -- 172.21.15.60:0/344229246 shutdown_connections 2024-04-14T23:58:12.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.739+0000 7f3e02800640 1 -- 172.21.15.60:0/344229246 wait complete. 2024-04-14T23:58:12.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.740+0000 7f3e02800640 1 Processor -- start 2024-04-14T23:58:12.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.741+0000 7f3e02800640 1 -- start start 2024-04-14T23:58:12.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.741+0000 7f3e02800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3dfc151b70 0x7f3dfc1f0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:12.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.741+0000 7f3e02800640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3dfc1581d0 0x7f3dfc1564e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:12.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.742+0000 7f3e02800640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3dfc0c9750 0x7f3dfc0c8c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:12.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.742+0000 7f3e02800640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f3dfc06f960 con 0x7f3dfc1581d0 2024-04-14T23:58:12.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.742+0000 7f3e02800640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3dfc0c4cd0 con 0x7f3dfc151b70 2024-04-14T23:58:12.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.742+0000 7f3e02800640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f3dfc0c7dc0 con 0x7f3dfc0c9750 2024-04-14T23:58:12.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.742+0000 7f3dfaa00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3dfc1581d0 0x7f3dfc1564e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:12.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.742+0000 7f3dfaa00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3dfc1581d0 0x7f3dfc1564e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59470/0 (socket says 172.21.15.60:59470) 2024-04-14T23:58:12.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.742+0000 7f3dfaa00640 1 -- 172.21.15.60:0/3815498909 learned_addr learned my addr 172.21.15.60:0/3815498909 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:58:12.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.742+0000 7f3dfbe00640 1 --2- 172.21.15.60:0/3815498909 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3dfc0c9750 0x7f3dfc0c8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:12.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.742+0000 7f3dfb400640 1 --2- 172.21.15.60:0/3815498909 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3dfc151b70 0x7f3dfc1f0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:12.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.742+0000 7f3dfaa00640 1 -- 172.21.15.60:0/3815498909 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3dfc0c9750 msgr2=0x7f3dfc0c8c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:12.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.742+0000 7f3dfaa00640 1 --2- 172.21.15.60:0/3815498909 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3dfc0c9750 0x7f3dfc0c8c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:12.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.743+0000 7f3dfaa00640 1 -- 172.21.15.60:0/3815498909 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3dfc151b70 msgr2=0x7f3dfc1f0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:12.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.743+0000 7f3dfaa00640 1 --2- 172.21.15.60:0/3815498909 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3dfc151b70 0x7f3dfc1f0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:12.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.743+0000 7f3dfaa00640 1 -- 172.21.15.60:0/3815498909 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dfc0c9c00 con 0x7f3dfc1581d0 2024-04-14T23:58:12.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.743+0000 7f3dfaa00640 1 --2- 172.21.15.60:0/3815498909 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3dfc1581d0 0x7f3dfc1564e0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f3de800c500 tx=0x7f3de800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:12.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.743+0000 7f3df3400640 1 -- 172.21.15.60:0/3815498909 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3de8018070 con 0x7f3dfc1581d0 2024-04-14T23:58:12.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.743+0000 7f3df3400640 1 -- 172.21.15.60:0/3815498909 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3de8007580 con 0x7f3dfc1581d0 2024-04-14T23:58:12.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.743+0000 7f3df3400640 1 -- 172.21.15.60:0/3815498909 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3de8004a20 con 0x7f3dfc1581d0 2024-04-14T23:58:12.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.743+0000 7f3e02800640 1 -- 172.21.15.60:0/3815498909 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3dfc0c9ee0 con 0x7f3dfc1581d0 2024-04-14T23:58:12.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.743+0000 7f3e02800640 1 -- 172.21.15.60:0/3815498909 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3dfc0c8400 con 0x7f3dfc1581d0 2024-04-14T23:58:12.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.744+0000 7f3e02800640 1 -- 172.21.15.60:0/3815498909 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3dfc0ca400 con 0x7f3dfc1581d0 2024-04-14T23:58:12.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.746+0000 7f3df3400640 1 -- 172.21.15.60:0/3815498909 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f3de801b360 con 0x7f3dfc1581d0 2024-04-14T23:58:12.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.746+0000 7f3df3400640 1 --2- 172.21.15.60:0/3815498909 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f3ddc06b990 0x7f3ddc06de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:12.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.747+0000 7f3df3400640 1 -- 172.21.15.60:0/3815498909 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6255+0+0 (secure 0 0 0) 0x7f3de808a1d0 con 0x7f3dfc1581d0 2024-04-14T23:58:12.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.747+0000 7f3df3400640 1 --2- 172.21.15.60:0/3815498909 >> [v2:172.21.15.60:6800/3115695676,v1:172.21.15.60:6801/3115695676] conn(0x7f3ddc074040 0x7f3ddc0764c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:12.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.747+0000 7f3df3400640 1 -- 172.21.15.60:0/3815498909 --> [v2:172.21.15.60:6800/3115695676,v1:172.21.15.60:6801/3115695676] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3de8006e50 con 0x7f3ddc074040 2024-04-14T23:58:12.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.747+0000 7f3df3400640 1 -- 172.21.15.60:0/3815498909 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f3de808a660 con 0x7f3dfc1581d0 2024-04-14T23:58:12.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.747+0000 7f3dfb400640 1 --2- 172.21.15.60:0/3815498909 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f3ddc06b990 0x7f3ddc06de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:12.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.747+0000 7f3dfbe00640 1 --2- 172.21.15.60:0/3815498909 >> [v2:172.21.15.60:6800/3115695676,v1:172.21.15.60:6801/3115695676] conn(0x7f3ddc074040 0x7f3ddc0764c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:12.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.748+0000 7f3dfb400640 1 --2- 172.21.15.60:0/3815498909 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f3ddc06b990 0x7f3ddc06de50 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f3dec009aa0 tx=0x7f3dec009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:12.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.748+0000 7f3dfbe00640 1 --2- 172.21.15.60:0/3815498909 >> [v2:172.21.15.60:6800/3115695676,v1:172.21.15.60:6801/3115695676] conn(0x7f3ddc074040 0x7f3ddc0764c0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:12.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.749+0000 7f3df3400640 1 -- 172.21.15.60:0/3815498909 <== osd.1 v2:172.21.15.60:6800/3115695676 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3de8006e50 con 0x7f3ddc074040 2024-04-14T23:58:12.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.764+0000 7f3e02800640 1 -- 172.21.15.60:0/3815498909 --> [v2:172.21.15.60:6800/3115695676,v1:172.21.15.60:6801/3115695676] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f3dfc070180 con 0x7f3ddc074040 2024-04-14T23:58:12.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.764+0000 7f3df3400640 1 -- 172.21.15.60:0/3815498909 <== osd.1 v2:172.21.15.60:6800/3115695676 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f3dfc070180 con 0x7f3ddc074040 2024-04-14T23:58:12.769 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-14T23:58:12.770 INFO:teuthology.orchestra.run.smithi060.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-14T23:58:12.770 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-14T23:58:12.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.764+0000 7f3e02800640 1 -- 172.21.15.60:0/3815498909 >> [v2:172.21.15.60:6800/3115695676,v1:172.21.15.60:6801/3115695676] conn(0x7f3ddc074040 msgr2=0x7f3ddc0764c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:12.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.764+0000 7f3e02800640 1 --2- 172.21.15.60:0/3815498909 >> [v2:172.21.15.60:6800/3115695676,v1:172.21.15.60:6801/3115695676] conn(0x7f3ddc074040 0x7f3ddc0764c0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:12.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.765+0000 7f3e02800640 1 -- 172.21.15.60:0/3815498909 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f3ddc06b990 msgr2=0x7f3ddc06de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:12.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.765+0000 7f3e02800640 1 --2- 172.21.15.60:0/3815498909 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f3ddc06b990 0x7f3ddc06de50 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f3dec009aa0 tx=0x7f3dec009190 comp rx=0 tx=0).stop 2024-04-14T23:58:12.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.765+0000 7f3e02800640 1 -- 172.21.15.60:0/3815498909 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3dfc1581d0 msgr2=0x7f3dfc1564e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:12.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.765+0000 7f3e02800640 1 --2- 172.21.15.60:0/3815498909 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3dfc1581d0 0x7f3dfc1564e0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f3de800c500 tx=0x7f3de800c9c0 comp rx=0 tx=0).stop 2024-04-14T23:58:12.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.765+0000 7f3dfbe00640 1 -- 172.21.15.60:0/3815498909 reap_dead start 2024-04-14T23:58:12.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.765+0000 7f3e02800640 1 -- 172.21.15.60:0/3815498909 shutdown_connections 2024-04-14T23:58:12.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.765+0000 7f3e02800640 1 -- 172.21.15.60:0/3815498909 >> 172.21.15.60:0/3815498909 conn(0x7f3dfc0cd210 msgr2=0x7f3dfc06b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:12.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.765+0000 7f3e02800640 1 -- 172.21.15.60:0/3815498909 shutdown_connections 2024-04-14T23:58:12.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:12.765+0000 7f3e02800640 1 -- 172.21.15.60:0/3815498909 wait complete. 2024-04-14T23:58:12.790 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-04-14T23:58:13.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.008+0000 7fdad9000640 1 Processor -- start 2024-04-14T23:58:13.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.009+0000 7fdad9000640 1 -- start start 2024-04-14T23:58:13.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.010+0000 7fdad9000640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fdad415c9b0 0x7fdad40c7280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:13.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.010+0000 7fdad9000640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdad41530d0 con 0x7fdad4152cf0 2024-04-14T23:58:13.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.010+0000 7fdad9000640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdad40751f0 con 0x7fdad40c63c0 2024-04-14T23:58:13.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.010+0000 7fdad9000640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fdad40c8370 con 0x7fdad415c9b0 2024-04-14T23:58:13.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.011+0000 7fdad1e00640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fdad4152cf0 0x7fdad415c2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53264/0 (socket says 172.21.15.60:53264) 2024-04-14T23:58:13.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.011+0000 7fdad1e00640 1 -- 172.21.15.60:0/3652188825 learned_addr learned my addr 172.21.15.60:0/3652188825 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:58:13.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.011+0000 7fdad2800640 1 --2- 172.21.15.60:0/3652188825 >> v2:172.21.15.186:3300/0 conn(0x7fdad415c9b0 0x7fdad40c7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:13.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.011+0000 7fdad1400640 1 -- 172.21.15.60:0/3652188825 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3525613111 0 0) 0x7fdad41530d0 con 0x7fdad4152cf0 2024-04-14T23:58:13.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.011+0000 7fdad1400640 1 -- 172.21.15.60:0/3652188825 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdab8003520 con 0x7fdad4152cf0 2024-04-14T23:58:13.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.011+0000 7fdad2800640 1 -- 172.21.15.60:0/3652188825 >> v1:172.21.15.186:6789/0 conn(0x7fdad40c63c0 legacy=0x7fdad40c6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:13.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.011+0000 7fdad2800640 1 -- 172.21.15.60:0/3652188825 >> v1:172.21.15.60:6789/0 conn(0x7fdad4152cf0 legacy=0x7fdad415c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:13.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.011+0000 7fdad2800640 1 -- 172.21.15.60:0/3652188825 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdad40c3810 con 0x7fdad415c9b0 2024-04-14T23:58:13.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.013+0000 7fdad2800640 1 --2- 172.21.15.60:0/3652188825 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdad415c9b0 0x7fdad40c7280 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fdacc0097a0 tx=0x7fdacc02b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=90d85b931569119c server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:13.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.013+0000 7fdad1400640 1 -- 172.21.15.60:0/3652188825 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdacc02f810 con 0x7fdad415c9b0 2024-04-14T23:58:13.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.013+0000 7fdad1400640 1 -- 172.21.15.60:0/3652188825 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdacc02f9b0 con 0x7fdad415c9b0 2024-04-14T23:58:13.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.013+0000 7fdad1400640 1 -- 172.21.15.60:0/3652188825 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdacc02fd60 con 0x7fdad415c9b0 2024-04-14T23:58:13.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.014+0000 7fdad9000640 1 -- 172.21.15.60:0/3652188825 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdad415c9b0 msgr2=0x7fdad40c7280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:13.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.014+0000 7fdad9000640 1 --2- 172.21.15.60:0/3652188825 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdad415c9b0 0x7fdad40c7280 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fdacc0097a0 tx=0x7fdacc02b9f0 comp rx=0 tx=0).stop 2024-04-14T23:58:13.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.014+0000 7fdad9000640 1 -- 172.21.15.60:0/3652188825 shutdown_connections 2024-04-14T23:58:13.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.014+0000 7fdad9000640 1 --2- 172.21.15.60:0/3652188825 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdad415c9b0 0x7fdad40c7280 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:13.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.014+0000 7fdad9000640 1 -- 172.21.15.60:0/3652188825 >> 172.21.15.60:0/3652188825 conn(0x7fdad40cc170 msgr2=0x7fdad40cc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:13.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.014+0000 7fdad9000640 1 -- 172.21.15.60:0/3652188825 shutdown_connections 2024-04-14T23:58:13.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.014+0000 7fdad9000640 1 -- 172.21.15.60:0/3652188825 wait complete. 2024-04-14T23:58:13.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.016+0000 7fdad9000640 1 Processor -- start 2024-04-14T23:58:13.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.016+0000 7fdad9000640 1 -- start start 2024-04-14T23:58:13.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.017+0000 7fdad9000640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdad40c63c0 0x7fdad4147160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:13.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.017+0000 7fdad9000640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdad4152cf0 0x7fdad4141060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:13.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.017+0000 7fdad9000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdad415c9b0 0x7fdad41415a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:13.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.017+0000 7fdad9000640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fdad40c88f0 con 0x7fdad40c63c0 2024-04-14T23:58:13.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.017+0000 7fdad9000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdad40c8610 con 0x7fdad415c9b0 2024-04-14T23:58:13.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.017+0000 7fdad9000640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fdad40c77c0 con 0x7fdad4152cf0 2024-04-14T23:58:13.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.017+0000 7fdad1e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdad40c63c0 0x7fdad4147160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:13.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.017+0000 7fdad1e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdad40c63c0 0x7fdad4147160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59476/0 (socket says 172.21.15.60:59476) 2024-04-14T23:58:13.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.017+0000 7fdad1e00640 1 -- 172.21.15.60:0/1640593671 learned_addr learned my addr 172.21.15.60:0/1640593671 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:58:13.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.017+0000 7fdad2800640 1 --2- 172.21.15.60:0/1640593671 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdad415c9b0 0x7fdad41415a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:13.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.017+0000 7fdac9400640 1 --2- 172.21.15.60:0/1640593671 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdad4152cf0 0x7fdad4141060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:13.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.018+0000 7fdad1e00640 1 -- 172.21.15.60:0/1640593671 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdad4152cf0 msgr2=0x7fdad4141060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:13.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.018+0000 7fdad1e00640 1 --2- 172.21.15.60:0/1640593671 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdad4152cf0 0x7fdad4141060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:13.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.018+0000 7fdad1e00640 1 -- 172.21.15.60:0/1640593671 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdad415c9b0 msgr2=0x7fdad41415a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:13.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.018+0000 7fdad1e00640 1 --2- 172.21.15.60:0/1640593671 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdad415c9b0 0x7fdad41415a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:13.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.018+0000 7fdad1e00640 1 -- 172.21.15.60:0/1640593671 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdad41f4b70 con 0x7fdad40c63c0 2024-04-14T23:58:13.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.018+0000 7fdad1e00640 1 --2- 172.21.15.60:0/1640593671 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdad40c63c0 0x7fdad4147160 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fdac400cbc0 tx=0x7fdac400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:13.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.018+0000 7fdacaa00640 1 -- 172.21.15.60:0/1640593671 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdac4017070 con 0x7fdad40c63c0 2024-04-14T23:58:13.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.018+0000 7fdacaa00640 1 -- 172.21.15.60:0/1640593671 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdac400ad40 con 0x7fdad40c63c0 2024-04-14T23:58:13.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.018+0000 7fdacaa00640 1 -- 172.21.15.60:0/1640593671 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdac4012600 con 0x7fdad40c63c0 2024-04-14T23:58:13.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.018+0000 7fdad9000640 1 -- 172.21.15.60:0/1640593671 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdad41f4e50 con 0x7fdad40c63c0 2024-04-14T23:58:13.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.019+0000 7fdad9000640 1 -- 172.21.15.60:0/1640593671 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdad41f5410 con 0x7fdad40c63c0 2024-04-14T23:58:13.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.020+0000 7fdad9000640 1 -- 172.21.15.60:0/1640593671 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdad41f5610 con 0x7fdad40c63c0 2024-04-14T23:58:13.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.020+0000 7fdacaa00640 1 -- 172.21.15.60:0/1640593671 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7fdac401a070 con 0x7fdad40c63c0 2024-04-14T23:58:13.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.021+0000 7fdacaa00640 1 --2- 172.21.15.60:0/1640593671 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fdab006b9c0 0x7fdab006de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:13.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.022+0000 7fdacaa00640 1 -- 172.21.15.60:0/1640593671 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6255+0+0 (secure 0 0 0) 0x7fdac408a930 con 0x7fdad40c63c0 2024-04-14T23:58:13.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.022+0000 7fdac9400640 1 --2- 172.21.15.60:0/1640593671 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fdab006b9c0 0x7fdab006de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:13.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.023+0000 7fdac9400640 1 --2- 172.21.15.60:0/1640593671 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fdab006b9c0 0x7fdab006de80 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fdad40c2710 tx=0x7fdabc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:13.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.026+0000 7fdacaa00640 1 -- 172.21.15.60:0/1640593671 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fdad41f5610 con 0x7fdad40c63c0 2024-04-14T23:58:13.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.228+0000 7fdad9000640 1 -- 172.21.15.60:0/1640593671 --> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7fdad4076970 con 0x7fdab006b9c0 2024-04-14T23:58:13.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.229+0000 7fdacaa00640 1 -- 172.21.15.60:0/1640593671 <== mgr.4100 v2:172.21.15.186:6832/714104835 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fdad4076970 con 0x7fdab006b9c0 2024-04-14T23:58:13.234 INFO:teuthology.orchestra.run.smithi060.stderr:instructed osd(s) 1 to deep-scrub 2024-04-14T23:58:13.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.231+0000 7fdad9000640 1 -- 172.21.15.60:0/1640593671 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fdab006b9c0 msgr2=0x7fdab006de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:13.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.231+0000 7fdad9000640 1 --2- 172.21.15.60:0/1640593671 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fdab006b9c0 0x7fdab006de80 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fdad40c2710 tx=0x7fdabc009040 comp rx=0 tx=0).stop 2024-04-14T23:58:13.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.231+0000 7fdad9000640 1 -- 172.21.15.60:0/1640593671 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdad40c63c0 msgr2=0x7fdad4147160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:13.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.231+0000 7fdad9000640 1 --2- 172.21.15.60:0/1640593671 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdad40c63c0 0x7fdad4147160 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fdac400cbc0 tx=0x7fdac400a430 comp rx=0 tx=0).stop 2024-04-14T23:58:13.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.232+0000 7fdad9000640 1 -- 172.21.15.60:0/1640593671 shutdown_connections 2024-04-14T23:58:13.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.232+0000 7fdad9000640 1 --2- 172.21.15.60:0/1640593671 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fdab006b9c0 0x7fdab006de80 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:13.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.232+0000 7fdad9000640 1 --2- 172.21.15.60:0/1640593671 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdad415c9b0 0x7fdad41415a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:13.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.232+0000 7fdad9000640 1 --2- 172.21.15.60:0/1640593671 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdad4152cf0 0x7fdad4141060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:13.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.232+0000 7fdad9000640 1 --2- 172.21.15.60:0/1640593671 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdad40c63c0 0x7fdad4147160 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:13.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.232+0000 7fdad9000640 1 -- 172.21.15.60:0/1640593671 >> 172.21.15.60:0/1640593671 conn(0x7fdad40cc170 msgr2=0x7fdad40c42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:13.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.232+0000 7fdad9000640 1 -- 172.21.15.60:0/1640593671 shutdown_connections 2024-04-14T23:58:13.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.232+0000 7fdad9000640 1 -- 172.21.15.60:0/1640593671 wait complete. 2024-04-14T23:58:13.257 INFO:tasks.ceph:Scrubbing osd.2 2024-04-14T23:58:13.257 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-14T23:58:13.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.482+0000 7f0d8b200640 1 Processor -- start 2024-04-14T23:58:13.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.482+0000 7f0d8b200640 1 -- start start 2024-04-14T23:58:13.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.484+0000 7f0d8b200640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f0d8c152ad0 0x7f0d8c0c8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:13.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.484+0000 7f0d8b200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0d8c0c4140 0x7f0d8c06e660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:13.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.484+0000 7f0d8b200640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d8c15c7c0 con 0x7f0d8c0cb790 2024-04-14T23:58:13.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.484+0000 7f0d8b200640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f0d8c15cd20 con 0x7f0d8c152ad0 2024-04-14T23:58:13.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.484+0000 7f0d8b200640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f0d8c06f430 con 0x7f0d8c0c4140 2024-04-14T23:58:13.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.485+0000 7f0d8a800640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0d8c0c4140 0x7f0d8c06e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:13.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.485+0000 7f0d89e00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f0d8c152ad0 0x7f0d8c0c8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:13.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.485+0000 7f0d89400640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f0d8c0cb790 0x7f0d8c0c3a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.60:41558/0 (socket says 172.21.15.60:41558) 2024-04-14T23:58:13.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.485+0000 7f0d89e00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f0d8c152ad0 0x7f0d8c0c8ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:57830/0 (socket says 172.21.15.60:57830) 2024-04-14T23:58:13.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.485+0000 7f0d89400640 1 -- 172.21.15.60:0/3973441426 learned_addr learned my addr 172.21.15.60:0/3973441426 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:58:13.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.485+0000 7f0d8a800640 1 -- 172.21.15.60:0/3973441426 >> v2:172.21.15.186:3300/0 conn(0x7f0d8c152ad0 msgr2=0x7f0d8c0c8ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:58:13.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.485+0000 7f0d8a800640 1 --2- 172.21.15.60:0/3973441426 >> v2:172.21.15.186:3300/0 conn(0x7f0d8c152ad0 0x7f0d8c0c8ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:13.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.485+0000 7f0d8a800640 1 -- 172.21.15.60:0/3973441426 >> v1:172.21.15.186:6789/0 conn(0x7f0d8c0cb790 legacy=0x7f0d8c0c3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:13.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.485+0000 7f0d8a800640 1 -- 172.21.15.60:0/3973441426 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d8c15ca40 con 0x7f0d8c0c4140 2024-04-14T23:58:13.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.485+0000 7f0d88a00640 1 -- 172.21.15.60:0/3973441426 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2315689234 0 0) 0x7f0d78001110 con 0x7f0d8c0cb790 2024-04-14T23:58:13.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.485+0000 7f0d89e00640 1 --2- 172.21.15.60:0/3973441426 >> v2:172.21.15.186:3300/0 conn(0x7f0d8c152ad0 0x7f0d8c0c8ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:58:13.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.487+0000 7f0d8a800640 1 --2- 172.21.15.60:0/3973441426 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0d8c0c4140 0x7f0d8c06e660 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f0d84009d90 tx=0x7f0d8402be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2cba7f0c5cecd04b server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:13.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.487+0000 7f0d88a00640 1 -- 172.21.15.60:0/3973441426 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d84039070 con 0x7f0d8c0c4140 2024-04-14T23:58:13.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.487+0000 7f0d88a00640 1 -- 172.21.15.60:0/3973441426 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d8402cbc0 con 0x7f0d8c0c4140 2024-04-14T23:58:13.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.487+0000 7f0d88a00640 1 -- 172.21.15.60:0/3973441426 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d84034440 con 0x7f0d8c0c4140 2024-04-14T23:58:13.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.488+0000 7f0d8b200640 1 -- 172.21.15.60:0/3973441426 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0d8c0c4140 msgr2=0x7f0d8c06e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:13.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.488+0000 7f0d8b200640 1 --2- 172.21.15.60:0/3973441426 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0d8c0c4140 0x7f0d8c06e660 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f0d84009d90 tx=0x7f0d8402be10 comp rx=0 tx=0).stop 2024-04-14T23:58:13.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.488+0000 7f0d8b200640 1 -- 172.21.15.60:0/3973441426 shutdown_connections 2024-04-14T23:58:13.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.488+0000 7f0d8b200640 1 --2- 172.21.15.60:0/3973441426 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0d8c0c4140 0x7f0d8c06e660 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:13.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.488+0000 7f0d8b200640 1 --2- 172.21.15.60:0/3973441426 >> v2:172.21.15.186:3300/0 conn(0x7f0d8c152ad0 0x7f0d8c0c8ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:13.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.488+0000 7f0d8b200640 1 -- 172.21.15.60:0/3973441426 >> 172.21.15.60:0/3973441426 conn(0x7f0d8c0ce0e0 msgr2=0x7f0d8c0ce4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:13.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.488+0000 7f0d8b200640 1 -- 172.21.15.60:0/3973441426 shutdown_connections 2024-04-14T23:58:13.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.488+0000 7f0d8b200640 1 -- 172.21.15.60:0/3973441426 wait complete. 2024-04-14T23:58:13.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.490+0000 7f0d8b200640 1 Processor -- start 2024-04-14T23:58:13.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.491+0000 7f0d8b200640 1 -- start start 2024-04-14T23:58:13.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.492+0000 7f0d8b200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0d8c0c4140 0x7f0d8c15b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:13.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.492+0000 7f0d8b200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0d8c0cb790 0x7f0d8c150350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:13.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.492+0000 7f0d8b200640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d8c152ad0 0x7f0d8c159c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:13.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.492+0000 7f0d8b200640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f0d8c152eb0 con 0x7f0d8c152ad0 2024-04-14T23:58:13.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.492+0000 7f0d8b200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0d8c0c3510 con 0x7f0d8c0cb790 2024-04-14T23:58:13.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.492+0000 7f0d8b200640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f0d8c0b3cb0 con 0x7f0d8c0c4140 2024-04-14T23:58:13.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.492+0000 7f0d89e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0d8c0c4140 0x7f0d8c15b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:13.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.492+0000 7f0d8a800640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d8c152ad0 0x7f0d8c159c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:13.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.492+0000 7f0d89400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0d8c0cb790 0x7f0d8c150350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:13.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.492+0000 7f0d89400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0d8c0cb790 0x7f0d8c150350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:57840/0 (socket says 172.21.15.60:57840) 2024-04-14T23:58:13.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.492+0000 7f0d89400640 1 -- 172.21.15.60:0/2482240636 learned_addr learned my addr 172.21.15.60:0/2482240636 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:58:13.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.492+0000 7f0d89e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0d8c0c4140 0x7f0d8c15b750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.60:53844/0 (socket says 172.21.15.60:53844) 2024-04-14T23:58:13.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.493+0000 7f0d89e00640 1 -- 172.21.15.60:0/2482240636 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0d8c0cb790 msgr2=0x7f0d8c150350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:13.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.493+0000 7f0d89e00640 1 --2- 172.21.15.60:0/2482240636 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0d8c0cb790 0x7f0d8c150350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:13.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.493+0000 7f0d89e00640 1 -- 172.21.15.60:0/2482240636 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d8c152ad0 msgr2=0x7f0d8c159c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:13.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.493+0000 7f0d89e00640 1 --2- 172.21.15.60:0/2482240636 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d8c152ad0 0x7f0d8c159c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:13.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.493+0000 7f0d89e00640 1 -- 172.21.15.60:0/2482240636 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d8c1599a0 con 0x7f0d8c0c4140 2024-04-14T23:58:13.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.493+0000 7f0d89e00640 1 --2- 172.21.15.60:0/2482240636 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0d8c0c4140 0x7f0d8c15b750 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c00e8a0 tx=0x7f0d7c00ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:13.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.493+0000 7f0d82000640 1 -- 172.21.15.60:0/2482240636 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d7c017070 con 0x7f0d8c0c4140 2024-04-14T23:58:13.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.494+0000 7f0d82000640 1 -- 172.21.15.60:0/2482240636 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d7c00c940 con 0x7f0d8c0c4140 2024-04-14T23:58:13.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.494+0000 7f0d82000640 1 -- 172.21.15.60:0/2482240636 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d7c00cd30 con 0x7f0d8c0c4140 2024-04-14T23:58:13.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.494+0000 7f0d8b200640 1 -- 172.21.15.60:0/2482240636 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d8c1f3d40 con 0x7f0d8c0c4140 2024-04-14T23:58:13.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.494+0000 7f0d8b200640 1 -- 172.21.15.60:0/2482240636 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d8c1f4300 con 0x7f0d8c0c4140 2024-04-14T23:58:13.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.495+0000 7f0d8b200640 1 -- 172.21.15.60:0/2482240636 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0d8c04f540 con 0x7f0d8c0c4140 2024-04-14T23:58:13.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.496+0000 7f0d82000640 1 -- 172.21.15.60:0/2482240636 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f0d7c01a070 con 0x7f0d8c0c4140 2024-04-14T23:58:13.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.497+0000 7f0d82000640 1 --2- 172.21.15.60:0/2482240636 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f0d6c06b9c0 0x7f0d6c06de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:13.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.497+0000 7f0d82000640 1 -- 172.21.15.60:0/2482240636 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(45..45 src has 1..45) ==== 6255+0+0 (secure 0 0 0) 0x7f0d7c08af80 con 0x7f0d8c0c4140 2024-04-14T23:58:13.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.497+0000 7f0d82000640 1 --2- 172.21.15.60:0/2482240636 >> [v2:172.21.15.60:6816/3475168698,v1:172.21.15.60:6819/3475168698] conn(0x7f0d6c074060 0x7f0d6c0764e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:13.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.497+0000 7f0d82000640 1 -- 172.21.15.60:0/2482240636 --> [v2:172.21.15.60:6816/3475168698,v1:172.21.15.60:6819/3475168698] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0d6c076bb0 con 0x7f0d6c074060 2024-04-14T23:58:13.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.497+0000 7f0d82000640 1 -- 172.21.15.60:0/2482240636 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f0d7c08b330 con 0x7f0d8c0c4140 2024-04-14T23:58:13.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.497+0000 7f0d89400640 1 --2- 172.21.15.60:0/2482240636 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f0d6c06b9c0 0x7f0d6c06de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:13.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.497+0000 7f0d8a800640 1 --2- 172.21.15.60:0/2482240636 >> [v2:172.21.15.60:6816/3475168698,v1:172.21.15.60:6819/3475168698] conn(0x7f0d6c074060 0x7f0d6c0764e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:13.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.498+0000 7f0d89400640 1 --2- 172.21.15.60:0/2482240636 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f0d6c06b9c0 0x7f0d6c06de80 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c1f1860 tx=0x7f0d78007ea0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:13.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.498+0000 7f0d8a800640 1 --2- 172.21.15.60:0/2482240636 >> [v2:172.21.15.60:6816/3475168698,v1:172.21.15.60:6819/3475168698] conn(0x7f0d6c074060 0x7f0d6c0764e0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:13.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.499+0000 7f0d82000640 1 -- 172.21.15.60:0/2482240636 <== osd.2 v2:172.21.15.60:6816/3475168698 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0d6c076bb0 con 0x7f0d6c074060 2024-04-14T23:58:13.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.514+0000 7f0d8b200640 1 -- 172.21.15.60:0/2482240636 --> [v2:172.21.15.60:6816/3475168698,v1:172.21.15.60:6819/3475168698] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f0d8c0c77a0 con 0x7f0d6c074060 2024-04-14T23:58:13.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.514+0000 7f0d82000640 1 -- 172.21.15.60:0/2482240636 <== osd.2 v2:172.21.15.60:6816/3475168698 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f0d8c0c77a0 con 0x7f0d6c074060 2024-04-14T23:58:13.520 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-14T23:58:13.520 INFO:teuthology.orchestra.run.smithi060.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-14T23:58:13.520 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-14T23:58:13.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.515+0000 7f0d8b200640 1 -- 172.21.15.60:0/2482240636 >> [v2:172.21.15.60:6816/3475168698,v1:172.21.15.60:6819/3475168698] conn(0x7f0d6c074060 msgr2=0x7f0d6c0764e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:13.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.515+0000 7f0d8b200640 1 --2- 172.21.15.60:0/2482240636 >> [v2:172.21.15.60:6816/3475168698,v1:172.21.15.60:6819/3475168698] conn(0x7f0d6c074060 0x7f0d6c0764e0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:13.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.515+0000 7f0d8b200640 1 -- 172.21.15.60:0/2482240636 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f0d6c06b9c0 msgr2=0x7f0d6c06de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:13.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.515+0000 7f0d8b200640 1 --2- 172.21.15.60:0/2482240636 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f0d6c06b9c0 0x7f0d6c06de80 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c1f1860 tx=0x7f0d78007ea0 comp rx=0 tx=0).stop 2024-04-14T23:58:13.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.515+0000 7f0d8b200640 1 -- 172.21.15.60:0/2482240636 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0d8c0c4140 msgr2=0x7f0d8c15b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:13.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.515+0000 7f0d8b200640 1 --2- 172.21.15.60:0/2482240636 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0d8c0c4140 0x7f0d8c15b750 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c00e8a0 tx=0x7f0d7c00ed60 comp rx=0 tx=0).stop 2024-04-14T23:58:13.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.515+0000 7f0d8a800640 1 -- 172.21.15.60:0/2482240636 reap_dead start 2024-04-14T23:58:13.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.515+0000 7f0d8b200640 1 -- 172.21.15.60:0/2482240636 shutdown_connections 2024-04-14T23:58:13.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.515+0000 7f0d8b200640 1 -- 172.21.15.60:0/2482240636 >> 172.21.15.60:0/2482240636 conn(0x7f0d8c0ce0e0 msgr2=0x7f0d8c14cda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:13.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.515+0000 7f0d8b200640 1 -- 172.21.15.60:0/2482240636 shutdown_connections 2024-04-14T23:58:13.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.515+0000 7f0d8b200640 1 -- 172.21.15.60:0/2482240636 wait complete. 2024-04-14T23:58:13.538 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-04-14T23:58:13.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.748+0000 7f2c4e800640 1 Processor -- start 2024-04-14T23:58:13.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.748+0000 7f2c4e800640 1 -- start start 2024-04-14T23:58:13.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.749+0000 7f2c4e800640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2c48158610 0x7f2c480b1740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:13.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.749+0000 7f2c4e800640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2c480c6800 0x7f2c480c6c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:13.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.749+0000 7f2c4e800640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c480b1fd0 con 0x7f2c480c3400 2024-04-14T23:58:13.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.749+0000 7f2c4e800640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f2c480c71d0 con 0x7f2c480c6800 2024-04-14T23:58:13.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.749+0000 7f2c4e800640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f2c4815b070 con 0x7f2c48158610 2024-04-14T23:58:13.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.750+0000 7f2c4de00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2c480c6800 0x7f2c480c6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:13.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.750+0000 7f2c4de00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2c480c6800 0x7f2c480c6c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59496/0 (socket says 172.21.15.60:59496) 2024-04-14T23:58:13.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.750+0000 7f2c4de00640 1 -- 172.21.15.60:0/3355144543 learned_addr learned my addr 172.21.15.60:0/3355144543 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:58:13.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.750+0000 7f2c4d400640 1 --2- 172.21.15.60:0/3355144543 >> v2:172.21.15.186:3300/0 conn(0x7f2c48158610 0x7f2c480b1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:13.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.750+0000 7f2c4de00640 1 -- 172.21.15.60:0/3355144543 >> v2:172.21.15.186:3300/0 conn(0x7f2c48158610 msgr2=0x7f2c480b1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:58:13.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.750+0000 7f2c4de00640 1 --2- 172.21.15.60:0/3355144543 >> v2:172.21.15.186:3300/0 conn(0x7f2c48158610 0x7f2c480b1740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:13.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.750+0000 7f2c4de00640 1 -- 172.21.15.60:0/3355144543 >> v1:172.21.15.60:6789/0 conn(0x7f2c480c3400 legacy=0x7f2c4806eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:13.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.750+0000 7f2c4de00640 1 -- 172.21.15.60:0/3355144543 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c4815b350 con 0x7f2c480c6800 2024-04-14T23:58:13.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.751+0000 7f2c4de00640 1 --2- 172.21.15.60:0/3355144543 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c480c6800 0x7f2c480c6c90 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f2c480ccfc0 tx=0x7f2c4002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5521fa09fec6c2d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:13.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.752+0000 7f2c47e00640 1 -- 172.21.15.60:0/3355144543 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c4002c960 con 0x7f2c480c6800 2024-04-14T23:58:13.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.752+0000 7f2c47e00640 1 -- 172.21.15.60:0/3355144543 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c4002cb00 con 0x7f2c480c6800 2024-04-14T23:58:13.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.752+0000 7f2c47e00640 1 -- 172.21.15.60:0/3355144543 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c400343f0 con 0x7f2c480c6800 2024-04-14T23:58:13.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.752+0000 7f2c4e800640 1 -- 172.21.15.60:0/3355144543 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c480c6800 msgr2=0x7f2c480c6c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:13.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.752+0000 7f2c4e800640 1 --2- 172.21.15.60:0/3355144543 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c480c6800 0x7f2c480c6c90 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f2c480ccfc0 tx=0x7f2c4002be40 comp rx=0 tx=0).stop 2024-04-14T23:58:13.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.752+0000 7f2c4e800640 1 -- 172.21.15.60:0/3355144543 shutdown_connections 2024-04-14T23:58:13.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.752+0000 7f2c4e800640 1 --2- 172.21.15.60:0/3355144543 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c480c6800 0x7f2c480c6c90 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:13.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.752+0000 7f2c4e800640 1 --2- 172.21.15.60:0/3355144543 >> v2:172.21.15.186:3300/0 conn(0x7f2c48158610 0x7f2c480b1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:13.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.752+0000 7f2c4e800640 1 -- 172.21.15.60:0/3355144543 >> 172.21.15.60:0/3355144543 conn(0x7f2c480cc2f0 msgr2=0x7f2c480cc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:13.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.753+0000 7f2c4e800640 1 -- 172.21.15.60:0/3355144543 shutdown_connections 2024-04-14T23:58:13.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.753+0000 7f2c4e800640 1 -- 172.21.15.60:0/3355144543 wait complete. 2024-04-14T23:58:13.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.754+0000 7f2c4e800640 1 Processor -- start 2024-04-14T23:58:13.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.755+0000 7f2c4e800640 1 -- start start 2024-04-14T23:58:13.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.755+0000 7f2c4e800640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2c480c3400 0x7f2c4815a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:13.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.755+0000 7f2c4e800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c480c6800 0x7f2c480ae8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:13.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.755+0000 7f2c4e800640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c48158610 0x7f2c48153120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:13.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.755+0000 7f2c4e800640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f2c4815bbf0 con 0x7f2c48158610 2024-04-14T23:58:13.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.755+0000 7f2c4e800640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2c4815b910 con 0x7f2c480c6800 2024-04-14T23:58:13.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.755+0000 7f2c4e800640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f2c480b1c80 con 0x7f2c480c3400 2024-04-14T23:58:13.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.756+0000 7f2c4de00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c48158610 0x7f2c48153120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:13.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.756+0000 7f2c4de00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c48158610 0x7f2c48153120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59504/0 (socket says 172.21.15.60:59504) 2024-04-14T23:58:13.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.756+0000 7f2c4de00640 1 -- 172.21.15.60:0/290308146 learned_addr learned my addr 172.21.15.60:0/290308146 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:58:13.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.756+0000 7f2c4d400640 1 --2- 172.21.15.60:0/290308146 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2c480c3400 0x7f2c4815a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:13.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.756+0000 7f2c4ca00640 1 --2- 172.21.15.60:0/290308146 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c480c6800 0x7f2c480ae8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:13.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.756+0000 7f2c4de00640 1 -- 172.21.15.60:0/290308146 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2c480c3400 msgr2=0x7f2c4815a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:13.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.756+0000 7f2c4de00640 1 --2- 172.21.15.60:0/290308146 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2c480c3400 0x7f2c4815a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:13.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.756+0000 7f2c4de00640 1 -- 172.21.15.60:0/290308146 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c480c6800 msgr2=0x7f2c480ae8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:13.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.756+0000 7f2c4de00640 1 --2- 172.21.15.60:0/290308146 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c480c6800 0x7f2c480ae8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:13.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.756+0000 7f2c4de00640 1 -- 172.21.15.60:0/290308146 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c480bffa0 con 0x7f2c48158610 2024-04-14T23:58:13.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.757+0000 7f2c4de00640 1 --2- 172.21.15.60:0/290308146 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c48158610 0x7f2c48153120 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f2c40009f40 tx=0x7f2c40004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:13.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.757+0000 7f2c45600640 1 -- 172.21.15.60:0/290308146 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c40043020 con 0x7f2c48158610 2024-04-14T23:58:13.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.757+0000 7f2c45600640 1 -- 172.21.15.60:0/290308146 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c4003e450 con 0x7f2c48158610 2024-04-14T23:58:13.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.757+0000 7f2c45600640 1 -- 172.21.15.60:0/290308146 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c40009510 con 0x7f2c48158610 2024-04-14T23:58:13.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.757+0000 7f2c4e800640 1 -- 172.21.15.60:0/290308146 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c480b0f20 con 0x7f2c48158610 2024-04-14T23:58:13.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.757+0000 7f2c4e800640 1 -- 172.21.15.60:0/290308146 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c480c0240 con 0x7f2c48158610 2024-04-14T23:58:13.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.758+0000 7f2c4e800640 1 -- 172.21.15.60:0/290308146 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c480c6410 con 0x7f2c48158610 2024-04-14T23:58:13.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.759+0000 7f2c45600640 1 -- 172.21.15.60:0/290308146 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f2c40046020 con 0x7f2c48158610 2024-04-14T23:58:13.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.760+0000 7f2c45600640 1 --2- 172.21.15.60:0/290308146 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f2c2806b990 0x7f2c2806de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:13.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.760+0000 7f2c45600640 1 -- 172.21.15.60:0/290308146 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6255+0+0 (secure 0 0 0) 0x7f2c400ad130 con 0x7f2c48158610 2024-04-14T23:58:13.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.761+0000 7f2c4d400640 1 --2- 172.21.15.60:0/290308146 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f2c2806b990 0x7f2c2806de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:13.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.761+0000 7f2c4d400640 1 --2- 172.21.15.60:0/290308146 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f2c2806b990 0x7f2c2806de50 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c009b40 tx=0x7f2c3c009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:13.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.765+0000 7f2c45600640 1 -- 172.21.15.60:0/290308146 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2c480c6410 con 0x7f2c48158610 2024-04-14T23:58:13.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.976+0000 7f2c4e800640 1 -- 172.21.15.60:0/290308146 --> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f2c4804f540 con 0x7f2c2806b990 2024-04-14T23:58:13.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.978+0000 7f2c45600640 1 -- 172.21.15.60:0/290308146 <== mgr.4100 v2:172.21.15.186:6832/714104835 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f2c4804f540 con 0x7f2c2806b990 2024-04-14T23:58:13.983 INFO:teuthology.orchestra.run.smithi060.stderr:instructed osd(s) 2 to deep-scrub 2024-04-14T23:58:13.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.980+0000 7f2c4e800640 1 -- 172.21.15.60:0/290308146 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f2c2806b990 msgr2=0x7f2c2806de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:13.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.980+0000 7f2c4e800640 1 --2- 172.21.15.60:0/290308146 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f2c2806b990 0x7f2c2806de50 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c009b40 tx=0x7f2c3c009190 comp rx=0 tx=0).stop 2024-04-14T23:58:13.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.980+0000 7f2c4e800640 1 -- 172.21.15.60:0/290308146 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c48158610 msgr2=0x7f2c48153120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:13.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.980+0000 7f2c4e800640 1 --2- 172.21.15.60:0/290308146 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c48158610 0x7f2c48153120 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f2c40009f40 tx=0x7f2c40004a40 comp rx=0 tx=0).stop 2024-04-14T23:58:13.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.981+0000 7f2c4e800640 1 -- 172.21.15.60:0/290308146 shutdown_connections 2024-04-14T23:58:13.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.981+0000 7f2c4e800640 1 --2- 172.21.15.60:0/290308146 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f2c2806b990 0x7f2c2806de50 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:13.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.981+0000 7f2c4e800640 1 --2- 172.21.15.60:0/290308146 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c48158610 0x7f2c48153120 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:13.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.981+0000 7f2c4e800640 1 --2- 172.21.15.60:0/290308146 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c480c6800 0x7f2c480ae8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:13.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.981+0000 7f2c4e800640 1 --2- 172.21.15.60:0/290308146 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2c480c3400 0x7f2c4815a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:13.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.981+0000 7f2c4e800640 1 -- 172.21.15.60:0/290308146 >> 172.21.15.60:0/290308146 conn(0x7f2c480cc2f0 msgr2=0x7f2c48071f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:13.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.981+0000 7f2c4e800640 1 -- 172.21.15.60:0/290308146 shutdown_connections 2024-04-14T23:58:13.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:13.981+0000 7f2c4e800640 1 -- 172.21.15.60:0/290308146 wait complete. 2024-04-14T23:58:14.007 INFO:tasks.ceph:Scrubbing osd.3 2024-04-14T23:58:14.007 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-04-14T23:58:14.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.230+0000 7f1945e00640 1 Processor -- start 2024-04-14T23:58:14.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.231+0000 7f1945e00640 1 -- start start 2024-04-14T23:58:14.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.232+0000 7f1945e00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f1940154ca0 0x7f19400cad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:14.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.233+0000 7f1945e00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f19400cda30 0x7f194006bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:14.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.233+0000 7f1945e00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f194015e460 con 0x7f194006c070 2024-04-14T23:58:14.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.233+0000 7f1945e00640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f194015bb20 con 0x7f19400cda30 2024-04-14T23:58:14.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.233+0000 7f1945e00640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f19400cbb90 con 0x7f1940154ca0 2024-04-14T23:58:14.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.233+0000 7f193fe00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f19400cda30 0x7f194006bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:14.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.233+0000 7f1945400640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f194006c070 0x7f19400cc020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53278/0 (socket says 172.21.15.60:53278) 2024-04-14T23:58:14.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.233+0000 7f1945400640 1 -- 172.21.15.60:0/325618483 learned_addr learned my addr 172.21.15.60:0/325618483 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:58:14.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.233+0000 7f1944a00640 1 --2- 172.21.15.60:0/325618483 >> v2:172.21.15.186:3300/0 conn(0x7f1940154ca0 0x7f19400cad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:14.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.234+0000 7f1944a00640 1 -- 172.21.15.60:0/325618483 >> v2:172.21.15.60:3300/0 conn(0x7f19400cda30 msgr2=0x7f194006bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:58:14.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.234+0000 7f1944a00640 1 --2- 172.21.15.60:0/325618483 >> v2:172.21.15.60:3300/0 conn(0x7f19400cda30 0x7f194006bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:14.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.234+0000 7f193f400640 1 -- 172.21.15.60:0/325618483 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2835866788 0 0) 0x7f194015e460 con 0x7f194006c070 2024-04-14T23:58:14.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.234+0000 7f1944a00640 1 -- 172.21.15.60:0/325618483 >> v1:172.21.15.60:6789/0 conn(0x7f194006c070 legacy=0x7f19400cc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:14.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.234+0000 7f1944a00640 1 -- 172.21.15.60:0/325618483 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f194015e700 con 0x7f1940154ca0 2024-04-14T23:58:14.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.235+0000 7f1944a00640 1 --2- 172.21.15.60:0/325618483 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1940154ca0 0x7f19400cad80 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f194006c750 tx=0x7f193402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56e8672b0bfa332c server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:14.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.235+0000 7f193f400640 1 -- 172.21.15.60:0/325618483 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1934035020 con 0x7f1940154ca0 2024-04-14T23:58:14.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.235+0000 7f193f400640 1 -- 172.21.15.60:0/325618483 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1934031830 con 0x7f1940154ca0 2024-04-14T23:58:14.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.235+0000 7f193f400640 1 -- 172.21.15.60:0/325618483 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1934031c00 con 0x7f1940154ca0 2024-04-14T23:58:14.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.236+0000 7f1945e00640 1 -- 172.21.15.60:0/325618483 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1940154ca0 msgr2=0x7f19400cad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:14.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.236+0000 7f1945e00640 1 --2- 172.21.15.60:0/325618483 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1940154ca0 0x7f19400cad80 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f194006c750 tx=0x7f193402b8d0 comp rx=0 tx=0).stop 2024-04-14T23:58:14.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.236+0000 7f1945e00640 1 -- 172.21.15.60:0/325618483 shutdown_connections 2024-04-14T23:58:14.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.236+0000 7f1945e00640 1 --2- 172.21.15.60:0/325618483 >> v2:172.21.15.60:3300/0 conn(0x7f19400cda30 0x7f194006bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:14.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.236+0000 7f1945e00640 1 --2- 172.21.15.60:0/325618483 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1940154ca0 0x7f19400cad80 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:14.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.236+0000 7f1945e00640 1 -- 172.21.15.60:0/325618483 >> 172.21.15.60:0/325618483 conn(0x7f194006d760 msgr2=0x7f194006db70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:14.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.236+0000 7f1945e00640 1 -- 172.21.15.60:0/325618483 shutdown_connections 2024-04-14T23:58:14.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.236+0000 7f1945e00640 1 -- 172.21.15.60:0/325618483 wait complete. 2024-04-14T23:58:14.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.239+0000 7f1945e00640 1 Processor -- start 2024-04-14T23:58:14.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.239+0000 7f1945e00640 1 -- start start 2024-04-14T23:58:14.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.240+0000 7f1945e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f194006c070 0x7f19400c9250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:14.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.240+0000 7f1945e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f19400cda30 0x7f1940157bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:14.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.240+0000 7f1945e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1940154ca0 0x7f19401580f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:14.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.241+0000 7f1945e00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f194015ef60 con 0x7f194006c070 2024-04-14T23:58:14.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.241+0000 7f1945e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f194015ec80 con 0x7f19400cda30 2024-04-14T23:58:14.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.241+0000 7f1945e00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f19400cb2c0 con 0x7f1940154ca0 2024-04-14T23:58:14.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.241+0000 7f1944a00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f194006c070 0x7f19400c9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:14.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.241+0000 7f1944a00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f194006c070 0x7f19400c9250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59516/0 (socket says 172.21.15.60:59516) 2024-04-14T23:58:14.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.241+0000 7f1944a00640 1 -- 172.21.15.60:0/389550190 learned_addr learned my addr 172.21.15.60:0/389550190 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:58:14.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.241+0000 7f193fe00640 1 --2- 172.21.15.60:0/389550190 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f19400cda30 0x7f1940157bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:14.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.241+0000 7f1945400640 1 --2- 172.21.15.60:0/389550190 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1940154ca0 0x7f19401580f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:14.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.241+0000 7f193fe00640 1 -- 172.21.15.60:0/389550190 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1940154ca0 msgr2=0x7f19401580f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:14.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.241+0000 7f193fe00640 1 --2- 172.21.15.60:0/389550190 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1940154ca0 0x7f19401580f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:14.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.241+0000 7f193fe00640 1 -- 172.21.15.60:0/389550190 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f194006c070 msgr2=0x7f19400c9250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:14.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.241+0000 7f193fe00640 1 --2- 172.21.15.60:0/389550190 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f194006c070 0x7f19400c9250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:14.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.241+0000 7f193fe00640 1 -- 172.21.15.60:0/389550190 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19400c9790 con 0x7f19400cda30 2024-04-14T23:58:14.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.242+0000 7f193fe00640 1 --2- 172.21.15.60:0/389550190 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f19400cda30 0x7f1940157bb0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f192c00e850 tx=0x7f192c00ed10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:14.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.242+0000 7f193cc00640 1 -- 172.21.15.60:0/389550190 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f192c017070 con 0x7f19400cda30 2024-04-14T23:58:14.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.242+0000 7f193cc00640 1 -- 172.21.15.60:0/389550190 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f192c00c940 con 0x7f19400cda30 2024-04-14T23:58:14.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.242+0000 7f193cc00640 1 -- 172.21.15.60:0/389550190 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f192c00cd10 con 0x7f19400cda30 2024-04-14T23:58:14.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.242+0000 7f1945e00640 1 -- 172.21.15.60:0/389550190 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19400c9a00 con 0x7f19400cda30 2024-04-14T23:58:14.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.242+0000 7f1945e00640 1 -- 172.21.15.60:0/389550190 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19401f17d0 con 0x7f19400cda30 2024-04-14T23:58:14.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.243+0000 7f1945e00640 1 -- 172.21.15.60:0/389550190 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f19400c65a0 con 0x7f19400cda30 2024-04-14T23:58:14.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.245+0000 7f193cc00640 1 -- 172.21.15.60:0/389550190 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f192c01a070 con 0x7f19400cda30 2024-04-14T23:58:14.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.245+0000 7f193cc00640 1 --2- 172.21.15.60:0/389550190 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f192006b990 0x7f192006de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:14.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.246+0000 7f193cc00640 1 -- 172.21.15.60:0/389550190 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6255+0+0 (secure 0 0 0) 0x7f192c08b190 con 0x7f19400cda30 2024-04-14T23:58:14.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.246+0000 7f193cc00640 1 --2- 172.21.15.60:0/389550190 >> [v2:172.21.15.60:6808/1730813674,v1:172.21.15.60:6809/1730813674] conn(0x7f1920074040 0x7f19200764c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:14.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.246+0000 7f193cc00640 1 -- 172.21.15.60:0/389550190 --> [v2:172.21.15.60:6808/1730813674,v1:172.21.15.60:6809/1730813674] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f192c026e50 con 0x7f1920074040 2024-04-14T23:58:14.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.246+0000 7f193cc00640 1 -- 172.21.15.60:0/389550190 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f192c08b540 con 0x7f19400cda30 2024-04-14T23:58:14.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.246+0000 7f1944a00640 1 --2- 172.21.15.60:0/389550190 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f192006b990 0x7f192006de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:14.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.246+0000 7f1945400640 1 --2- 172.21.15.60:0/389550190 >> [v2:172.21.15.60:6808/1730813674,v1:172.21.15.60:6809/1730813674] conn(0x7f1920074040 0x7f19200764c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:14.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.247+0000 7f1945400640 1 --2- 172.21.15.60:0/389550190 >> [v2:172.21.15.60:6808/1730813674,v1:172.21.15.60:6809/1730813674] conn(0x7f1920074040 0x7f19200764c0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:14.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.247+0000 7f1944a00640 1 --2- 172.21.15.60:0/389550190 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f192006b990 0x7f192006de50 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f193402b910 tx=0x7f1934008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:14.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.248+0000 7f193cc00640 1 -- 172.21.15.60:0/389550190 <== osd.3 v2:172.21.15.60:6808/1730813674 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f192c026e50 con 0x7f1920074040 2024-04-14T23:58:14.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.263+0000 7f1945e00640 1 -- 172.21.15.60:0/389550190 --> [v2:172.21.15.60:6808/1730813674,v1:172.21.15.60:6809/1730813674] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f194004f540 con 0x7f1920074040 2024-04-14T23:58:14.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.263+0000 7f193cc00640 1 -- 172.21.15.60:0/389550190 <== osd.3 v2:172.21.15.60:6808/1730813674 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f194004f540 con 0x7f1920074040 2024-04-14T23:58:14.270 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-14T23:58:14.271 INFO:teuthology.orchestra.run.smithi060.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-14T23:58:14.271 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-14T23:58:14.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.264+0000 7f1945e00640 1 -- 172.21.15.60:0/389550190 >> [v2:172.21.15.60:6808/1730813674,v1:172.21.15.60:6809/1730813674] conn(0x7f1920074040 msgr2=0x7f19200764c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:14.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.264+0000 7f1945e00640 1 --2- 172.21.15.60:0/389550190 >> [v2:172.21.15.60:6808/1730813674,v1:172.21.15.60:6809/1730813674] conn(0x7f1920074040 0x7f19200764c0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:14.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.264+0000 7f1945e00640 1 -- 172.21.15.60:0/389550190 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f192006b990 msgr2=0x7f192006de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:14.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.264+0000 7f1945e00640 1 --2- 172.21.15.60:0/389550190 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f192006b990 0x7f192006de50 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f193402b910 tx=0x7f1934008030 comp rx=0 tx=0).stop 2024-04-14T23:58:14.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.264+0000 7f1945e00640 1 -- 172.21.15.60:0/389550190 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f19400cda30 msgr2=0x7f1940157bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:14.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.264+0000 7f1945e00640 1 --2- 172.21.15.60:0/389550190 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f19400cda30 0x7f1940157bb0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f192c00e850 tx=0x7f192c00ed10 comp rx=0 tx=0).stop 2024-04-14T23:58:14.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.264+0000 7f1945400640 1 -- 172.21.15.60:0/389550190 reap_dead start 2024-04-14T23:58:14.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.264+0000 7f1945e00640 1 -- 172.21.15.60:0/389550190 shutdown_connections 2024-04-14T23:58:14.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.264+0000 7f1945e00640 1 -- 172.21.15.60:0/389550190 >> 172.21.15.60:0/389550190 conn(0x7f194006d760 msgr2=0x7f1940147850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:14.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.264+0000 7f1945e00640 1 -- 172.21.15.60:0/389550190 shutdown_connections 2024-04-14T23:58:14.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.264+0000 7f1945e00640 1 -- 172.21.15.60:0/389550190 wait complete. 2024-04-14T23:58:14.289 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-04-14T23:58:14.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.506+0000 7f3d81c00640 1 Processor -- start 2024-04-14T23:58:14.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.506+0000 7f3d81c00640 1 -- start start 2024-04-14T23:58:14.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.507+0000 7f3d81c00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3d7c158610 0x7f3d7c0b1740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:14.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.507+0000 7f3d81c00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3d7c0c6800 0x7f3d7c0c6c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:14.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.507+0000 7f3d81c00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d7c0b1fd0 con 0x7f3d7c0c3400 2024-04-14T23:58:14.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.507+0000 7f3d81c00640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f3d7c0c71d0 con 0x7f3d7c158610 2024-04-14T23:58:14.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.507+0000 7f3d81c00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f3d7c15b070 con 0x7f3d7c0c6800 2024-04-14T23:58:14.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.508+0000 7f3d7be00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3d7c158610 0x7f3d7c0b1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:14.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.508+0000 7f3d7be00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3d7c158610 0x7f3d7c0b1740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59520/0 (socket says 172.21.15.60:59520) 2024-04-14T23:58:14.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.508+0000 7f3d81200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3d7c0c6800 0x7f3d7c0c6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:14.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.508+0000 7f3d7be00640 1 -- 172.21.15.60:0/331120475 learned_addr learned my addr 172.21.15.60:0/331120475 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:58:14.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.508+0000 7f3d81200640 1 -- 172.21.15.60:0/331120475 >> v2:172.21.15.60:3300/0 conn(0x7f3d7c158610 msgr2=0x7f3d7c0b1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:58:14.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.508+0000 7f3d81200640 1 --2- 172.21.15.60:0/331120475 >> v2:172.21.15.60:3300/0 conn(0x7f3d7c158610 0x7f3d7c0b1740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:14.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.508+0000 7f3d81200640 1 -- 172.21.15.60:0/331120475 >> v1:172.21.15.60:6789/0 conn(0x7f3d7c0c3400 legacy=0x7f3d7c06eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:14.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.508+0000 7f3d81200640 1 -- 172.21.15.60:0/331120475 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d7c15b350 con 0x7f3d7c0c6800 2024-04-14T23:58:14.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.508+0000 7f3d7aa00640 1 -- 172.21.15.60:0/331120475 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1366202238 0 0) 0x7f3d7c0b1fd0 con 0x7f3d7c0c3400 2024-04-14T23:58:14.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.508+0000 7f3d7be00640 1 --2- 172.21.15.60:0/331120475 >> v2:172.21.15.60:3300/0 conn(0x7f3d7c158610 0x7f3d7c0b1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:58:14.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.510+0000 7f3d81200640 1 --2- 172.21.15.60:0/331120475 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3d7c0c6800 0x7f3d7c0c6c90 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f3d74009d90 tx=0x7f3d7402bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9d1eaf7c1edf18c server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:14.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.510+0000 7f3d7aa00640 1 -- 172.21.15.60:0/331120475 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d74039070 con 0x7f3d7c0c6800 2024-04-14T23:58:14.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.510+0000 7f3d7aa00640 1 -- 172.21.15.60:0/331120475 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d7402cce0 con 0x7f3d7c0c6800 2024-04-14T23:58:14.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.510+0000 7f3d7aa00640 1 -- 172.21.15.60:0/331120475 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d74034660 con 0x7f3d7c0c6800 2024-04-14T23:58:14.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.511+0000 7f3d81c00640 1 -- 172.21.15.60:0/331120475 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3d7c0c6800 msgr2=0x7f3d7c0c6c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:14.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.511+0000 7f3d81c00640 1 --2- 172.21.15.60:0/331120475 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3d7c0c6800 0x7f3d7c0c6c90 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f3d74009d90 tx=0x7f3d7402bf30 comp rx=0 tx=0).stop 2024-04-14T23:58:14.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.511+0000 7f3d81c00640 1 -- 172.21.15.60:0/331120475 shutdown_connections 2024-04-14T23:58:14.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.511+0000 7f3d81c00640 1 --2- 172.21.15.60:0/331120475 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3d7c0c6800 0x7f3d7c0c6c90 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:14.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.511+0000 7f3d81c00640 1 --2- 172.21.15.60:0/331120475 >> v2:172.21.15.60:3300/0 conn(0x7f3d7c158610 0x7f3d7c0b1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:14.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.511+0000 7f3d81c00640 1 -- 172.21.15.60:0/331120475 >> 172.21.15.60:0/331120475 conn(0x7f3d7c0cc2f0 msgr2=0x7f3d7c0cc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:14.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.511+0000 7f3d81c00640 1 -- 172.21.15.60:0/331120475 shutdown_connections 2024-04-14T23:58:14.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.511+0000 7f3d81c00640 1 -- 172.21.15.60:0/331120475 wait complete. 2024-04-14T23:58:14.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.513+0000 7f3d81c00640 1 Processor -- start 2024-04-14T23:58:14.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.513+0000 7f3d81c00640 1 -- start start 2024-04-14T23:58:14.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.514+0000 7f3d81c00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3d7c0c3400 0x7f3d7c0c2820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:14.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.514+0000 7f3d81c00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3d7c0c6800 0x7f3d7c0c01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:14.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.514+0000 7f3d81c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d7c158610 0x7f3d7c0c0700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:14.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.514+0000 7f3d81c00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f3d7c15bbf0 con 0x7f3d7c0c3400 2024-04-14T23:58:14.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.514+0000 7f3d81c00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3d7c15b910 con 0x7f3d7c158610 2024-04-14T23:58:14.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.514+0000 7f3d81c00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f3d7c0b1c80 con 0x7f3d7c0c6800 2024-04-14T23:58:14.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.514+0000 7f3d7be00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3d7c0c3400 0x7f3d7c0c2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:14.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.514+0000 7f3d7b400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3d7c0c6800 0x7f3d7c0c01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:14.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.514+0000 7f3d7be00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3d7c0c3400 0x7f3d7c0c2820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59526/0 (socket says 172.21.15.60:59526) 2024-04-14T23:58:14.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.514+0000 7f3d7be00640 1 -- 172.21.15.60:0/1474326558 learned_addr learned my addr 172.21.15.60:0/1474326558 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:58:14.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.514+0000 7f3d81200640 1 --2- 172.21.15.60:0/1474326558 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d7c158610 0x7f3d7c0c0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:14.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.515+0000 7f3d7be00640 1 -- 172.21.15.60:0/1474326558 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3d7c0c6800 msgr2=0x7f3d7c0c01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:14.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.515+0000 7f3d7be00640 1 --2- 172.21.15.60:0/1474326558 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3d7c0c6800 0x7f3d7c0c01c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:14.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.515+0000 7f3d7be00640 1 -- 172.21.15.60:0/1474326558 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d7c158610 msgr2=0x7f3d7c0c0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:14.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.515+0000 7f3d7be00640 1 --2- 172.21.15.60:0/1474326558 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d7c158610 0x7f3d7c0c0700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:14.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.515+0000 7f3d7be00640 1 -- 172.21.15.60:0/1474326558 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d7c1f0920 con 0x7f3d7c0c3400 2024-04-14T23:58:14.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.515+0000 7f3d7be00640 1 --2- 172.21.15.60:0/1474326558 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3d7c0c3400 0x7f3d7c0c2820 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c00ed70 tx=0x7f3d6c00d720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:14.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.515+0000 7f3d73e00640 1 -- 172.21.15.60:0/1474326558 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d6c010850 con 0x7f3d7c0c3400 2024-04-14T23:58:14.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.515+0000 7f3d73e00640 1 -- 172.21.15.60:0/1474326558 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d6c0109f0 con 0x7f3d7c0c3400 2024-04-14T23:58:14.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.515+0000 7f3d73e00640 1 -- 172.21.15.60:0/1474326558 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d6c010da0 con 0x7f3d7c0c3400 2024-04-14T23:58:14.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.515+0000 7f3d81c00640 1 -- 172.21.15.60:0/1474326558 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d7c1f0be0 con 0x7f3d7c0c3400 2024-04-14T23:58:14.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.516+0000 7f3d81c00640 1 -- 172.21.15.60:0/1474326558 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d7c1f11a0 con 0x7f3d7c0c3400 2024-04-14T23:58:14.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.517+0000 7f3d81c00640 1 -- 172.21.15.60:0/1474326558 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d7c06c930 con 0x7f3d7c0c3400 2024-04-14T23:58:14.524 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.518+0000 7f3d73e00640 1 -- 172.21.15.60:0/1474326558 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f3d6c0223f0 con 0x7f3d7c0c3400 2024-04-14T23:58:14.524 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.519+0000 7f3d73e00640 1 --2- 172.21.15.60:0/1474326558 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f3d5806b990 0x7f3d5806de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:14.525 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.519+0000 7f3d73e00640 1 -- 172.21.15.60:0/1474326558 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6255+0+0 (secure 0 0 0) 0x7f3d6c08b590 con 0x7f3d7c0c3400 2024-04-14T23:58:14.525 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.519+0000 7f3d7b400640 1 --2- 172.21.15.60:0/1474326558 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f3d5806b990 0x7f3d5806de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:14.525 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.520+0000 7f3d7b400640 1 --2- 172.21.15.60:0/1474326558 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f3d5806b990 0x7f3d5806de50 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c1e6fa0 tx=0x7f3d64009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:14.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.523+0000 7f3d73e00640 1 -- 172.21.15.60:0/1474326558 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3d7c06c930 con 0x7f3d7c0c3400 2024-04-14T23:58:14.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.728+0000 7f3d81c00640 1 -- 172.21.15.60:0/1474326558 --> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f3d7c04f540 con 0x7f3d5806b990 2024-04-14T23:58:14.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.730+0000 7f3d73e00640 1 -- 172.21.15.60:0/1474326558 <== mgr.4100 v2:172.21.15.186:6832/714104835 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f3d7c04f540 con 0x7f3d5806b990 2024-04-14T23:58:14.735 INFO:teuthology.orchestra.run.smithi060.stderr:instructed osd(s) 3 to deep-scrub 2024-04-14T23:58:14.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.732+0000 7f3d81c00640 1 -- 172.21.15.60:0/1474326558 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f3d5806b990 msgr2=0x7f3d5806de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:14.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.732+0000 7f3d81c00640 1 --2- 172.21.15.60:0/1474326558 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f3d5806b990 0x7f3d5806de50 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c1e6fa0 tx=0x7f3d64009040 comp rx=0 tx=0).stop 2024-04-14T23:58:14.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.732+0000 7f3d81c00640 1 -- 172.21.15.60:0/1474326558 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3d7c0c3400 msgr2=0x7f3d7c0c2820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:14.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.732+0000 7f3d81c00640 1 --2- 172.21.15.60:0/1474326558 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3d7c0c3400 0x7f3d7c0c2820 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c00ed70 tx=0x7f3d6c00d720 comp rx=0 tx=0).stop 2024-04-14T23:58:14.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.733+0000 7f3d81c00640 1 -- 172.21.15.60:0/1474326558 shutdown_connections 2024-04-14T23:58:14.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.733+0000 7f3d81c00640 1 --2- 172.21.15.60:0/1474326558 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f3d5806b990 0x7f3d5806de50 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:14.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.733+0000 7f3d81c00640 1 --2- 172.21.15.60:0/1474326558 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d7c158610 0x7f3d7c0c0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:14.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.733+0000 7f3d81c00640 1 --2- 172.21.15.60:0/1474326558 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3d7c0c6800 0x7f3d7c0c01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:14.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.733+0000 7f3d81c00640 1 --2- 172.21.15.60:0/1474326558 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3d7c0c3400 0x7f3d7c0c2820 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:14.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.733+0000 7f3d81c00640 1 -- 172.21.15.60:0/1474326558 >> 172.21.15.60:0/1474326558 conn(0x7f3d7c0cc2f0 msgr2=0x7f3d7c0c4c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:14.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.733+0000 7f3d81c00640 1 -- 172.21.15.60:0/1474326558 shutdown_connections 2024-04-14T23:58:14.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.733+0000 7f3d81c00640 1 -- 172.21.15.60:0/1474326558 wait complete. 2024-04-14T23:58:14.757 INFO:tasks.ceph:Scrubbing osd.4 2024-04-14T23:58:14.757 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-04-14T23:58:14.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.980+0000 7f6154a00640 1 Processor -- start 2024-04-14T23:58:14.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.981+0000 7f6154a00640 1 -- start start 2024-04-14T23:58:14.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.982+0000 7f6154a00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6150070fc0 0x7f61500c7ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:14.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.982+0000 7f6154a00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f61500c61f0 0x7f61500c80e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:14.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.982+0000 7f6154a00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f615015f010 con 0x7f61501592e0 2024-04-14T23:58:14.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.982+0000 7f6154a00640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f61500c5e30 con 0x7f61500c61f0 2024-04-14T23:58:14.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.982+0000 7f6154a00640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f61500c9430 con 0x7f6150070fc0 2024-04-14T23:58:14.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.983+0000 7f614e800640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f61500c61f0 0x7f61500c80e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:14.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.983+0000 7f614e800640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f61500c61f0 0x7f61500c80e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59528/0 (socket says 172.21.15.60:59528) 2024-04-14T23:58:14.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.983+0000 7f614e800640 1 -- 172.21.15.60:0/3177721777 learned_addr learned my addr 172.21.15.60:0/3177721777 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:58:14.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.983+0000 7f614de00640 1 --1- 172.21.15.60:0/3177721777 >> v1:172.21.15.60:6789/0 conn(0x7f61501592e0 0x7f61501625e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53284/0 (socket says 172.21.15.60:53284) 2024-04-14T23:58:14.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.983+0000 7f614d400640 1 --2- 172.21.15.60:0/3177721777 >> v2:172.21.15.186:3300/0 conn(0x7f6150070fc0 0x7f61500c7ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:14.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.983+0000 7f614e800640 1 -- 172.21.15.60:0/3177721777 >> v2:172.21.15.186:3300/0 conn(0x7f6150070fc0 msgr2=0x7f61500c7ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:58:14.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.983+0000 7f614e800640 1 --2- 172.21.15.60:0/3177721777 >> v2:172.21.15.186:3300/0 conn(0x7f6150070fc0 0x7f61500c7ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:14.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.983+0000 7f614e800640 1 -- 172.21.15.60:0/3177721777 >> v1:172.21.15.60:6789/0 conn(0x7f61501592e0 legacy=0x7f61501625e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:14.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.983+0000 7f614e800640 1 -- 172.21.15.60:0/3177721777 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f615015f290 con 0x7f61500c61f0 2024-04-14T23:58:14.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.984+0000 7f614e800640 1 --2- 172.21.15.60:0/3177721777 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61500c61f0 0x7f61500c80e0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f61500c3580 tx=0x7f614802ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b6211861951421f server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:14.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.985+0000 7f614ca00640 1 -- 172.21.15.60:0/3177721777 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f614802f850 con 0x7f61500c61f0 2024-04-14T23:58:14.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.985+0000 7f614ca00640 1 -- 172.21.15.60:0/3177721777 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f614802f9f0 con 0x7f61500c61f0 2024-04-14T23:58:14.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.985+0000 7f614ca00640 1 -- 172.21.15.60:0/3177721777 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f614802fdc0 con 0x7f61500c61f0 2024-04-14T23:58:14.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.985+0000 7f6154a00640 1 -- 172.21.15.60:0/3177721777 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61500c61f0 msgr2=0x7f61500c80e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:14.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.985+0000 7f6154a00640 1 --2- 172.21.15.60:0/3177721777 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61500c61f0 0x7f61500c80e0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f61500c3580 tx=0x7f614802ba30 comp rx=0 tx=0).stop 2024-04-14T23:58:14.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.986+0000 7f6154a00640 1 -- 172.21.15.60:0/3177721777 shutdown_connections 2024-04-14T23:58:14.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.986+0000 7f6154a00640 1 --2- 172.21.15.60:0/3177721777 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61500c61f0 0x7f61500c80e0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:14.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.986+0000 7f6154a00640 1 --2- 172.21.15.60:0/3177721777 >> v2:172.21.15.186:3300/0 conn(0x7f6150070fc0 0x7f61500c7ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:14.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.986+0000 7f6154a00640 1 -- 172.21.15.60:0/3177721777 >> 172.21.15.60:0/3177721777 conn(0x7f6150067540 msgr2=0x7f615006b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:14.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.986+0000 7f6154a00640 1 -- 172.21.15.60:0/3177721777 shutdown_connections 2024-04-14T23:58:14.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.986+0000 7f6154a00640 1 -- 172.21.15.60:0/3177721777 wait complete. 2024-04-14T23:58:14.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.988+0000 7f6154a00640 1 Processor -- start 2024-04-14T23:58:14.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.988+0000 7f6154a00640 1 -- start start 2024-04-14T23:58:14.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.989+0000 7f6154a00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6150070fc0 0x7f61500cc070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:14.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.989+0000 7f6154a00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f61500c61f0 0x7f6150153bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:14.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.989+0000 7f6154a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61501592e0 0x7f61501f8860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:14.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.989+0000 7f6154a00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f615015fb30 con 0x7f6150070fc0 2024-04-14T23:58:14.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.989+0000 7f6154a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f615015f850 con 0x7f61501592e0 2024-04-14T23:58:14.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.989+0000 7f6154a00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f61500c90f0 con 0x7f61500c61f0 2024-04-14T23:58:14.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.989+0000 7f614de00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6150070fc0 0x7f61500cc070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:14.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.989+0000 7f614de00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6150070fc0 0x7f61500cc070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59536/0 (socket says 172.21.15.60:59536) 2024-04-14T23:58:14.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.989+0000 7f614de00640 1 -- 172.21.15.60:0/1385229782 learned_addr learned my addr 172.21.15.60:0/1385229782 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:58:14.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.989+0000 7f614e800640 1 --2- 172.21.15.60:0/1385229782 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61501592e0 0x7f61501f8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:14.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.989+0000 7f614d400640 1 --2- 172.21.15.60:0/1385229782 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f61500c61f0 0x7f6150153bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:14.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.990+0000 7f614de00640 1 -- 172.21.15.60:0/1385229782 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f61500c61f0 msgr2=0x7f6150153bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:14.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.990+0000 7f614de00640 1 --2- 172.21.15.60:0/1385229782 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f61500c61f0 0x7f6150153bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:14.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.990+0000 7f614de00640 1 -- 172.21.15.60:0/1385229782 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61501592e0 msgr2=0x7f61501f8860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:14.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.990+0000 7f614de00640 1 --2- 172.21.15.60:0/1385229782 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61501592e0 0x7f61501f8860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:14.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.990+0000 7f614de00640 1 -- 172.21.15.60:0/1385229782 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61500cbd90 con 0x7f6150070fc0 2024-04-14T23:58:14.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.990+0000 7f614de00640 1 --2- 172.21.15.60:0/1385229782 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6150070fc0 0x7f61500cc070 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f614000c730 tx=0x7f614000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:14.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.990+0000 7f6146000640 1 -- 172.21.15.60:0/1385229782 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6140017070 con 0x7f6150070fc0 2024-04-14T23:58:14.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.990+0000 7f6146000640 1 -- 172.21.15.60:0/1385229782 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f614000ce40 con 0x7f6150070fc0 2024-04-14T23:58:14.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.990+0000 7f6146000640 1 -- 172.21.15.60:0/1385229782 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f614000a860 con 0x7f6150070fc0 2024-04-14T23:58:14.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.990+0000 7f6154a00640 1 -- 172.21.15.60:0/1385229782 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f615015c3a0 con 0x7f6150070fc0 2024-04-14T23:58:14.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.991+0000 7f6154a00640 1 -- 172.21.15.60:0/1385229782 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f615015c960 con 0x7f6150070fc0 2024-04-14T23:58:14.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.992+0000 7f6154a00640 1 -- 172.21.15.60:0/1385229782 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f61501f8410 con 0x7f6150070fc0 2024-04-14T23:58:14.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.993+0000 7f6146000640 1 -- 172.21.15.60:0/1385229782 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f614001a070 con 0x7f6150070fc0 2024-04-14T23:58:15.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.994+0000 7f6146000640 1 --2- 172.21.15.60:0/1385229782 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f613006b990 0x7f613006de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:15.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.994+0000 7f6146000640 1 -- 172.21.15.60:0/1385229782 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6255+0+0 (secure 0 0 0) 0x7f614008a0e0 con 0x7f6150070fc0 2024-04-14T23:58:15.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.994+0000 7f6146000640 1 --2- 172.21.15.60:0/1385229782 >> [v2:172.21.15.186:6800/1464587038,v1:172.21.15.186:6805/1464587038] conn(0x7f6130074030 0x7f61300764b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:15.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.994+0000 7f6146000640 1 -- 172.21.15.60:0/1385229782 --> [v2:172.21.15.186:6800/1464587038,v1:172.21.15.186:6805/1464587038] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6130076b80 con 0x7f6130074030 2024-04-14T23:58:15.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.994+0000 7f6146000640 1 -- 172.21.15.60:0/1385229782 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f6140091050 con 0x7f6150070fc0 2024-04-14T23:58:15.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.994+0000 7f614d400640 1 --2- 172.21.15.60:0/1385229782 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f613006b990 0x7f613006de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:15.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.995+0000 7f614e800640 1 --2- 172.21.15.60:0/1385229782 >> [v2:172.21.15.186:6800/1464587038,v1:172.21.15.186:6805/1464587038] conn(0x7f6130074030 0x7f61300764b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:15.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.995+0000 7f614d400640 1 --2- 172.21.15.60:0/1385229782 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f613006b990 0x7f613006de50 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f6150154a50 tx=0x7f613c009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:15.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.995+0000 7f614e800640 1 --2- 172.21.15.60:0/1385229782 >> [v2:172.21.15.186:6800/1464587038,v1:172.21.15.186:6805/1464587038] conn(0x7f6130074030 0x7f61300764b0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:15.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:14.996+0000 7f6146000640 1 -- 172.21.15.60:0/1385229782 <== osd.4 v2:172.21.15.186:6800/1464587038 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6130076b80 con 0x7f6130074030 2024-04-14T23:58:15.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.011+0000 7f6154a00640 1 -- 172.21.15.60:0/1385229782 --> [v2:172.21.15.186:6800/1464587038,v1:172.21.15.186:6805/1464587038] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f61500c7980 con 0x7f6130074030 2024-04-14T23:58:15.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.012+0000 7f6146000640 1 -- 172.21.15.60:0/1385229782 <== osd.4 v2:172.21.15.186:6800/1464587038 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f61500c7980 con 0x7f6130074030 2024-04-14T23:58:15.017 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-14T23:58:15.017 INFO:teuthology.orchestra.run.smithi060.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-14T23:58:15.017 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-14T23:58:15.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.012+0000 7f6154a00640 1 -- 172.21.15.60:0/1385229782 >> [v2:172.21.15.186:6800/1464587038,v1:172.21.15.186:6805/1464587038] conn(0x7f6130074030 msgr2=0x7f61300764b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:15.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.012+0000 7f6154a00640 1 --2- 172.21.15.60:0/1385229782 >> [v2:172.21.15.186:6800/1464587038,v1:172.21.15.186:6805/1464587038] conn(0x7f6130074030 0x7f61300764b0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:15.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.012+0000 7f6154a00640 1 -- 172.21.15.60:0/1385229782 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f613006b990 msgr2=0x7f613006de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:15.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.012+0000 7f6154a00640 1 --2- 172.21.15.60:0/1385229782 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f613006b990 0x7f613006de50 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f6150154a50 tx=0x7f613c009190 comp rx=0 tx=0).stop 2024-04-14T23:58:15.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.012+0000 7f6154a00640 1 -- 172.21.15.60:0/1385229782 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6150070fc0 msgr2=0x7f61500cc070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:15.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.012+0000 7f6154a00640 1 --2- 172.21.15.60:0/1385229782 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6150070fc0 0x7f61500cc070 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f614000c730 tx=0x7f614000cbf0 comp rx=0 tx=0).stop 2024-04-14T23:58:15.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.012+0000 7f614e800640 1 -- 172.21.15.60:0/1385229782 reap_dead start 2024-04-14T23:58:15.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.012+0000 7f6154a00640 1 -- 172.21.15.60:0/1385229782 shutdown_connections 2024-04-14T23:58:15.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.012+0000 7f6154a00640 1 -- 172.21.15.60:0/1385229782 >> 172.21.15.60:0/1385229782 conn(0x7f6150067540 msgr2=0x7f6150071510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:15.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.013+0000 7f6154a00640 1 -- 172.21.15.60:0/1385229782 shutdown_connections 2024-04-14T23:58:15.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.013+0000 7f6154a00640 1 -- 172.21.15.60:0/1385229782 wait complete. 2024-04-14T23:58:15.036 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-04-14T23:58:15.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.251+0000 7fec97e00640 1 Processor -- start 2024-04-14T23:58:15.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.252+0000 7fec97e00640 1 -- start start 2024-04-14T23:58:15.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.253+0000 7fec97e00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fec98154d50 0x7fec980cb0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:15.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.253+0000 7fec97e00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec9815baa0 con 0x7fec980d01d0 2024-04-14T23:58:15.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.253+0000 7fec97e00640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec980cc710 con 0x7fec980cbe60 2024-04-14T23:58:15.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.253+0000 7fec97e00640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fec980cc9f0 con 0x7fec98154d50 2024-04-14T23:58:15.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.253+0000 7fec97400640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fec980d01d0 0x7fec98165440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53294/0 (socket says 172.21.15.60:53294) 2024-04-14T23:58:15.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.253+0000 7fec97400640 1 -- 172.21.15.60:0/2092536555 learned_addr learned my addr 172.21.15.60:0/2092536555 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:58:15.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.253+0000 7fec96a00640 1 --2- 172.21.15.60:0/2092536555 >> v2:172.21.15.186:3300/0 conn(0x7fec98154d50 0x7fec980cb0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:15.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.254+0000 7fec96000640 1 -- 172.21.15.60:0/2092536555 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1645179845 0 0) 0x7fec980cc710 con 0x7fec980cbe60 2024-04-14T23:58:15.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.254+0000 7fec96000640 1 -- 172.21.15.60:0/2092536555 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec7c003520 con 0x7fec980cbe60 2024-04-14T23:58:15.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.254+0000 7fec96000640 1 -- 172.21.15.60:0/2092536555 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2323697113 0 0) 0x7fec9815baa0 con 0x7fec980d01d0 2024-04-14T23:58:15.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.254+0000 7fec96000640 1 -- 172.21.15.60:0/2092536555 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec980cc710 con 0x7fec980d01d0 2024-04-14T23:58:15.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.254+0000 7fec96a00640 1 -- 172.21.15.60:0/2092536555 >> v1:172.21.15.186:6789/0 conn(0x7fec980cbe60 legacy=0x7fec9815e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:15.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.254+0000 7fec96a00640 1 -- 172.21.15.60:0/2092536555 >> v1:172.21.15.60:6789/0 conn(0x7fec980d01d0 legacy=0x7fec98165440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:15.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.254+0000 7fec96a00640 1 -- 172.21.15.60:0/2092536555 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec9815ace0 con 0x7fec98154d50 2024-04-14T23:58:15.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.255+0000 7fec96a00640 1 --2- 172.21.15.60:0/2092536555 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec98154d50 0x7fec980cb0e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fec880097a0 tx=0x7fec8802b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9310f7f832204a server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:15.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.256+0000 7fec96000640 1 -- 172.21.15.60:0/2092536555 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec8802f810 con 0x7fec98154d50 2024-04-14T23:58:15.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.256+0000 7fec96000640 1 -- 172.21.15.60:0/2092536555 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec8802f9b0 con 0x7fec98154d50 2024-04-14T23:58:15.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.256+0000 7fec96000640 1 -- 172.21.15.60:0/2092536555 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec8802fd80 con 0x7fec98154d50 2024-04-14T23:58:15.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.257+0000 7fec97e00640 1 -- 172.21.15.60:0/2092536555 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec98154d50 msgr2=0x7fec980cb0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:15.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.257+0000 7fec97e00640 1 --2- 172.21.15.60:0/2092536555 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec98154d50 0x7fec980cb0e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fec880097a0 tx=0x7fec8802b9f0 comp rx=0 tx=0).stop 2024-04-14T23:58:15.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.257+0000 7fec97e00640 1 -- 172.21.15.60:0/2092536555 shutdown_connections 2024-04-14T23:58:15.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.257+0000 7fec97e00640 1 --2- 172.21.15.60:0/2092536555 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec98154d50 0x7fec980cb0e0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:15.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.257+0000 7fec97e00640 1 -- 172.21.15.60:0/2092536555 >> 172.21.15.60:0/2092536555 conn(0x7fec98072b30 msgr2=0x7fec9806d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:15.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.257+0000 7fec97e00640 1 -- 172.21.15.60:0/2092536555 shutdown_connections 2024-04-14T23:58:15.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.257+0000 7fec97e00640 1 -- 172.21.15.60:0/2092536555 wait complete. 2024-04-14T23:58:15.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.259+0000 7fec97e00640 1 Processor -- start 2024-04-14T23:58:15.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.259+0000 7fec97e00640 1 -- start start 2024-04-14T23:58:15.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.260+0000 7fec97e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fec980cbe60 0x7fec980bafb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:15.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.260+0000 7fec97e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec980d01d0 0x7fec98152330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:15.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.261+0000 7fec97e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec98154d50 0x7fec981f0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:15.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.261+0000 7fec97e00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fec9815b580 con 0x7fec98154d50 2024-04-14T23:58:15.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.261+0000 7fec97e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fec9815b2a0 con 0x7fec980d01d0 2024-04-14T23:58:15.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.261+0000 7fec97e00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fec980cb620 con 0x7fec980cbe60 2024-04-14T23:58:15.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.261+0000 7fec97400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec98154d50 0x7fec981f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:15.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.261+0000 7fec96a00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fec980cbe60 0x7fec980bafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:15.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.261+0000 7fec8fe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec980d01d0 0x7fec98152330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:15.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.261+0000 7fec96a00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fec980cbe60 0x7fec980bafb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.60:53904/0 (socket says 172.21.15.60:53904) 2024-04-14T23:58:15.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.261+0000 7fec96a00640 1 -- 172.21.15.60:0/2865707898 learned_addr learned my addr 172.21.15.60:0/2865707898 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:58:15.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.262+0000 7fec96a00640 1 -- 172.21.15.60:0/2865707898 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec980d01d0 msgr2=0x7fec98152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:15.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.262+0000 7fec96a00640 1 --2- 172.21.15.60:0/2865707898 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec980d01d0 0x7fec98152330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:15.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.262+0000 7fec96a00640 1 -- 172.21.15.60:0/2865707898 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec98154d50 msgr2=0x7fec981f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:15.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.262+0000 7fec96a00640 1 --2- 172.21.15.60:0/2865707898 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec98154d50 0x7fec981f0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:15.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.262+0000 7fec96a00640 1 -- 172.21.15.60:0/2865707898 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec980cf310 con 0x7fec980cbe60 2024-04-14T23:58:15.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.262+0000 7fec8fe00640 1 --2- 172.21.15.60:0/2865707898 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec980d01d0 0x7fec98152330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:58:15.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.262+0000 7fec97400640 1 --2- 172.21.15.60:0/2865707898 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec98154d50 0x7fec981f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:58:15.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.262+0000 7fec96a00640 1 --2- 172.21.15.60:0/2865707898 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fec980cbe60 0x7fec980bafb0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fec88009890 tx=0x7fec88004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:15.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.262+0000 7fec8ea00640 1 -- 172.21.15.60:0/2865707898 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec88046070 con 0x7fec980cbe60 2024-04-14T23:58:15.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.262+0000 7fec8ea00640 1 -- 172.21.15.60:0/2865707898 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec880378f0 con 0x7fec980cbe60 2024-04-14T23:58:15.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.262+0000 7fec8ea00640 1 -- 172.21.15.60:0/2865707898 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec88037ce0 con 0x7fec980cbe60 2024-04-14T23:58:15.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.262+0000 7fec97e00640 1 -- 172.21.15.60:0/2865707898 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec9815a490 con 0x7fec980cbe60 2024-04-14T23:58:15.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.263+0000 7fec97e00640 1 -- 172.21.15.60:0/2865707898 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fec9815aa50 con 0x7fec980cbe60 2024-04-14T23:58:15.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.264+0000 7fec97e00640 1 -- 172.21.15.60:0/2865707898 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec9804f540 con 0x7fec980cbe60 2024-04-14T23:58:15.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.265+0000 7fec8ea00640 1 -- 172.21.15.60:0/2865707898 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7fec88049070 con 0x7fec980cbe60 2024-04-14T23:58:15.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.266+0000 7fec8ea00640 1 --2- 172.21.15.60:0/2865707898 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fec7406b990 0x7fec7406de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:15.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.266+0000 7fec8ea00640 1 -- 172.21.15.60:0/2865707898 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(45..45 src has 1..45) ==== 6255+0+0 (secure 0 0 0) 0x7fec880ac380 con 0x7fec980cbe60 2024-04-14T23:58:15.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.266+0000 7fec8fe00640 1 --2- 172.21.15.60:0/2865707898 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fec7406b990 0x7fec7406de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:15.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.267+0000 7fec8fe00640 1 --2- 172.21.15.60:0/2865707898 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fec7406b990 0x7fec7406de50 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fec981536c0 tx=0x7fec80009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:15.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.271+0000 7fec8ea00640 1 -- 172.21.15.60:0/2865707898 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fec88033030 con 0x7fec980cbe60 2024-04-14T23:58:15.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.474+0000 7fec97e00640 1 -- 172.21.15.60:0/2865707898 --> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7fec980c93b0 con 0x7fec7406b990 2024-04-14T23:58:15.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.475+0000 7fec8ea00640 1 -- 172.21.15.60:0/2865707898 <== mgr.4100 v2:172.21.15.186:6832/714104835 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fec980c93b0 con 0x7fec7406b990 2024-04-14T23:58:15.480 INFO:teuthology.orchestra.run.smithi060.stderr:instructed osd(s) 4 to deep-scrub 2024-04-14T23:58:15.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.477+0000 7fec97e00640 1 -- 172.21.15.60:0/2865707898 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fec7406b990 msgr2=0x7fec7406de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:15.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.477+0000 7fec97e00640 1 --2- 172.21.15.60:0/2865707898 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fec7406b990 0x7fec7406de50 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fec981536c0 tx=0x7fec80009040 comp rx=0 tx=0).stop 2024-04-14T23:58:15.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.477+0000 7fec97e00640 1 -- 172.21.15.60:0/2865707898 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fec980cbe60 msgr2=0x7fec980bafb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:15.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.477+0000 7fec97e00640 1 --2- 172.21.15.60:0/2865707898 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fec980cbe60 0x7fec980bafb0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fec88009890 tx=0x7fec88004730 comp rx=0 tx=0).stop 2024-04-14T23:58:15.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.477+0000 7fec97e00640 1 -- 172.21.15.60:0/2865707898 shutdown_connections 2024-04-14T23:58:15.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.477+0000 7fec97e00640 1 --2- 172.21.15.60:0/2865707898 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fec7406b990 0x7fec7406de50 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:15.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.477+0000 7fec97e00640 1 --2- 172.21.15.60:0/2865707898 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec98154d50 0x7fec981f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:15.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.477+0000 7fec97e00640 1 --2- 172.21.15.60:0/2865707898 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec980d01d0 0x7fec98152330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:15.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.477+0000 7fec97e00640 1 --2- 172.21.15.60:0/2865707898 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fec980cbe60 0x7fec980bafb0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:15.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.477+0000 7fec97e00640 1 -- 172.21.15.60:0/2865707898 >> 172.21.15.60:0/2865707898 conn(0x7fec98072b30 msgr2=0x7fec980cd4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:15.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.478+0000 7fec97e00640 1 -- 172.21.15.60:0/2865707898 shutdown_connections 2024-04-14T23:58:15.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.478+0000 7fec97e00640 1 -- 172.21.15.60:0/2865707898 wait complete. 2024-04-14T23:58:15.503 INFO:tasks.ceph:Scrubbing osd.5 2024-04-14T23:58:15.504 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-04-14T23:58:15.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.730+0000 7efc3be00640 1 Processor -- start 2024-04-14T23:58:15.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.730+0000 7efc3be00640 1 -- start start 2024-04-14T23:58:15.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.732+0000 7efc3be00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efc3c0c7570 0x7efc3c154910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:15.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.732+0000 7efc3be00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7efc3c0c7950 0x7efc3c154e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:15.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.732+0000 7efc3be00640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc3c15e070 con 0x7efc3c0728f0 2024-04-14T23:58:15.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.732+0000 7efc3be00640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7efc3c15e350 con 0x7efc3c0c7950 2024-04-14T23:58:15.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.732+0000 7efc3be00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7efc3c15e630 con 0x7efc3c0c7570 2024-04-14T23:58:15.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.732+0000 7efc3a000640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7efc3c0c7950 0x7efc3c154e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:15.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.732+0000 7efc3aa00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efc3c0c7570 0x7efc3c154910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:15.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.732+0000 7efc3b400640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7efc3c0728f0 0x7efc3c0cc5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.60:41588/0 (socket says 172.21.15.60:41588) 2024-04-14T23:58:15.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.732+0000 7efc3b400640 1 -- 172.21.15.60:0/3808903099 learned_addr learned my addr 172.21.15.60:0/3808903099 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:58:15.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.733+0000 7efc39600640 1 -- 172.21.15.60:0/3808903099 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 558697897 0 0) 0x7efc3c15e070 con 0x7efc3c0728f0 2024-04-14T23:58:15.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.733+0000 7efc3aa00640 1 -- 172.21.15.60:0/3808903099 >> v2:172.21.15.186:3300/0 conn(0x7efc3c0c7950 msgr2=0x7efc3c154e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:58:15.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.733+0000 7efc3aa00640 1 --2- 172.21.15.60:0/3808903099 >> v2:172.21.15.186:3300/0 conn(0x7efc3c0c7950 0x7efc3c154e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:15.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.733+0000 7efc3aa00640 1 -- 172.21.15.60:0/3808903099 >> v1:172.21.15.186:6789/0 conn(0x7efc3c0728f0 legacy=0x7efc3c0cc5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:15.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.733+0000 7efc3aa00640 1 -- 172.21.15.60:0/3808903099 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc3c15e910 con 0x7efc3c0c7570 2024-04-14T23:58:15.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.734+0000 7efc3aa00640 1 --2- 172.21.15.60:0/3808903099 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efc3c0c7570 0x7efc3c154910 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7efc2c0097d0 tx=0x7efc2c02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ccb00eff42dafd server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:15.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.734+0000 7efc39600640 1 -- 172.21.15.60:0/3808903099 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc2c035020 con 0x7efc3c0c7570 2024-04-14T23:58:15.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.735+0000 7efc39600640 1 -- 172.21.15.60:0/3808903099 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc2c031810 con 0x7efc3c0c7570 2024-04-14T23:58:15.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.735+0000 7efc39600640 1 -- 172.21.15.60:0/3808903099 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc2c031be0 con 0x7efc3c0c7570 2024-04-14T23:58:15.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.735+0000 7efc3be00640 1 -- 172.21.15.60:0/3808903099 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efc3c0c7570 msgr2=0x7efc3c154910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:15.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.735+0000 7efc3be00640 1 --2- 172.21.15.60:0/3808903099 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efc3c0c7570 0x7efc3c154910 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7efc2c0097d0 tx=0x7efc2c02b8b0 comp rx=0 tx=0).stop 2024-04-14T23:58:15.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.735+0000 7efc3be00640 1 -- 172.21.15.60:0/3808903099 shutdown_connections 2024-04-14T23:58:15.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.735+0000 7efc3be00640 1 --2- 172.21.15.60:0/3808903099 >> v2:172.21.15.186:3300/0 conn(0x7efc3c0c7950 0x7efc3c154e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:15.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.735+0000 7efc3be00640 1 --2- 172.21.15.60:0/3808903099 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efc3c0c7570 0x7efc3c154910 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:15.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.735+0000 7efc3be00640 1 -- 172.21.15.60:0/3808903099 >> 172.21.15.60:0/3808903099 conn(0x7efc3c06b1c0 msgr2=0x7efc3c06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:15.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.735+0000 7efc3be00640 1 -- 172.21.15.60:0/3808903099 shutdown_connections 2024-04-14T23:58:15.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.735+0000 7efc3be00640 1 -- 172.21.15.60:0/3808903099 wait complete. 2024-04-14T23:58:15.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.737+0000 7efc3be00640 1 Processor -- start 2024-04-14T23:58:15.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.737+0000 7efc3be00640 1 -- start start 2024-04-14T23:58:15.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.738+0000 7efc3be00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc3c0728f0 0x7efc3c151140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:15.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.738+0000 7efc3be00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efc3c0c7570 0x7efc3c156710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:15.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.738+0000 7efc3be00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efc3c0c7950 0x7efc3c14dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:15.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.738+0000 7efc3be00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7efc3c0cce80 con 0x7efc3c0728f0 2024-04-14T23:58:15.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.738+0000 7efc3be00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7efc3c0ccd00 con 0x7efc3c0c7950 2024-04-14T23:58:15.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.738+0000 7efc3be00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7efc3c15ae00 con 0x7efc3c0c7570 2024-04-14T23:58:15.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.739+0000 7efc3aa00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc3c0728f0 0x7efc3c151140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:15.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.739+0000 7efc3aa00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc3c0728f0 0x7efc3c151140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59558/0 (socket says 172.21.15.60:59558) 2024-04-14T23:58:15.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.739+0000 7efc3a000640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efc3c0c7570 0x7efc3c156710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:15.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.739+0000 7efc3aa00640 1 -- 172.21.15.60:0/38433523 learned_addr learned my addr 172.21.15.60:0/38433523 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:58:15.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.739+0000 7efc3b400640 1 --2- 172.21.15.60:0/38433523 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efc3c0c7950 0x7efc3c14dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:15.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.739+0000 7efc3aa00640 1 -- 172.21.15.60:0/38433523 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efc3c0c7570 msgr2=0x7efc3c156710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:15.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.739+0000 7efc3aa00640 1 --2- 172.21.15.60:0/38433523 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efc3c0c7570 0x7efc3c156710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:15.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.739+0000 7efc3aa00640 1 -- 172.21.15.60:0/38433523 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efc3c0c7950 msgr2=0x7efc3c14dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:15.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.739+0000 7efc3aa00640 1 --2- 172.21.15.60:0/38433523 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efc3c0c7950 0x7efc3c14dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:15.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.739+0000 7efc3aa00640 1 -- 172.21.15.60:0/38433523 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc3c15daf0 con 0x7efc3c0728f0 2024-04-14T23:58:15.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.739+0000 7efc3aa00640 1 --2- 172.21.15.60:0/38433523 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc3c0728f0 0x7efc3c151140 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7efc2c0098f0 tx=0x7efc2c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:15.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.740+0000 7efc32a00640 1 -- 172.21.15.60:0/38433523 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc2c041070 con 0x7efc3c0728f0 2024-04-14T23:58:15.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.740+0000 7efc32a00640 1 -- 172.21.15.60:0/38433523 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc2c03ad00 con 0x7efc3c0728f0 2024-04-14T23:58:15.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.740+0000 7efc32a00640 1 -- 172.21.15.60:0/38433523 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc2c035040 con 0x7efc3c0728f0 2024-04-14T23:58:15.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.740+0000 7efc3be00640 1 -- 172.21.15.60:0/38433523 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc3c0c4800 con 0x7efc3c0728f0 2024-04-14T23:58:15.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.740+0000 7efc3be00640 1 -- 172.21.15.60:0/38433523 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc3c0c4dc0 con 0x7efc3c0728f0 2024-04-14T23:58:15.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.741+0000 7efc3be00640 1 -- 172.21.15.60:0/38433523 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7efc3c0c5760 con 0x7efc3c0728f0 2024-04-14T23:58:15.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.742+0000 7efc32a00640 1 -- 172.21.15.60:0/38433523 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7efc2c038020 con 0x7efc3c0728f0 2024-04-14T23:58:15.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.743+0000 7efc32a00640 1 --2- 172.21.15.60:0/38433523 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7efc1c06b990 0x7efc1c06de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:15.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.743+0000 7efc32a00640 1 -- 172.21.15.60:0/38433523 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6255+0+0 (secure 0 0 0) 0x7efc2c0ac720 con 0x7efc3c0728f0 2024-04-14T23:58:15.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.743+0000 7efc32a00640 1 --2- 172.21.15.60:0/38433523 >> [v2:172.21.15.186:6801/1697098067,v1:172.21.15.186:6803/1697098067] conn(0x7efc1c074040 0x7efc1c0764c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:15.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.743+0000 7efc32a00640 1 -- 172.21.15.60:0/38433523 --> [v2:172.21.15.186:6801/1697098067,v1:172.21.15.186:6803/1697098067] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efc2c02eea0 con 0x7efc1c074040 2024-04-14T23:58:15.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.743+0000 7efc32a00640 1 -- 172.21.15.60:0/38433523 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7efc2c0b3050 con 0x7efc3c0728f0 2024-04-14T23:58:15.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.743+0000 7efc3a000640 1 --2- 172.21.15.60:0/38433523 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7efc1c06b990 0x7efc1c06de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:15.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.744+0000 7efc3b400640 1 --2- 172.21.15.60:0/38433523 >> [v2:172.21.15.186:6801/1697098067,v1:172.21.15.186:6803/1697098067] conn(0x7efc1c074040 0x7efc1c0764c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:15.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.744+0000 7efc3b400640 1 --2- 172.21.15.60:0/38433523 >> [v2:172.21.15.186:6801/1697098067,v1:172.21.15.186:6803/1697098067] conn(0x7efc1c074040 0x7efc1c0764c0 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:15.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.744+0000 7efc3a000640 1 --2- 172.21.15.60:0/38433523 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7efc1c06b990 0x7efc1c06de50 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7efc3c151d80 tx=0x7efc28009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:15.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.745+0000 7efc32a00640 1 -- 172.21.15.60:0/38433523 <== osd.5 v2:172.21.15.186:6801/1697098067 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7efc2c02eea0 con 0x7efc1c074040 2024-04-14T23:58:15.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.764+0000 7efc3be00640 1 -- 172.21.15.60:0/38433523 --> [v2:172.21.15.186:6801/1697098067,v1:172.21.15.186:6803/1697098067] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7efc3c0ca050 con 0x7efc1c074040 2024-04-14T23:58:15.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.765+0000 7efc32a00640 1 -- 172.21.15.60:0/38433523 <== osd.5 v2:172.21.15.186:6801/1697098067 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7efc3c0ca050 con 0x7efc1c074040 2024-04-14T23:58:15.770 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-14T23:58:15.770 INFO:teuthology.orchestra.run.smithi060.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-14T23:58:15.771 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-14T23:58:15.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.766+0000 7efc3be00640 1 -- 172.21.15.60:0/38433523 >> [v2:172.21.15.186:6801/1697098067,v1:172.21.15.186:6803/1697098067] conn(0x7efc1c074040 msgr2=0x7efc1c0764c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:15.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.766+0000 7efc3be00640 1 --2- 172.21.15.60:0/38433523 >> [v2:172.21.15.186:6801/1697098067,v1:172.21.15.186:6803/1697098067] conn(0x7efc1c074040 0x7efc1c0764c0 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:15.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.766+0000 7efc3be00640 1 -- 172.21.15.60:0/38433523 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7efc1c06b990 msgr2=0x7efc1c06de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:15.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.766+0000 7efc3be00640 1 --2- 172.21.15.60:0/38433523 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7efc1c06b990 0x7efc1c06de50 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7efc3c151d80 tx=0x7efc28009190 comp rx=0 tx=0).stop 2024-04-14T23:58:15.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.766+0000 7efc3be00640 1 -- 172.21.15.60:0/38433523 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc3c0728f0 msgr2=0x7efc3c151140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:15.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.766+0000 7efc3be00640 1 --2- 172.21.15.60:0/38433523 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc3c0728f0 0x7efc3c151140 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7efc2c0098f0 tx=0x7efc2c0046c0 comp rx=0 tx=0).stop 2024-04-14T23:58:15.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.766+0000 7efc3b400640 1 -- 172.21.15.60:0/38433523 reap_dead start 2024-04-14T23:58:15.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.766+0000 7efc3be00640 1 -- 172.21.15.60:0/38433523 shutdown_connections 2024-04-14T23:58:15.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.766+0000 7efc3be00640 1 -- 172.21.15.60:0/38433523 >> 172.21.15.60:0/38433523 conn(0x7efc3c06b1c0 msgr2=0x7efc3c14ed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:15.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.766+0000 7efc3be00640 1 -- 172.21.15.60:0/38433523 shutdown_connections 2024-04-14T23:58:15.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.766+0000 7efc3be00640 1 -- 172.21.15.60:0/38433523 wait complete. 2024-04-14T23:58:15.789 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-04-14T23:58:16.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.999+0000 7f3511000640 1 Processor -- start 2024-04-14T23:58:16.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:15.999+0000 7f3511000640 1 -- start start 2024-04-14T23:58:16.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.000+0000 7f3511000640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f350c152cf0 0x7f350c15c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:16.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.001+0000 7f3511000640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f350c0c63c0 0x7f350c15c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:16.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.001+0000 7f3511000640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f350c1530d0 con 0x7f350c159450 2024-04-14T23:58:16.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.001+0000 7f3511000640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f350c0751f0 con 0x7f350c0c63c0 2024-04-14T23:58:16.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.001+0000 7f3511000640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f350c0c8370 con 0x7f350c152cf0 2024-04-14T23:58:16.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.001+0000 7f3509e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f350c152cf0 0x7f350c15c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:16.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.001+0000 7f3509e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f350c152cf0 0x7f350c15c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.60:53946/0 (socket says 172.21.15.60:53946) 2024-04-14T23:58:16.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.001+0000 7f3509e00640 1 -- 172.21.15.60:0/378464084 learned_addr learned my addr 172.21.15.60:0/378464084 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:58:16.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.001+0000 7f3509400640 1 --2- 172.21.15.60:0/378464084 >> v2:172.21.15.60:3300/0 conn(0x7f350c0c63c0 0x7f350c15c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:16.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.001+0000 7f3509e00640 1 -- 172.21.15.60:0/378464084 >> v2:172.21.15.60:3300/0 conn(0x7f350c0c63c0 msgr2=0x7f350c15c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:58:16.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.001+0000 7f3509e00640 1 --2- 172.21.15.60:0/378464084 >> v2:172.21.15.60:3300/0 conn(0x7f350c0c63c0 0x7f350c15c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:16.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.001+0000 7f3509e00640 1 -- 172.21.15.60:0/378464084 >> v1:172.21.15.186:6789/0 conn(0x7f350c159450 legacy=0x7f350c0c3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:16.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.001+0000 7f3509e00640 1 -- 172.21.15.60:0/378464084 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f350c0c3810 con 0x7f350c152cf0 2024-04-14T23:58:16.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.003+0000 7f3509e00640 1 --2- 172.21.15.60:0/378464084 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f350c152cf0 0x7f350c15c2a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f34fc009870 tx=0x7f34fc02ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83cb1b5887b683d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:16.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.003+0000 7f3508a00640 1 -- 172.21.15.60:0/378464084 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34fc02f840 con 0x7f350c152cf0 2024-04-14T23:58:16.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.003+0000 7f3508a00640 1 -- 172.21.15.60:0/378464084 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34fc02f9e0 con 0x7f350c152cf0 2024-04-14T23:58:16.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.003+0000 7f3508a00640 1 -- 172.21.15.60:0/378464084 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34fc02fdb0 con 0x7f350c152cf0 2024-04-14T23:58:16.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.004+0000 7f3511000640 1 -- 172.21.15.60:0/378464084 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f350c152cf0 msgr2=0x7f350c15c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:16.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.004+0000 7f3511000640 1 --2- 172.21.15.60:0/378464084 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f350c152cf0 0x7f350c15c2a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f34fc009870 tx=0x7f34fc02ba20 comp rx=0 tx=0).stop 2024-04-14T23:58:16.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.004+0000 7f3511000640 1 -- 172.21.15.60:0/378464084 shutdown_connections 2024-04-14T23:58:16.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.004+0000 7f3511000640 1 --2- 172.21.15.60:0/378464084 >> v2:172.21.15.60:3300/0 conn(0x7f350c0c63c0 0x7f350c15c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:16.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.004+0000 7f3511000640 1 --2- 172.21.15.60:0/378464084 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f350c152cf0 0x7f350c15c2a0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:16.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.004+0000 7f3511000640 1 -- 172.21.15.60:0/378464084 >> 172.21.15.60:0/378464084 conn(0x7f350c0cc170 msgr2=0x7f350c0cc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:16.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.004+0000 7f3511000640 1 -- 172.21.15.60:0/378464084 shutdown_connections 2024-04-14T23:58:16.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.004+0000 7f3511000640 1 -- 172.21.15.60:0/378464084 wait complete. 2024-04-14T23:58:16.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.006+0000 7f3511000640 1 Processor -- start 2024-04-14T23:58:16.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.006+0000 7f3511000640 1 -- start start 2024-04-14T23:58:16.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.007+0000 7f3511000640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f350c0c63c0 0x7f350c15acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:16.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.007+0000 7f3511000640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f350c152cf0 0x7f350c157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:16.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.007+0000 7f3511000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f350c159450 0x7f350c1506d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:16.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.007+0000 7f3511000640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f350c0c88f0 con 0x7f350c0c63c0 2024-04-14T23:58:16.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.007+0000 7f3511000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f350c0c8610 con 0x7f350c159450 2024-04-14T23:58:16.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.007+0000 7f3511000640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f350c0c4690 con 0x7f350c152cf0 2024-04-14T23:58:16.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.007+0000 7f3509400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f350c152cf0 0x7f350c157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:16.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.007+0000 7f3509e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f350c0c63c0 0x7f350c15acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:16.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.007+0000 7f3509400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f350c152cf0 0x7f350c157fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.60:53950/0 (socket says 172.21.15.60:53950) 2024-04-14T23:58:16.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.007+0000 7f3509400640 1 -- 172.21.15.60:0/1779342606 learned_addr learned my addr 172.21.15.60:0/1779342606 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:58:16.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.008+0000 7f350a800640 1 --2- 172.21.15.60:0/1779342606 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f350c159450 0x7f350c1506d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:16.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.008+0000 7f3509400640 1 -- 172.21.15.60:0/1779342606 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f350c159450 msgr2=0x7f350c1506d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:16.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.008+0000 7f3509400640 1 --2- 172.21.15.60:0/1779342606 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f350c159450 0x7f350c1506d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:16.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.008+0000 7f3509400640 1 -- 172.21.15.60:0/1779342606 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f350c0c63c0 msgr2=0x7f350c15acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:16.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.008+0000 7f3509400640 1 --2- 172.21.15.60:0/1779342606 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f350c0c63c0 0x7f350c15acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:16.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.008+0000 7f3509400640 1 -- 172.21.15.60:0/1779342606 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f350c15b700 con 0x7f350c152cf0 2024-04-14T23:58:16.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.008+0000 7f3509400640 1 --2- 172.21.15.60:0/1779342606 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f350c152cf0 0x7f350c157fd0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f34f4009db0 tx=0x7f34f400c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:16.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.009+0000 7f3502000640 1 -- 172.21.15.60:0/1779342606 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34f4018020 con 0x7f350c152cf0 2024-04-14T23:58:16.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.009+0000 7f3502000640 1 -- 172.21.15.60:0/1779342606 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34f4014480 con 0x7f350c152cf0 2024-04-14T23:58:16.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.009+0000 7f3502000640 1 -- 172.21.15.60:0/1779342606 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34f4014850 con 0x7f350c152cf0 2024-04-14T23:58:16.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.009+0000 7f3511000640 1 -- 172.21.15.60:0/1779342606 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f350c15b230 con 0x7f350c152cf0 2024-04-14T23:58:16.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.009+0000 7f3511000640 1 -- 172.21.15.60:0/1779342606 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f350c15bc50 con 0x7f350c152cf0 2024-04-14T23:58:16.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.010+0000 7f3511000640 1 -- 172.21.15.60:0/1779342606 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f350c0c60c0 con 0x7f350c152cf0 2024-04-14T23:58:16.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.012+0000 7f3502000640 1 -- 172.21.15.60:0/1779342606 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f34f4005d10 con 0x7f350c152cf0 2024-04-14T23:58:16.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.013+0000 7f3502000640 1 --2- 172.21.15.60:0/1779342606 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f34e806b9c0 0x7f34e806de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:16.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.013+0000 7f3502000640 1 -- 172.21.15.60:0/1779342606 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(45..45 src has 1..45) ==== 6255+0+0 (secure 0 0 0) 0x7f34f408b6d0 con 0x7f350c152cf0 2024-04-14T23:58:16.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.013+0000 7f3509e00640 1 --2- 172.21.15.60:0/1779342606 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f34e806b9c0 0x7f34e806de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:16.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.014+0000 7f3509e00640 1 --2- 172.21.15.60:0/1779342606 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f34e806b9c0 0x7f34e806de80 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f34fc036000 tx=0x7f34fc008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:16.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.017+0000 7f3502000640 1 -- 172.21.15.60:0/1779342606 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f350c0c60c0 con 0x7f350c152cf0 2024-04-14T23:58:16.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.224+0000 7f3511000640 1 -- 172.21.15.60:0/1779342606 --> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f350c0c5be0 con 0x7f34e806b9c0 2024-04-14T23:58:16.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.225+0000 7f3502000640 1 -- 172.21.15.60:0/1779342606 <== mgr.4100 v2:172.21.15.186:6832/714104835 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f350c0c5be0 con 0x7f34e806b9c0 2024-04-14T23:58:16.230 INFO:teuthology.orchestra.run.smithi060.stderr:instructed osd(s) 5 to deep-scrub 2024-04-14T23:58:16.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.228+0000 7f3511000640 1 -- 172.21.15.60:0/1779342606 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f34e806b9c0 msgr2=0x7f34e806de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:16.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.228+0000 7f3511000640 1 --2- 172.21.15.60:0/1779342606 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f34e806b9c0 0x7f34e806de80 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f34fc036000 tx=0x7f34fc008030 comp rx=0 tx=0).stop 2024-04-14T23:58:16.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.228+0000 7f3511000640 1 -- 172.21.15.60:0/1779342606 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f350c152cf0 msgr2=0x7f350c157fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:16.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.228+0000 7f3511000640 1 --2- 172.21.15.60:0/1779342606 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f350c152cf0 0x7f350c157fd0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f34f4009db0 tx=0x7f34f400c490 comp rx=0 tx=0).stop 2024-04-14T23:58:16.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.228+0000 7f3511000640 1 -- 172.21.15.60:0/1779342606 shutdown_connections 2024-04-14T23:58:16.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.228+0000 7f3511000640 1 --2- 172.21.15.60:0/1779342606 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f34e806b9c0 0x7f34e806de80 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:16.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.228+0000 7f3511000640 1 --2- 172.21.15.60:0/1779342606 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f350c159450 0x7f350c1506d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:16.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.228+0000 7f3511000640 1 --2- 172.21.15.60:0/1779342606 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f350c152cf0 0x7f350c157fd0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:16.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.228+0000 7f3511000640 1 --2- 172.21.15.60:0/1779342606 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f350c0c63c0 0x7f350c15acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:16.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.228+0000 7f3511000640 1 -- 172.21.15.60:0/1779342606 >> 172.21.15.60:0/1779342606 conn(0x7f350c0cc170 msgr2=0x7f350c158c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:16.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.228+0000 7f3511000640 1 -- 172.21.15.60:0/1779342606 shutdown_connections 2024-04-14T23:58:16.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.228+0000 7f3511000640 1 -- 172.21.15.60:0/1779342606 wait complete. 2024-04-14T23:58:16.254 INFO:tasks.ceph:Scrubbing osd.6 2024-04-14T23:58:16.254 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-04-14T23:58:16.470 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-14T23:58:16.474 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-14T23:58:16.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.475+0000 7f91ba600640 1 Processor -- start 2024-04-14T23:58:16.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.475+0000 7f91ba600640 1 -- start start 2024-04-14T23:58:16.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.477+0000 7f91ba600640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f91b415b890 0x7f91b415ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:16.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.477+0000 7f91ba600640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f91b406d940 0x7f91b406be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:16.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.477+0000 7f91ba600640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91b415f1c0 con 0x7f91b40c8330 2024-04-14T23:58:16.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.477+0000 7f91ba600640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f91b40c7f80 con 0x7f91b406d940 2024-04-14T23:58:16.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.477+0000 7f91ba600640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f91b40c75a0 con 0x7f91b415b890 2024-04-14T23:58:16.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.477+0000 7f91b9200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f91b415b890 0x7f91b415ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:16.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.477+0000 7f91b3e00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f91b406d940 0x7f91b406be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:16.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.477+0000 7f91b9c00640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f91b40c8330 0x7f91b40c9af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53302/0 (socket says 172.21.15.60:53302) 2024-04-14T23:58:16.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.477+0000 7f91b9c00640 1 -- 172.21.15.60:0/3845930651 learned_addr learned my addr 172.21.15.60:0/3845930651 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:58:16.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.478+0000 7f91b9200640 1 -- 172.21.15.60:0/3845930651 >> v2:172.21.15.186:3300/0 conn(0x7f91b406d940 msgr2=0x7f91b406be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:58:16.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.478+0000 7f91b9200640 1 --2- 172.21.15.60:0/3845930651 >> v2:172.21.15.186:3300/0 conn(0x7f91b406d940 0x7f91b406be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:16.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.478+0000 7f91b9200640 1 -- 172.21.15.60:0/3845930651 >> v1:172.21.15.60:6789/0 conn(0x7f91b40c8330 legacy=0x7f91b40c9af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:16.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.478+0000 7f91b9200640 1 -- 172.21.15.60:0/3845930651 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91b40c7a00 con 0x7f91b415b890 2024-04-14T23:58:16.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.478+0000 7f91b3400640 1 -- 172.21.15.60:0/3845930651 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 274629844 0 0) 0x7f91b415f1c0 con 0x7f91b40c8330 2024-04-14T23:58:16.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.479+0000 7f91b9200640 1 --2- 172.21.15.60:0/3845930651 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f91b415b890 0x7f91b415ec80 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f91a80097a0 tx=0x7f91a802b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b0af87d4e551eb4 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:16.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.479+0000 7f91b3400640 1 -- 172.21.15.60:0/3845930651 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91a8034070 con 0x7f91b415b890 2024-04-14T23:58:16.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.479+0000 7f91b3400640 1 -- 172.21.15.60:0/3845930651 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91a8030610 con 0x7f91b415b890 2024-04-14T23:58:16.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.479+0000 7f91b3400640 1 -- 172.21.15.60:0/3845930651 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91a80309e0 con 0x7f91b415b890 2024-04-14T23:58:16.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.480+0000 7f91ba600640 1 -- 172.21.15.60:0/3845930651 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f91b415b890 msgr2=0x7f91b415ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:16.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.480+0000 7f91ba600640 1 --2- 172.21.15.60:0/3845930651 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f91b415b890 0x7f91b415ec80 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f91a80097a0 tx=0x7f91a802b960 comp rx=0 tx=0).stop 2024-04-14T23:58:16.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.480+0000 7f91ba600640 1 -- 172.21.15.60:0/3845930651 shutdown_connections 2024-04-14T23:58:16.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.480+0000 7f91ba600640 1 --2- 172.21.15.60:0/3845930651 >> v2:172.21.15.186:3300/0 conn(0x7f91b406d940 0x7f91b406be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:16.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.480+0000 7f91ba600640 1 --2- 172.21.15.60:0/3845930651 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f91b415b890 0x7f91b415ec80 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:16.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.480+0000 7f91ba600640 1 -- 172.21.15.60:0/3845930651 >> 172.21.15.60:0/3845930651 conn(0x7f91b406f9e0 msgr2=0x7f91b406fdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:16.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.480+0000 7f91ba600640 1 -- 172.21.15.60:0/3845930651 shutdown_connections 2024-04-14T23:58:16.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.480+0000 7f91ba600640 1 -- 172.21.15.60:0/3845930651 wait complete. 2024-04-14T23:58:16.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.482+0000 7f91ba600640 1 Processor -- start 2024-04-14T23:58:16.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.482+0000 7f91ba600640 1 -- start start 2024-04-14T23:58:16.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.483+0000 7f91ba600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f91b406d940 0x7f91b40cb670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:16.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.483+0000 7f91ba600640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f91b40c8330 0x7f91b41f4040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:16.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.483+0000 7f91ba600640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f91b415b890 0x7f91b41f4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:16.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.483+0000 7f91ba600640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f91b406b360 con 0x7f91b40c8330 2024-04-14T23:58:16.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.483+0000 7f91ba600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f91b40c7ca0 con 0x7f91b406d940 2024-04-14T23:58:16.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.483+0000 7f91ba600640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f91b406b640 con 0x7f91b415b890 2024-04-14T23:58:16.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.484+0000 7f91b3e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f91b40c8330 0x7f91b41f4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:16.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.484+0000 7f91b9200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f91b406d940 0x7f91b40cb670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:16.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.484+0000 7f91b9c00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f91b415b890 0x7f91b41f4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:16.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.484+0000 7f91b3e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f91b40c8330 0x7f91b41f4040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59590/0 (socket says 172.21.15.60:59590) 2024-04-14T23:58:16.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.484+0000 7f91b3e00640 1 -- 172.21.15.60:0/857644053 learned_addr learned my addr 172.21.15.60:0/857644053 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:58:16.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.484+0000 7f91b3e00640 1 -- 172.21.15.60:0/857644053 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f91b415b890 msgr2=0x7f91b41f4580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:16.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.484+0000 7f91b3e00640 1 --2- 172.21.15.60:0/857644053 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f91b415b890 0x7f91b41f4580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:16.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.484+0000 7f91b3e00640 1 -- 172.21.15.60:0/857644053 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f91b406d940 msgr2=0x7f91b40cb670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:16.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.484+0000 7f91b3e00640 1 --2- 172.21.15.60:0/857644053 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f91b406d940 0x7f91b40cb670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:16.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.484+0000 7f91b3e00640 1 -- 172.21.15.60:0/857644053 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91b40c3360 con 0x7f91b40c8330 2024-04-14T23:58:16.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.484+0000 7f91b3e00640 1 --2- 172.21.15.60:0/857644053 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f91b40c8330 0x7f91b41f4040 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f91a000e9b0 tx=0x7f91a000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:16.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.485+0000 7f91b0c00640 1 -- 172.21.15.60:0/857644053 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91a0017070 con 0x7f91b40c8330 2024-04-14T23:58:16.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.485+0000 7f91b0c00640 1 -- 172.21.15.60:0/857644053 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91a000c9a0 con 0x7f91b40c8330 2024-04-14T23:58:16.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.485+0000 7f91b0c00640 1 -- 172.21.15.60:0/857644053 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91a000cd70 con 0x7f91b40c8330 2024-04-14T23:58:16.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.485+0000 7f91ba600640 1 -- 172.21.15.60:0/857644053 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91b4154ed0 con 0x7f91b40c8330 2024-04-14T23:58:16.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.485+0000 7f91ba600640 1 -- 172.21.15.60:0/857644053 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91b4155490 con 0x7f91b40c8330 2024-04-14T23:58:16.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.486+0000 7f91ba600640 1 -- 172.21.15.60:0/857644053 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f91b4154c80 con 0x7f91b40c8330 2024-04-14T23:58:16.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.487+0000 7f91b0c00640 1 -- 172.21.15.60:0/857644053 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f91a001a070 con 0x7f91b40c8330 2024-04-14T23:58:16.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.488+0000 7f91b0c00640 1 --2- 172.21.15.60:0/857644053 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f919406b990 0x7f919406de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:16.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.488+0000 7f91b0c00640 1 -- 172.21.15.60:0/857644053 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6255+0+0 (secure 0 0 0) 0x7f91a008aa50 con 0x7f91b40c8330 2024-04-14T23:58:16.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.489+0000 7f91b0c00640 1 --2- 172.21.15.60:0/857644053 >> [v2:172.21.15.186:6802/1837108559,v1:172.21.15.186:6804/1837108559] conn(0x7f9194074030 0x7f91940764b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:16.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.489+0000 7f91b0c00640 1 -- 172.21.15.60:0/857644053 --> [v2:172.21.15.186:6802/1837108559,v1:172.21.15.186:6804/1837108559] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f91a0015e50 con 0x7f9194074030 2024-04-14T23:58:16.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.489+0000 7f91b0c00640 1 -- 172.21.15.60:0/857644053 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f91a008f2a0 con 0x7f91b40c8330 2024-04-14T23:58:16.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.489+0000 7f91b9200640 1 --2- 172.21.15.60:0/857644053 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f919406b990 0x7f919406de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:16.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.489+0000 7f91b9c00640 1 --2- 172.21.15.60:0/857644053 >> [v2:172.21.15.186:6802/1837108559,v1:172.21.15.186:6804/1837108559] conn(0x7f9194074030 0x7f91940764b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:16.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.489+0000 7f91b9200640 1 --2- 172.21.15.60:0/857644053 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f919406b990 0x7f919406de50 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f91a8036000 tx=0x7f91a8008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:16.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.490+0000 7f91b9c00640 1 --2- 172.21.15.60:0/857644053 >> [v2:172.21.15.186:6802/1837108559,v1:172.21.15.186:6804/1837108559] conn(0x7f9194074030 0x7f91940764b0 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:16.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.490+0000 7f91b0c00640 1 -- 172.21.15.60:0/857644053 <== osd.6 v2:172.21.15.186:6802/1837108559 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f91a0015e50 con 0x7f9194074030 2024-04-14T23:58:16.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.505+0000 7f91ba600640 1 -- 172.21.15.60:0/857644053 --> [v2:172.21.15.186:6802/1837108559,v1:172.21.15.186:6804/1837108559] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f91b404f540 con 0x7f9194074030 2024-04-14T23:58:16.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.506+0000 7f91b0c00640 1 -- 172.21.15.60:0/857644053 <== osd.6 v2:172.21.15.186:6802/1837108559 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f91b404f540 con 0x7f9194074030 2024-04-14T23:58:16.511 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-14T23:58:16.511 INFO:teuthology.orchestra.run.smithi060.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-14T23:58:16.511 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-14T23:58:16.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.506+0000 7f91ba600640 1 -- 172.21.15.60:0/857644053 >> [v2:172.21.15.186:6802/1837108559,v1:172.21.15.186:6804/1837108559] conn(0x7f9194074030 msgr2=0x7f91940764b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:16.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.506+0000 7f91ba600640 1 --2- 172.21.15.60:0/857644053 >> [v2:172.21.15.186:6802/1837108559,v1:172.21.15.186:6804/1837108559] conn(0x7f9194074030 0x7f91940764b0 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:16.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.506+0000 7f91ba600640 1 -- 172.21.15.60:0/857644053 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f919406b990 msgr2=0x7f919406de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:16.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.506+0000 7f91ba600640 1 --2- 172.21.15.60:0/857644053 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f919406b990 0x7f919406de50 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f91a8036000 tx=0x7f91a8008030 comp rx=0 tx=0).stop 2024-04-14T23:58:16.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.506+0000 7f91ba600640 1 -- 172.21.15.60:0/857644053 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f91b40c8330 msgr2=0x7f91b41f4040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:16.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.506+0000 7f91ba600640 1 --2- 172.21.15.60:0/857644053 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f91b40c8330 0x7f91b41f4040 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f91a000e9b0 tx=0x7f91a000ee70 comp rx=0 tx=0).stop 2024-04-14T23:58:16.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.506+0000 7f91b9c00640 1 -- 172.21.15.60:0/857644053 reap_dead start 2024-04-14T23:58:16.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.507+0000 7f91ba600640 1 -- 172.21.15.60:0/857644053 shutdown_connections 2024-04-14T23:58:16.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.507+0000 7f91ba600640 1 -- 172.21.15.60:0/857644053 >> 172.21.15.60:0/857644053 conn(0x7f91b406f9e0 msgr2=0x7f91b40c9af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:16.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.507+0000 7f91ba600640 1 -- 172.21.15.60:0/857644053 shutdown_connections 2024-04-14T23:58:16.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.507+0000 7f91ba600640 1 -- 172.21.15.60:0/857644053 wait complete. 2024-04-14T23:58:16.529 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-04-14T23:58:16.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.795+0000 7f7c6ba00640 1 Processor -- start 2024-04-14T23:58:16.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.796+0000 7f7c6ba00640 1 -- start start 2024-04-14T23:58:16.802 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.797+0000 7f7c6ba00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f7c6406fc10 0x7f7c640b0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:16.802 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.797+0000 7f7c6ba00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c6415eb00 con 0x7f7c64154fd0 2024-04-14T23:58:16.802 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.797+0000 7f7c6ba00640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c6415f060 con 0x7f7c6406cff0 2024-04-14T23:58:16.802 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.797+0000 7f7c6ba00640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f7c6406c450 con 0x7f7c6406fc10 2024-04-14T23:58:16.803 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.797+0000 7f7c69000640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f7c64154fd0 0x7f7c6406c8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53316/0 (socket says 172.21.15.60:53316) 2024-04-14T23:58:16.803 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.797+0000 7f7c69000640 1 -- 172.21.15.60:0/3103980671 learned_addr learned my addr 172.21.15.60:0/3103980671 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:58:16.803 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.797+0000 7f7c5be00640 1 --2- 172.21.15.60:0/3103980671 >> v2:172.21.15.186:3300/0 conn(0x7f7c6406fc10 0x7f7c640b0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:16.803 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.798+0000 7f7c5be00640 1 -- 172.21.15.60:0/3103980671 >> v1:172.21.15.186:6789/0 conn(0x7f7c6406cff0 legacy=0x7f7c6406e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:16.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.798+0000 7f7c5be00640 1 -- 172.21.15.60:0/3103980671 >> v1:172.21.15.60:6789/0 conn(0x7f7c64154fd0 legacy=0x7f7c6406c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:16.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.798+0000 7f7c5be00640 1 -- 172.21.15.60:0/3103980671 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c640c97f0 con 0x7f7c6406fc10 2024-04-14T23:58:16.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.799+0000 7f7c5be00640 1 --2- 172.21.15.60:0/3103980671 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c6406fc10 0x7f7c640b0ed0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c0097a0 tx=0x7f7c4c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2eec3e1cf4137ed0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:16.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.799+0000 7f7c63e00640 1 -- 172.21.15.60:0/3103980671 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c4c034070 con 0x7f7c6406fc10 2024-04-14T23:58:16.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.799+0000 7f7c63e00640 1 -- 172.21.15.60:0/3103980671 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c4c030610 con 0x7f7c6406fc10 2024-04-14T23:58:16.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.799+0000 7f7c63e00640 1 -- 172.21.15.60:0/3103980671 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c4c0309e0 con 0x7f7c6406fc10 2024-04-14T23:58:16.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.800+0000 7f7c6ba00640 1 -- 172.21.15.60:0/3103980671 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c6406fc10 msgr2=0x7f7c640b0ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:16.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.800+0000 7f7c6ba00640 1 --2- 172.21.15.60:0/3103980671 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c6406fc10 0x7f7c640b0ed0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c0097a0 tx=0x7f7c4c02b960 comp rx=0 tx=0).stop 2024-04-14T23:58:16.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.800+0000 7f7c6ba00640 1 -- 172.21.15.60:0/3103980671 shutdown_connections 2024-04-14T23:58:16.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.800+0000 7f7c6ba00640 1 --2- 172.21.15.60:0/3103980671 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c6406fc10 0x7f7c640b0ed0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:16.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.800+0000 7f7c6ba00640 1 -- 172.21.15.60:0/3103980671 >> 172.21.15.60:0/3103980671 conn(0x7f7c64070d20 msgr2=0x7f7c64071100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:16.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.800+0000 7f7c6ba00640 1 -- 172.21.15.60:0/3103980671 shutdown_connections 2024-04-14T23:58:16.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.800+0000 7f7c6ba00640 1 -- 172.21.15.60:0/3103980671 wait complete. 2024-04-14T23:58:16.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.802+0000 7f7c6ba00640 1 Processor -- start 2024-04-14T23:58:16.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.802+0000 7f7c6ba00640 1 -- start start 2024-04-14T23:58:16.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.803+0000 7f7c6ba00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c6406cff0 0x7f7c64153450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:16.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.803+0000 7f7c6ba00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7c6406fc10 0x7f7c64153990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:16.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.803+0000 7f7c6ba00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c64154fd0 0x7f7c641f2830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:16.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.803+0000 7f7c6ba00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f7c6406eca0 con 0x7f7c6406cff0 2024-04-14T23:58:16.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.803+0000 7f7c6ba00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7c6415ed80 con 0x7f7c64154fd0 2024-04-14T23:58:16.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.803+0000 7f7c6ba00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f7c6406ee20 con 0x7f7c6406fc10 2024-04-14T23:58:16.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.803+0000 7f7c69000640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c6406cff0 0x7f7c64153450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:16.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.803+0000 7f7c5be00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7c6406fc10 0x7f7c64153990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:16.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.803+0000 7f7c69a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c64154fd0 0x7f7c641f2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:16.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.803+0000 7f7c69000640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c6406cff0 0x7f7c64153450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59600/0 (socket says 172.21.15.60:59600) 2024-04-14T23:58:16.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.803+0000 7f7c69000640 1 -- 172.21.15.60:0/4194610782 learned_addr learned my addr 172.21.15.60:0/4194610782 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:58:16.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.804+0000 7f7c5be00640 1 -- 172.21.15.60:0/4194610782 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c64154fd0 msgr2=0x7f7c641f2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:16.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.804+0000 7f7c5be00640 1 --2- 172.21.15.60:0/4194610782 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c64154fd0 0x7f7c641f2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:16.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.804+0000 7f7c5be00640 1 -- 172.21.15.60:0/4194610782 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c6406cff0 msgr2=0x7f7c64153450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:16.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.804+0000 7f7c5be00640 1 --2- 172.21.15.60:0/4194610782 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c6406cff0 0x7f7c64153450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:16.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.804+0000 7f7c5be00640 1 -- 172.21.15.60:0/4194610782 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c640afb40 con 0x7f7c6406fc10 2024-04-14T23:58:16.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.804+0000 7f7c69000640 1 --2- 172.21.15.60:0/4194610782 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c6406cff0 0x7f7c64153450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:58:16.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.804+0000 7f7c5be00640 1 --2- 172.21.15.60:0/4194610782 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7c6406fc10 0x7f7c64153990 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7c64071b80 tx=0x7f7c4c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:16.812 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.804+0000 7f7c61600640 1 -- 172.21.15.60:0/4194610782 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c4c034040 con 0x7f7c6406fc10 2024-04-14T23:58:16.812 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.805+0000 7f7c61600640 1 -- 172.21.15.60:0/4194610782 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c4c039a10 con 0x7f7c6406fc10 2024-04-14T23:58:16.812 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.805+0000 7f7c61600640 1 -- 172.21.15.60:0/4194610782 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c4c039e20 con 0x7f7c6406fc10 2024-04-14T23:58:16.812 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.805+0000 7f7c6ba00640 1 -- 172.21.15.60:0/4194610782 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c640b3c20 con 0x7f7c6406fc10 2024-04-14T23:58:16.812 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.805+0000 7f7c6ba00640 1 -- 172.21.15.60:0/4194610782 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c641f1a70 con 0x7f7c6406fc10 2024-04-14T23:58:16.813 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.806+0000 7f7c6ba00640 1 -- 172.21.15.60:0/4194610782 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c64076970 con 0x7f7c6406fc10 2024-04-14T23:58:16.813 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.807+0000 7f7c61600640 1 -- 172.21.15.60:0/4194610782 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f7c4c037070 con 0x7f7c6406fc10 2024-04-14T23:58:16.813 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.807+0000 7f7c61600640 1 --2- 172.21.15.60:0/4194610782 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f7c4006b990 0x7f7c4006de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:16.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.808+0000 7f7c61600640 1 -- 172.21.15.60:0/4194610782 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(45..45 src has 1..45) ==== 6255+0+0 (secure 0 0 0) 0x7f7c4c0aca30 con 0x7f7c6406fc10 2024-04-14T23:58:16.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.808+0000 7f7c69000640 1 --2- 172.21.15.60:0/4194610782 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f7c4006b990 0x7f7c4006de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:16.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.808+0000 7f7c69000640 1 --2- 172.21.15.60:0/4194610782 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f7c4006b990 0x7f7c4006de50 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f7c54005880 tx=0x7f7c54009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:16.817 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:16.812+0000 7f7c61600640 1 -- 172.21.15.60:0/4194610782 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7c4c07ad20 con 0x7f7c6406fc10 2024-04-14T23:58:17.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.015+0000 7f7c6ba00640 1 -- 172.21.15.60:0/4194610782 --> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7f7c640c9530 con 0x7f7c4006b990 2024-04-14T23:58:17.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.016+0000 7f7c61600640 1 -- 172.21.15.60:0/4194610782 <== mgr.4100 v2:172.21.15.186:6832/714104835 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f7c640c9530 con 0x7f7c4006b990 2024-04-14T23:58:17.021 INFO:teuthology.orchestra.run.smithi060.stderr:instructed osd(s) 6 to deep-scrub 2024-04-14T23:58:17.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.018+0000 7f7c6ba00640 1 -- 172.21.15.60:0/4194610782 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f7c4006b990 msgr2=0x7f7c4006de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:17.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.018+0000 7f7c6ba00640 1 --2- 172.21.15.60:0/4194610782 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f7c4006b990 0x7f7c4006de50 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f7c54005880 tx=0x7f7c54009040 comp rx=0 tx=0).stop 2024-04-14T23:58:17.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.019+0000 7f7c6ba00640 1 -- 172.21.15.60:0/4194610782 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7c6406fc10 msgr2=0x7f7c64153990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:17.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.019+0000 7f7c6ba00640 1 --2- 172.21.15.60:0/4194610782 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7c6406fc10 0x7f7c64153990 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7c64071b80 tx=0x7f7c4c0046c0 comp rx=0 tx=0).stop 2024-04-14T23:58:17.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.019+0000 7f7c6ba00640 1 -- 172.21.15.60:0/4194610782 shutdown_connections 2024-04-14T23:58:17.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.019+0000 7f7c6ba00640 1 --2- 172.21.15.60:0/4194610782 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f7c4006b990 0x7f7c4006de50 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:17.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.019+0000 7f7c6ba00640 1 --2- 172.21.15.60:0/4194610782 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c64154fd0 0x7f7c641f2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:17.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.019+0000 7f7c6ba00640 1 --2- 172.21.15.60:0/4194610782 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7c6406fc10 0x7f7c64153990 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:17.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.019+0000 7f7c6ba00640 1 --2- 172.21.15.60:0/4194610782 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c6406cff0 0x7f7c64153450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:17.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.019+0000 7f7c6ba00640 1 -- 172.21.15.60:0/4194610782 >> 172.21.15.60:0/4194610782 conn(0x7f7c64070d20 msgr2=0x7f7c640ccce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:17.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.019+0000 7f7c6ba00640 1 -- 172.21.15.60:0/4194610782 shutdown_connections 2024-04-14T23:58:17.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.019+0000 7f7c6ba00640 1 -- 172.21.15.60:0/4194610782 wait complete. 2024-04-14T23:58:17.048 INFO:tasks.ceph:Scrubbing osd.7 2024-04-14T23:58:17.049 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-04-14T23:58:17.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.264+0000 7f73dbe00640 1 Processor -- start 2024-04-14T23:58:17.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.265+0000 7f73dbe00640 1 -- start start 2024-04-14T23:58:17.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.266+0000 7f73dbe00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f73dc154ca0 0x7f73dc06bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:17.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.266+0000 7f73dbe00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73dc0ccbb0 0x7f73dc15e190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:17.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.266+0000 7f73dbe00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73dc155080 con 0x7f73dc0cda40 2024-04-14T23:58:17.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.266+0000 7f73dbe00640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f73dc15b790 con 0x7f73dc154ca0 2024-04-14T23:58:17.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.266+0000 7f73dbe00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f73dc06c2d0 con 0x7f73dc0ccbb0 2024-04-14T23:58:17.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.267+0000 7f73daa00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f73dc154ca0 0x7f73dc06bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:17.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.267+0000 7f73da000640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73dc0ccbb0 0x7f73dc15e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:17.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.267+0000 7f73db400640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f73dc0cda40 0x7f73dc0cb640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53320/0 (socket says 172.21.15.60:53320) 2024-04-14T23:58:17.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.267+0000 7f73db400640 1 -- 172.21.15.60:0/4186982188 learned_addr learned my addr 172.21.15.60:0/4186982188 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:58:17.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.267+0000 7f73daa00640 1 -- 172.21.15.60:0/4186982188 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73dc0ccbb0 msgr2=0x7f73dc15e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:58:17.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.267+0000 7f73daa00640 1 --2- 172.21.15.60:0/4186982188 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73dc0ccbb0 0x7f73dc15e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:17.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.267+0000 7f73daa00640 1 -- 172.21.15.60:0/4186982188 >> v1:172.21.15.60:6789/0 conn(0x7f73dc0cda40 legacy=0x7f73dc0cb640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:17.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.267+0000 7f73daa00640 1 -- 172.21.15.60:0/4186982188 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73dc15aba0 con 0x7f73dc154ca0 2024-04-14T23:58:17.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.267+0000 7f73d9600640 1 -- 172.21.15.60:0/4186982188 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3807493109 0 0) 0x7f73d4001110 con 0x7f73dc0cda40 2024-04-14T23:58:17.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.268+0000 7f73daa00640 1 --2- 172.21.15.60:0/4186982188 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73dc154ca0 0x7f73dc06bab0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f73c40097a0 tx=0x7f73c402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f2e75e939c4a9c6e server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:17.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.269+0000 7f73d9600640 1 -- 172.21.15.60:0/4186982188 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73c4034070 con 0x7f73dc154ca0 2024-04-14T23:58:17.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.269+0000 7f73d9600640 1 -- 172.21.15.60:0/4186982188 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73c4030610 con 0x7f73dc154ca0 2024-04-14T23:58:17.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.269+0000 7f73d9600640 1 -- 172.21.15.60:0/4186982188 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73c40309e0 con 0x7f73dc154ca0 2024-04-14T23:58:17.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.269+0000 7f73dbe00640 1 -- 172.21.15.60:0/4186982188 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73dc154ca0 msgr2=0x7f73dc06bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:17.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.269+0000 7f73dbe00640 1 --2- 172.21.15.60:0/4186982188 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73dc154ca0 0x7f73dc06bab0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f73c40097a0 tx=0x7f73c402b960 comp rx=0 tx=0).stop 2024-04-14T23:58:17.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.269+0000 7f73dbe00640 1 -- 172.21.15.60:0/4186982188 shutdown_connections 2024-04-14T23:58:17.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.269+0000 7f73dbe00640 1 --2- 172.21.15.60:0/4186982188 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73dc0ccbb0 0x7f73dc15e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:17.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.269+0000 7f73dbe00640 1 --2- 172.21.15.60:0/4186982188 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73dc154ca0 0x7f73dc06bab0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:17.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.269+0000 7f73dbe00640 1 -- 172.21.15.60:0/4186982188 >> 172.21.15.60:0/4186982188 conn(0x7f73dc06d760 msgr2=0x7f73dc06db70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:17.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.269+0000 7f73dbe00640 1 -- 172.21.15.60:0/4186982188 shutdown_connections 2024-04-14T23:58:17.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.269+0000 7f73dbe00640 1 -- 172.21.15.60:0/4186982188 wait complete. 2024-04-14T23:58:17.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.271+0000 7f73dbe00640 1 Processor -- start 2024-04-14T23:58:17.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.271+0000 7f73dbe00640 1 -- start start 2024-04-14T23:58:17.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.272+0000 7f73dbe00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73dc0ccbb0 0x7f73dc1e3300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:17.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.272+0000 7f73dbe00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73dc0cda40 0x7f73dc1537f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:17.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.272+0000 7f73dbe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73dc154ca0 0x7f73dc153d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:17.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.272+0000 7f73dbe00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f73dc15b420 con 0x7f73dc0ccbb0 2024-04-14T23:58:17.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.272+0000 7f73dbe00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f73dc15b140 con 0x7f73dc154ca0 2024-04-14T23:58:17.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.272+0000 7f73dbe00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f73dc06bff0 con 0x7f73dc0cda40 2024-04-14T23:58:17.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.272+0000 7f73da000640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73dc0cda40 0x7f73dc1537f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:17.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.272+0000 7f73db400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73dc154ca0 0x7f73dc153d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:17.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.272+0000 7f73daa00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73dc0ccbb0 0x7f73dc1e3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:17.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.272+0000 7f73da000640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73dc0cda40 0x7f73dc1537f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.60:53982/0 (socket says 172.21.15.60:53982) 2024-04-14T23:58:17.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.272+0000 7f73da000640 1 -- 172.21.15.60:0/2504360851 learned_addr learned my addr 172.21.15.60:0/2504360851 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:58:17.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.273+0000 7f73da000640 1 -- 172.21.15.60:0/2504360851 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73dc154ca0 msgr2=0x7f73dc153d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:17.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.273+0000 7f73da000640 1 --2- 172.21.15.60:0/2504360851 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73dc154ca0 0x7f73dc153d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:17.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.273+0000 7f73da000640 1 -- 172.21.15.60:0/2504360851 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73dc0ccbb0 msgr2=0x7f73dc1e3300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:17.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.273+0000 7f73da000640 1 --2- 172.21.15.60:0/2504360851 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73dc0ccbb0 0x7f73dc1e3300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:17.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.273+0000 7f73da000640 1 -- 172.21.15.60:0/2504360851 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73dc1f3af0 con 0x7f73dc0cda40 2024-04-14T23:58:17.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.273+0000 7f73daa00640 1 --2- 172.21.15.60:0/2504360851 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73dc0ccbb0 0x7f73dc1e3300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:58:17.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.273+0000 7f73da000640 1 --2- 172.21.15.60:0/2504360851 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73dc0cda40 0x7f73dc1537f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f73cc009db0 tx=0x7f73cc00c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:17.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.274+0000 7f73d3400640 1 -- 172.21.15.60:0/2504360851 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73cc018020 con 0x7f73dc0cda40 2024-04-14T23:58:17.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.274+0000 7f73d3400640 1 -- 172.21.15.60:0/2504360851 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73cc014480 con 0x7f73dc0cda40 2024-04-14T23:58:17.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.274+0000 7f73d3400640 1 -- 172.21.15.60:0/2504360851 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73cc014850 con 0x7f73dc0cda40 2024-04-14T23:58:17.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.274+0000 7f73dbe00640 1 -- 172.21.15.60:0/2504360851 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73dc0c7540 con 0x7f73dc0cda40 2024-04-14T23:58:17.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.274+0000 7f73dbe00640 1 -- 172.21.15.60:0/2504360851 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f73dc0c7b00 con 0x7f73dc0cda40 2024-04-14T23:58:17.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.275+0000 7f73dbe00640 1 -- 172.21.15.60:0/2504360851 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f73dc04f540 con 0x7f73dc0cda40 2024-04-14T23:58:17.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.276+0000 7f73d3400640 1 -- 172.21.15.60:0/2504360851 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f73cc005d10 con 0x7f73dc0cda40 2024-04-14T23:58:17.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.277+0000 7f73d3400640 1 --2- 172.21.15.60:0/2504360851 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f73b806b9e0 0x7f73b806dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:17.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.277+0000 7f73d3400640 1 -- 172.21.15.60:0/2504360851 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(45..45 src has 1..45) ==== 6255+0+0 (secure 0 0 0) 0x7f73cc08bfe0 con 0x7f73dc0cda40 2024-04-14T23:58:17.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.277+0000 7f73d3400640 1 --2- 172.21.15.60:0/2504360851 >> [v2:172.21.15.186:6824/842537875,v1:172.21.15.186:6825/842537875] conn(0x7f73b8074110 0x7f73b8076590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:17.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.277+0000 7f73daa00640 1 --2- 172.21.15.60:0/2504360851 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f73b806b9e0 0x7f73b806dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:17.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.277+0000 7f73d3400640 1 -- 172.21.15.60:0/2504360851 --> [v2:172.21.15.186:6824/842537875,v1:172.21.15.186:6825/842537875] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f73cc00ae90 con 0x7f73b8074110 2024-04-14T23:58:17.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.277+0000 7f73d3400640 1 -- 172.21.15.60:0/2504360851 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f73cc08c410 con 0x7f73dc0cda40 2024-04-14T23:58:17.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.278+0000 7f73db400640 1 --2- 172.21.15.60:0/2504360851 >> [v2:172.21.15.186:6824/842537875,v1:172.21.15.186:6825/842537875] conn(0x7f73b8074110 0x7f73b8076590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:17.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.278+0000 7f73daa00640 1 --2- 172.21.15.60:0/2504360851 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f73b806b9e0 0x7f73b806dea0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f73c4036000 tx=0x7f73c403b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:17.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.278+0000 7f73db400640 1 --2- 172.21.15.60:0/2504360851 >> [v2:172.21.15.186:6824/842537875,v1:172.21.15.186:6825/842537875] conn(0x7f73b8074110 0x7f73b8076590 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:17.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.279+0000 7f73d3400640 1 -- 172.21.15.60:0/2504360851 <== osd.7 v2:172.21.15.186:6824/842537875 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f73cc00ae90 con 0x7f73b8074110 2024-04-14T23:58:17.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.295+0000 7f73dbe00640 1 -- 172.21.15.60:0/2504360851 --> [v2:172.21.15.186:6824/842537875,v1:172.21.15.186:6825/842537875] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f73dc0cb420 con 0x7f73b8074110 2024-04-14T23:58:17.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.295+0000 7f73d3400640 1 -- 172.21.15.60:0/2504360851 <== osd.7 v2:172.21.15.186:6824/842537875 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f73dc0cb420 con 0x7f73b8074110 2024-04-14T23:58:17.301 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-14T23:58:17.301 INFO:teuthology.orchestra.run.smithi060.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-14T23:58:17.302 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-14T23:58:17.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.296+0000 7f73dbe00640 1 -- 172.21.15.60:0/2504360851 >> [v2:172.21.15.186:6824/842537875,v1:172.21.15.186:6825/842537875] conn(0x7f73b8074110 msgr2=0x7f73b8076590 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:17.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.296+0000 7f73dbe00640 1 --2- 172.21.15.60:0/2504360851 >> [v2:172.21.15.186:6824/842537875,v1:172.21.15.186:6825/842537875] conn(0x7f73b8074110 0x7f73b8076590 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:17.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.296+0000 7f73dbe00640 1 -- 172.21.15.60:0/2504360851 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f73b806b9e0 msgr2=0x7f73b806dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:17.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.296+0000 7f73dbe00640 1 --2- 172.21.15.60:0/2504360851 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f73b806b9e0 0x7f73b806dea0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f73c4036000 tx=0x7f73c403b040 comp rx=0 tx=0).stop 2024-04-14T23:58:17.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.296+0000 7f73dbe00640 1 -- 172.21.15.60:0/2504360851 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73dc0cda40 msgr2=0x7f73dc1537f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:17.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.296+0000 7f73dbe00640 1 --2- 172.21.15.60:0/2504360851 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73dc0cda40 0x7f73dc1537f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f73cc009db0 tx=0x7f73cc00c490 comp rx=0 tx=0).stop 2024-04-14T23:58:17.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.297+0000 7f73db400640 1 -- 172.21.15.60:0/2504360851 reap_dead start 2024-04-14T23:58:17.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.297+0000 7f73dbe00640 1 -- 172.21.15.60:0/2504360851 shutdown_connections 2024-04-14T23:58:17.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.297+0000 7f73dbe00640 1 -- 172.21.15.60:0/2504360851 >> 172.21.15.60:0/2504360851 conn(0x7f73dc06d760 msgr2=0x7f73dc15a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:17.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.297+0000 7f73dbe00640 1 -- 172.21.15.60:0/2504360851 shutdown_connections 2024-04-14T23:58:17.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.297+0000 7f73dbe00640 1 -- 172.21.15.60:0/2504360851 wait complete. 2024-04-14T23:58:17.321 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-04-14T23:58:17.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.540+0000 7fabaf200640 1 Processor -- start 2024-04-14T23:58:17.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.540+0000 7fabaf200640 1 -- start start 2024-04-14T23:58:17.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.542+0000 7fabaf200640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fabb00c6c70 0x7fabb00b1c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:17.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.542+0000 7fabaf200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fabb006c6b0 0x7fabb006eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:17.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.542+0000 7fabaf200640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fabb00c6160 con 0x7fabb00b1860 2024-04-14T23:58:17.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.542+0000 7fabaf200640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fabb0159930 con 0x7fabb00c6c70 2024-04-14T23:58:17.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.542+0000 7fabaf200640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fabb006cb00 con 0x7fabb006c6b0 2024-04-14T23:58:17.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.542+0000 7faba7e00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fabb00c6c70 0x7fabb00b1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:17.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.542+0000 7fabade00640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fabb00b1860 0x7fabb006b1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.60:41622/0 (socket says 172.21.15.60:41622) 2024-04-14T23:58:17.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.542+0000 7fabade00640 1 -- 172.21.15.60:0/1998406664 learned_addr learned my addr 172.21.15.60:0/1998406664 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:58:17.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.542+0000 7fabae800640 1 --2- 172.21.15.60:0/1998406664 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fabb006c6b0 0x7fabb006eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:17.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.543+0000 7fabad400640 1 -- 172.21.15.60:0/1998406664 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2328423294 0 0) 0x7fabb00c6160 con 0x7fabb00b1860 2024-04-14T23:58:17.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.543+0000 7faba7e00640 1 -- 172.21.15.60:0/1998406664 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fabb006c6b0 msgr2=0x7fabb006eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:58:17.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.543+0000 7faba7e00640 1 --2- 172.21.15.60:0/1998406664 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fabb006c6b0 0x7fabb006eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:17.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.543+0000 7faba7e00640 1 -- 172.21.15.60:0/1998406664 >> v1:172.21.15.186:6789/0 conn(0x7fabb00b1860 legacy=0x7fabb006b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:17.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.543+0000 7faba7e00640 1 -- 172.21.15.60:0/1998406664 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabb0158b70 con 0x7fabb00c6c70 2024-04-14T23:58:17.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.544+0000 7faba7e00640 1 --2- 172.21.15.60:0/1998406664 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fabb00c6c70 0x7fabb00b1c40 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fab980097a0 tx=0x7fab9802b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a7b5001419641c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:17.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.544+0000 7fabad400640 1 -- 172.21.15.60:0/1998406664 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab98034070 con 0x7fabb00c6c70 2024-04-14T23:58:17.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.545+0000 7fabad400640 1 -- 172.21.15.60:0/1998406664 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab98030800 con 0x7fabb00c6c70 2024-04-14T23:58:17.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.545+0000 7fabad400640 1 -- 172.21.15.60:0/1998406664 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab98030bd0 con 0x7fabb00c6c70 2024-04-14T23:58:17.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.545+0000 7fabaf200640 1 -- 172.21.15.60:0/1998406664 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fabb00c6c70 msgr2=0x7fabb00b1c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:17.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.545+0000 7fabaf200640 1 --2- 172.21.15.60:0/1998406664 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fabb00c6c70 0x7fabb00b1c40 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fab980097a0 tx=0x7fab9802b940 comp rx=0 tx=0).stop 2024-04-14T23:58:17.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.545+0000 7fabaf200640 1 -- 172.21.15.60:0/1998406664 shutdown_connections 2024-04-14T23:58:17.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.545+0000 7fabaf200640 1 --2- 172.21.15.60:0/1998406664 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fabb006c6b0 0x7fabb006eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:17.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.545+0000 7fabaf200640 1 --2- 172.21.15.60:0/1998406664 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fabb00c6c70 0x7fabb00b1c40 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:17.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.545+0000 7fabaf200640 1 -- 172.21.15.60:0/1998406664 >> 172.21.15.60:0/1998406664 conn(0x7fabb00cc2f0 msgr2=0x7fabb00cc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:17.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.546+0000 7fabaf200640 1 -- 172.21.15.60:0/1998406664 shutdown_connections 2024-04-14T23:58:17.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.546+0000 7fabaf200640 1 -- 172.21.15.60:0/1998406664 wait complete. 2024-04-14T23:58:17.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.547+0000 7fabaf200640 1 Processor -- start 2024-04-14T23:58:17.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.547+0000 7fabaf200640 1 -- start start 2024-04-14T23:58:17.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.548+0000 7fabaf200640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fabb006c6b0 0x7fabb0154cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:17.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.548+0000 7fabaf200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fabb00b1860 0x7fabb0147fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:17.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.548+0000 7fabaf200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fabb00c6c70 0x7fabb014d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:17.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.548+0000 7fabaf200640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fabb0159130 con 0x7fabb006c6b0 2024-04-14T23:58:17.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.548+0000 7fabaf200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fabb00c6440 con 0x7fabb00b1860 2024-04-14T23:58:17.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.548+0000 7fabaf200640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fabb0159410 con 0x7fabb00c6c70 2024-04-14T23:58:17.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.549+0000 7fabade00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fabb006c6b0 0x7fabb0154cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:17.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.549+0000 7fabade00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fabb006c6b0 0x7fabb0154cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59630/0 (socket says 172.21.15.60:59630) 2024-04-14T23:58:17.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.549+0000 7fabade00640 1 -- 172.21.15.60:0/4164366480 learned_addr learned my addr 172.21.15.60:0/4164366480 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:58:17.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.549+0000 7faba7e00640 1 --2- 172.21.15.60:0/4164366480 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fabb00b1860 0x7fabb0147fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:17.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.549+0000 7fabae800640 1 --2- 172.21.15.60:0/4164366480 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fabb00c6c70 0x7fabb014d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:17.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.549+0000 7fabade00640 1 -- 172.21.15.60:0/4164366480 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fabb00c6c70 msgr2=0x7fabb014d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:17.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.549+0000 7fabade00640 1 --2- 172.21.15.60:0/4164366480 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fabb00c6c70 0x7fabb014d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:17.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.549+0000 7fabade00640 1 -- 172.21.15.60:0/4164366480 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fabb00b1860 msgr2=0x7fabb0147fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:17.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.549+0000 7fabade00640 1 --2- 172.21.15.60:0/4164366480 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fabb00b1860 0x7fabb0147fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:17.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.549+0000 7fabade00640 1 -- 172.21.15.60:0/4164366480 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabb0153450 con 0x7fabb006c6b0 2024-04-14T23:58:17.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.550+0000 7fabade00640 1 --2- 172.21.15.60:0/4164366480 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fabb006c6b0 0x7fabb0154cb0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7faba000c460 tx=0x7faba000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:17.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.550+0000 7faba6000640 1 -- 172.21.15.60:0/4164366480 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faba0017070 con 0x7fabb006c6b0 2024-04-14T23:58:17.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.550+0000 7faba6000640 1 -- 172.21.15.60:0/4164366480 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faba0005d20 con 0x7fabb006c6b0 2024-04-14T23:58:17.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.550+0000 7faba6000640 1 -- 172.21.15.60:0/4164366480 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faba0007310 con 0x7fabb006c6b0 2024-04-14T23:58:17.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.550+0000 7fabaf200640 1 -- 172.21.15.60:0/4164366480 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabb01536f0 con 0x7fabb006c6b0 2024-04-14T23:58:17.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.550+0000 7fabaf200640 1 -- 172.21.15.60:0/4164366480 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabb0153cb0 con 0x7fabb006c6b0 2024-04-14T23:58:17.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.551+0000 7fabaf200640 1 -- 172.21.15.60:0/4164366480 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabb004f540 con 0x7fabb006c6b0 2024-04-14T23:58:17.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.552+0000 7faba6000640 1 -- 172.21.15.60:0/4164366480 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7faba001a070 con 0x7fabb006c6b0 2024-04-14T23:58:17.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.553+0000 7faba6000640 1 --2- 172.21.15.60:0/4164366480 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fab8c06b9c0 0x7fab8c06de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:17.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.553+0000 7faba6000640 1 -- 172.21.15.60:0/4164366480 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6255+0+0 (secure 0 0 0) 0x7faba0089cf0 con 0x7fabb006c6b0 2024-04-14T23:58:17.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.554+0000 7faba7e00640 1 --2- 172.21.15.60:0/4164366480 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fab8c06b9c0 0x7fab8c06de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:17.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.554+0000 7faba7e00640 1 --2- 172.21.15.60:0/4164366480 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fab8c06b9c0 0x7fab8c06de80 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fab98036000 tx=0x7fab98008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:17.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.558+0000 7faba6000640 1 -- 172.21.15.60:0/4164366480 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7faba0058080 con 0x7fabb006c6b0 2024-04-14T23:58:17.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.762+0000 7fabaf200640 1 -- 172.21.15.60:0/4164366480 --> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7fabb014bf90 con 0x7fab8c06b9c0 2024-04-14T23:58:17.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.763+0000 7faba6000640 1 -- 172.21.15.60:0/4164366480 <== mgr.4100 v2:172.21.15.186:6832/714104835 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fabb014bf90 con 0x7fab8c06b9c0 2024-04-14T23:58:17.768 INFO:teuthology.orchestra.run.smithi060.stderr:instructed osd(s) 7 to deep-scrub 2024-04-14T23:58:17.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.766+0000 7fabaf200640 1 -- 172.21.15.60:0/4164366480 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fab8c06b9c0 msgr2=0x7fab8c06de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:17.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.766+0000 7fabaf200640 1 --2- 172.21.15.60:0/4164366480 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fab8c06b9c0 0x7fab8c06de80 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fab98036000 tx=0x7fab98008030 comp rx=0 tx=0).stop 2024-04-14T23:58:17.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.766+0000 7fabaf200640 1 -- 172.21.15.60:0/4164366480 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fabb006c6b0 msgr2=0x7fabb0154cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:17.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.766+0000 7fabaf200640 1 --2- 172.21.15.60:0/4164366480 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fabb006c6b0 0x7fabb0154cb0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7faba000c460 tx=0x7faba000c920 comp rx=0 tx=0).stop 2024-04-14T23:58:17.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.766+0000 7fabaf200640 1 -- 172.21.15.60:0/4164366480 shutdown_connections 2024-04-14T23:58:17.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.766+0000 7fabaf200640 1 --2- 172.21.15.60:0/4164366480 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fab8c06b9c0 0x7fab8c06de80 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:17.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.766+0000 7fabaf200640 1 --2- 172.21.15.60:0/4164366480 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fabb00c6c70 0x7fabb014d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:17.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.766+0000 7fabaf200640 1 --2- 172.21.15.60:0/4164366480 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fabb00b1860 0x7fabb0147fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:17.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.766+0000 7fabaf200640 1 --2- 172.21.15.60:0/4164366480 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fabb006c6b0 0x7fabb0154cb0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:17.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.766+0000 7fabaf200640 1 -- 172.21.15.60:0/4164366480 >> 172.21.15.60:0/4164366480 conn(0x7fabb00cc2f0 msgr2=0x7fabb0156cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:17.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.766+0000 7fabaf200640 1 -- 172.21.15.60:0/4164366480 shutdown_connections 2024-04-14T23:58:17.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:17.766+0000 7fabaf200640 1 -- 172.21.15.60:0/4164366480 wait complete. 2024-04-14T23:58:17.793 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-14T23:58:18.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.000+0000 7fefa9000640 1 Processor -- start 2024-04-14T23:58:18.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.000+0000 7fefa9000640 1 -- start start 2024-04-14T23:58:18.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.001+0000 7fefa9000640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fefa4152cf0 0x7fefa415c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:18.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.001+0000 7fefa9000640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fefa4159450 0x7fefa415c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:18.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.001+0000 7fefa9000640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefa41530d0 con 0x7fefa40c63c0 2024-04-14T23:58:18.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.001+0000 7fefa9000640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fefa40751f0 con 0x7fefa4159450 2024-04-14T23:58:18.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.001+0000 7fefa9000640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fefa40c8370 con 0x7fefa4152cf0 2024-04-14T23:58:18.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.001+0000 7fefa1e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fefa4152cf0 0x7fefa415c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:18.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.001+0000 7fefa1e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fefa4152cf0 0x7fefa415c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.60:54002/0 (socket says 172.21.15.60:54002) 2024-04-14T23:58:18.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.001+0000 7fefa1e00640 1 -- 172.21.15.60:0/2704538656 learned_addr learned my addr 172.21.15.60:0/2704538656 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:58:18.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.001+0000 7fefa2800640 1 --2- 172.21.15.60:0/2704538656 >> v2:172.21.15.186:3300/0 conn(0x7fefa4159450 0x7fefa415c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:18.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.002+0000 7fefa1e00640 1 -- 172.21.15.60:0/2704538656 >> v2:172.21.15.186:3300/0 conn(0x7fefa4159450 msgr2=0x7fefa415c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:58:18.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.002+0000 7fefa1e00640 1 --2- 172.21.15.60:0/2704538656 >> v2:172.21.15.186:3300/0 conn(0x7fefa4159450 0x7fefa415c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:18.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.002+0000 7fefa1e00640 1 -- 172.21.15.60:0/2704538656 >> v1:172.21.15.186:6789/0 conn(0x7fefa40c63c0 legacy=0x7fefa40c3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:18.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.002+0000 7fefa1e00640 1 -- 172.21.15.60:0/2704538656 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefa40c3810 con 0x7fefa4152cf0 2024-04-14T23:58:18.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.003+0000 7fefa1e00640 1 --2- 172.21.15.60:0/2704538656 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fefa4152cf0 0x7fefa415c2a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fef94009e50 tx=0x7fef9402bff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35ba18e2c2881507 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:18.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.003+0000 7fefa0a00640 1 -- 172.21.15.60:0/2704538656 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef9402cb90 con 0x7fefa4152cf0 2024-04-14T23:58:18.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.003+0000 7fefa0a00640 1 -- 172.21.15.60:0/2704538656 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef9402cd30 con 0x7fefa4152cf0 2024-04-14T23:58:18.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.003+0000 7fefa0a00640 1 -- 172.21.15.60:0/2704538656 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef94034610 con 0x7fefa4152cf0 2024-04-14T23:58:18.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.004+0000 7fefa9000640 1 -- 172.21.15.60:0/2704538656 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fefa4152cf0 msgr2=0x7fefa415c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:18.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.004+0000 7fefa9000640 1 --2- 172.21.15.60:0/2704538656 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fefa4152cf0 0x7fefa415c2a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fef94009e50 tx=0x7fef9402bff0 comp rx=0 tx=0).stop 2024-04-14T23:58:18.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.004+0000 7fefa9000640 1 -- 172.21.15.60:0/2704538656 shutdown_connections 2024-04-14T23:58:18.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.004+0000 7fefa9000640 1 --2- 172.21.15.60:0/2704538656 >> v2:172.21.15.186:3300/0 conn(0x7fefa4159450 0x7fefa415c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:18.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.004+0000 7fefa9000640 1 --2- 172.21.15.60:0/2704538656 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fefa4152cf0 0x7fefa415c2a0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:18.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.004+0000 7fefa9000640 1 -- 172.21.15.60:0/2704538656 >> 172.21.15.60:0/2704538656 conn(0x7fefa40cc170 msgr2=0x7fefa40cc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:18.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.004+0000 7fefa9000640 1 -- 172.21.15.60:0/2704538656 shutdown_connections 2024-04-14T23:58:18.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.004+0000 7fefa9000640 1 -- 172.21.15.60:0/2704538656 wait complete. 2024-04-14T23:58:18.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.006+0000 7fefa9000640 1 Processor -- start 2024-04-14T23:58:18.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.006+0000 7fefa9000640 1 -- start start 2024-04-14T23:58:18.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.007+0000 7fefa9000640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fefa40c63c0 0x7fefa4072420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:18.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.007+0000 7fefa9000640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefa4152cf0 0x7fefa41f0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:18.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.007+0000 7fefa9000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fefa4159450 0x7fefa41f13a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:18.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.007+0000 7fefa9000640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fefa40c88f0 con 0x7fefa4152cf0 2024-04-14T23:58:18.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.007+0000 7fefa9000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fefa40c8610 con 0x7fefa4159450 2024-04-14T23:58:18.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.007+0000 7fefa9000640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fefa40c4690 con 0x7fefa40c63c0 2024-04-14T23:58:18.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.007+0000 7fefa1400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefa4152cf0 0x7fefa41f0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:18.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.007+0000 7fefa1e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fefa40c63c0 0x7fefa4072420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:18.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.007+0000 7fefa2800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fefa4159450 0x7fefa41f13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:18.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.007+0000 7fefa1400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefa4152cf0 0x7fefa41f0e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59642/0 (socket says 172.21.15.60:59642) 2024-04-14T23:58:18.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.007+0000 7fefa1400640 1 -- 172.21.15.60:0/572443719 learned_addr learned my addr 172.21.15.60:0/572443719 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:58:18.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.007+0000 7fefa1e00640 1 -- 172.21.15.60:0/572443719 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fefa4159450 msgr2=0x7fefa41f13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:18.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.007+0000 7fefa1e00640 1 --2- 172.21.15.60:0/572443719 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fefa4159450 0x7fefa41f13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:18.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.007+0000 7fefa1e00640 1 -- 172.21.15.60:0/572443719 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefa4152cf0 msgr2=0x7fefa41f0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:18.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.007+0000 7fefa1e00640 1 --2- 172.21.15.60:0/572443719 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefa4152cf0 0x7fefa41f0e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:18.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.007+0000 7fefa1e00640 1 -- 172.21.15.60:0/572443719 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefa4145dc0 con 0x7fefa40c63c0 2024-04-14T23:58:18.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.007+0000 7fefa1400640 1 --2- 172.21.15.60:0/572443719 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefa4152cf0 0x7fefa41f0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:58:18.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.008+0000 7fefa1e00640 1 --2- 172.21.15.60:0/572443719 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fefa40c63c0 0x7fefa4072420 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fef94032000 tx=0x7fef94004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:18.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.008+0000 7fef9a000640 1 -- 172.21.15.60:0/572443719 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef94043020 con 0x7fefa40c63c0 2024-04-14T23:58:18.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.008+0000 7fef9a000640 1 -- 172.21.15.60:0/572443719 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef9403e450 con 0x7fefa40c63c0 2024-04-14T23:58:18.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.008+0000 7fef9a000640 1 -- 172.21.15.60:0/572443719 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef94009510 con 0x7fefa40c63c0 2024-04-14T23:58:18.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.008+0000 7fefa9000640 1 -- 172.21.15.60:0/572443719 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefa4151a00 con 0x7fefa40c63c0 2024-04-14T23:58:18.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.008+0000 7fefa9000640 1 -- 172.21.15.60:0/572443719 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fefa4151fc0 con 0x7fefa40c63c0 2024-04-14T23:58:18.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.009+0000 7fefa9000640 1 -- 172.21.15.60:0/572443719 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefa4076970 con 0x7fefa40c63c0 2024-04-14T23:58:18.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.010+0000 7fef9a000640 1 -- 172.21.15.60:0/572443719 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7fef94046020 con 0x7fefa40c63c0 2024-04-14T23:58:18.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.011+0000 7fef9a000640 1 --2- 172.21.15.60:0/572443719 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fef8406b9c0 0x7fef8406de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:18.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.011+0000 7fef9a000640 1 -- 172.21.15.60:0/572443719 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(45..45 src has 1..45) ==== 6255+0+0 (secure 0 0 0) 0x7fef940aca00 con 0x7fefa40c63c0 2024-04-14T23:58:18.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.012+0000 7fefa1400640 1 --2- 172.21.15.60:0/572443719 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fef8406b9c0 0x7fef8406de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:18.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.012+0000 7fefa1400640 1 --2- 172.21.15.60:0/572443719 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fef8406b9c0 0x7fef8406de80 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fefa4072cd0 tx=0x7fef90009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:18.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.016+0000 7fef9a000640 1 -- 172.21.15.60:0/572443719 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fef9404e020 con 0x7fefa40c63c0 2024-04-14T23:58:18.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.219+0000 7fefa9000640 1 -- 172.21.15.60:0/572443719 --> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fefa40c7f40 con 0x7fef8406b9c0 2024-04-14T23:58:18.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.228+0000 7fef9a000640 1 -- 172.21.15.60:0/572443719 <== mgr.4100 v2:172.21.15.186:6832/714104835 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+253420 (secure 0 0 0) 0x7fefa40c7f40 con 0x7fef8406b9c0 2024-04-14T23:58:18.237 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:58:18.238 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-04-14T23:58:18.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.234+0000 7fefa9000640 1 -- 172.21.15.60:0/572443719 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fef8406b9c0 msgr2=0x7fef8406de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:18.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.234+0000 7fefa9000640 1 --2- 172.21.15.60:0/572443719 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fef8406b9c0 0x7fef8406de80 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fefa4072cd0 tx=0x7fef90009040 comp rx=0 tx=0).stop 2024-04-14T23:58:18.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.234+0000 7fefa9000640 1 -- 172.21.15.60:0/572443719 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fefa40c63c0 msgr2=0x7fefa4072420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:18.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.234+0000 7fefa9000640 1 --2- 172.21.15.60:0/572443719 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fefa40c63c0 0x7fefa4072420 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fef94032000 tx=0x7fef94004a40 comp rx=0 tx=0).stop 2024-04-14T23:58:18.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.235+0000 7fefa9000640 1 -- 172.21.15.60:0/572443719 shutdown_connections 2024-04-14T23:58:18.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.235+0000 7fefa9000640 1 --2- 172.21.15.60:0/572443719 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fef8406b9c0 0x7fef8406de80 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:18.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.235+0000 7fefa9000640 1 --2- 172.21.15.60:0/572443719 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fefa4159450 0x7fefa41f13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:18.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.235+0000 7fefa9000640 1 --2- 172.21.15.60:0/572443719 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefa4152cf0 0x7fefa41f0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:18.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.235+0000 7fefa9000640 1 --2- 172.21.15.60:0/572443719 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fefa40c63c0 0x7fefa4072420 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:18.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.235+0000 7fefa9000640 1 -- 172.21.15.60:0/572443719 >> 172.21.15.60:0/572443719 conn(0x7fefa40cc170 msgr2=0x7fefa4148ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:18.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.235+0000 7fefa9000640 1 -- 172.21.15.60:0/572443719 shutdown_connections 2024-04-14T23:58:18.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:18.235+0000 7fefa9000640 1 -- 172.21.15.60:0/572443719 wait complete. 2024-04-14T23:58:18.260 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":329,"stamp":"2024-04-14T23:58:16.315979+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":152616787,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":2895,"num_read_kb":545038,"num_write":14631,"num_write_kb":441831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14565,"ondisk_log_size":14565,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":533432,"kb_used_data":67448,"kb_used_omap":12,"kb_used_meta":465907,"kb_avail":754441288,"statfs":{"total":773094113280,"available":772547878912,"internally_reserved":0,"allocated":69066752,"data_stored":307148518,"data_compressed":34495944,"data_compressed_allocated":44949504,"data_compressed_original":285278208,"omap_allocated":12752,"internal_metadata":477089328},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-49568041,"num_objects":-29,"num_object_clones":0,"num_object_copies":-58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-29,"num_whiteouts":0,"num_read":53,"num_read_kb":42,"num_write":120,"num_write_kb":12326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002826"},"pg_stats":[{"pgid":"3.3f","version":"36'158","reported_seq":2280,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.085799+0000","last_change":"2024-04-14T23:55:41.994501+0000","last_active":"2024-04-14T23:57:39.085799+0000","last_peered":"2024-04-14T23:57:39.085799+0000","last_clean":"2024-04-14T23:57:39.085799+0000","last_became_active":"2024-04-14T23:48:43.222278+0000","last_became_peered":"2024-04-14T23:48:43.222278+0000","last_unstale":"2024-04-14T23:57:39.085799+0000","last_undegraded":"2024-04-14T23:57:39.085799+0000","last_fullsized":"2024-04-14T23:57:39.085799+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T04:04:17.956339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025657449999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":29,"num_read_kb":15,"num_write":158,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"44'142","reported_seq":2270,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.088726+0000","last_change":"2024-04-14T23:55:42.310665+0000","last_active":"2024-04-14T23:57:39.088726+0000","last_peered":"2024-04-14T23:57:39.088726+0000","last_clean":"2024-04-14T23:57:39.088726+0000","last_became_active":"2024-04-14T23:48:43.226426+0000","last_became_peered":"2024-04-14T23:48:43.226426+0000","last_unstale":"2024-04-14T23:57:39.088726+0000","last_undegraded":"2024-04-14T23:57:39.088726+0000","last_fullsized":"2024-04-14T23:57:39.088726+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011741033999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":16,"num_write":142,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"37'171","reported_seq":2300,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.088112+0000","last_change":"2024-04-14T23:55:42.315743+0000","last_active":"2024-04-14T23:57:39.088112+0000","last_peered":"2024-04-14T23:57:39.088112+0000","last_clean":"2024-04-14T23:57:39.088112+0000","last_became_active":"2024-04-14T23:48:43.236616+0000","last_became_peered":"2024-04-14T23:48:43.236616+0000","last_unstale":"2024-04-14T23:57:39.088112+0000","last_undegraded":"2024-04-14T23:57:39.088112+0000","last_fullsized":"2024-04-14T23:57:39.088112+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01445958,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":38,"num_read_kb":8211,"num_write":164,"num_write_kb":8272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"45'141","reported_seq":2254,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:02.544309+0000","last_change":"2024-04-14T23:55:42.084275+0000","last_active":"2024-04-14T23:58:02.544309+0000","last_peered":"2024-04-14T23:58:02.544309+0000","last_clean":"2024-04-14T23:58:02.544309+0000","last_became_active":"2024-04-14T23:48:43.242000+0000","last_became_peered":"2024-04-14T23:48:43.242000+0000","last_unstale":"2024-04-14T23:58:02.544309+0000","last_undegraded":"2024-04-14T23:58:02.544309+0000","last_fullsized":"2024-04-14T23:58:02.544309+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:06:54.726051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020739690000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":9,"num_write":140,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"35'197","reported_seq":2311,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.086576+0000","last_change":"2024-04-14T23:55:42.308076+0000","last_active":"2024-04-14T23:57:39.086576+0000","last_peered":"2024-04-14T23:57:39.086576+0000","last_clean":"2024-04-14T23:57:39.086576+0000","last_became_active":"2024-04-14T23:49:29.448078+0000","last_became_peered":"2024-04-14T23:49:29.448078+0000","last_unstale":"2024-04-14T23:57:39.086576+0000","last_undegraded":"2024-04-14T23:57:39.086576+0000","last_fullsized":"2024-04-14T23:57:39.086576+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":197,"log_dups_size":0,"ondisk_log_size":197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011461469,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":27,"num_read_kb":18,"num_write":196,"num_write_kb":98,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3a","version":"35'170","reported_seq":2294,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.083401+0000","last_change":"2024-04-14T23:55:42.084342+0000","last_active":"2024-04-14T23:57:39.083401+0000","last_peered":"2024-04-14T23:57:39.083401+0000","last_clean":"2024-04-14T23:57:39.083401+0000","last_became_active":"2024-04-14T23:48:43.230405+0000","last_became_peered":"2024-04-14T23:48:43.230405+0000","last_unstale":"2024-04-14T23:57:39.083401+0000","last_undegraded":"2024-04-14T23:57:39.083401+0000","last_fullsized":"2024-04-14T23:57:39.083401+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:40:55.017190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002322771,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":32,"num_read_kb":19,"num_write":170,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"44'198","reported_seq":2327,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.084946+0000","last_change":"2024-04-14T23:55:41.992292+0000","last_active":"2024-04-14T23:57:39.084946+0000","last_peered":"2024-04-14T23:57:39.084946+0000","last_clean":"2024-04-14T23:57:39.084946+0000","last_became_active":"2024-04-14T23:48:43.230784+0000","last_became_peered":"2024-04-14T23:48:43.230784+0000","last_unstale":"2024-04-14T23:57:39.084946+0000","last_undegraded":"2024-04-14T23:57:39.084946+0000","last_fullsized":"2024-04-14T23:57:39.084946+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":198,"log_dups_size":0,"ondisk_log_size":198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T04:49:40.437771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023027080000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":37,"num_read_kb":16,"num_write":198,"num_write_kb":4194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"45'135","reported_seq":2254,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:08.588786+0000","last_change":"2024-04-14T23:55:42.317071+0000","last_active":"2024-04-14T23:58:08.588786+0000","last_peered":"2024-04-14T23:58:08.588786+0000","last_clean":"2024-04-14T23:58:08.588786+0000","last_became_active":"2024-04-14T23:48:43.235719+0000","last_became_peered":"2024-04-14T23:48:43.235719+0000","last_unstale":"2024-04-14T23:58:08.588786+0000","last_undegraded":"2024-04-14T23:58:08.588786+0000","last_fullsized":"2024-04-14T23:58:08.588786+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015616025,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":26,"num_read_kb":10,"num_write":134,"num_write_kb":12352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"44'157","reported_seq":2283,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.085798+0000","last_change":"2024-04-14T23:55:41.994669+0000","last_active":"2024-04-14T23:57:39.085798+0000","last_peered":"2024-04-14T23:57:39.085798+0000","last_clean":"2024-04-14T23:57:39.085798+0000","last_became_active":"2024-04-14T23:48:43.231751+0000","last_became_peered":"2024-04-14T23:48:43.231751+0000","last_unstale":"2024-04-14T23:57:39.085798+0000","last_undegraded":"2024-04-14T23:57:39.085798+0000","last_fullsized":"2024-04-14T23:57:39.085798+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:44:15.830503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025112200000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":4114,"num_write":154,"num_write_kb":8271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"35'164","reported_seq":2283,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.010548+0000","last_change":"2024-04-14T23:55:41.992813+0000","last_active":"2024-04-14T23:57:39.010548+0000","last_peered":"2024-04-14T23:57:39.010548+0000","last_clean":"2024-04-14T23:57:39.010548+0000","last_became_active":"2024-04-14T23:48:43.224592+0000","last_became_peered":"2024-04-14T23:48:43.224592+0000","last_unstale":"2024-04-14T23:57:39.010548+0000","last_undegraded":"2024-04-14T23:57:39.010548+0000","last_fullsized":"2024-04-14T23:57:39.010548+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:36:03.843739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014741629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":27,"num_read_kb":16,"num_write":162,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"45'173","reported_seq":2300,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:07.582110+0000","last_change":"2024-04-14T23:55:41.991958+0000","last_active":"2024-04-14T23:58:07.582110+0000","last_peered":"2024-04-14T23:58:07.582110+0000","last_clean":"2024-04-14T23:58:07.582110+0000","last_became_active":"2024-04-14T23:48:43.224989+0000","last_became_peered":"2024-04-14T23:48:43.224989+0000","last_unstale":"2024-04-14T23:58:07.582110+0000","last_undegraded":"2024-04-14T23:58:07.582110+0000","last_fullsized":"2024-04-14T23:58:07.582110+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":173,"log_dups_size":0,"ondisk_log_size":173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:31:12.485950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001581301,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":18,"num_write":172,"num_write_kb":4183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"45'147","reported_seq":2263,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:02.543716+0000","last_change":"2024-04-14T23:55:42.224637+0000","last_active":"2024-04-14T23:58:02.543716+0000","last_peered":"2024-04-14T23:58:02.543716+0000","last_clean":"2024-04-14T23:58:02.543716+0000","last_became_active":"2024-04-14T23:48:43.225892+0000","last_became_peered":"2024-04-14T23:48:43.225892+0000","last_unstale":"2024-04-14T23:58:02.543716+0000","last_undegraded":"2024-04-14T23:58:02.543716+0000","last_fullsized":"2024-04-14T23:58:02.543716+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:30:03.715201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011254640000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":24,"num_read_kb":10,"num_write":146,"num_write_kb":8263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"37'146","reported_seq":2268,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.092849+0000","last_change":"2024-04-14T23:55:42.317691+0000","last_active":"2024-04-14T23:57:39.092849+0000","last_peered":"2024-04-14T23:57:39.092849+0000","last_clean":"2024-04-14T23:57:39.092849+0000","last_became_active":"2024-04-14T23:48:43.219614+0000","last_became_peered":"2024-04-14T23:48:43.219614+0000","last_unstale":"2024-04-14T23:57:39.092849+0000","last_undegraded":"2024-04-14T23:57:39.092849+0000","last_fullsized":"2024-04-14T23:57:39.092849+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":146,"log_dups_size":0,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012872641000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":30,"num_read_kb":4115,"num_write":142,"num_write_kb":8261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"44'175","reported_seq":2309,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:04.378092+0000","last_change":"2024-04-14T23:55:42.319365+0000","last_active":"2024-04-14T23:58:04.378092+0000","last_peered":"2024-04-14T23:58:04.378092+0000","last_clean":"2024-04-14T23:58:04.378092+0000","last_became_active":"2024-04-14T23:48:43.239136+0000","last_became_peered":"2024-04-14T23:48:43.239136+0000","last_unstale":"2024-04-14T23:58:04.378092+0000","last_undegraded":"2024-04-14T23:58:04.378092+0000","last_fullsized":"2024-04-14T23:58:04.378092+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015761819,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":40,"num_read_kb":19,"num_write":174,"num_write_kb":12372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"44'159","reported_seq":2290,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:04.349306+0000","last_change":"2024-04-14T23:55:42.308265+0000","last_active":"2024-04-14T23:58:04.349306+0000","last_peered":"2024-04-14T23:58:04.349306+0000","last_clean":"2024-04-14T23:58:04.349306+0000","last_became_active":"2024-04-14T23:48:43.227898+0000","last_became_peered":"2024-04-14T23:48:43.227898+0000","last_unstale":"2024-04-14T23:58:04.349306+0000","last_undegraded":"2024-04-14T23:58:04.349306+0000","last_fullsized":"2024-04-14T23:58:04.349306+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011044834999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":37,"num_read_kb":19,"num_write":159,"num_write_kb":4178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"44'184","reported_seq":2317,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.083264+0000","last_change":"2024-04-14T23:55:42.083783+0000","last_active":"2024-04-14T23:57:39.083264+0000","last_peered":"2024-04-14T23:57:39.083264+0000","last_clean":"2024-04-14T23:57:39.083264+0000","last_became_active":"2024-04-14T23:48:43.238714+0000","last_became_peered":"2024-04-14T23:48:43.238714+0000","last_unstale":"2024-04-14T23:57:39.083264+0000","last_undegraded":"2024-04-14T23:57:39.083264+0000","last_fullsized":"2024-04-14T23:57:39.083264+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":184,"log_dups_size":0,"ondisk_log_size":184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:37:44.837525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019546920000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":40,"num_read_kb":19,"num_write":182,"num_write_kb":7459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"44'174","reported_seq":2306,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.083777+0000","last_change":"2024-04-14T23:55:42.085173+0000","last_active":"2024-04-14T23:57:39.083777+0000","last_peered":"2024-04-14T23:57:39.083777+0000","last_clean":"2024-04-14T23:57:39.083777+0000","last_became_active":"2024-04-14T23:48:43.228618+0000","last_became_peered":"2024-04-14T23:48:43.228618+0000","last_unstale":"2024-04-14T23:57:39.083777+0000","last_undegraded":"2024-04-14T23:57:39.083777+0000","last_fullsized":"2024-04-14T23:57:39.083777+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":174,"log_dups_size":0,"ondisk_log_size":174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T09:07:18.121366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002424158,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":40,"num_read_kb":4117,"num_write":170,"num_write_kb":4181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"37'153","reported_seq":2294,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:04.373979+0000","last_change":"2024-04-14T23:55:42.175749+0000","last_active":"2024-04-14T23:58:04.373979+0000","last_peered":"2024-04-14T23:58:04.373979+0000","last_clean":"2024-04-14T23:58:04.373979+0000","last_became_active":"2024-04-14T23:48:43.232971+0000","last_became_peered":"2024-04-14T23:48:43.232971+0000","last_unstale":"2024-04-14T23:58:04.373979+0000","last_undegraded":"2024-04-14T23:58:04.373979+0000","last_fullsized":"2024-04-14T23:58:04.373979+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030089610000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":47,"num_read_kb":26,"num_write":152,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"45'164","reported_seq":2291,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:45.262194+0000","last_change":"2024-04-14T23:55:41.994445+0000","last_active":"2024-04-14T23:57:45.262194+0000","last_peered":"2024-04-14T23:57:45.262194+0000","last_clean":"2024-04-14T23:57:45.262194+0000","last_became_active":"2024-04-14T23:48:43.232135+0000","last_became_peered":"2024-04-14T23:48:43.232135+0000","last_unstale":"2024-04-14T23:57:45.262194+0000","last_undegraded":"2024-04-14T23:57:45.262194+0000","last_fullsized":"2024-04-14T23:57:45.262194+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:37:12.273966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029988570000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":4112,"num_write":160,"num_write_kb":8270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"45'197","reported_seq":2331,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:08.581562+0000","last_change":"2024-04-14T23:55:41.992691+0000","last_active":"2024-04-14T23:58:08.581562+0000","last_peered":"2024-04-14T23:58:08.581562+0000","last_clean":"2024-04-14T23:58:08.581562+0000","last_became_active":"2024-04-14T23:48:43.226225+0000","last_became_peered":"2024-04-14T23:48:43.226225+0000","last_unstale":"2024-04-14T23:58:08.581562+0000","last_undegraded":"2024-04-14T23:58:08.581562+0000","last_fullsized":"2024-04-14T23:58:08.581562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":197,"log_dups_size":0,"ondisk_log_size":197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:43:42.017755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019674760000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":42,"num_read_kb":4116,"num_write":194,"num_write_kb":16477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"44'154","reported_seq":2292,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:44.374810+0000","last_change":"2024-04-14T23:55:42.316046+0000","last_active":"2024-04-14T23:57:44.374810+0000","last_peered":"2024-04-14T23:57:44.374810+0000","last_clean":"2024-04-14T23:57:44.374810+0000","last_became_active":"2024-04-14T23:48:43.230056+0000","last_became_peered":"2024-04-14T23:48:43.230056+0000","last_unstale":"2024-04-14T23:57:44.374810+0000","last_undegraded":"2024-04-14T23:57:44.374810+0000","last_fullsized":"2024-04-14T23:57:44.374810+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015336659000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":44,"num_read_kb":22,"num_write":154,"num_write_kb":77,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"44'175","reported_seq":2304,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.083498+0000","last_change":"2024-04-14T23:55:42.084499+0000","last_active":"2024-04-14T23:57:39.083498+0000","last_peered":"2024-04-14T23:57:39.083498+0000","last_clean":"2024-04-14T23:57:39.083498+0000","last_became_active":"2024-04-14T23:48:43.229110+0000","last_became_peered":"2024-04-14T23:48:43.229110+0000","last_unstale":"2024-04-14T23:57:39.083498+0000","last_undegraded":"2024-04-14T23:57:39.083498+0000","last_fullsized":"2024-04-14T23:57:39.083498+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T09:03:22.176338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022905320000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":36,"num_read_kb":17,"num_write":174,"num_write_kb":4183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"45'171","reported_seq":2303,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:07.587258+0000","last_change":"2024-04-14T23:55:42.314003+0000","last_active":"2024-04-14T23:58:07.587258+0000","last_peered":"2024-04-14T23:58:07.587258+0000","last_clean":"2024-04-14T23:58:07.587258+0000","last_became_active":"2024-04-14T23:48:43.233853+0000","last_became_peered":"2024-04-14T23:48:43.233853+0000","last_unstale":"2024-04-14T23:58:07.587258+0000","last_undegraded":"2024-04-14T23:58:07.587258+0000","last_fullsized":"2024-04-14T23:58:07.587258+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012858206,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":39,"num_read_kb":4117,"num_write":168,"num_write_kb":4180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"45'174","reported_seq":2298,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:09.374480+0000","last_change":"2024-04-14T23:55:42.175389+0000","last_active":"2024-04-14T23:58:09.374480+0000","last_peered":"2024-04-14T23:58:09.374480+0000","last_clean":"2024-04-14T23:58:09.374480+0000","last_became_active":"2024-04-14T23:49:29.512499+0000","last_became_peered":"2024-04-14T23:49:29.512499+0000","last_unstale":"2024-04-14T23:58:09.374480+0000","last_undegraded":"2024-04-14T23:58:09.374480+0000","last_fullsized":"2024-04-14T23:58:09.374480+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":174,"log_dups_size":0,"ondisk_log_size":174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037213820000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":36,"num_read_kb":16,"num_write":174,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"37'160","reported_seq":2288,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.093496+0000","last_change":"2024-04-14T23:55:42.319252+0000","last_active":"2024-04-14T23:57:39.093496+0000","last_peered":"2024-04-14T23:57:39.093496+0000","last_clean":"2024-04-14T23:57:39.093496+0000","last_became_active":"2024-04-14T23:48:43.233238+0000","last_became_peered":"2024-04-14T23:48:43.233238+0000","last_unstale":"2024-04-14T23:57:39.093496+0000","last_undegraded":"2024-04-14T23:57:39.093496+0000","last_fullsized":"2024-04-14T23:57:39.093496+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":160,"log_dups_size":0,"ondisk_log_size":160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012037971,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":18,"num_write":158,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"36'184","reported_seq":2312,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.010657+0000","last_change":"2024-04-14T23:55:41.993037+0000","last_active":"2024-04-14T23:57:39.010657+0000","last_peered":"2024-04-14T23:57:39.010657+0000","last_clean":"2024-04-14T23:57:39.010657+0000","last_became_active":"2024-04-14T23:48:43.222624+0000","last_became_peered":"2024-04-14T23:48:43.222624+0000","last_unstale":"2024-04-14T23:57:39.010657+0000","last_undegraded":"2024-04-14T23:57:39.010657+0000","last_fullsized":"2024-04-14T23:57:39.010657+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":184,"log_dups_size":0,"ondisk_log_size":184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:43:37.661814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001456707,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":19,"num_write":184,"num_write_kb":92,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"44'140","reported_seq":2258,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.086407+0000","last_change":"2024-04-14T23:55:42.309950+0000","last_active":"2024-04-14T23:57:39.086407+0000","last_peered":"2024-04-14T23:57:39.086407+0000","last_clean":"2024-04-14T23:57:39.086407+0000","last_became_active":"2024-04-14T23:48:43.232277+0000","last_became_peered":"2024-04-14T23:48:43.232277+0000","last_unstale":"2024-04-14T23:57:39.086407+0000","last_undegraded":"2024-04-14T23:57:39.086407+0000","last_fullsized":"2024-04-14T23:57:39.086407+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011200412,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":138,"num_write_kb":4167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"45'157","reported_seq":2281,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:07.588628+0000","last_change":"2024-04-14T23:55:42.319096+0000","last_active":"2024-04-14T23:58:07.588628+0000","last_peered":"2024-04-14T23:58:07.588628+0000","last_clean":"2024-04-14T23:58:07.588628+0000","last_became_active":"2024-04-14T23:48:43.229172+0000","last_became_peered":"2024-04-14T23:48:43.229172+0000","last_unstale":"2024-04-14T23:58:07.588628+0000","last_undegraded":"2024-04-14T23:58:07.588628+0000","last_fullsized":"2024-04-14T23:58:07.588628+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011889788,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":31,"num_read_kb":3287,"num_write":154,"num_write_kb":3350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"45'186","reported_seq":2324,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:08.587942+0000","last_change":"2024-04-14T23:55:42.317181+0000","last_active":"2024-04-14T23:58:08.587942+0000","last_peered":"2024-04-14T23:58:08.587942+0000","last_clean":"2024-04-14T23:58:08.587942+0000","last_became_active":"2024-04-14T23:49:29.449548+0000","last_became_peered":"2024-04-14T23:49:29.449548+0000","last_unstale":"2024-04-14T23:58:08.587942+0000","last_undegraded":"2024-04-14T23:58:08.587942+0000","last_fullsized":"2024-04-14T23:58:08.587942+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":186,"log_dups_size":0,"ondisk_log_size":186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015790789,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":39,"num_read_kb":18,"num_write":186,"num_write_kb":4188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"35'188","reported_seq":2319,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:49.371699+0000","last_change":"2024-04-14T23:55:42.174354+0000","last_active":"2024-04-14T23:57:49.371699+0000","last_peered":"2024-04-14T23:57:49.371699+0000","last_clean":"2024-04-14T23:57:49.371699+0000","last_became_active":"2024-04-14T23:49:29.512523+0000","last_became_peered":"2024-04-14T23:49:29.512523+0000","last_unstale":"2024-04-14T23:57:49.371699+0000","last_undegraded":"2024-04-14T23:57:49.371699+0000","last_fullsized":"2024-04-14T23:57:49.371699+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":188,"log_dups_size":0,"ondisk_log_size":188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026059680000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":43,"num_read_kb":22,"num_write":188,"num_write_kb":94,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"45'206","reported_seq":2335,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:07.587198+0000","last_change":"2024-04-14T23:55:42.175131+0000","last_active":"2024-04-14T23:58:07.587198+0000","last_peered":"2024-04-14T23:58:07.587198+0000","last_clean":"2024-04-14T23:58:07.587198+0000","last_became_active":"2024-04-14T23:48:43.233567+0000","last_became_peered":"2024-04-14T23:48:43.233567+0000","last_unstale":"2024-04-14T23:58:07.587198+0000","last_undegraded":"2024-04-14T23:58:07.587198+0000","last_fullsized":"2024-04-14T23:58:07.587198+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":206,"log_dups_size":0,"ondisk_log_size":206,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003348115,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":17,"num_write":206,"num_write_kb":14807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"45'202","reported_seq":2342,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:08.008795+0000","last_change":"2024-04-14T23:55:41.992153+0000","last_active":"2024-04-14T23:58:08.008795+0000","last_peered":"2024-04-14T23:58:08.008795+0000","last_clean":"2024-04-14T23:58:08.008795+0000","last_became_active":"2024-04-14T23:48:43.230626+0000","last_became_peered":"2024-04-14T23:48:43.230626+0000","last_unstale":"2024-04-14T23:58:08.008795+0000","last_undegraded":"2024-04-14T23:58:08.008795+0000","last_fullsized":"2024-04-14T23:58:08.008795+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":202,"log_dups_size":0,"ondisk_log_size":202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T23:53:31.323623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001517437,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":47,"num_read_kb":27,"num_write":201,"num_write_kb":12389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"44'158","reported_seq":2278,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.084838+0000","last_change":"2024-04-14T23:55:42.224072+0000","last_active":"2024-04-14T23:57:39.084838+0000","last_peered":"2024-04-14T23:57:39.084838+0000","last_clean":"2024-04-14T23:57:39.084838+0000","last_became_active":"2024-04-14T23:48:43.226773+0000","last_became_peered":"2024-04-14T23:48:43.226773+0000","last_unstale":"2024-04-14T23:57:39.084838+0000","last_undegraded":"2024-04-14T23:57:39.084838+0000","last_fullsized":"2024-04-14T23:57:39.084838+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:04:13.345776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00101596,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":28,"num_read_kb":16,"num_write":158,"num_write_kb":4174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"45'21","reported_seq":92,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:07.587084+0000","last_change":"2024-04-14T23:48:42.239254+0000","last_active":"2024-04-14T23:58:07.587084+0000","last_peered":"2024-04-14T23:58:07.587084+0000","last_clean":"2024-04-14T23:58:07.587084+0000","last_became_active":"2024-04-14T23:48:42.238847+0000","last_became_peered":"2024-04-14T23:48:42.238847+0000","last_unstale":"2024-04-14T23:58:07.587084+0000","last_undegraded":"2024-04-14T23:58:07.587084+0000","last_fullsized":"2024-04-14T23:58:07.587084+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":16386,"num_write":22,"num_write_kb":4233,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"44'168","reported_seq":2297,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.091905+0000","last_change":"2024-04-14T23:55:42.317522+0000","last_active":"2024-04-14T23:57:39.091905+0000","last_peered":"2024-04-14T23:57:39.091905+0000","last_clean":"2024-04-14T23:57:39.091905+0000","last_became_active":"2024-04-14T23:48:43.237391+0000","last_became_peered":"2024-04-14T23:48:43.237391+0000","last_unstale":"2024-04-14T23:57:39.091905+0000","last_undegraded":"2024-04-14T23:57:39.091905+0000","last_fullsized":"2024-04-14T23:57:39.091905+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01542551,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":36,"num_read_kb":15,"num_write":168,"num_write_kb":4179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"34'164","reported_seq":240,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:45.069610+0000","last_change":"2024-04-14T23:48:42.219277+0000","last_active":"2024-04-14T23:57:45.069610+0000","last_peered":"2024-04-14T23:57:45.069610+0000","last_clean":"2024-04-14T23:57:45.069610+0000","last_became_active":"2024-04-14T23:48:42.218995+0000","last_became_peered":"2024-04-14T23:48:42.218995+0000","last_unstale":"2024-04-14T23:57:45.069610+0000","last_undegraded":"2024-04-14T23:57:45.069610+0000","last_fullsized":"2024-04-14T23:57:45.069610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T23:51:33.370857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":11,"num_read_kb":40960,"num_write":162,"num_write_kb":12391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"45'25","reported_seq":102,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:14.765400+0000","last_change":"2024-04-14T23:58:14.765434+0000","last_active":"2024-04-14T23:58:14.765400+0000","last_peered":"2024-04-14T23:58:14.765400+0000","last_clean":"2024-04-14T23:58:14.765400+0000","last_became_active":"2024-04-14T23:48:42.225937+0000","last_became_peered":"2024-04-14T23:48:42.225937+0000","last_unstale":"2024-04-14T23:58:14.765400+0000","last_undegraded":"2024-04-14T23:58:14.765400+0000","last_fullsized":"2024-04-14T23:58:14.765400+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'25","last_scrub_stamp":"2024-04-14T23:58:14.765308+0000","last_deep_scrub":"45'25","last_deep_scrub_stamp":"2024-04-14T23:58:14.765308+0000","last_clean_scrub_stamp":"2024-04-14T23:58:14.765308+0000","objects_scrubbed":1,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:47:25.922937+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":22,"num_write_kb":4105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"35'209","reported_seq":2341,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.091716+0000","last_change":"2024-04-14T23:55:42.316735+0000","last_active":"2024-04-14T23:57:39.091716+0000","last_peered":"2024-04-14T23:57:39.091716+0000","last_clean":"2024-04-14T23:57:39.091716+0000","last_became_active":"2024-04-14T23:48:43.227356+0000","last_became_peered":"2024-04-14T23:48:43.227356+0000","last_unstale":"2024-04-14T23:57:39.091716+0000","last_undegraded":"2024-04-14T23:57:39.091716+0000","last_fullsized":"2024-04-14T23:57:39.091716+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":209,"log_dups_size":0,"ondisk_log_size":209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014613774,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":40,"num_read_kb":4118,"num_write":206,"num_write_kb":8293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"45'189","reported_seq":260,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:44.808805+0000","last_change":"2024-04-14T23:48:42.210883+0000","last_active":"2024-04-14T23:57:44.808805+0000","last_peered":"2024-04-14T23:57:44.808805+0000","last_clean":"2024-04-14T23:57:44.808805+0000","last_became_active":"2024-04-14T23:48:42.210607+0000","last_became_peered":"2024-04-14T23:48:42.210607+0000","last_unstale":"2024-04-14T23:57:44.808805+0000","last_undegraded":"2024-04-14T23:57:44.808805+0000","last_fullsized":"2024-04-14T23:57:44.808805+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":189,"log_dups_size":0,"ondisk_log_size":189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:33:32.526223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":186,"num_write_kb":8305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"42'164","reported_seq":2297,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.086105+0000","last_change":"2024-04-14T23:55:42.310398+0000","last_active":"2024-04-14T23:57:39.086105+0000","last_peered":"2024-04-14T23:57:39.086105+0000","last_clean":"2024-04-14T23:57:39.086105+0000","last_became_active":"2024-04-14T23:48:43.217801+0000","last_became_peered":"2024-04-14T23:48:43.217801+0000","last_unstale":"2024-04-14T23:57:39.086105+0000","last_undegraded":"2024-04-14T23:57:39.086105+0000","last_fullsized":"2024-04-14T23:57:39.086105+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012539378,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":41,"num_read_kb":4114,"num_write":158,"num_write_kb":4174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"45'6","reported_seq":71,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.010421+0000","last_change":"2024-04-14T23:48:42.218652+0000","last_active":"2024-04-14T23:57:39.010421+0000","last_peered":"2024-04-14T23:57:39.010421+0000","last_clean":"2024-04-14T23:57:39.010421+0000","last_became_active":"2024-04-14T23:48:42.218321+0000","last_became_peered":"2024-04-14T23:48:42.218321+0000","last_unstale":"2024-04-14T23:57:39.010421+0000","last_undegraded":"2024-04-14T23:57:39.010421+0000","last_fullsized":"2024-04-14T23:57:39.010421+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T04:22:06.957747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"45'190","reported_seq":2327,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.083810+0000","last_change":"2024-04-14T23:55:42.084801+0000","last_active":"2024-04-14T23:57:39.083810+0000","last_peered":"2024-04-14T23:57:39.083810+0000","last_clean":"2024-04-14T23:57:39.083810+0000","last_became_active":"2024-04-14T23:48:43.228805+0000","last_became_peered":"2024-04-14T23:48:43.228805+0000","last_unstale":"2024-04-14T23:57:39.083810+0000","last_undegraded":"2024-04-14T23:57:39.083810+0000","last_fullsized":"2024-04-14T23:57:39.083810+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":190,"log_dups_size":0,"ondisk_log_size":190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:59:51.260017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022812650000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":45,"num_read_kb":20,"num_write":190,"num_write_kb":4190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"44'27","reported_seq":95,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.083988+0000","last_change":"2024-04-14T23:48:42.221104+0000","last_active":"2024-04-14T23:57:39.083988+0000","last_peered":"2024-04-14T23:57:39.083988+0000","last_clean":"2024-04-14T23:57:39.083988+0000","last_became_active":"2024-04-14T23:48:42.220767+0000","last_became_peered":"2024-04-14T23:48:42.220767+0000","last_unstale":"2024-04-14T23:57:39.083988+0000","last_undegraded":"2024-04-14T23:57:39.083988+0000","last_fullsized":"2024-04-14T23:57:39.083988+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:23:57.213764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":24,"num_write_kb":4108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"45'163","reported_seq":2289,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:07.589532+0000","last_change":"2024-04-14T23:55:42.316479+0000","last_active":"2024-04-14T23:58:07.589532+0000","last_peered":"2024-04-14T23:58:07.589532+0000","last_clean":"2024-04-14T23:58:07.589532+0000","last_became_active":"2024-04-14T23:48:43.236168+0000","last_became_peered":"2024-04-14T23:48:43.236168+0000","last_unstale":"2024-04-14T23:58:07.589532+0000","last_undegraded":"2024-04-14T23:58:07.589532+0000","last_fullsized":"2024-04-14T23:58:07.589532+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":163,"log_dups_size":0,"ondisk_log_size":163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015342639999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":33,"num_read_kb":18,"num_write":162,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"45'148","reported_seq":230,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:08.590614+0000","last_change":"2024-04-14T23:48:42.231329+0000","last_active":"2024-04-14T23:58:08.590614+0000","last_peered":"2024-04-14T23:58:08.590614+0000","last_clean":"2024-04-14T23:58:08.590614+0000","last_became_active":"2024-04-14T23:48:42.230969+0000","last_became_peered":"2024-04-14T23:48:42.230969+0000","last_unstale":"2024-04-14T23:58:08.590614+0000","last_undegraded":"2024-04-14T23:58:08.590614+0000","last_fullsized":"2024-04-14T23:58:08.590614+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194394,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":30,"num_read_kb":9177,"num_write":147,"num_write_kb":4202,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"44'151","reported_seq":2272,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.086423+0000","last_change":"2024-04-14T23:55:42.224786+0000","last_active":"2024-04-14T23:57:39.086423+0000","last_peered":"2024-04-14T23:57:39.086423+0000","last_clean":"2024-04-14T23:57:39.086423+0000","last_became_active":"2024-04-14T23:48:43.230253+0000","last_became_peered":"2024-04-14T23:48:43.230253+0000","last_unstale":"2024-04-14T23:57:39.086423+0000","last_undegraded":"2024-04-14T23:57:39.086423+0000","last_fullsized":"2024-04-14T23:57:39.086423+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":151,"log_dups_size":0,"ondisk_log_size":151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:04:12.081827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011901889999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":28,"num_read_kb":14,"num_write":150,"num_write_kb":4170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"45'707","reported_seq":803,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:07.570329+0000","last_change":"2024-04-14T23:48:42.237069+0000","last_active":"2024-04-14T23:58:07.570329+0000","last_peered":"2024-04-14T23:58:07.570329+0000","last_clean":"2024-04-14T23:58:07.570329+0000","last_became_active":"2024-04-14T23:48:42.236804+0000","last_became_peered":"2024-04-14T23:48:42.236804+0000","last_unstale":"2024-04-14T23:58:07.570329+0000","last_undegraded":"2024-04-14T23:58:07.570329+0000","last_fullsized":"2024-04-14T23:58:07.570329+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":707,"log_dups_size":0,"ondisk_log_size":707,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:25:37.318379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583584,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":41866,"num_write":698,"num_write_kb":12861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"44'154","reported_seq":2275,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.085057+0000","last_change":"2024-04-14T23:55:41.994228+0000","last_active":"2024-04-14T23:57:39.085057+0000","last_peered":"2024-04-14T23:57:39.085057+0000","last_clean":"2024-04-14T23:57:39.085057+0000","last_became_active":"2024-04-14T23:48:43.221462+0000","last_became_peered":"2024-04-14T23:48:43.221462+0000","last_unstale":"2024-04-14T23:57:39.085057+0000","last_undegraded":"2024-04-14T23:57:39.085057+0000","last_fullsized":"2024-04-14T23:57:39.085057+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:09:26.659126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033453850000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":154,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"45'104","reported_seq":197,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:13.839366+0000","last_change":"2024-04-14T23:58:13.839436+0000","last_active":"2024-04-14T23:58:13.839366+0000","last_peered":"2024-04-14T23:58:13.839366+0000","last_clean":"2024-04-14T23:58:13.839366+0000","last_became_active":"2024-04-14T23:48:42.229758+0000","last_became_peered":"2024-04-14T23:48:42.229758+0000","last_unstale":"2024-04-14T23:58:13.839366+0000","last_undegraded":"2024-04-14T23:58:13.839366+0000","last_fullsized":"2024-04-14T23:58:13.839366+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'104","last_scrub_stamp":"2024-04-14T23:58:13.839195+0000","last_deep_scrub":"45'104","last_deep_scrub_stamp":"2024-04-14T23:58:13.839195+0000","last_clean_scrub_stamp":"2024-04-14T23:58:13.839195+0000","objects_scrubbed":4,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T09:26:22.202312+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388642,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":28,"num_read_kb":32808,"num_write":95,"num_write_kb":8238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"45'163","reported_seq":2293,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:08.586016+0000","last_change":"2024-04-14T23:55:42.319479+0000","last_active":"2024-04-14T23:58:08.586016+0000","last_peered":"2024-04-14T23:58:08.586016+0000","last_clean":"2024-04-14T23:58:08.586016+0000","last_became_active":"2024-04-14T23:48:43.229476+0000","last_became_peered":"2024-04-14T23:48:43.229476+0000","last_unstale":"2024-04-14T23:58:08.586016+0000","last_undegraded":"2024-04-14T23:58:08.586016+0000","last_fullsized":"2024-04-14T23:58:08.586016+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":163,"log_dups_size":0,"ondisk_log_size":163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015192783,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":37,"num_read_kb":4115,"num_write":158,"num_write_kb":12367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":66,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.084193+0000","last_change":"2024-04-14T23:48:42.225404+0000","last_active":"2024-04-14T23:57:39.084193+0000","last_peered":"2024-04-14T23:57:39.084193+0000","last_clean":"2024-04-14T23:57:39.084193+0000","last_became_active":"2024-04-14T23:48:42.225211+0000","last_became_peered":"2024-04-14T23:48:42.225211+0000","last_unstale":"2024-04-14T23:57:39.084193+0000","last_undegraded":"2024-04-14T23:57:39.084193+0000","last_fullsized":"2024-04-14T23:57:39.084193+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:49:45.282259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"36'172","reported_seq":2308,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.085033+0000","last_change":"2024-04-14T23:55:41.993454+0000","last_active":"2024-04-14T23:57:39.085033+0000","last_peered":"2024-04-14T23:57:39.085033+0000","last_clean":"2024-04-14T23:57:39.085033+0000","last_became_active":"2024-04-14T23:48:43.229297+0000","last_became_peered":"2024-04-14T23:48:43.229297+0000","last_unstale":"2024-04-14T23:57:39.085033+0000","last_undegraded":"2024-04-14T23:57:39.085033+0000","last_fullsized":"2024-04-14T23:57:39.085033+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:21:50.218321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029516669999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":43,"num_read_kb":23,"num_write":172,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"45'4","reported_seq":70,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:07.579346+0000","last_change":"2024-04-14T23:48:42.220492+0000","last_active":"2024-04-14T23:58:07.579346+0000","last_peered":"2024-04-14T23:58:07.579346+0000","last_clean":"2024-04-14T23:58:07.579346+0000","last_became_active":"2024-04-14T23:48:42.220145+0000","last_became_peered":"2024-04-14T23:48:42.220145+0000","last_unstale":"2024-04-14T23:58:07.579346+0000","last_undegraded":"2024-04-14T23:58:07.579346+0000","last_fullsized":"2024-04-14T23:58:07.579346+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:17:45.278993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"45'169","reported_seq":2297,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:05.264010+0000","last_change":"2024-04-14T23:55:41.992426+0000","last_active":"2024-04-14T23:58:05.264010+0000","last_peered":"2024-04-14T23:58:05.264010+0000","last_clean":"2024-04-14T23:58:05.264010+0000","last_became_active":"2024-04-14T23:48:43.235319+0000","last_became_peered":"2024-04-14T23:48:43.235319+0000","last_unstale":"2024-04-14T23:58:05.264010+0000","last_undegraded":"2024-04-14T23:58:05.264010+0000","last_fullsized":"2024-04-14T23:58:05.264010+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":169,"log_dups_size":0,"ondisk_log_size":169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:38:08.635546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017199240000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":36,"num_read_kb":14,"num_write":164,"num_write_kb":82,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"35'101","reported_seq":200,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:04.373922+0000","last_change":"2024-04-14T23:48:42.231178+0000","last_active":"2024-04-14T23:58:04.373922+0000","last_peered":"2024-04-14T23:58:04.373922+0000","last_clean":"2024-04-14T23:58:04.373922+0000","last_became_active":"2024-04-14T23:48:42.230792+0000","last_became_peered":"2024-04-14T23:48:42.230792+0000","last_unstale":"2024-04-14T23:58:04.373922+0000","last_undegraded":"2024-04-14T23:58:04.373922+0000","last_fullsized":"2024-04-14T23:58:04.373922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":84,"num_read_kb":20558,"num_write":102,"num_write_kb":8239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"45'182","reported_seq":2316,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:05.264346+0000","last_change":"2024-04-14T23:55:41.993119+0000","last_active":"2024-04-14T23:58:05.264346+0000","last_peered":"2024-04-14T23:58:05.264346+0000","last_clean":"2024-04-14T23:58:05.264346+0000","last_became_active":"2024-04-14T23:48:43.228004+0000","last_became_peered":"2024-04-14T23:48:43.228004+0000","last_unstale":"2024-04-14T23:58:05.264346+0000","last_undegraded":"2024-04-14T23:58:05.264346+0000","last_fullsized":"2024-04-14T23:58:05.264346+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":182,"log_dups_size":0,"ondisk_log_size":182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:09:06.909970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003032402,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":44,"num_read_kb":8209,"num_write":176,"num_write_kb":12374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"34'22","reported_seq":96,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:44.962330+0000","last_change":"2024-04-14T23:48:42.231936+0000","last_active":"2024-04-14T23:57:44.962330+0000","last_peered":"2024-04-14T23:57:44.962330+0000","last_clean":"2024-04-14T23:57:44.962330+0000","last_became_active":"2024-04-14T23:48:42.231612+0000","last_became_peered":"2024-04-14T23:48:42.231612+0000","last_unstale":"2024-04-14T23:57:44.962330+0000","last_undegraded":"2024-04-14T23:57:44.962330+0000","last_fullsized":"2024-04-14T23:57:44.962330+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:45:03.098024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194326,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":12293,"num_write":21,"num_write_kb":4101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"36'127","reported_seq":2246,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.086202+0000","last_change":"2024-04-14T23:55:41.993658+0000","last_active":"2024-04-14T23:57:39.086202+0000","last_peered":"2024-04-14T23:57:39.086202+0000","last_clean":"2024-04-14T23:57:39.086202+0000","last_became_active":"2024-04-14T23:48:43.228665+0000","last_became_peered":"2024-04-14T23:48:43.228665+0000","last_unstale":"2024-04-14T23:57:39.086202+0000","last_undegraded":"2024-04-14T23:57:39.086202+0000","last_fullsized":"2024-04-14T23:57:39.086202+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T23:49:58.331034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003047124,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":27,"num_read_kb":4107,"num_write":124,"num_write_kb":4157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"45'101","reported_seq":178,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:07.590474+0000","last_change":"2024-04-14T23:48:42.239794+0000","last_active":"2024-04-14T23:58:07.590474+0000","last_peered":"2024-04-14T23:58:07.590474+0000","last_clean":"2024-04-14T23:58:07.590474+0000","last_became_active":"2024-04-14T23:48:42.239505+0000","last_became_peered":"2024-04-14T23:48:42.239505+0000","last_unstale":"2024-04-14T23:58:07.590474+0000","last_undegraded":"2024-04-14T23:58:07.590474+0000","last_fullsized":"2024-04-14T23:58:07.590474+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":24,"num_read_kb":12309,"num_write":100,"num_write_kb":4158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"45'165","reported_seq":2294,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:02.544601+0000","last_change":"2024-04-14T23:55:41.994666+0000","last_active":"2024-04-14T23:58:02.544601+0000","last_peered":"2024-04-14T23:58:02.544601+0000","last_clean":"2024-04-14T23:58:02.544601+0000","last_became_active":"2024-04-14T23:48:43.229949+0000","last_became_peered":"2024-04-14T23:48:43.229949+0000","last_unstale":"2024-04-14T23:58:02.544601+0000","last_undegraded":"2024-04-14T23:58:02.544601+0000","last_fullsized":"2024-04-14T23:58:02.544601+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:26:59.738426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002291033,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":37,"num_read_kb":4115,"num_write":160,"num_write_kb":4175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"45'20","reported_seq":104,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:14.041076+0000","last_change":"2024-04-14T23:58:14.041148+0000","last_active":"2024-04-14T23:58:14.041076+0000","last_peered":"2024-04-14T23:58:14.041076+0000","last_clean":"2024-04-14T23:58:14.041076+0000","last_became_active":"2024-04-14T23:48:42.226845+0000","last_became_peered":"2024-04-14T23:48:42.226845+0000","last_unstale":"2024-04-14T23:58:14.041076+0000","last_undegraded":"2024-04-14T23:58:14.041076+0000","last_fullsized":"2024-04-14T23:58:14.041076+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'20","last_scrub_stamp":"2024-04-14T23:58:14.040948+0000","last_deep_scrub":"45'20","last_deep_scrub_stamp":"2024-04-14T23:58:14.040948+0000","last_clean_scrub_stamp":"2024-04-14T23:58:14.040948+0000","objects_scrubbed":2,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:46:15.430380+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194374,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":9,"num_read_kb":16389,"num_write":18,"num_write_kb":4108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"43'183","reported_seq":2304,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.084349+0000","last_change":"2024-04-14T23:55:42.084740+0000","last_active":"2024-04-14T23:57:39.084349+0000","last_peered":"2024-04-14T23:57:39.084349+0000","last_clean":"2024-04-14T23:57:39.084349+0000","last_became_active":"2024-04-14T23:48:43.226717+0000","last_became_peered":"2024-04-14T23:48:43.226717+0000","last_unstale":"2024-04-14T23:57:39.084349+0000","last_undegraded":"2024-04-14T23:57:39.084349+0000","last_fullsized":"2024-04-14T23:57:39.084349+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:44:01.195501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022579950000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":28,"num_read_kb":14,"num_write":180,"num_write_kb":90,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"34'2","reported_seq":69,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:44.370805+0000","last_change":"2024-04-14T23:48:42.232164+0000","last_active":"2024-04-14T23:57:44.370805+0000","last_peered":"2024-04-14T23:57:44.370805+0000","last_clean":"2024-04-14T23:57:44.370805+0000","last_became_active":"2024-04-14T23:48:42.229785+0000","last_became_peered":"2024-04-14T23:48:42.229785+0000","last_unstale":"2024-04-14T23:57:44.370805+0000","last_undegraded":"2024-04-14T23:57:44.370805+0000","last_fullsized":"2024-04-14T23:57:44.370805+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"45'174","reported_seq":2298,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:08.136883+0000","last_change":"2024-04-14T23:55:41.994270+0000","last_active":"2024-04-14T23:58:08.136883+0000","last_peered":"2024-04-14T23:58:08.136883+0000","last_clean":"2024-04-14T23:58:08.136883+0000","last_became_active":"2024-04-14T23:48:43.232612+0000","last_became_peered":"2024-04-14T23:48:43.232612+0000","last_unstale":"2024-04-14T23:58:08.136883+0000","last_undegraded":"2024-04-14T23:58:08.136883+0000","last_fullsized":"2024-04-14T23:58:08.136883+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":174,"log_dups_size":0,"ondisk_log_size":174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T04:34:43.974518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031586029999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":31,"num_read_kb":17,"num_write":174,"num_write_kb":7493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":222,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.085334+0000","last_change":"2024-04-14T23:48:38.150200+0000","last_active":"2024-04-14T23:57:39.085334+0000","last_peered":"2024-04-14T23:57:39.085334+0000","last_clean":"2024-04-14T23:57:39.085334+0000","last_became_active":"2024-04-14T23:48:38.149301+0000","last_became_peered":"2024-04-14T23:48:38.149301+0000","last_unstale":"2024-04-14T23:57:39.085334+0000","last_undegraded":"2024-04-14T23:57:39.085334+0000","last_fullsized":"2024-04-14T23:57:39.085334+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:37.129663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:37.129663+0000","last_clean_scrub_stamp":"2024-04-14T23:48:37.129663+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T23:58:01.303851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"45'4","reported_seq":70,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.083463+0000","last_change":"2024-04-14T23:48:42.229238+0000","last_active":"2024-04-14T23:57:39.083463+0000","last_peered":"2024-04-14T23:57:39.083463+0000","last_clean":"2024-04-14T23:57:39.083463+0000","last_became_active":"2024-04-14T23:48:42.228994+0000","last_became_peered":"2024-04-14T23:48:42.228994+0000","last_unstale":"2024-04-14T23:57:39.083463+0000","last_undegraded":"2024-04-14T23:57:39.083463+0000","last_fullsized":"2024-04-14T23:57:39.083463+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:47:15.406228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"45'164","reported_seq":2291,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:07.588184+0000","last_change":"2024-04-14T23:55:42.318283+0000","last_active":"2024-04-14T23:58:07.588184+0000","last_peered":"2024-04-14T23:58:07.588184+0000","last_clean":"2024-04-14T23:58:07.588184+0000","last_became_active":"2024-04-14T23:48:43.223665+0000","last_became_peered":"2024-04-14T23:48:43.223665+0000","last_unstale":"2024-04-14T23:58:07.588184+0000","last_undegraded":"2024-04-14T23:58:07.588184+0000","last_fullsized":"2024-04-14T23:58:07.588184+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012914624,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":33,"num_read_kb":14,"num_write":162,"num_write_kb":4180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"45'101","reported_seq":173,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:12.569341+0000","last_change":"2024-04-14T23:48:42.230404+0000","last_active":"2024-04-14T23:58:12.569341+0000","last_peered":"2024-04-14T23:58:12.569341+0000","last_clean":"2024-04-14T23:58:12.569341+0000","last_became_active":"2024-04-14T23:48:42.230173+0000","last_became_peered":"2024-04-14T23:48:42.230173+0000","last_unstale":"2024-04-14T23:58:12.569341+0000","last_undegraded":"2024-04-14T23:58:12.569341+0000","last_fullsized":"2024-04-14T23:58:12.569341+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:23:45.666680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8882093,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":92,"num_write_kb":8721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"42'152","reported_seq":2273,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.088134+0000","last_change":"2024-04-14T23:55:42.315208+0000","last_active":"2024-04-14T23:57:39.088134+0000","last_peered":"2024-04-14T23:57:39.088134+0000","last_clean":"2024-04-14T23:57:39.088134+0000","last_became_active":"2024-04-14T23:48:43.231025+0000","last_became_peered":"2024-04-14T23:48:43.231025+0000","last_unstale":"2024-04-14T23:57:39.088134+0000","last_undegraded":"2024-04-14T23:57:39.088134+0000","last_fullsized":"2024-04-14T23:57:39.088134+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014709888000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":27,"num_read_kb":10,"num_write":150,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"45'135","reported_seq":336,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:08.577623+0000","last_change":"2024-04-14T23:48:42.240651+0000","last_active":"2024-04-14T23:58:08.577623+0000","last_peered":"2024-04-14T23:58:08.577623+0000","last_clean":"2024-04-14T23:58:08.577623+0000","last_became_active":"2024-04-14T23:48:42.240351+0000","last_became_peered":"2024-04-14T23:48:42.240351+0000","last_unstale":"2024-04-14T23:58:08.577623+0000","last_undegraded":"2024-04-14T23:58:08.577623+0000","last_fullsized":"2024-04-14T23:58:08.577623+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536520,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":134,"num_read_kb":610,"num_write":129,"num_write_kb":610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"44'152","reported_seq":2275,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.086360+0000","last_change":"2024-04-14T23:55:42.306590+0000","last_active":"2024-04-14T23:57:39.086360+0000","last_peered":"2024-04-14T23:57:39.086360+0000","last_clean":"2024-04-14T23:57:39.086360+0000","last_became_active":"2024-04-14T23:48:43.232120+0000","last_became_peered":"2024-04-14T23:48:43.232120+0000","last_unstale":"2024-04-14T23:57:39.086360+0000","last_undegraded":"2024-04-14T23:57:39.086360+0000","last_fullsized":"2024-04-14T23:57:39.086360+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0096232650000000006,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":29,"num_read_kb":17,"num_write":152,"num_write_kb":4171,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"34'156","reported_seq":228,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:44.101825+0000","last_change":"2024-04-14T23:48:42.233743+0000","last_active":"2024-04-14T23:57:44.101825+0000","last_peered":"2024-04-14T23:57:44.101825+0000","last_clean":"2024-04-14T23:57:44.101825+0000","last_became_active":"2024-04-14T23:48:42.233354+0000","last_became_peered":"2024-04-14T23:48:42.233354+0000","last_unstale":"2024-04-14T23:57:44.101825+0000","last_undegraded":"2024-04-14T23:57:44.101825+0000","last_fullsized":"2024-04-14T23:57:44.101825+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:49:23.045968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":154,"num_write_kb":8284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"45'184","reported_seq":2315,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:02.542594+0000","last_change":"2024-04-14T23:55:42.224478+0000","last_active":"2024-04-14T23:58:02.542594+0000","last_peered":"2024-04-14T23:58:02.542594+0000","last_clean":"2024-04-14T23:58:02.542594+0000","last_became_active":"2024-04-14T23:48:43.232119+0000","last_became_peered":"2024-04-14T23:48:43.232119+0000","last_unstale":"2024-04-14T23:58:02.542594+0000","last_undegraded":"2024-04-14T23:58:02.542594+0000","last_fullsized":"2024-04-14T23:58:02.542594+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":184,"log_dups_size":0,"ondisk_log_size":184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:37:54.631325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012271420000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":38,"num_read_kb":16,"num_write":184,"num_write_kb":92,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"45'4","reported_seq":71,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:07.591646+0000","last_change":"2024-04-14T23:48:42.242249+0000","last_active":"2024-04-14T23:58:07.591646+0000","last_peered":"2024-04-14T23:58:07.591646+0000","last_clean":"2024-04-14T23:58:07.591646+0000","last_became_active":"2024-04-14T23:48:42.242039+0000","last_became_peered":"2024-04-14T23:48:42.242039+0000","last_unstale":"2024-04-14T23:58:07.591646+0000","last_undegraded":"2024-04-14T23:58:07.591646+0000","last_fullsized":"2024-04-14T23:58:07.591646+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"44'162","reported_seq":2291,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.084067+0000","last_change":"2024-04-14T23:55:42.085024+0000","last_active":"2024-04-14T23:57:39.084067+0000","last_peered":"2024-04-14T23:57:39.084067+0000","last_clean":"2024-04-14T23:57:39.084067+0000","last_became_active":"2024-04-14T23:48:43.241776+0000","last_became_peered":"2024-04-14T23:48:43.241776+0000","last_unstale":"2024-04-14T23:57:39.084067+0000","last_undegraded":"2024-04-14T23:57:39.084067+0000","last_fullsized":"2024-04-14T23:57:39.084067+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:05:29.929626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024520940000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":36,"num_read_kb":17,"num_write":162,"num_write_kb":8271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"45'30","reported_seq":99,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:46.454715+0000","last_change":"2024-04-14T23:48:42.232537+0000","last_active":"2024-04-14T23:57:46.454715+0000","last_peered":"2024-04-14T23:57:46.454715+0000","last_clean":"2024-04-14T23:57:46.454715+0000","last_became_active":"2024-04-14T23:48:42.232325+0000","last_became_peered":"2024-04-14T23:48:42.232325+0000","last_unstale":"2024-04-14T23:57:46.454715+0000","last_undegraded":"2024-04-14T23:57:46.454715+0000","last_fullsized":"2024-04-14T23:57:46.454715+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:34:37.448301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":25,"num_write_kb":4108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"44'166","reported_seq":2289,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.086595+0000","last_change":"2024-04-14T23:55:42.308941+0000","last_active":"2024-04-14T23:57:39.086595+0000","last_peered":"2024-04-14T23:57:39.086595+0000","last_clean":"2024-04-14T23:57:39.086595+0000","last_became_active":"2024-04-14T23:48:43.234150+0000","last_became_peered":"2024-04-14T23:48:43.234150+0000","last_unstale":"2024-04-14T23:57:39.086595+0000","last_undegraded":"2024-04-14T23:57:39.086595+0000","last_fullsized":"2024-04-14T23:57:39.086595+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":166,"log_dups_size":0,"ondisk_log_size":166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011050757,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":29,"num_read_kb":14,"num_write":166,"num_write_kb":4178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"45'186","reported_seq":259,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:08.555123+0000","last_change":"2024-04-14T23:48:42.230014+0000","last_active":"2024-04-14T23:58:08.555123+0000","last_peered":"2024-04-14T23:58:08.555123+0000","last_clean":"2024-04-14T23:58:08.555123+0000","last_became_active":"2024-04-14T23:48:42.229712+0000","last_became_peered":"2024-04-14T23:48:42.229712+0000","last_unstale":"2024-04-14T23:58:08.555123+0000","last_undegraded":"2024-04-14T23:58:08.555123+0000","last_fullsized":"2024-04-14T23:58:08.555123+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":186,"log_dups_size":0,"ondisk_log_size":186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":142,"num_write":420,"num_write_kb":618,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"44'166","reported_seq":2298,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.094147+0000","last_change":"2024-04-14T23:55:42.320181+0000","last_active":"2024-04-14T23:57:39.094147+0000","last_peered":"2024-04-14T23:57:39.094147+0000","last_clean":"2024-04-14T23:57:39.094147+0000","last_became_active":"2024-04-14T23:48:43.228739+0000","last_became_peered":"2024-04-14T23:48:43.228739+0000","last_unstale":"2024-04-14T23:57:39.094147+0000","last_undegraded":"2024-04-14T23:57:39.094147+0000","last_fullsized":"2024-04-14T23:57:39.094147+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":166,"log_dups_size":0,"ondisk_log_size":166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012444432,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":38,"num_read_kb":16,"num_write":166,"num_write_kb":4178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"45'209","reported_seq":286,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:03.088717+0000","last_change":"2024-04-14T23:48:42.220035+0000","last_active":"2024-04-14T23:58:03.088717+0000","last_peered":"2024-04-14T23:58:03.088717+0000","last_clean":"2024-04-14T23:58:03.088717+0000","last_became_active":"2024-04-14T23:48:42.219750+0000","last_became_peered":"2024-04-14T23:48:42.219750+0000","last_unstale":"2024-04-14T23:58:03.088717+0000","last_undegraded":"2024-04-14T23:58:03.088717+0000","last_fullsized":"2024-04-14T23:58:03.088717+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":209,"log_dups_size":0,"ondisk_log_size":209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:00:58.928166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":25,"num_read_kb":6094,"num_write":202,"num_write_kb":4203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"45'178","reported_seq":2316,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:08.586465+0000","last_change":"2024-04-14T23:55:42.319020+0000","last_active":"2024-04-14T23:58:08.586465+0000","last_peered":"2024-04-14T23:58:08.586465+0000","last_clean":"2024-04-14T23:58:08.586465+0000","last_became_active":"2024-04-14T23:48:43.232730+0000","last_became_peered":"2024-04-14T23:48:43.232730+0000","last_unstale":"2024-04-14T23:58:08.586465+0000","last_undegraded":"2024-04-14T23:58:08.586465+0000","last_fullsized":"2024-04-14T23:58:08.586465+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":178,"log_dups_size":0,"ondisk_log_size":178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014562113999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":45,"num_read_kb":26,"num_write":173,"num_write_kb":8277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'28","reported_seq":97,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.084866+0000","last_change":"2024-04-14T23:48:42.232087+0000","last_active":"2024-04-14T23:57:39.084866+0000","last_peered":"2024-04-14T23:57:39.084866+0000","last_clean":"2024-04-14T23:57:39.084866+0000","last_became_active":"2024-04-14T23:48:42.231796+0000","last_became_peered":"2024-04-14T23:48:42.231796+0000","last_unstale":"2024-04-14T23:57:39.084866+0000","last_undegraded":"2024-04-14T23:57:39.084866+0000","last_fullsized":"2024-04-14T23:57:39.084866+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:14:35.050093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":27,"num_write_kb":4121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"43'183","reported_seq":2313,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.085628+0000","last_change":"2024-04-14T23:55:42.224366+0000","last_active":"2024-04-14T23:57:39.085628+0000","last_peered":"2024-04-14T23:57:39.085628+0000","last_clean":"2024-04-14T23:57:39.085628+0000","last_became_active":"2024-04-14T23:48:43.232594+0000","last_became_peered":"2024-04-14T23:48:43.232594+0000","last_unstale":"2024-04-14T23:57:39.085628+0000","last_undegraded":"2024-04-14T23:57:39.085628+0000","last_fullsized":"2024-04-14T23:57:39.085628+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T09:41:55.211912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001144237,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":37,"num_read_kb":20,"num_write":182,"num_write_kb":91,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"44'240","reported_seq":309,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:45.632033+0000","last_change":"2024-04-14T23:48:42.231237+0000","last_active":"2024-04-14T23:57:45.632033+0000","last_peered":"2024-04-14T23:57:45.632033+0000","last_clean":"2024-04-14T23:57:45.632033+0000","last_became_active":"2024-04-14T23:48:42.230902+0000","last_became_peered":"2024-04-14T23:48:42.230902+0000","last_unstale":"2024-04-14T23:57:45.632033+0000","last_undegraded":"2024-04-14T23:57:45.632033+0000","last_fullsized":"2024-04-14T23:57:45.632033+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":240,"log_dups_size":0,"ondisk_log_size":240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":238,"num_write_kb":4263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"44'168","reported_seq":2295,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.092325+0000","last_change":"2024-04-14T23:55:42.316834+0000","last_active":"2024-04-14T23:57:39.092325+0000","last_peered":"2024-04-14T23:57:39.092325+0000","last_clean":"2024-04-14T23:57:39.092325+0000","last_became_active":"2024-04-14T23:48:43.232747+0000","last_became_peered":"2024-04-14T23:48:43.232747+0000","last_unstale":"2024-04-14T23:57:39.092325+0000","last_undegraded":"2024-04-14T23:57:39.092325+0000","last_fullsized":"2024-04-14T23:57:39.092325+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014090612000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":33,"num_read_kb":16,"num_write":168,"num_write_kb":8274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"45'25","reported_seq":96,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:07.588062+0000","last_change":"2024-04-14T23:48:42.240496+0000","last_active":"2024-04-14T23:58:07.588062+0000","last_peered":"2024-04-14T23:58:07.588062+0000","last_clean":"2024-04-14T23:58:07.588062+0000","last_became_active":"2024-04-14T23:48:42.238524+0000","last_became_peered":"2024-04-14T23:48:42.238524+0000","last_unstale":"2024-04-14T23:58:07.588062+0000","last_undegraded":"2024-04-14T23:58:07.588062+0000","last_fullsized":"2024-04-14T23:58:07.588062+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":23,"num_write_kb":4102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"45'166","reported_seq":2295,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:08.587115+0000","last_change":"2024-04-14T23:55:42.318594+0000","last_active":"2024-04-14T23:58:08.587115+0000","last_peered":"2024-04-14T23:58:08.587115+0000","last_clean":"2024-04-14T23:58:08.587115+0000","last_became_active":"2024-04-14T23:48:43.233651+0000","last_became_peered":"2024-04-14T23:48:43.233651+0000","last_unstale":"2024-04-14T23:58:08.587115+0000","last_undegraded":"2024-04-14T23:58:08.587115+0000","last_fullsized":"2024-04-14T23:58:08.587115+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":166,"log_dups_size":0,"ondisk_log_size":166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015058134000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":13,"num_write":166,"num_write_kb":8274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"45'71","reported_seq":149,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:45.262156+0000","last_change":"2024-04-14T23:48:42.218897+0000","last_active":"2024-04-14T23:57:45.262156+0000","last_peered":"2024-04-14T23:57:45.262156+0000","last_clean":"2024-04-14T23:57:45.262156+0000","last_became_active":"2024-04-14T23:48:42.218580+0000","last_became_peered":"2024-04-14T23:48:42.218580+0000","last_unstale":"2024-04-14T23:57:45.262156+0000","last_undegraded":"2024-04-14T23:57:45.262156+0000","last_fullsized":"2024-04-14T23:57:45.262156+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:26:44.477638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":28682,"num_write":62,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"35'166","reported_seq":2282,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.010030+0000","last_change":"2024-04-14T23:55:41.991867+0000","last_active":"2024-04-14T23:57:39.010030+0000","last_peered":"2024-04-14T23:57:39.010030+0000","last_clean":"2024-04-14T23:57:39.010030+0000","last_became_active":"2024-04-14T23:48:43.219601+0000","last_became_peered":"2024-04-14T23:48:43.219601+0000","last_unstale":"2024-04-14T23:57:39.010030+0000","last_undegraded":"2024-04-14T23:57:39.010030+0000","last_fullsized":"2024-04-14T23:57:39.010030+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":166,"log_dups_size":0,"ondisk_log_size":166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:23:08.643833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001540055,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":23,"num_read_kb":13,"num_write":166,"num_write_kb":83,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"44'151","reported_seq":2286,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.083755+0000","last_change":"2024-04-14T23:55:42.084976+0000","last_active":"2024-04-14T23:57:39.083755+0000","last_peered":"2024-04-14T23:57:39.083755+0000","last_clean":"2024-04-14T23:57:39.083755+0000","last_became_active":"2024-04-14T23:48:43.221842+0000","last_became_peered":"2024-04-14T23:48:43.221842+0000","last_unstale":"2024-04-14T23:57:39.083755+0000","last_undegraded":"2024-04-14T23:57:39.083755+0000","last_fullsized":"2024-04-14T23:57:39.083755+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":151,"log_dups_size":0,"ondisk_log_size":151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:58:33.633046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024436879999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":43,"num_read_kb":23,"num_write":150,"num_write_kb":4170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"33'62","reported_seq":137,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:44.931184+0000","last_change":"2024-04-14T23:48:42.221314+0000","last_active":"2024-04-14T23:57:44.931184+0000","last_peered":"2024-04-14T23:57:44.931184+0000","last_clean":"2024-04-14T23:57:44.931184+0000","last_became_active":"2024-04-14T23:48:42.221028+0000","last_became_peered":"2024-04-14T23:48:42.221028+0000","last_unstale":"2024-04-14T23:57:44.931184+0000","last_undegraded":"2024-04-14T23:57:44.931184+0000","last_fullsized":"2024-04-14T23:57:44.931184+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T09:58:18.195400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":24,"num_read_kb":12309,"num_write":62,"num_write_kb":4128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"45'166","reported_seq":2289,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.083883+0000","last_change":"2024-04-14T23:55:42.084580+0000","last_active":"2024-04-14T23:57:39.083883+0000","last_peered":"2024-04-14T23:57:39.083883+0000","last_clean":"2024-04-14T23:57:39.083883+0000","last_became_active":"2024-04-14T23:48:43.224312+0000","last_became_peered":"2024-04-14T23:48:43.224312+0000","last_unstale":"2024-04-14T23:57:39.083883+0000","last_undegraded":"2024-04-14T23:57:39.083883+0000","last_fullsized":"2024-04-14T23:57:39.083883+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":166,"log_dups_size":0,"ondisk_log_size":166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:55:12.275566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00232957,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":31,"num_read_kb":13,"num_write":166,"num_write_kb":4182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"45'670","reported_seq":755,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:07.574713+0000","last_change":"2024-04-14T23:48:42.221000+0000","last_active":"2024-04-14T23:58:07.574713+0000","last_peered":"2024-04-14T23:58:07.574713+0000","last_clean":"2024-04-14T23:58:07.574713+0000","last_became_active":"2024-04-14T23:48:42.219813+0000","last_became_peered":"2024-04-14T23:48:42.219813+0000","last_unstale":"2024-04-14T23:58:07.574713+0000","last_undegraded":"2024-04-14T23:58:07.574713+0000","last_fullsized":"2024-04-14T23:58:07.574713+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":670,"log_dups_size":0,"ondisk_log_size":670,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:00:38.107953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":44198,"num_write":659,"num_write_kb":12787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"37'165","reported_seq":2294,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:57:39.088103+0000","last_change":"2024-04-14T23:55:42.313777+0000","last_active":"2024-04-14T23:57:39.088103+0000","last_peered":"2024-04-14T23:57:39.088103+0000","last_clean":"2024-04-14T23:57:39.088103+0000","last_became_active":"2024-04-14T23:48:43.236595+0000","last_became_peered":"2024-04-14T23:48:43.236595+0000","last_unstale":"2024-04-14T23:57:39.088103+0000","last_undegraded":"2024-04-14T23:57:39.088103+0000","last_fullsized":"2024-04-14T23:57:39.088103+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013391734000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":19,"num_write":164,"num_write_kb":82,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"45'48","reported_seq":120,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:07.590416+0000","last_change":"2024-04-14T23:48:42.239255+0000","last_active":"2024-04-14T23:58:07.590416+0000","last_peered":"2024-04-14T23:58:07.590416+0000","last_clean":"2024-04-14T23:58:07.590416+0000","last_became_active":"2024-04-14T23:48:42.238892+0000","last_became_peered":"2024-04-14T23:48:42.238892+0000","last_unstale":"2024-04-14T23:58:07.590416+0000","last_undegraded":"2024-04-14T23:58:07.590416+0000","last_fullsized":"2024-04-14T23:58:07.590416+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:41.176196+0000","last_clean_scrub_stamp":"2024-04-14T23:48:41.176196+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":16384,"num_write":44,"num_write_kb":4126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"45'176","reported_seq":2300,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:08.585228+0000","last_change":"2024-04-14T23:55:42.319487+0000","last_active":"2024-04-14T23:58:08.585228+0000","last_peered":"2024-04-14T23:58:08.585228+0000","last_clean":"2024-04-14T23:58:08.585228+0000","last_became_active":"2024-04-14T23:48:43.222707+0000","last_became_peered":"2024-04-14T23:48:43.222707+0000","last_unstale":"2024-04-14T23:58:08.585228+0000","last_undegraded":"2024-04-14T23:58:08.585228+0000","last_fullsized":"2024-04-14T23:58:08.585228+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:48:42.189846+0000","last_clean_scrub_stamp":"2024-04-14T23:48:42.189846+0000","objects_scrubbed":0,"log_size":176,"log_dups_size":0,"ondisk_log_size":176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011942019,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":30,"num_read_kb":12,"num_write":176,"num_write_kb":4183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"45'79","reported_seq":169,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:12.850896+0000","last_change":"2024-04-14T23:58:12.850977+0000","last_active":"2024-04-14T23:58:12.850896+0000","last_peered":"2024-04-14T23:58:12.850896+0000","last_clean":"2024-04-14T23:58:12.850896+0000","last_became_active":"2024-04-14T23:48:42.226238+0000","last_became_peered":"2024-04-14T23:48:42.226238+0000","last_unstale":"2024-04-14T23:58:12.850896+0000","last_undegraded":"2024-04-14T23:58:12.850896+0000","last_fullsized":"2024-04-14T23:58:12.850896+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'79","last_scrub_stamp":"2024-04-14T23:58:12.850700+0000","last_deep_scrub":"45'79","last_deep_scrub_stamp":"2024-04-14T23:58:12.850700+0000","last_clean_scrub_stamp":"2024-04-14T23:58:12.850700+0000","objects_scrubbed":3,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:09:52.524693+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":25,"num_read_kb":20500,"num_write":71,"num_write_kb":4136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2223,"num_read_kb":65817,"num_write":10661,"num_write_kb":287920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10759,"ondisk_log_size":10759,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":152026419,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":556,"num_read_kb":479121,"num_write":3811,"num_write_kb":151495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":64839680,"data_stored":304052838,"data_compressed":34430816,"data_compressed_allocated":44859392,"data_compressed_original":284131328,"omap_allocated":0,"internal_metadata":0},"log_size":3689,"ondisk_log_size":3689,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65128,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574967,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76420,"kb_used_data":13308,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94295420,"statfs":{"total":96636764160,"available":96558510080,"internally_reserved":0,"allocated":13627392,"data_stored":60044059,"data_compressed":6906850,"data_compressed_allocated":9056256,"data_compressed_original":55754752,"omap_allocated":1590,"internal_metadata":64616906},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574968,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57228,"kb_used_data":12164,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94314612,"statfs":{"total":96636764160,"available":96578162688,"internally_reserved":0,"allocated":12455936,"data_stored":50571688,"data_compressed":5492002,"data_compressed_allocated":7053312,"data_compressed_original":45449216,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574967,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73208,"kb_used_data":5936,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94298632,"statfs":{"total":96636764160,"available":96561799168,"internally_reserved":0,"allocated":6078464,"data_stored":29599586,"data_compressed":3110985,"data_compressed_allocated":4100096,"data_compressed_original":27897856,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574968,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76316,"kb_used_data":9044,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94295524,"statfs":{"total":96636764160,"available":96558616576,"internally_reserved":0,"allocated":9261056,"data_stored":46376838,"data_compressed":5258839,"data_compressed_allocated":6774784,"data_compressed_original":44175360,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574966,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":75612,"kb_used_data":8340,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94296228,"statfs":{"total":96636764160,"available":96559337472,"internally_reserved":0,"allocated":8540160,"data_stored":34330992,"data_compressed":3819253,"data_compressed_allocated":4960256,"data_compressed_original":31031296,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574967,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47116,"kb_used_data":3460,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94324724,"statfs":{"total":96636764160,"available":96588517376,"internally_reserved":0,"allocated":3543040,"data_stored":13358800,"data_compressed":1445694,"data_compressed_allocated":2031616,"data_compressed_original":12120064,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574967,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54240,"kb_used_data":9176,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94317600,"statfs":{"total":96636764160,"available":96581222400,"internally_reserved":0,"allocated":9396224,"data_stored":42676053,"data_compressed":4921274,"data_compressed_allocated":6492160,"data_compressed_original":40054784,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574966,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73292,"kb_used_data":6020,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94298548,"statfs":{"total":96636764160,"available":96561713152,"internally_reserved":0,"allocated":6164480,"data_stored":30190502,"data_compressed":3541047,"data_compressed_allocated":4481024,"data_compressed_original":28794880,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32564,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32564,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5586944,"data_stored":29360766,"data_compressed":3508483,"data_compressed_allocated":4435968,"data_compressed_original":28221440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8884224,"data_stored":42436685,"data_compressed":4921274,"data_compressed_allocated":6492160,"data_compressed_original":40054784,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3031040,"data_stored":13119432,"data_compressed":1445694,"data_compressed_allocated":2031616,"data_compressed_original":12120064,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8028160,"data_stored":34091624,"data_compressed":3819253,"data_compressed_allocated":4960256,"data_compressed_original":31031296,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8749056,"data_stored":46137470,"data_compressed":5258839,"data_compressed_allocated":6774784,"data_compressed_original":44175360,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":5566464,"data_stored":29360218,"data_compressed":3110985,"data_compressed_allocated":4100096,"data_compressed_original":27897856,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":11943936,"data_stored":50332320,"data_compressed":5492002,"data_compressed_allocated":7053312,"data_compressed_original":45449216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":13049856,"data_stored":59214323,"data_compressed":6874286,"data_compressed_allocated":9011200,"data_compressed_original":55181312,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-14T23:58:18.266 INFO:tasks.ceph:pgid 3.3f last_scrub_stamp 2024-04-14T23:48:42.189846+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.266 INFO:tasks.ceph:pgid 3.3e last_scrub_stamp 2024-04-14T23:48:42.189846+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.266 INFO:tasks.ceph:pgid 3.3d last_scrub_stamp 2024-04-14T23:48:42.189846+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.266 INFO:tasks.ceph:pgid 3.3c last_scrub_stamp 2024-04-14T23:48:42.189846+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.267 INFO:tasks.ceph:pgid 3.3b last_scrub_stamp 2024-04-14T23:48:42.189846+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.267 INFO:tasks.ceph:pgid 3.3a last_scrub_stamp 2024-04-14T23:48:42.189846+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.267 INFO:tasks.ceph:pgid 3.39 last_scrub_stamp 2024-04-14T23:48:42.189846+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.267 INFO:tasks.ceph:pgid 3.38 last_scrub_stamp 2024-04-14T23:48:42.189846+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.267 INFO:tasks.ceph:pgid 3.37 last_scrub_stamp 2024-04-14T23:48:42.189846+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.267 INFO:tasks.ceph:pgid 3.36 last_scrub_stamp 2024-04-14T23:48:42.189846+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.268 INFO:tasks.ceph:pgid 3.35 last_scrub_stamp 2024-04-14T23:48:42.189846+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.268 INFO:tasks.ceph:pgid 3.34 last_scrub_stamp 2024-04-14T23:48:42.189846+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.268 INFO:tasks.ceph:pgid 3.33 last_scrub_stamp 2024-04-14T23:48:42.189846+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.268 INFO:tasks.ceph:pgid 3.32 last_scrub_stamp 2024-04-14T23:48:42.189846+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.268 INFO:tasks.ceph:pgid 3.31 last_scrub_stamp 2024-04-14T23:48:42.189846+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.268 INFO:tasks.ceph:pgid 3.30 last_scrub_stamp 2024-04-14T23:48:42.189846+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.269 INFO:tasks.ceph:pgid 3.2f last_scrub_stamp 2024-04-14T23:48:42.189846+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.269 INFO:tasks.ceph:pgid 3.2e last_scrub_stamp 2024-04-14T23:48:42.189846+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.269 INFO:tasks.ceph:pgid 3.2d last_scrub_stamp 2024-04-14T23:48:42.189846+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.269 INFO:tasks.ceph:pgid 3.2c last_scrub_stamp 2024-04-14T23:48:42.189846+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.269 INFO:tasks.ceph:pgid 3.2b last_scrub_stamp 2024-04-14T23:48:42.189846+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.269 INFO:tasks.ceph:pgid 3.2a last_scrub_stamp 2024-04-14T23:48:42.189846+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.270 INFO:tasks.ceph:pgid 3.29 last_scrub_stamp 2024-04-14T23:48:42.189846+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.270 INFO:tasks.ceph:pgid 3.28 last_scrub_stamp 2024-04-14T23:48:42.189846+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.270 INFO:tasks.ceph:pgid 3.27 last_scrub_stamp 2024-04-14T23:48:42.189846+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.270 INFO:tasks.ceph:pgid 3.26 last_scrub_stamp 2024-04-14T23:48:42.189846+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.270 INFO:tasks.ceph:pgid 3.25 last_scrub_stamp 2024-04-14T23:48:42.189846+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.270 INFO:tasks.ceph:pgid 3.24 last_scrub_stamp 2024-04-14T23:48:42.189846+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.271 INFO:tasks.ceph:pgid 3.23 last_scrub_stamp 2024-04-14T23:48:42.189846+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.271 INFO:tasks.ceph:pgid 3.22 last_scrub_stamp 2024-04-14T23:48:42.189846+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.271 INFO:tasks.ceph:pgid 3.21 last_scrub_stamp 2024-04-14T23:48:42.189846+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.271 INFO:tasks.ceph:pgid 3.20 last_scrub_stamp 2024-04-14T23:48:42.189846+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.271 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2024-04-14T23:48:42.189846+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.271 INFO:tasks.ceph:pgid 2.1b last_scrub_stamp 2024-04-14T23:48:41.176196+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=41, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.272 INFO:tasks.ceph:pgid 3.1b last_scrub_stamp 2024-04-14T23:48:42.189846+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.272 INFO:tasks.ceph:pgid 2.1a last_scrub_stamp 2024-04-14T23:48:41.176196+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=41, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.272 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2024-04-14T23:48:42.189846+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.272 INFO:tasks.ceph:pgid 2.18 last_scrub_stamp 2024-04-14T23:48:41.176196+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=41, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.272 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2024-04-14T23:48:42.189846+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.272 INFO:tasks.ceph:pgid 2.17 last_scrub_stamp 2024-04-14T23:48:41.176196+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=41, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.273 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2024-04-14T23:48:42.189846+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.273 INFO:tasks.ceph:pgid 2.16 last_scrub_stamp 2024-04-14T23:48:41.176196+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=41, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.273 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2024-04-14T23:48:42.189846+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.273 INFO:tasks.ceph:pgid 2.15 last_scrub_stamp 2024-04-14T23:48:41.176196+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=41, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.273 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2024-04-14T23:48:42.189846+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.273 INFO:tasks.ceph:pgid 2.14 last_scrub_stamp 2024-04-14T23:48:41.176196+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=41, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.273 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2024-04-14T23:48:42.189846+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.274 INFO:tasks.ceph:pgid 3.12 last_scrub_stamp 2024-04-14T23:48:42.189846+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.274 INFO:tasks.ceph:pgid 2.12 last_scrub_stamp 2024-04-14T23:48:41.176196+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=41, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.274 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2024-04-14T23:48:42.189846+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.274 INFO:tasks.ceph:pgid 2.11 last_scrub_stamp 2024-04-14T23:48:41.176196+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=41, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.274 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2024-04-14T23:48:42.189846+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.275 INFO:tasks.ceph:pgid 2.10 last_scrub_stamp 2024-04-14T23:48:41.176196+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=41, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.275 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2024-04-14T23:48:42.189846+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.275 INFO:tasks.ceph:pgid 2.f last_scrub_stamp 2024-04-14T23:48:41.176196+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=41, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.275 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2024-04-14T23:48:42.189846+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.275 INFO:tasks.ceph:pgid 2.e last_scrub_stamp 2024-04-14T23:48:41.176196+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=41, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.275 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2024-04-14T23:48:42.189846+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.275 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2024-04-14T23:48:42.189846+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.276 INFO:tasks.ceph:pgid 2.c last_scrub_stamp 2024-04-14T23:48:41.176196+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=41, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.276 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2024-04-14T23:48:42.189846+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.276 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-14T23:48:37.129663+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=37, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.276 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-04-14T23:48:41.176196+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=41, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.276 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-04-14T23:48:42.189846+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.276 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-04-14T23:48:41.176196+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=41, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.277 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-04-14T23:48:42.189846+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.277 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-04-14T23:48:41.176196+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=41, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.277 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-04-14T23:48:42.189846+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.277 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-04-14T23:48:41.176196+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=41, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.277 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-04-14T23:48:42.189846+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.277 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-04-14T23:48:41.176196+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=41, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.278 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2024-04-14T23:48:42.189846+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.279 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-04-14T23:48:41.176196+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=41, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.279 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2024-04-14T23:48:42.189846+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.279 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-04-14T23:48:41.176196+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=41, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.279 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2024-04-14T23:48:42.189846+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.279 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-04-14T23:48:41.176196+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=41, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.279 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2024-04-14T23:48:42.189846+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.280 INFO:tasks.ceph:pgid 2.8 last_scrub_stamp 2024-04-14T23:48:41.176196+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=41, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.280 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2024-04-14T23:48:42.189846+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.280 INFO:tasks.ceph:pgid 2.9 last_scrub_stamp 2024-04-14T23:48:41.176196+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=41, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.280 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-04-14T23:48:42.189846+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.280 INFO:tasks.ceph:pgid 2.a last_scrub_stamp 2024-04-14T23:48:41.176196+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=41, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.280 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2024-04-14T23:48:42.189846+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.281 INFO:tasks.ceph:pgid 2.b last_scrub_stamp 2024-04-14T23:48:41.176196+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=41, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.281 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2024-04-14T23:48:42.189846+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.281 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2024-04-14T23:48:42.189846+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.281 INFO:tasks.ceph:pgid 2.1c last_scrub_stamp 2024-04-14T23:48:41.176196+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=41, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.281 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2024-04-14T23:48:42.189846+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.281 INFO:tasks.ceph:pgid 2.1d last_scrub_stamp 2024-04-14T23:48:41.176196+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=41, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.281 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2024-04-14T23:48:42.189846+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.282 INFO:tasks.ceph:pgid 2.1e last_scrub_stamp 2024-04-14T23:48:41.176196+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=41, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.282 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2024-04-14T23:48:42.189846+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=48, tm_sec=42, tm_wday=6, tm_yday=105, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=14, tm_hour=23, tm_min=58, tm_sec=10, tm_wday=6, tm_yday=105, tm_isdst=0) 2024-04-14T23:58:18.282 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-14T23:58:38.283 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-14T23:58:38.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.493+0000 7ff7a4a00640 1 Processor -- start 2024-04-14T23:58:38.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.494+0000 7ff7a4a00640 1 -- start start 2024-04-14T23:58:38.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.495+0000 7ff7a4a00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff7a0154b20 0x7ff7a00cbf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:38.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.495+0000 7ff7a4a00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff7a015e930 0x7ff7a015ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:38.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.495+0000 7ff7a4a00640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7a006cda0 con 0x7ff7a00cc720 2024-04-14T23:58:38.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.495+0000 7ff7a4a00640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7ff7a006d2f0 con 0x7ff7a0154b20 2024-04-14T23:58:38.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.495+0000 7ff7a4a00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7ff7a00cbc20 con 0x7ff7a015e930 2024-04-14T23:58:38.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.496+0000 7ff79f400640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff7a0154b20 0x7ff7a00cbf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:38.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.496+0000 7ff796a00640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7ff7a00cc720 0x7ff7a00d0230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.60:53766/0 (socket says 172.21.15.60:53766) 2024-04-14T23:58:38.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.496+0000 7ff796a00640 1 -- 172.21.15.60:0/1847420304 learned_addr learned my addr 172.21.15.60:0/1847420304 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:58:38.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.496+0000 7ff79fe00640 1 --2- 172.21.15.60:0/1847420304 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff7a015e930 0x7ff7a015ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:38.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.496+0000 7ff79ea00640 1 -- 172.21.15.60:0/1847420304 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4184349432 0 0) 0x7ff7a006cda0 con 0x7ff7a00cc720 2024-04-14T23:58:38.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.496+0000 7ff79f400640 1 -- 172.21.15.60:0/1847420304 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff7a015e930 msgr2=0x7ff7a015ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:58:38.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.496+0000 7ff79f400640 1 --2- 172.21.15.60:0/1847420304 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff7a015e930 0x7ff7a015ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:38.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.496+0000 7ff79f400640 1 -- 172.21.15.60:0/1847420304 >> v1:172.21.15.186:6789/0 conn(0x7ff7a00cc720 legacy=0x7ff7a00d0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:38.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.496+0000 7ff79f400640 1 -- 172.21.15.60:0/1847420304 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7a006d010 con 0x7ff7a0154b20 2024-04-14T23:58:38.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.496+0000 7ff79fe00640 1 --2- 172.21.15.60:0/1847420304 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff7a015e930 0x7ff7a015ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:58:38.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.498+0000 7ff79f400640 1 --2- 172.21.15.60:0/1847420304 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7a0154b20 0x7ff7a00cbf00 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7ff7900097a0 tx=0x7ff79002b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e46f92e3208091 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:38.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.498+0000 7ff79ea00640 1 -- 172.21.15.60:0/1847420304 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff790031780 con 0x7ff7a0154b20 2024-04-14T23:58:38.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.498+0000 7ff79ea00640 1 -- 172.21.15.60:0/1847420304 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff790037070 con 0x7ff7a0154b20 2024-04-14T23:58:38.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.498+0000 7ff79ea00640 1 -- 172.21.15.60:0/1847420304 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff790031b40 con 0x7ff7a0154b20 2024-04-14T23:58:38.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.499+0000 7ff7a4a00640 1 -- 172.21.15.60:0/1847420304 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7a0154b20 msgr2=0x7ff7a00cbf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:38.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.499+0000 7ff7a4a00640 1 --2- 172.21.15.60:0/1847420304 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7a0154b20 0x7ff7a00cbf00 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7ff7900097a0 tx=0x7ff79002b820 comp rx=0 tx=0).stop 2024-04-14T23:58:38.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.499+0000 7ff7a4a00640 1 -- 172.21.15.60:0/1847420304 shutdown_connections 2024-04-14T23:58:38.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.499+0000 7ff7a4a00640 1 --2- 172.21.15.60:0/1847420304 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff7a015e930 0x7ff7a015ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:38.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.499+0000 7ff7a4a00640 1 --2- 172.21.15.60:0/1847420304 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7a0154b20 0x7ff7a00cbf00 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:38.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.499+0000 7ff7a4a00640 1 -- 172.21.15.60:0/1847420304 >> 172.21.15.60:0/1847420304 conn(0x7ff7a006ddb0 msgr2=0x7ff7a0066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:38.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.499+0000 7ff7a4a00640 1 -- 172.21.15.60:0/1847420304 shutdown_connections 2024-04-14T23:58:38.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.499+0000 7ff7a4a00640 1 -- 172.21.15.60:0/1847420304 wait complete. 2024-04-14T23:58:38.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.501+0000 7ff7a4a00640 1 Processor -- start 2024-04-14T23:58:38.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.501+0000 7ff7a4a00640 1 -- start start 2024-04-14T23:58:38.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.502+0000 7ff7a4a00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff7a00cc720 0x7ff7a0155250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:38.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.502+0000 7ff7a4a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff7a0154b20 0x7ff7a015c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:38.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.502+0000 7ff7a4a00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7a015e930 0x7ff7a0158c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:38.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.502+0000 7ff7a4a00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff7a0155080 con 0x7ff7a015e930 2024-04-14T23:58:38.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.502+0000 7ff7a4a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff7a0154f00 con 0x7ff7a0154b20 2024-04-14T23:58:38.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.502+0000 7ff7a4a00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7ff7a00d0940 con 0x7ff7a00cc720 2024-04-14T23:58:38.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.502+0000 7ff79fe00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7a015e930 0x7ff7a0158c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:38.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.502+0000 7ff79fe00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7a015e930 0x7ff7a0158c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48658/0 (socket says 172.21.15.60:48658) 2024-04-14T23:58:38.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.502+0000 7ff79fe00640 1 -- 172.21.15.60:0/3269233744 learned_addr learned my addr 172.21.15.60:0/3269233744 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:58:38.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.502+0000 7ff79f400640 1 --2- 172.21.15.60:0/3269233744 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff7a00cc720 0x7ff7a0155250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:38.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.502+0000 7ff796a00640 1 --2- 172.21.15.60:0/3269233744 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff7a0154b20 0x7ff7a015c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:38.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.503+0000 7ff79fe00640 1 -- 172.21.15.60:0/3269233744 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff7a00cc720 msgr2=0x7ff7a0155250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:38.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.503+0000 7ff79fe00640 1 --2- 172.21.15.60:0/3269233744 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff7a00cc720 0x7ff7a0155250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:38.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.503+0000 7ff79fe00640 1 -- 172.21.15.60:0/3269233744 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff7a0154b20 msgr2=0x7ff7a015c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:38.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.503+0000 7ff79fe00640 1 --2- 172.21.15.60:0/3269233744 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff7a0154b20 0x7ff7a015c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:38.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.503+0000 7ff79fe00640 1 -- 172.21.15.60:0/3269233744 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7a01f0820 con 0x7ff7a015e930 2024-04-14T23:58:38.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.503+0000 7ff79fe00640 1 --2- 172.21.15.60:0/3269233744 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7a015e930 0x7ff7a0158c40 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7ff79800efc0 tx=0x7ff79800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:38.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.503+0000 7ff797e00640 1 -- 172.21.15.60:0/3269233744 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff798018030 con 0x7ff7a015e930 2024-04-14T23:58:38.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.503+0000 7ff797e00640 1 -- 172.21.15.60:0/3269233744 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff79800cd90 con 0x7ff7a015e930 2024-04-14T23:58:38.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.503+0000 7ff797e00640 1 -- 172.21.15.60:0/3269233744 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff798003bb0 con 0x7ff7a015e930 2024-04-14T23:58:38.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.503+0000 7ff7a4a00640 1 -- 172.21.15.60:0/3269233744 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7a01f0ae0 con 0x7ff7a015e930 2024-04-14T23:58:38.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.504+0000 7ff7a4a00640 1 -- 172.21.15.60:0/3269233744 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7a01f10a0 con 0x7ff7a015e930 2024-04-14T23:58:38.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.505+0000 7ff7a4a00640 1 -- 172.21.15.60:0/3269233744 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7a004f540 con 0x7ff7a015e930 2024-04-14T23:58:38.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.506+0000 7ff797e00640 1 -- 172.21.15.60:0/3269233744 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7ff79801b070 con 0x7ff7a015e930 2024-04-14T23:58:38.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.507+0000 7ff797e00640 1 --2- 172.21.15.60:0/3269233744 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff77c06b990 0x7ff77c06de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:38.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.507+0000 7ff797e00640 1 -- 172.21.15.60:0/3269233744 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6255+0+0 (secure 0 0 0) 0x7ff79808b370 con 0x7ff7a015e930 2024-04-14T23:58:38.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.507+0000 7ff79f400640 1 --2- 172.21.15.60:0/3269233744 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff77c06b990 0x7ff77c06de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:38.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.508+0000 7ff79f400640 1 --2- 172.21.15.60:0/3269233744 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff77c06b990 0x7ff77c06de50 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff790036000 tx=0x7ff79003d040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:38.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.511+0000 7ff797e00640 1 -- 172.21.15.60:0/3269233744 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff798010040 con 0x7ff7a015e930 2024-04-14T23:58:38.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.712+0000 7ff7a4a00640 1 -- 172.21.15.60:0/3269233744 --> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff7a01583a0 con 0x7ff77c06b990 2024-04-14T23:58:38.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.720+0000 7ff797e00640 1 -- 172.21.15.60:0/3269233744 <== mgr.4100 v2:172.21.15.186:6832/714104835 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+255502 (secure 0 0 0) 0x7ff7a01583a0 con 0x7ff77c06b990 2024-04-14T23:58:38.730 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:58:38.731 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-04-14T23:58:38.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.725+0000 7ff7a4a00640 1 -- 172.21.15.60:0/3269233744 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff77c06b990 msgr2=0x7ff77c06de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:38.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.725+0000 7ff7a4a00640 1 --2- 172.21.15.60:0/3269233744 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff77c06b990 0x7ff77c06de50 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff790036000 tx=0x7ff79003d040 comp rx=0 tx=0).stop 2024-04-14T23:58:38.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.725+0000 7ff7a4a00640 1 -- 172.21.15.60:0/3269233744 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7a015e930 msgr2=0x7ff7a0158c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:38.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.725+0000 7ff7a4a00640 1 --2- 172.21.15.60:0/3269233744 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7a015e930 0x7ff7a0158c40 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7ff79800efc0 tx=0x7ff79800c490 comp rx=0 tx=0).stop 2024-04-14T23:58:38.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.725+0000 7ff7a4a00640 1 -- 172.21.15.60:0/3269233744 shutdown_connections 2024-04-14T23:58:38.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.725+0000 7ff7a4a00640 1 --2- 172.21.15.60:0/3269233744 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7ff77c06b990 0x7ff77c06de50 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:38.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.725+0000 7ff7a4a00640 1 --2- 172.21.15.60:0/3269233744 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7a015e930 0x7ff7a0158c40 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:38.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.725+0000 7ff7a4a00640 1 --2- 172.21.15.60:0/3269233744 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff7a0154b20 0x7ff7a015c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:38.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.725+0000 7ff7a4a00640 1 --2- 172.21.15.60:0/3269233744 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff7a00cc720 0x7ff7a0155250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:38.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.725+0000 7ff7a4a00640 1 -- 172.21.15.60:0/3269233744 >> 172.21.15.60:0/3269233744 conn(0x7ff7a006ddb0 msgr2=0x7ff7a0066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:38.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.726+0000 7ff7a4a00640 1 -- 172.21.15.60:0/3269233744 shutdown_connections 2024-04-14T23:58:38.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:38.726+0000 7ff7a4a00640 1 -- 172.21.15.60:0/3269233744 wait complete. 2024-04-14T23:58:38.751 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":340,"stamp":"2024-04-14T23:58:38.321643+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":152616787,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":2895,"num_read_kb":545038,"num_write":14631,"num_write_kb":441831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5561,"num_omap_keys":11,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14611,"ondisk_log_size":14611,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":533432,"kb_used_data":67448,"kb_used_omap":12,"kb_used_meta":465907,"kb_avail":754441288,"statfs":{"total":773094113280,"available":772547878912,"internally_reserved":0,"allocated":69066752,"data_stored":307148518,"data_compressed":34495944,"data_compressed_allocated":44949504,"data_compressed_original":285278208,"omap_allocated":12752,"internal_metadata":477089328},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003197"},"pg_stats":[{"pgid":"3.3f","version":"36'158","reported_seq":2289,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:18.327271+0000","last_change":"2024-04-14T23:58:18.327314+0000","last_active":"2024-04-14T23:58:18.327271+0000","last_peered":"2024-04-14T23:58:18.327271+0000","last_clean":"2024-04-14T23:58:18.327271+0000","last_became_active":"2024-04-14T23:48:43.222278+0000","last_became_peered":"2024-04-14T23:48:43.222278+0000","last_unstale":"2024-04-14T23:58:18.327271+0000","last_undegraded":"2024-04-14T23:58:18.327271+0000","last_fullsized":"2024-04-14T23:58:18.327271+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'158","last_scrub_stamp":"2024-04-14T23:58:18.327156+0000","last_deep_scrub":"36'158","last_deep_scrub_stamp":"2024-04-14T23:58:18.327156+0000","last_clean_scrub_stamp":"2024-04-14T23:58:18.327156+0000","objects_scrubbed":0,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:13:55.757306+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0025657449999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":29,"num_read_kb":15,"num_write":158,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"44'142","reported_seq":2278,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:21.079656+0000","last_change":"2024-04-14T23:58:21.079715+0000","last_active":"2024-04-14T23:58:21.079656+0000","last_peered":"2024-04-14T23:58:21.079656+0000","last_clean":"2024-04-14T23:58:21.079656+0000","last_became_active":"2024-04-14T23:48:43.226426+0000","last_became_peered":"2024-04-14T23:48:43.226426+0000","last_unstale":"2024-04-14T23:58:21.079656+0000","last_undegraded":"2024-04-14T23:58:21.079656+0000","last_fullsized":"2024-04-14T23:58:21.079656+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'142","last_scrub_stamp":"2024-04-14T23:58:21.079474+0000","last_deep_scrub":"44'142","last_deep_scrub_stamp":"2024-04-14T23:58:21.079474+0000","last_clean_scrub_stamp":"2024-04-14T23:58:21.079474+0000","objects_scrubbed":0,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:20:04.301136+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.011741033999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":16,"num_write":142,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"37'171","reported_seq":2308,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:18.752668+0000","last_change":"2024-04-14T23:58:18.752785+0000","last_active":"2024-04-14T23:58:18.752668+0000","last_peered":"2024-04-14T23:58:18.752668+0000","last_clean":"2024-04-14T23:58:18.752668+0000","last_became_active":"2024-04-14T23:48:43.236616+0000","last_became_peered":"2024-04-14T23:48:43.236616+0000","last_unstale":"2024-04-14T23:58:18.752668+0000","last_undegraded":"2024-04-14T23:58:18.752668+0000","last_fullsized":"2024-04-14T23:58:18.752668+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'171","last_scrub_stamp":"2024-04-14T23:58:18.752420+0000","last_deep_scrub":"37'171","last_deep_scrub_stamp":"2024-04-14T23:58:18.752420+0000","last_clean_scrub_stamp":"2024-04-14T23:58:18.752420+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:03:27.827653+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.01445958,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":38,"num_read_kb":8211,"num_write":164,"num_write_kb":8272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"45'141","reported_seq":2263,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:30.816778+0000","last_change":"2024-04-14T23:58:30.816803+0000","last_active":"2024-04-14T23:58:30.816778+0000","last_peered":"2024-04-14T23:58:30.816778+0000","last_clean":"2024-04-14T23:58:30.816778+0000","last_became_active":"2024-04-14T23:48:43.242000+0000","last_became_peered":"2024-04-14T23:48:43.242000+0000","last_unstale":"2024-04-14T23:58:30.816778+0000","last_undegraded":"2024-04-14T23:58:30.816778+0000","last_fullsized":"2024-04-14T23:58:30.816778+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'141","last_scrub_stamp":"2024-04-14T23:58:30.816679+0000","last_deep_scrub":"45'141","last_deep_scrub_stamp":"2024-04-14T23:58:30.816679+0000","last_clean_scrub_stamp":"2024-04-14T23:58:30.816679+0000","objects_scrubbed":0,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:00:48.795222+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0020739690000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":9,"num_write":140,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"35'197","reported_seq":2319,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:25.993673+0000","last_change":"2024-04-14T23:58:25.993730+0000","last_active":"2024-04-14T23:58:25.993673+0000","last_peered":"2024-04-14T23:58:25.993673+0000","last_clean":"2024-04-14T23:58:25.993673+0000","last_became_active":"2024-04-14T23:49:29.448078+0000","last_became_peered":"2024-04-14T23:49:29.448078+0000","last_unstale":"2024-04-14T23:58:25.993673+0000","last_undegraded":"2024-04-14T23:58:25.993673+0000","last_fullsized":"2024-04-14T23:58:25.993673+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'197","last_scrub_stamp":"2024-04-14T23:58:25.993497+0000","last_deep_scrub":"35'197","last_deep_scrub_stamp":"2024-04-14T23:58:25.993497+0000","last_clean_scrub_stamp":"2024-04-14T23:58:25.993497+0000","objects_scrubbed":0,"log_size":197,"log_dups_size":0,"ondisk_log_size":197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:39:43.733871+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.011461469,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":27,"num_read_kb":18,"num_write":196,"num_write_kb":98,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3a","version":"35'170","reported_seq":2303,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:27.873480+0000","last_change":"2024-04-14T23:58:27.873505+0000","last_active":"2024-04-14T23:58:27.873480+0000","last_peered":"2024-04-14T23:58:27.873480+0000","last_clean":"2024-04-14T23:58:27.873480+0000","last_became_active":"2024-04-14T23:48:43.230405+0000","last_became_peered":"2024-04-14T23:48:43.230405+0000","last_unstale":"2024-04-14T23:58:27.873480+0000","last_undegraded":"2024-04-14T23:58:27.873480+0000","last_fullsized":"2024-04-14T23:58:27.873480+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'170","last_scrub_stamp":"2024-04-14T23:58:27.873377+0000","last_deep_scrub":"35'170","last_deep_scrub_stamp":"2024-04-14T23:58:27.873377+0000","last_clean_scrub_stamp":"2024-04-14T23:58:27.873377+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:17:26.929046+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.002322771,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":32,"num_read_kb":19,"num_write":170,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"44'198","reported_seq":2336,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:31.210915+0000","last_change":"2024-04-14T23:58:31.210960+0000","last_active":"2024-04-14T23:58:31.210915+0000","last_peered":"2024-04-14T23:58:31.210915+0000","last_clean":"2024-04-14T23:58:31.210915+0000","last_became_active":"2024-04-14T23:48:43.230784+0000","last_became_peered":"2024-04-14T23:48:43.230784+0000","last_unstale":"2024-04-14T23:58:31.210915+0000","last_undegraded":"2024-04-14T23:58:31.210915+0000","last_fullsized":"2024-04-14T23:58:31.210915+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'198","last_scrub_stamp":"2024-04-14T23:58:31.210770+0000","last_deep_scrub":"44'198","last_deep_scrub_stamp":"2024-04-14T23:58:31.210770+0000","last_clean_scrub_stamp":"2024-04-14T23:58:31.210770+0000","objects_scrubbed":0,"log_size":198,"log_dups_size":0,"ondisk_log_size":198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:33:58.319776+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0023027080000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":37,"num_read_kb":16,"num_write":198,"num_write_kb":4194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"45'135","reported_seq":2262,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:27.872867+0000","last_change":"2024-04-14T23:58:27.872926+0000","last_active":"2024-04-14T23:58:27.872867+0000","last_peered":"2024-04-14T23:58:27.872867+0000","last_clean":"2024-04-14T23:58:27.872867+0000","last_became_active":"2024-04-14T23:48:43.235719+0000","last_became_peered":"2024-04-14T23:48:43.235719+0000","last_unstale":"2024-04-14T23:58:27.872867+0000","last_undegraded":"2024-04-14T23:58:27.872867+0000","last_fullsized":"2024-04-14T23:58:27.872867+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'135","last_scrub_stamp":"2024-04-14T23:58:27.872675+0000","last_deep_scrub":"45'135","last_deep_scrub_stamp":"2024-04-14T23:58:27.872675+0000","last_clean_scrub_stamp":"2024-04-14T23:58:27.872675+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T04:01:24.449341+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.015616025,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":26,"num_read_kb":10,"num_write":134,"num_write_kb":12352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"44'157","reported_seq":2293,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:32.190852+0000","last_change":"2024-04-14T23:58:32.190880+0000","last_active":"2024-04-14T23:58:32.190852+0000","last_peered":"2024-04-14T23:58:32.190852+0000","last_clean":"2024-04-14T23:58:32.190852+0000","last_became_active":"2024-04-14T23:48:43.231751+0000","last_became_peered":"2024-04-14T23:48:43.231751+0000","last_unstale":"2024-04-14T23:58:32.190852+0000","last_undegraded":"2024-04-14T23:58:32.190852+0000","last_fullsized":"2024-04-14T23:58:32.190852+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'157","last_scrub_stamp":"2024-04-14T23:58:32.190751+0000","last_deep_scrub":"44'157","last_deep_scrub_stamp":"2024-04-14T23:58:32.190751+0000","last_clean_scrub_stamp":"2024-04-14T23:58:32.190751+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:30:28.830370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025112200000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":4114,"num_write":154,"num_write_kb":8271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"35'164","reported_seq":2293,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:24.915201+0000","last_change":"2024-04-14T23:58:24.915201+0000","last_active":"2024-04-14T23:58:24.915201+0000","last_peered":"2024-04-14T23:58:24.915201+0000","last_clean":"2024-04-14T23:58:24.915201+0000","last_became_active":"2024-04-14T23:48:43.224592+0000","last_became_peered":"2024-04-14T23:48:43.224592+0000","last_unstale":"2024-04-14T23:58:24.915201+0000","last_undegraded":"2024-04-14T23:58:24.915201+0000","last_fullsized":"2024-04-14T23:58:24.915201+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'164","last_scrub_stamp":"2024-04-14T23:58:24.915072+0000","last_deep_scrub":"35'164","last_deep_scrub_stamp":"2024-04-14T23:58:24.915072+0000","last_clean_scrub_stamp":"2024-04-14T23:58:24.915072+0000","objects_scrubbed":0,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:46:38.376637+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0014741629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":27,"num_read_kb":16,"num_write":162,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"45'173","reported_seq":2310,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:25.946671+0000","last_change":"2024-04-14T23:58:25.946698+0000","last_active":"2024-04-14T23:58:25.946671+0000","last_peered":"2024-04-14T23:58:25.946671+0000","last_clean":"2024-04-14T23:58:25.946671+0000","last_became_active":"2024-04-14T23:48:43.224989+0000","last_became_peered":"2024-04-14T23:48:43.224989+0000","last_unstale":"2024-04-14T23:58:25.946671+0000","last_undegraded":"2024-04-14T23:58:25.946671+0000","last_fullsized":"2024-04-14T23:58:25.946671+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'173","last_scrub_stamp":"2024-04-14T23:58:25.946583+0000","last_deep_scrub":"45'173","last_deep_scrub_stamp":"2024-04-14T23:58:25.946583+0000","last_clean_scrub_stamp":"2024-04-14T23:58:25.946583+0000","objects_scrubbed":0,"log_size":173,"log_dups_size":0,"ondisk_log_size":173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:30:44.142676+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001581301,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":18,"num_write":172,"num_write_kb":4183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"45'147","reported_seq":2273,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:17.151685+0000","last_change":"2024-04-14T23:58:17.151685+0000","last_active":"2024-04-14T23:58:17.151685+0000","last_peered":"2024-04-14T23:58:17.151685+0000","last_clean":"2024-04-14T23:58:17.151685+0000","last_became_active":"2024-04-14T23:48:43.225892+0000","last_became_peered":"2024-04-14T23:48:43.225892+0000","last_unstale":"2024-04-14T23:58:17.151685+0000","last_undegraded":"2024-04-14T23:58:17.151685+0000","last_fullsized":"2024-04-14T23:58:17.151685+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'147","last_scrub_stamp":"2024-04-14T23:58:17.151569+0000","last_deep_scrub":"45'147","last_deep_scrub_stamp":"2024-04-14T23:58:17.151569+0000","last_clean_scrub_stamp":"2024-04-14T23:58:17.151569+0000","objects_scrubbed":0,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:04:00.040301+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0011254640000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":24,"num_read_kb":10,"num_write":146,"num_write_kb":8263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"37'146","reported_seq":2277,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:17.746653+0000","last_change":"2024-04-14T23:58:17.746653+0000","last_active":"2024-04-14T23:58:17.746653+0000","last_peered":"2024-04-14T23:58:17.746653+0000","last_clean":"2024-04-14T23:58:17.746653+0000","last_became_active":"2024-04-14T23:48:43.219614+0000","last_became_peered":"2024-04-14T23:48:43.219614+0000","last_unstale":"2024-04-14T23:58:17.746653+0000","last_undegraded":"2024-04-14T23:58:17.746653+0000","last_fullsized":"2024-04-14T23:58:17.746653+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'146","last_scrub_stamp":"2024-04-14T23:58:17.746395+0000","last_deep_scrub":"37'146","last_deep_scrub_stamp":"2024-04-14T23:58:17.746395+0000","last_clean_scrub_stamp":"2024-04-14T23:58:17.746395+0000","objects_scrubbed":0,"log_size":146,"log_dups_size":0,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:35:50.250946+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.012872641000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":30,"num_read_kb":4115,"num_write":142,"num_write_kb":8261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"44'175","reported_seq":2317,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:22.710894+0000","last_change":"2024-04-14T23:58:22.710955+0000","last_active":"2024-04-14T23:58:22.710894+0000","last_peered":"2024-04-14T23:58:22.710894+0000","last_clean":"2024-04-14T23:58:22.710894+0000","last_became_active":"2024-04-14T23:48:43.239136+0000","last_became_peered":"2024-04-14T23:48:43.239136+0000","last_unstale":"2024-04-14T23:58:22.710894+0000","last_undegraded":"2024-04-14T23:58:22.710894+0000","last_fullsized":"2024-04-14T23:58:22.710894+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'175","last_scrub_stamp":"2024-04-14T23:58:22.710640+0000","last_deep_scrub":"44'175","last_deep_scrub_stamp":"2024-04-14T23:58:22.710640+0000","last_clean_scrub_stamp":"2024-04-14T23:58:22.710640+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:57:42.324508+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.015761819,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":40,"num_read_kb":19,"num_write":174,"num_write_kb":12372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"44'159","reported_seq":2298,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:22.087310+0000","last_change":"2024-04-14T23:58:22.087368+0000","last_active":"2024-04-14T23:58:22.087310+0000","last_peered":"2024-04-14T23:58:22.087310+0000","last_clean":"2024-04-14T23:58:22.087310+0000","last_became_active":"2024-04-14T23:48:43.227898+0000","last_became_peered":"2024-04-14T23:48:43.227898+0000","last_unstale":"2024-04-14T23:58:22.087310+0000","last_undegraded":"2024-04-14T23:58:22.087310+0000","last_fullsized":"2024-04-14T23:58:22.087310+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'159","last_scrub_stamp":"2024-04-14T23:58:22.087128+0000","last_deep_scrub":"44'159","last_deep_scrub_stamp":"2024-04-14T23:58:22.087128+0000","last_clean_scrub_stamp":"2024-04-14T23:58:22.087128+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:50:30.333096+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.011044834999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":37,"num_read_kb":19,"num_write":159,"num_write_kb":4178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"44'184","reported_seq":2326,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:28.884772+0000","last_change":"2024-04-14T23:58:28.884814+0000","last_active":"2024-04-14T23:58:28.884772+0000","last_peered":"2024-04-14T23:58:28.884772+0000","last_clean":"2024-04-14T23:58:28.884772+0000","last_became_active":"2024-04-14T23:48:43.238714+0000","last_became_peered":"2024-04-14T23:48:43.238714+0000","last_unstale":"2024-04-14T23:58:28.884772+0000","last_undegraded":"2024-04-14T23:58:28.884772+0000","last_fullsized":"2024-04-14T23:58:28.884772+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'184","last_scrub_stamp":"2024-04-14T23:58:28.884653+0000","last_deep_scrub":"44'184","last_deep_scrub_stamp":"2024-04-14T23:58:28.884653+0000","last_clean_scrub_stamp":"2024-04-14T23:58:28.884653+0000","objects_scrubbed":0,"log_size":184,"log_dups_size":0,"ondisk_log_size":184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:33:42.269566+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0019546920000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":40,"num_read_kb":19,"num_write":182,"num_write_kb":7459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"44'174","reported_seq":2315,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:25.962965+0000","last_change":"2024-04-14T23:58:25.962989+0000","last_active":"2024-04-14T23:58:25.962965+0000","last_peered":"2024-04-14T23:58:25.962965+0000","last_clean":"2024-04-14T23:58:25.962965+0000","last_became_active":"2024-04-14T23:48:43.228618+0000","last_became_peered":"2024-04-14T23:48:43.228618+0000","last_unstale":"2024-04-14T23:58:25.962965+0000","last_undegraded":"2024-04-14T23:58:25.962965+0000","last_fullsized":"2024-04-14T23:58:25.962965+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'174","last_scrub_stamp":"2024-04-14T23:58:25.962881+0000","last_deep_scrub":"44'174","last_deep_scrub_stamp":"2024-04-14T23:58:25.962881+0000","last_clean_scrub_stamp":"2024-04-14T23:58:25.962881+0000","objects_scrubbed":0,"log_size":174,"log_dups_size":0,"ondisk_log_size":174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:21:18.645881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002424158,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":40,"num_read_kb":4117,"num_write":170,"num_write_kb":4181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"37'153","reported_seq":2302,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:18.747159+0000","last_change":"2024-04-14T23:58:18.747284+0000","last_active":"2024-04-14T23:58:18.747159+0000","last_peered":"2024-04-14T23:58:18.747159+0000","last_clean":"2024-04-14T23:58:18.747159+0000","last_became_active":"2024-04-14T23:48:43.232971+0000","last_became_peered":"2024-04-14T23:48:43.232971+0000","last_unstale":"2024-04-14T23:58:18.747159+0000","last_undegraded":"2024-04-14T23:58:18.747159+0000","last_fullsized":"2024-04-14T23:58:18.747159+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'153","last_scrub_stamp":"2024-04-14T23:58:18.746930+0000","last_deep_scrub":"37'153","last_deep_scrub_stamp":"2024-04-14T23:58:18.746930+0000","last_clean_scrub_stamp":"2024-04-14T23:58:18.746930+0000","objects_scrubbed":0,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:15:43.060805+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0030089610000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":47,"num_read_kb":26,"num_write":152,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"45'164","reported_seq":2300,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:34.175055+0000","last_change":"2024-04-14T23:58:34.175083+0000","last_active":"2024-04-14T23:58:34.175055+0000","last_peered":"2024-04-14T23:58:34.175055+0000","last_clean":"2024-04-14T23:58:34.175055+0000","last_became_active":"2024-04-14T23:48:43.232135+0000","last_became_peered":"2024-04-14T23:48:43.232135+0000","last_unstale":"2024-04-14T23:58:34.175055+0000","last_undegraded":"2024-04-14T23:58:34.175055+0000","last_fullsized":"2024-04-14T23:58:34.175055+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'164","last_scrub_stamp":"2024-04-14T23:58:34.174954+0000","last_deep_scrub":"45'164","last_deep_scrub_stamp":"2024-04-14T23:58:34.174954+0000","last_clean_scrub_stamp":"2024-04-14T23:58:34.174954+0000","objects_scrubbed":0,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:20:22.522414+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0029988570000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":4112,"num_write":160,"num_write_kb":8270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"45'197","reported_seq":2340,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:26.985232+0000","last_change":"2024-04-14T23:58:26.985258+0000","last_active":"2024-04-14T23:58:26.985232+0000","last_peered":"2024-04-14T23:58:26.985232+0000","last_clean":"2024-04-14T23:58:26.985232+0000","last_became_active":"2024-04-14T23:48:43.226225+0000","last_became_peered":"2024-04-14T23:48:43.226225+0000","last_unstale":"2024-04-14T23:58:26.985232+0000","last_undegraded":"2024-04-14T23:58:26.985232+0000","last_fullsized":"2024-04-14T23:58:26.985232+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'197","last_scrub_stamp":"2024-04-14T23:58:26.985153+0000","last_deep_scrub":"45'197","last_deep_scrub_stamp":"2024-04-14T23:58:26.985153+0000","last_clean_scrub_stamp":"2024-04-14T23:58:26.985153+0000","objects_scrubbed":0,"log_size":197,"log_dups_size":0,"ondisk_log_size":197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T09:06:59.954041+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0019674760000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":42,"num_read_kb":4116,"num_write":194,"num_write_kb":16477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"44'154","reported_seq":2300,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:20.673252+0000","last_change":"2024-04-14T23:58:20.673313+0000","last_active":"2024-04-14T23:58:20.673252+0000","last_peered":"2024-04-14T23:58:20.673252+0000","last_clean":"2024-04-14T23:58:20.673252+0000","last_became_active":"2024-04-14T23:48:43.230056+0000","last_became_peered":"2024-04-14T23:48:43.230056+0000","last_unstale":"2024-04-14T23:58:20.673252+0000","last_undegraded":"2024-04-14T23:58:20.673252+0000","last_fullsized":"2024-04-14T23:58:20.673252+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'154","last_scrub_stamp":"2024-04-14T23:58:20.673062+0000","last_deep_scrub":"44'154","last_deep_scrub_stamp":"2024-04-14T23:58:20.673062+0000","last_clean_scrub_stamp":"2024-04-14T23:58:20.673062+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:46:08.415575+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.015336659000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":44,"num_read_kb":22,"num_write":154,"num_write_kb":77,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"44'175","reported_seq":2313,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:26.921502+0000","last_change":"2024-04-14T23:58:26.921540+0000","last_active":"2024-04-14T23:58:26.921502+0000","last_peered":"2024-04-14T23:58:26.921502+0000","last_clean":"2024-04-14T23:58:26.921502+0000","last_became_active":"2024-04-14T23:48:43.229110+0000","last_became_peered":"2024-04-14T23:48:43.229110+0000","last_unstale":"2024-04-14T23:58:26.921502+0000","last_undegraded":"2024-04-14T23:58:26.921502+0000","last_fullsized":"2024-04-14T23:58:26.921502+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'175","last_scrub_stamp":"2024-04-14T23:58:26.921412+0000","last_deep_scrub":"44'175","last_deep_scrub_stamp":"2024-04-14T23:58:26.921412+0000","last_clean_scrub_stamp":"2024-04-14T23:58:26.921412+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:30:00.903633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022905320000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":36,"num_read_kb":17,"num_write":174,"num_write_kb":4183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"45'171","reported_seq":2311,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:26.904979+0000","last_change":"2024-04-14T23:58:26.905040+0000","last_active":"2024-04-14T23:58:26.904979+0000","last_peered":"2024-04-14T23:58:26.904979+0000","last_clean":"2024-04-14T23:58:26.904979+0000","last_became_active":"2024-04-14T23:48:43.233853+0000","last_became_peered":"2024-04-14T23:48:43.233853+0000","last_unstale":"2024-04-14T23:58:26.904979+0000","last_undegraded":"2024-04-14T23:58:26.904979+0000","last_fullsized":"2024-04-14T23:58:26.904979+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'171","last_scrub_stamp":"2024-04-14T23:58:26.904692+0000","last_deep_scrub":"45'171","last_deep_scrub_stamp":"2024-04-14T23:58:26.904692+0000","last_clean_scrub_stamp":"2024-04-14T23:58:26.904692+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:16:25.197412+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.012858206,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":39,"num_read_kb":4117,"num_write":168,"num_write_kb":4180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"45'174","reported_seq":2306,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:20.686514+0000","last_change":"2024-04-14T23:58:20.686577+0000","last_active":"2024-04-14T23:58:20.686514+0000","last_peered":"2024-04-14T23:58:20.686514+0000","last_clean":"2024-04-14T23:58:20.686514+0000","last_became_active":"2024-04-14T23:49:29.512499+0000","last_became_peered":"2024-04-14T23:49:29.512499+0000","last_unstale":"2024-04-14T23:58:20.686514+0000","last_undegraded":"2024-04-14T23:58:20.686514+0000","last_fullsized":"2024-04-14T23:58:20.686514+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'174","last_scrub_stamp":"2024-04-14T23:58:20.686291+0000","last_deep_scrub":"45'174","last_deep_scrub_stamp":"2024-04-14T23:58:20.686291+0000","last_clean_scrub_stamp":"2024-04-14T23:58:20.686291+0000","objects_scrubbed":0,"log_size":174,"log_dups_size":0,"ondisk_log_size":174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:21:53.481714+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0037213820000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":36,"num_read_kb":16,"num_write":174,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"37'160","reported_seq":2296,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:24.831187+0000","last_change":"2024-04-14T23:58:24.831245+0000","last_active":"2024-04-14T23:58:24.831187+0000","last_peered":"2024-04-14T23:58:24.831187+0000","last_clean":"2024-04-14T23:58:24.831187+0000","last_became_active":"2024-04-14T23:48:43.233238+0000","last_became_peered":"2024-04-14T23:48:43.233238+0000","last_unstale":"2024-04-14T23:58:24.831187+0000","last_undegraded":"2024-04-14T23:58:24.831187+0000","last_fullsized":"2024-04-14T23:58:24.831187+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'160","last_scrub_stamp":"2024-04-14T23:58:24.831007+0000","last_deep_scrub":"37'160","last_deep_scrub_stamp":"2024-04-14T23:58:24.831007+0000","last_clean_scrub_stamp":"2024-04-14T23:58:24.831007+0000","objects_scrubbed":0,"log_size":160,"log_dups_size":0,"ondisk_log_size":160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:59:16.965861+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.012037971,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":18,"num_write":158,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"36'184","reported_seq":2321,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:23.951721+0000","last_change":"2024-04-14T23:58:23.951752+0000","last_active":"2024-04-14T23:58:23.951721+0000","last_peered":"2024-04-14T23:58:23.951721+0000","last_clean":"2024-04-14T23:58:23.951721+0000","last_became_active":"2024-04-14T23:48:43.222624+0000","last_became_peered":"2024-04-14T23:48:43.222624+0000","last_unstale":"2024-04-14T23:58:23.951721+0000","last_undegraded":"2024-04-14T23:58:23.951721+0000","last_fullsized":"2024-04-14T23:58:23.951721+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'184","last_scrub_stamp":"2024-04-14T23:58:23.951623+0000","last_deep_scrub":"36'184","last_deep_scrub_stamp":"2024-04-14T23:58:23.951623+0000","last_clean_scrub_stamp":"2024-04-14T23:58:23.951623+0000","objects_scrubbed":0,"log_size":184,"log_dups_size":0,"ondisk_log_size":184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:12:22.241724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001456707,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":19,"num_write":184,"num_write_kb":92,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"44'140","reported_seq":2266,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:23.037973+0000","last_change":"2024-04-14T23:58:23.038083+0000","last_active":"2024-04-14T23:58:23.037973+0000","last_peered":"2024-04-14T23:58:23.037973+0000","last_clean":"2024-04-14T23:58:23.037973+0000","last_became_active":"2024-04-14T23:48:43.232277+0000","last_became_peered":"2024-04-14T23:48:43.232277+0000","last_unstale":"2024-04-14T23:58:23.037973+0000","last_undegraded":"2024-04-14T23:58:23.037973+0000","last_fullsized":"2024-04-14T23:58:23.037973+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'140","last_scrub_stamp":"2024-04-14T23:58:23.037761+0000","last_deep_scrub":"44'140","last_deep_scrub_stamp":"2024-04-14T23:58:23.037761+0000","last_clean_scrub_stamp":"2024-04-14T23:58:23.037761+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:43:25.128157+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.011200412,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":138,"num_write_kb":4167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"45'157","reported_seq":2289,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:21.777408+0000","last_change":"2024-04-14T23:58:21.777489+0000","last_active":"2024-04-14T23:58:21.777408+0000","last_peered":"2024-04-14T23:58:21.777408+0000","last_clean":"2024-04-14T23:58:21.777408+0000","last_became_active":"2024-04-14T23:48:43.229172+0000","last_became_peered":"2024-04-14T23:48:43.229172+0000","last_unstale":"2024-04-14T23:58:21.777408+0000","last_undegraded":"2024-04-14T23:58:21.777408+0000","last_fullsized":"2024-04-14T23:58:21.777408+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'157","last_scrub_stamp":"2024-04-14T23:58:21.777136+0000","last_deep_scrub":"45'157","last_deep_scrub_stamp":"2024-04-14T23:58:21.777136+0000","last_clean_scrub_stamp":"2024-04-14T23:58:21.777136+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:22:30.327045+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.011889788,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":31,"num_read_kb":3287,"num_write":154,"num_write_kb":3350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"45'186","reported_seq":2332,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:23.746718+0000","last_change":"2024-04-14T23:58:23.746829+0000","last_active":"2024-04-14T23:58:23.746718+0000","last_peered":"2024-04-14T23:58:23.746718+0000","last_clean":"2024-04-14T23:58:23.746718+0000","last_became_active":"2024-04-14T23:49:29.449548+0000","last_became_peered":"2024-04-14T23:49:29.449548+0000","last_unstale":"2024-04-14T23:58:23.746718+0000","last_undegraded":"2024-04-14T23:58:23.746718+0000","last_fullsized":"2024-04-14T23:58:23.746718+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'186","last_scrub_stamp":"2024-04-14T23:58:23.746405+0000","last_deep_scrub":"45'186","last_deep_scrub_stamp":"2024-04-14T23:58:23.746405+0000","last_clean_scrub_stamp":"2024-04-14T23:58:23.746405+0000","objects_scrubbed":0,"log_size":186,"log_dups_size":0,"ondisk_log_size":186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:26:35.374136+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.015790789,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":39,"num_read_kb":18,"num_write":186,"num_write_kb":4188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"35'188","reported_seq":2327,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:21.691800+0000","last_change":"2024-04-14T23:58:21.691858+0000","last_active":"2024-04-14T23:58:21.691800+0000","last_peered":"2024-04-14T23:58:21.691800+0000","last_clean":"2024-04-14T23:58:21.691800+0000","last_became_active":"2024-04-14T23:49:29.512523+0000","last_became_peered":"2024-04-14T23:49:29.512523+0000","last_unstale":"2024-04-14T23:58:21.691800+0000","last_undegraded":"2024-04-14T23:58:21.691800+0000","last_fullsized":"2024-04-14T23:58:21.691800+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'188","last_scrub_stamp":"2024-04-14T23:58:21.691610+0000","last_deep_scrub":"35'188","last_deep_scrub_stamp":"2024-04-14T23:58:21.691610+0000","last_clean_scrub_stamp":"2024-04-14T23:58:21.691610+0000","objects_scrubbed":0,"log_size":188,"log_dups_size":0,"ondisk_log_size":188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:48:27.271314+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0026059680000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":43,"num_read_kb":22,"num_write":188,"num_write_kb":94,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"45'206","reported_seq":2343,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:19.707716+0000","last_change":"2024-04-14T23:58:19.707778+0000","last_active":"2024-04-14T23:58:19.707716+0000","last_peered":"2024-04-14T23:58:19.707716+0000","last_clean":"2024-04-14T23:58:19.707716+0000","last_became_active":"2024-04-14T23:48:43.233567+0000","last_became_peered":"2024-04-14T23:48:43.233567+0000","last_unstale":"2024-04-14T23:58:19.707716+0000","last_undegraded":"2024-04-14T23:58:19.707716+0000","last_fullsized":"2024-04-14T23:58:19.707716+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'206","last_scrub_stamp":"2024-04-14T23:58:19.707534+0000","last_deep_scrub":"45'206","last_deep_scrub_stamp":"2024-04-14T23:58:19.707534+0000","last_clean_scrub_stamp":"2024-04-14T23:58:19.707534+0000","objects_scrubbed":0,"log_size":206,"log_dups_size":0,"ondisk_log_size":206,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:28:10.154171+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.003348115,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":17,"num_write":206,"num_write_kb":14807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"45'202","reported_seq":2351,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:28.036085+0000","last_change":"2024-04-14T23:58:28.036114+0000","last_active":"2024-04-14T23:58:28.036085+0000","last_peered":"2024-04-14T23:58:28.036085+0000","last_clean":"2024-04-14T23:58:28.036085+0000","last_became_active":"2024-04-14T23:48:43.230626+0000","last_became_peered":"2024-04-14T23:48:43.230626+0000","last_unstale":"2024-04-14T23:58:28.036085+0000","last_undegraded":"2024-04-14T23:58:28.036085+0000","last_fullsized":"2024-04-14T23:58:28.036085+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'202","last_scrub_stamp":"2024-04-14T23:58:28.035967+0000","last_deep_scrub":"45'202","last_deep_scrub_stamp":"2024-04-14T23:58:28.035967+0000","last_clean_scrub_stamp":"2024-04-14T23:58:28.035967+0000","objects_scrubbed":0,"log_size":202,"log_dups_size":0,"ondisk_log_size":202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:21:02.418550+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001517437,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":47,"num_read_kb":27,"num_write":201,"num_write_kb":12389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"44'158","reported_seq":2287,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:18.103631+0000","last_change":"2024-04-14T23:58:18.103659+0000","last_active":"2024-04-14T23:58:18.103631+0000","last_peered":"2024-04-14T23:58:18.103631+0000","last_clean":"2024-04-14T23:58:18.103631+0000","last_became_active":"2024-04-14T23:48:43.226773+0000","last_became_peered":"2024-04-14T23:48:43.226773+0000","last_unstale":"2024-04-14T23:58:18.103631+0000","last_undegraded":"2024-04-14T23:58:18.103631+0000","last_fullsized":"2024-04-14T23:58:18.103631+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'158","last_scrub_stamp":"2024-04-14T23:58:18.103529+0000","last_deep_scrub":"44'158","last_deep_scrub_stamp":"2024-04-14T23:58:18.103529+0000","last_clean_scrub_stamp":"2024-04-14T23:58:18.103529+0000","objects_scrubbed":0,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:50:38.289113+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00101596,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":28,"num_read_kb":16,"num_write":158,"num_write_kb":4174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"45'23","reported_seq":102,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:15.757841+0000","last_change":"2024-04-14T23:58:15.757900+0000","last_active":"2024-04-14T23:58:15.757841+0000","last_peered":"2024-04-14T23:58:15.757841+0000","last_clean":"2024-04-14T23:58:15.757841+0000","last_became_active":"2024-04-14T23:48:42.238847+0000","last_became_peered":"2024-04-14T23:48:42.238847+0000","last_unstale":"2024-04-14T23:58:15.757841+0000","last_undegraded":"2024-04-14T23:58:15.757841+0000","last_fullsized":"2024-04-14T23:58:15.757841+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'23","last_scrub_stamp":"2024-04-14T23:58:15.757745+0000","last_deep_scrub":"45'23","last_deep_scrub_stamp":"2024-04-14T23:58:15.757745+0000","last_clean_scrub_stamp":"2024-04-14T23:58:15.757745+0000","objects_scrubbed":2,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:00:25.253826+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":16386,"num_write":22,"num_write_kb":4233,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":561,"num_omap_keys":1,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"44'168","reported_seq":2305,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:21.711507+0000","last_change":"2024-04-14T23:58:21.711629+0000","last_active":"2024-04-14T23:58:21.711507+0000","last_peered":"2024-04-14T23:58:21.711507+0000","last_clean":"2024-04-14T23:58:21.711507+0000","last_became_active":"2024-04-14T23:48:43.237391+0000","last_became_peered":"2024-04-14T23:48:43.237391+0000","last_unstale":"2024-04-14T23:58:21.711507+0000","last_undegraded":"2024-04-14T23:58:21.711507+0000","last_fullsized":"2024-04-14T23:58:21.711507+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'168","last_scrub_stamp":"2024-04-14T23:58:21.711258+0000","last_deep_scrub":"44'168","last_deep_scrub_stamp":"2024-04-14T23:58:21.711258+0000","last_clean_scrub_stamp":"2024-04-14T23:58:21.711258+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:51:16.261420+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.01542551,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":36,"num_read_kb":15,"num_write":168,"num_write_kb":4179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"45'167","reported_seq":252,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:18.902219+0000","last_change":"2024-04-14T23:58:18.902257+0000","last_active":"2024-04-14T23:58:18.902219+0000","last_peered":"2024-04-14T23:58:18.902219+0000","last_clean":"2024-04-14T23:58:18.902219+0000","last_became_active":"2024-04-14T23:48:42.218995+0000","last_became_peered":"2024-04-14T23:48:42.218995+0000","last_unstale":"2024-04-14T23:58:18.902219+0000","last_undegraded":"2024-04-14T23:58:18.902219+0000","last_fullsized":"2024-04-14T23:58:18.902219+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'167","last_scrub_stamp":"2024-04-14T23:58:18.902137+0000","last_deep_scrub":"45'167","last_deep_scrub_stamp":"2024-04-14T23:58:18.902137+0000","last_clean_scrub_stamp":"2024-04-14T23:58:18.902137+0000","objects_scrubbed":3,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:40:45.866142+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":11,"num_read_kb":40960,"num_write":162,"num_write_kb":12391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"45'25","reported_seq":103,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:14.765434+0000","last_change":"2024-04-14T23:58:14.765434+0000","last_active":"2024-04-14T23:58:14.765434+0000","last_peered":"2024-04-14T23:58:14.765434+0000","last_clean":"2024-04-14T23:58:14.765434+0000","last_became_active":"2024-04-14T23:48:42.225937+0000","last_became_peered":"2024-04-14T23:48:42.225937+0000","last_unstale":"2024-04-14T23:58:14.765434+0000","last_undegraded":"2024-04-14T23:58:14.765434+0000","last_fullsized":"2024-04-14T23:58:14.765434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'25","last_scrub_stamp":"2024-04-14T23:58:14.765308+0000","last_deep_scrub":"45'25","last_deep_scrub_stamp":"2024-04-14T23:58:14.765308+0000","last_clean_scrub_stamp":"2024-04-14T23:58:14.765308+0000","objects_scrubbed":1,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:47:25.922937+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":22,"num_write_kb":4105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"35'209","reported_seq":2350,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:22.795850+0000","last_change":"2024-04-14T23:58:22.795850+0000","last_active":"2024-04-14T23:58:22.795850+0000","last_peered":"2024-04-14T23:58:22.795850+0000","last_clean":"2024-04-14T23:58:22.795850+0000","last_became_active":"2024-04-14T23:48:43.227356+0000","last_became_peered":"2024-04-14T23:48:43.227356+0000","last_unstale":"2024-04-14T23:58:22.795850+0000","last_undegraded":"2024-04-14T23:58:22.795850+0000","last_fullsized":"2024-04-14T23:58:22.795850+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'209","last_scrub_stamp":"2024-04-14T23:58:22.795604+0000","last_deep_scrub":"35'209","last_deep_scrub_stamp":"2024-04-14T23:58:22.795604+0000","last_clean_scrub_stamp":"2024-04-14T23:58:22.795604+0000","objects_scrubbed":0,"log_size":209,"log_dups_size":0,"ondisk_log_size":209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T09:01:25.807501+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.014613774,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":40,"num_read_kb":4118,"num_write":206,"num_write_kb":8293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"45'191","reported_seq":272,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:17.009463+0000","last_change":"2024-04-14T23:58:17.009463+0000","last_active":"2024-04-14T23:58:17.009463+0000","last_peered":"2024-04-14T23:58:17.009463+0000","last_clean":"2024-04-14T23:58:17.009463+0000","last_became_active":"2024-04-14T23:48:42.210607+0000","last_became_peered":"2024-04-14T23:48:42.210607+0000","last_unstale":"2024-04-14T23:58:17.009463+0000","last_undegraded":"2024-04-14T23:58:17.009463+0000","last_fullsized":"2024-04-14T23:58:17.009463+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'191","last_scrub_stamp":"2024-04-14T23:58:17.009354+0000","last_deep_scrub":"45'191","last_deep_scrub_stamp":"2024-04-14T23:58:17.009354+0000","last_clean_scrub_stamp":"2024-04-14T23:58:17.009354+0000","objects_scrubbed":2,"log_size":191,"log_dups_size":0,"ondisk_log_size":191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:02:58.552434+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":186,"num_write_kb":8305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"42'164","reported_seq":2306,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:20.121053+0000","last_change":"2024-04-14T23:58:20.121053+0000","last_active":"2024-04-14T23:58:20.121053+0000","last_peered":"2024-04-14T23:58:20.121053+0000","last_clean":"2024-04-14T23:58:20.121053+0000","last_became_active":"2024-04-14T23:48:43.217801+0000","last_became_peered":"2024-04-14T23:48:43.217801+0000","last_unstale":"2024-04-14T23:58:20.121053+0000","last_undegraded":"2024-04-14T23:58:20.121053+0000","last_fullsized":"2024-04-14T23:58:20.121053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'164","last_scrub_stamp":"2024-04-14T23:58:20.120732+0000","last_deep_scrub":"42'164","last_deep_scrub_stamp":"2024-04-14T23:58:20.120732+0000","last_clean_scrub_stamp":"2024-04-14T23:58:20.120732+0000","objects_scrubbed":0,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:38:06.380512+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.012539378,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":41,"num_read_kb":4114,"num_write":158,"num_write_kb":4174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"45'6","reported_seq":81,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:17.898603+0000","last_change":"2024-04-14T23:58:17.898603+0000","last_active":"2024-04-14T23:58:17.898603+0000","last_peered":"2024-04-14T23:58:17.898603+0000","last_clean":"2024-04-14T23:58:17.898603+0000","last_became_active":"2024-04-14T23:48:42.218321+0000","last_became_peered":"2024-04-14T23:48:42.218321+0000","last_unstale":"2024-04-14T23:58:17.898603+0000","last_undegraded":"2024-04-14T23:58:17.898603+0000","last_fullsized":"2024-04-14T23:58:17.898603+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'6","last_scrub_stamp":"2024-04-14T23:58:17.898492+0000","last_deep_scrub":"45'6","last_deep_scrub_stamp":"2024-04-14T23:58:17.898492+0000","last_clean_scrub_stamp":"2024-04-14T23:58:17.898492+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:27:22.284664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"45'190","reported_seq":2337,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:24.951435+0000","last_change":"2024-04-14T23:58:24.951435+0000","last_active":"2024-04-14T23:58:24.951435+0000","last_peered":"2024-04-14T23:58:24.951435+0000","last_clean":"2024-04-14T23:58:24.951435+0000","last_became_active":"2024-04-14T23:48:43.228805+0000","last_became_peered":"2024-04-14T23:48:43.228805+0000","last_unstale":"2024-04-14T23:58:24.951435+0000","last_undegraded":"2024-04-14T23:58:24.951435+0000","last_fullsized":"2024-04-14T23:58:24.951435+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'190","last_scrub_stamp":"2024-04-14T23:58:24.951296+0000","last_deep_scrub":"45'190","last_deep_scrub_stamp":"2024-04-14T23:58:24.951296+0000","last_clean_scrub_stamp":"2024-04-14T23:58:24.951296+0000","objects_scrubbed":0,"log_size":190,"log_dups_size":0,"ondisk_log_size":190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:26:17.864333+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0022812650000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":45,"num_read_kb":20,"num_write":190,"num_write_kb":4190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"45'28","reported_seq":106,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:18.002614+0000","last_change":"2024-04-14T23:58:18.002614+0000","last_active":"2024-04-14T23:58:18.002614+0000","last_peered":"2024-04-14T23:58:18.002614+0000","last_clean":"2024-04-14T23:58:18.002614+0000","last_became_active":"2024-04-14T23:48:42.220767+0000","last_became_peered":"2024-04-14T23:48:42.220767+0000","last_unstale":"2024-04-14T23:58:18.002614+0000","last_undegraded":"2024-04-14T23:58:18.002614+0000","last_fullsized":"2024-04-14T23:58:18.002614+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'28","last_scrub_stamp":"2024-04-14T23:58:18.002489+0000","last_deep_scrub":"45'28","last_deep_scrub_stamp":"2024-04-14T23:58:18.002489+0000","last_clean_scrub_stamp":"2024-04-14T23:58:18.002489+0000","objects_scrubbed":1,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:20:25.057821+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":24,"num_write_kb":4108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"45'163","reported_seq":2298,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:17.769733+0000","last_change":"2024-04-14T23:58:17.769733+0000","last_active":"2024-04-14T23:58:17.769733+0000","last_peered":"2024-04-14T23:58:17.769733+0000","last_clean":"2024-04-14T23:58:17.769733+0000","last_became_active":"2024-04-14T23:48:43.236168+0000","last_became_peered":"2024-04-14T23:48:43.236168+0000","last_unstale":"2024-04-14T23:58:17.769733+0000","last_undegraded":"2024-04-14T23:58:17.769733+0000","last_fullsized":"2024-04-14T23:58:17.769733+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'163","last_scrub_stamp":"2024-04-14T23:58:17.769582+0000","last_deep_scrub":"45'163","last_deep_scrub_stamp":"2024-04-14T23:58:17.769582+0000","last_clean_scrub_stamp":"2024-04-14T23:58:17.769582+0000","objects_scrubbed":0,"log_size":163,"log_dups_size":0,"ondisk_log_size":163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:51:27.820287+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.015342639999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":33,"num_read_kb":18,"num_write":162,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"45'150","reported_seq":241,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:17.038764+0000","last_change":"2024-04-14T23:58:17.038764+0000","last_active":"2024-04-14T23:58:17.038764+0000","last_peered":"2024-04-14T23:58:17.038764+0000","last_clean":"2024-04-14T23:58:17.038764+0000","last_became_active":"2024-04-14T23:48:42.230969+0000","last_became_peered":"2024-04-14T23:48:42.230969+0000","last_unstale":"2024-04-14T23:58:17.038764+0000","last_undegraded":"2024-04-14T23:58:17.038764+0000","last_fullsized":"2024-04-14T23:58:17.038764+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'150","last_scrub_stamp":"2024-04-14T23:58:17.038667+0000","last_deep_scrub":"45'150","last_deep_scrub_stamp":"2024-04-14T23:58:17.038667+0000","last_clean_scrub_stamp":"2024-04-14T23:58:17.038667+0000","objects_scrubbed":3,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:41:37.820990+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194394,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":30,"num_read_kb":9177,"num_write":147,"num_write_kb":4202,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"44'151","reported_seq":2281,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:19.061545+0000","last_change":"2024-04-14T23:58:19.061571+0000","last_active":"2024-04-14T23:58:19.061545+0000","last_peered":"2024-04-14T23:58:19.061545+0000","last_clean":"2024-04-14T23:58:19.061545+0000","last_became_active":"2024-04-14T23:48:43.230253+0000","last_became_peered":"2024-04-14T23:48:43.230253+0000","last_unstale":"2024-04-14T23:58:19.061545+0000","last_undegraded":"2024-04-14T23:58:19.061545+0000","last_fullsized":"2024-04-14T23:58:19.061545+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'151","last_scrub_stamp":"2024-04-14T23:58:19.061447+0000","last_deep_scrub":"44'151","last_deep_scrub_stamp":"2024-04-14T23:58:19.061447+0000","last_clean_scrub_stamp":"2024-04-14T23:58:19.061447+0000","objects_scrubbed":0,"log_size":151,"log_dups_size":0,"ondisk_log_size":151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T04:15:14.500632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011901889999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":28,"num_read_kb":14,"num_write":150,"num_write_kb":4170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"45'712","reported_seq":817,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:22.009932+0000","last_change":"2024-04-14T23:58:22.009970+0000","last_active":"2024-04-14T23:58:22.009932+0000","last_peered":"2024-04-14T23:58:22.009932+0000","last_clean":"2024-04-14T23:58:22.009932+0000","last_became_active":"2024-04-14T23:48:42.236804+0000","last_became_peered":"2024-04-14T23:48:42.236804+0000","last_unstale":"2024-04-14T23:58:22.009932+0000","last_undegraded":"2024-04-14T23:58:22.009932+0000","last_fullsized":"2024-04-14T23:58:22.009932+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'712","last_scrub_stamp":"2024-04-14T23:58:22.009859+0000","last_deep_scrub":"45'712","last_deep_scrub_stamp":"2024-04-14T23:58:22.009859+0000","last_clean_scrub_stamp":"2024-04-14T23:58:22.009859+0000","objects_scrubbed":6,"log_size":712,"log_dups_size":0,"ondisk_log_size":712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T09:22:34.504411+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583584,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":41866,"num_write":698,"num_write_kb":12861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"44'154","reported_seq":2285,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:26.250834+0000","last_change":"2024-04-14T23:58:26.250877+0000","last_active":"2024-04-14T23:58:26.250834+0000","last_peered":"2024-04-14T23:58:26.250834+0000","last_clean":"2024-04-14T23:58:26.250834+0000","last_became_active":"2024-04-14T23:48:43.221462+0000","last_became_peered":"2024-04-14T23:48:43.221462+0000","last_unstale":"2024-04-14T23:58:26.250834+0000","last_undegraded":"2024-04-14T23:58:26.250834+0000","last_fullsized":"2024-04-14T23:58:26.250834+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'154","last_scrub_stamp":"2024-04-14T23:58:26.250688+0000","last_deep_scrub":"44'154","last_deep_scrub_stamp":"2024-04-14T23:58:26.250688+0000","last_clean_scrub_stamp":"2024-04-14T23:58:26.250688+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:22:44.985705+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0033453850000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":154,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"45'104","reported_seq":197,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:13.839366+0000","last_change":"2024-04-14T23:58:13.839436+0000","last_active":"2024-04-14T23:58:13.839366+0000","last_peered":"2024-04-14T23:58:13.839366+0000","last_clean":"2024-04-14T23:58:13.839366+0000","last_became_active":"2024-04-14T23:48:42.229758+0000","last_became_peered":"2024-04-14T23:48:42.229758+0000","last_unstale":"2024-04-14T23:58:13.839366+0000","last_undegraded":"2024-04-14T23:58:13.839366+0000","last_fullsized":"2024-04-14T23:58:13.839366+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'104","last_scrub_stamp":"2024-04-14T23:58:13.839195+0000","last_deep_scrub":"45'104","last_deep_scrub_stamp":"2024-04-14T23:58:13.839195+0000","last_clean_scrub_stamp":"2024-04-14T23:58:13.839195+0000","objects_scrubbed":4,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T09:26:22.202312+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388642,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":28,"num_read_kb":32808,"num_write":95,"num_write_kb":8238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"45'163","reported_seq":2301,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:14.863122+0000","last_change":"2024-04-14T23:58:14.863158+0000","last_active":"2024-04-14T23:58:14.863122+0000","last_peered":"2024-04-14T23:58:14.863122+0000","last_clean":"2024-04-14T23:58:14.863122+0000","last_became_active":"2024-04-14T23:48:43.229476+0000","last_became_peered":"2024-04-14T23:48:43.229476+0000","last_unstale":"2024-04-14T23:58:14.863122+0000","last_undegraded":"2024-04-14T23:58:14.863122+0000","last_fullsized":"2024-04-14T23:58:14.863122+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'163","last_scrub_stamp":"2024-04-14T23:58:14.863021+0000","last_deep_scrub":"45'163","last_deep_scrub_stamp":"2024-04-14T23:58:14.863021+0000","last_clean_scrub_stamp":"2024-04-14T23:58:14.863021+0000","objects_scrubbed":0,"log_size":163,"log_dups_size":0,"ondisk_log_size":163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T04:55:25.415340+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.015192783,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":37,"num_read_kb":4115,"num_write":158,"num_write_kb":12367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":75,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:18.979387+0000","last_change":"2024-04-14T23:58:18.979422+0000","last_active":"2024-04-14T23:58:18.979387+0000","last_peered":"2024-04-14T23:58:18.979387+0000","last_clean":"2024-04-14T23:58:18.979387+0000","last_became_active":"2024-04-14T23:48:42.225211+0000","last_became_peered":"2024-04-14T23:48:42.225211+0000","last_unstale":"2024-04-14T23:58:18.979387+0000","last_undegraded":"2024-04-14T23:58:18.979387+0000","last_fullsized":"2024-04-14T23:58:18.979387+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T23:58:18.979226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T23:58:18.979226+0000","last_clean_scrub_stamp":"2024-04-14T23:58:18.979226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:36:52.747927+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"36'172","reported_seq":2317,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:29.274851+0000","last_change":"2024-04-14T23:58:29.274876+0000","last_active":"2024-04-14T23:58:29.274851+0000","last_peered":"2024-04-14T23:58:29.274851+0000","last_clean":"2024-04-14T23:58:29.274851+0000","last_became_active":"2024-04-14T23:48:43.229297+0000","last_became_peered":"2024-04-14T23:48:43.229297+0000","last_unstale":"2024-04-14T23:58:29.274851+0000","last_undegraded":"2024-04-14T23:58:29.274851+0000","last_fullsized":"2024-04-14T23:58:29.274851+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'172","last_scrub_stamp":"2024-04-14T23:58:29.274763+0000","last_deep_scrub":"36'172","last_deep_scrub_stamp":"2024-04-14T23:58:29.274763+0000","last_clean_scrub_stamp":"2024-04-14T23:58:29.274763+0000","objects_scrubbed":0,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:31:45.503020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029516669999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":43,"num_read_kb":23,"num_write":172,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"45'4","reported_seq":79,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:20.916869+0000","last_change":"2024-04-14T23:58:20.916923+0000","last_active":"2024-04-14T23:58:20.916869+0000","last_peered":"2024-04-14T23:58:20.916869+0000","last_clean":"2024-04-14T23:58:20.916869+0000","last_became_active":"2024-04-14T23:48:42.220145+0000","last_became_peered":"2024-04-14T23:48:42.220145+0000","last_unstale":"2024-04-14T23:58:20.916869+0000","last_undegraded":"2024-04-14T23:58:20.916869+0000","last_fullsized":"2024-04-14T23:58:20.916869+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'4","last_scrub_stamp":"2024-04-14T23:58:20.916767+0000","last_deep_scrub":"45'4","last_deep_scrub_stamp":"2024-04-14T23:58:20.916767+0000","last_clean_scrub_stamp":"2024-04-14T23:58:20.916767+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:21:14.823161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"45'169","reported_seq":2306,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:29.082415+0000","last_change":"2024-04-14T23:58:29.082442+0000","last_active":"2024-04-14T23:58:29.082415+0000","last_peered":"2024-04-14T23:58:29.082415+0000","last_clean":"2024-04-14T23:58:29.082415+0000","last_became_active":"2024-04-14T23:48:43.235319+0000","last_became_peered":"2024-04-14T23:48:43.235319+0000","last_unstale":"2024-04-14T23:58:29.082415+0000","last_undegraded":"2024-04-14T23:58:29.082415+0000","last_fullsized":"2024-04-14T23:58:29.082415+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'169","last_scrub_stamp":"2024-04-14T23:58:29.082332+0000","last_deep_scrub":"45'169","last_deep_scrub_stamp":"2024-04-14T23:58:29.082332+0000","last_clean_scrub_stamp":"2024-04-14T23:58:29.082332+0000","objects_scrubbed":0,"log_size":169,"log_dups_size":0,"ondisk_log_size":169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T04:01:24.282441+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0017199240000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":36,"num_read_kb":14,"num_write":164,"num_write_kb":82,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"45'107","reported_seq":214,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:14.813752+0000","last_change":"2024-04-14T23:58:14.813791+0000","last_active":"2024-04-14T23:58:14.813752+0000","last_peered":"2024-04-14T23:58:14.813752+0000","last_clean":"2024-04-14T23:58:14.813752+0000","last_became_active":"2024-04-14T23:48:42.230792+0000","last_became_peered":"2024-04-14T23:48:42.230792+0000","last_unstale":"2024-04-14T23:58:14.813752+0000","last_undegraded":"2024-04-14T23:58:14.813752+0000","last_fullsized":"2024-04-14T23:58:14.813752+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'107","last_scrub_stamp":"2024-04-14T23:58:14.813678+0000","last_deep_scrub":"45'107","last_deep_scrub_stamp":"2024-04-14T23:58:14.813678+0000","last_clean_scrub_stamp":"2024-04-14T23:58:14.813678+0000","objects_scrubbed":6,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:26:27.281587+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":84,"num_read_kb":20558,"num_write":102,"num_write_kb":8239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"45'182","reported_seq":2325,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:27.295668+0000","last_change":"2024-04-14T23:58:27.295695+0000","last_active":"2024-04-14T23:58:27.295668+0000","last_peered":"2024-04-14T23:58:27.295668+0000","last_clean":"2024-04-14T23:58:27.295668+0000","last_became_active":"2024-04-14T23:48:43.228004+0000","last_became_peered":"2024-04-14T23:48:43.228004+0000","last_unstale":"2024-04-14T23:58:27.295668+0000","last_undegraded":"2024-04-14T23:58:27.295668+0000","last_fullsized":"2024-04-14T23:58:27.295668+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'182","last_scrub_stamp":"2024-04-14T23:58:27.295581+0000","last_deep_scrub":"45'182","last_deep_scrub_stamp":"2024-04-14T23:58:27.295581+0000","last_clean_scrub_stamp":"2024-04-14T23:58:27.295581+0000","objects_scrubbed":0,"log_size":182,"log_dups_size":0,"ondisk_log_size":182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T04:36:57.840939+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.003032402,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":44,"num_read_kb":8209,"num_write":176,"num_write_kb":12374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"45'24","reported_seq":107,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:16.219780+0000","last_change":"2024-04-14T23:58:16.219815+0000","last_active":"2024-04-14T23:58:16.219780+0000","last_peered":"2024-04-14T23:58:16.219780+0000","last_clean":"2024-04-14T23:58:16.219780+0000","last_became_active":"2024-04-14T23:48:42.231612+0000","last_became_peered":"2024-04-14T23:48:42.231612+0000","last_unstale":"2024-04-14T23:58:16.219780+0000","last_undegraded":"2024-04-14T23:58:16.219780+0000","last_fullsized":"2024-04-14T23:58:16.219780+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'24","last_scrub_stamp":"2024-04-14T23:58:16.219700+0000","last_deep_scrub":"45'24","last_deep_scrub_stamp":"2024-04-14T23:58:16.219700+0000","last_clean_scrub_stamp":"2024-04-14T23:58:16.219700+0000","objects_scrubbed":2,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:26:31.980326+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194326,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":12293,"num_write":21,"num_write_kb":4101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"36'127","reported_seq":2255,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:28.315880+0000","last_change":"2024-04-14T23:58:28.315905+0000","last_active":"2024-04-14T23:58:28.315880+0000","last_peered":"2024-04-14T23:58:28.315880+0000","last_clean":"2024-04-14T23:58:28.315880+0000","last_became_active":"2024-04-14T23:48:43.228665+0000","last_became_peered":"2024-04-14T23:48:43.228665+0000","last_unstale":"2024-04-14T23:58:28.315880+0000","last_undegraded":"2024-04-14T23:58:28.315880+0000","last_fullsized":"2024-04-14T23:58:28.315880+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"36'127","last_scrub_stamp":"2024-04-14T23:58:28.315773+0000","last_deep_scrub":"36'127","last_deep_scrub_stamp":"2024-04-14T23:58:28.315773+0000","last_clean_scrub_stamp":"2024-04-14T23:58:28.315773+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:48:22.206749+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.003047124,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":27,"num_read_kb":4107,"num_write":124,"num_write_kb":4157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"45'103","reported_seq":188,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:16.795680+0000","last_change":"2024-04-14T23:58:16.795708+0000","last_active":"2024-04-14T23:58:16.795680+0000","last_peered":"2024-04-14T23:58:16.795680+0000","last_clean":"2024-04-14T23:58:16.795680+0000","last_became_active":"2024-04-14T23:48:42.239505+0000","last_became_peered":"2024-04-14T23:48:42.239505+0000","last_unstale":"2024-04-14T23:58:16.795680+0000","last_undegraded":"2024-04-14T23:58:16.795680+0000","last_fullsized":"2024-04-14T23:58:16.795680+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'103","last_scrub_stamp":"2024-04-14T23:58:16.795612+0000","last_deep_scrub":"45'103","last_deep_scrub_stamp":"2024-04-14T23:58:16.795612+0000","last_clean_scrub_stamp":"2024-04-14T23:58:16.795612+0000","objects_scrubbed":2,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T04:11:21.971833+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":24,"num_read_kb":12309,"num_write":100,"num_write_kb":4158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"45'165","reported_seq":2303,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:30.257933+0000","last_change":"2024-04-14T23:58:30.257959+0000","last_active":"2024-04-14T23:58:30.257933+0000","last_peered":"2024-04-14T23:58:30.257933+0000","last_clean":"2024-04-14T23:58:30.257933+0000","last_became_active":"2024-04-14T23:48:43.229949+0000","last_became_peered":"2024-04-14T23:48:43.229949+0000","last_unstale":"2024-04-14T23:58:30.257933+0000","last_undegraded":"2024-04-14T23:58:30.257933+0000","last_fullsized":"2024-04-14T23:58:30.257933+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'165","last_scrub_stamp":"2024-04-14T23:58:30.257849+0000","last_deep_scrub":"45'165","last_deep_scrub_stamp":"2024-04-14T23:58:30.257849+0000","last_clean_scrub_stamp":"2024-04-14T23:58:30.257849+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:36:21.363154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002291033,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":37,"num_read_kb":4115,"num_write":160,"num_write_kb":4175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"45'20","reported_seq":104,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:14.041076+0000","last_change":"2024-04-14T23:58:14.041148+0000","last_active":"2024-04-14T23:58:14.041076+0000","last_peered":"2024-04-14T23:58:14.041076+0000","last_clean":"2024-04-14T23:58:14.041076+0000","last_became_active":"2024-04-14T23:48:42.226845+0000","last_became_peered":"2024-04-14T23:48:42.226845+0000","last_unstale":"2024-04-14T23:58:14.041076+0000","last_undegraded":"2024-04-14T23:58:14.041076+0000","last_fullsized":"2024-04-14T23:58:14.041076+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'20","last_scrub_stamp":"2024-04-14T23:58:14.040948+0000","last_deep_scrub":"45'20","last_deep_scrub_stamp":"2024-04-14T23:58:14.040948+0000","last_clean_scrub_stamp":"2024-04-14T23:58:14.040948+0000","objects_scrubbed":2,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:46:15.430380+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194374,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":9,"num_read_kb":16389,"num_write":18,"num_write_kb":4108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"43'183","reported_seq":2313,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:23.987704+0000","last_change":"2024-04-14T23:58:23.987729+0000","last_active":"2024-04-14T23:58:23.987704+0000","last_peered":"2024-04-14T23:58:23.987704+0000","last_clean":"2024-04-14T23:58:23.987704+0000","last_became_active":"2024-04-14T23:48:43.226717+0000","last_became_peered":"2024-04-14T23:48:43.226717+0000","last_unstale":"2024-04-14T23:58:23.987704+0000","last_undegraded":"2024-04-14T23:58:23.987704+0000","last_fullsized":"2024-04-14T23:58:23.987704+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'183","last_scrub_stamp":"2024-04-14T23:58:23.987581+0000","last_deep_scrub":"43'183","last_deep_scrub_stamp":"2024-04-14T23:58:23.987581+0000","last_clean_scrub_stamp":"2024-04-14T23:58:23.987581+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:42:05.247888+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0022579950000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":28,"num_read_kb":14,"num_write":180,"num_write_kb":90,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"34'2","reported_seq":77,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:15.777831+0000","last_change":"2024-04-14T23:58:15.777855+0000","last_active":"2024-04-14T23:58:15.777831+0000","last_peered":"2024-04-14T23:58:15.777831+0000","last_clean":"2024-04-14T23:58:15.777831+0000","last_became_active":"2024-04-14T23:48:42.229785+0000","last_became_peered":"2024-04-14T23:48:42.229785+0000","last_unstale":"2024-04-14T23:58:15.777831+0000","last_undegraded":"2024-04-14T23:58:15.777831+0000","last_fullsized":"2024-04-14T23:58:15.777831+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'2","last_scrub_stamp":"2024-04-14T23:58:15.777751+0000","last_deep_scrub":"34'2","last_deep_scrub_stamp":"2024-04-14T23:58:15.777751+0000","last_clean_scrub_stamp":"2024-04-14T23:58:15.777751+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:23:35.526087+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"45'174","reported_seq":2307,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:33.154777+0000","last_change":"2024-04-14T23:58:33.154824+0000","last_active":"2024-04-14T23:58:33.154777+0000","last_peered":"2024-04-14T23:58:33.154777+0000","last_clean":"2024-04-14T23:58:33.154777+0000","last_became_active":"2024-04-14T23:48:43.232612+0000","last_became_peered":"2024-04-14T23:48:43.232612+0000","last_unstale":"2024-04-14T23:58:33.154777+0000","last_undegraded":"2024-04-14T23:58:33.154777+0000","last_fullsized":"2024-04-14T23:58:33.154777+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'174","last_scrub_stamp":"2024-04-14T23:58:33.154677+0000","last_deep_scrub":"45'174","last_deep_scrub_stamp":"2024-04-14T23:58:33.154677+0000","last_clean_scrub_stamp":"2024-04-14T23:58:33.154677+0000","objects_scrubbed":0,"log_size":174,"log_dups_size":0,"ondisk_log_size":174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:28:31.149650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031586029999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":31,"num_read_kb":17,"num_write":174,"num_write_kb":7493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"45'119","reported_seq":233,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:19.285334+0000","last_change":"2024-04-14T23:58:19.285388+0000","last_active":"2024-04-14T23:58:19.285334+0000","last_peered":"2024-04-14T23:58:19.285334+0000","last_clean":"2024-04-14T23:58:19.285334+0000","last_became_active":"2024-04-14T23:48:38.149301+0000","last_became_peered":"2024-04-14T23:48:38.149301+0000","last_unstale":"2024-04-14T23:58:19.285334+0000","last_undegraded":"2024-04-14T23:58:19.285334+0000","last_fullsized":"2024-04-14T23:58:19.285334+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'119","last_scrub_stamp":"2024-04-14T23:58:19.285239+0000","last_deep_scrub":"45'119","last_deep_scrub_stamp":"2024-04-14T23:58:19.285239+0000","last_clean_scrub_stamp":"2024-04-14T23:58:19.285239+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:46:15.812438+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"45'4","reported_seq":79,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:19.941407+0000","last_change":"2024-04-14T23:58:19.941431+0000","last_active":"2024-04-14T23:58:19.941407+0000","last_peered":"2024-04-14T23:58:19.941407+0000","last_clean":"2024-04-14T23:58:19.941407+0000","last_became_active":"2024-04-14T23:48:42.228994+0000","last_became_peered":"2024-04-14T23:48:42.228994+0000","last_unstale":"2024-04-14T23:58:19.941407+0000","last_undegraded":"2024-04-14T23:58:19.941407+0000","last_fullsized":"2024-04-14T23:58:19.941407+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'4","last_scrub_stamp":"2024-04-14T23:58:19.941295+0000","last_deep_scrub":"45'4","last_deep_scrub_stamp":"2024-04-14T23:58:19.941295+0000","last_clean_scrub_stamp":"2024-04-14T23:58:19.941295+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T09:29:26.488781+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"45'164","reported_seq":2299,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:18.700691+0000","last_change":"2024-04-14T23:58:18.700751+0000","last_active":"2024-04-14T23:58:18.700691+0000","last_peered":"2024-04-14T23:58:18.700691+0000","last_clean":"2024-04-14T23:58:18.700691+0000","last_became_active":"2024-04-14T23:48:43.223665+0000","last_became_peered":"2024-04-14T23:48:43.223665+0000","last_unstale":"2024-04-14T23:58:18.700691+0000","last_undegraded":"2024-04-14T23:58:18.700691+0000","last_fullsized":"2024-04-14T23:58:18.700691+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'164","last_scrub_stamp":"2024-04-14T23:58:18.700496+0000","last_deep_scrub":"45'164","last_deep_scrub_stamp":"2024-04-14T23:58:18.700496+0000","last_clean_scrub_stamp":"2024-04-14T23:58:18.700496+0000","objects_scrubbed":0,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T04:15:10.865167+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.012914624,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":33,"num_read_kb":14,"num_write":162,"num_write_kb":4180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"45'103","reported_seq":184,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:23.281566+0000","last_change":"2024-04-14T23:58:23.281617+0000","last_active":"2024-04-14T23:58:23.281566+0000","last_peered":"2024-04-14T23:58:23.281566+0000","last_clean":"2024-04-14T23:58:23.281566+0000","last_became_active":"2024-04-14T23:48:42.230173+0000","last_became_peered":"2024-04-14T23:48:42.230173+0000","last_unstale":"2024-04-14T23:58:23.281566+0000","last_undegraded":"2024-04-14T23:58:23.281566+0000","last_fullsized":"2024-04-14T23:58:23.281566+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'103","last_scrub_stamp":"2024-04-14T23:58:23.281432+0000","last_deep_scrub":"45'103","last_deep_scrub_stamp":"2024-04-14T23:58:23.281432+0000","last_clean_scrub_stamp":"2024-04-14T23:58:23.281432+0000","objects_scrubbed":3,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:56:04.430946+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8882093,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":92,"num_write_kb":8721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"42'152","reported_seq":2281,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:15.863020+0000","last_change":"2024-04-14T23:58:15.863134+0000","last_active":"2024-04-14T23:58:15.863020+0000","last_peered":"2024-04-14T23:58:15.863020+0000","last_clean":"2024-04-14T23:58:15.863020+0000","last_became_active":"2024-04-14T23:48:43.231025+0000","last_became_peered":"2024-04-14T23:48:43.231025+0000","last_unstale":"2024-04-14T23:58:15.863020+0000","last_undegraded":"2024-04-14T23:58:15.863020+0000","last_fullsized":"2024-04-14T23:58:15.863020+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"42'152","last_scrub_stamp":"2024-04-14T23:58:15.862782+0000","last_deep_scrub":"42'152","last_deep_scrub_stamp":"2024-04-14T23:58:15.862782+0000","last_clean_scrub_stamp":"2024-04-14T23:58:15.862782+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:30:29.895971+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.014709888000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":27,"num_read_kb":10,"num_write":150,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"45'135","reported_seq":344,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:17.793410+0000","last_change":"2024-04-14T23:58:17.793444+0000","last_active":"2024-04-14T23:58:17.793410+0000","last_peered":"2024-04-14T23:58:17.793410+0000","last_clean":"2024-04-14T23:58:17.793410+0000","last_became_active":"2024-04-14T23:48:42.240351+0000","last_became_peered":"2024-04-14T23:48:42.240351+0000","last_unstale":"2024-04-14T23:58:17.793410+0000","last_undegraded":"2024-04-14T23:58:17.793410+0000","last_fullsized":"2024-04-14T23:58:17.793410+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'135","last_scrub_stamp":"2024-04-14T23:58:17.793309+0000","last_deep_scrub":"45'135","last_deep_scrub_stamp":"2024-04-14T23:58:17.793309+0000","last_clean_scrub_stamp":"2024-04-14T23:58:17.793309+0000","objects_scrubbed":1,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:32:49.226309+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536520,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":134,"num_read_kb":610,"num_write":129,"num_write_kb":610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"44'152","reported_seq":2283,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:25.039386+0000","last_change":"2024-04-14T23:58:25.039445+0000","last_active":"2024-04-14T23:58:25.039386+0000","last_peered":"2024-04-14T23:58:25.039386+0000","last_clean":"2024-04-14T23:58:25.039386+0000","last_became_active":"2024-04-14T23:48:43.232120+0000","last_became_peered":"2024-04-14T23:48:43.232120+0000","last_unstale":"2024-04-14T23:58:25.039386+0000","last_undegraded":"2024-04-14T23:58:25.039386+0000","last_fullsized":"2024-04-14T23:58:25.039386+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'152","last_scrub_stamp":"2024-04-14T23:58:25.039188+0000","last_deep_scrub":"44'152","last_deep_scrub_stamp":"2024-04-14T23:58:25.039188+0000","last_clean_scrub_stamp":"2024-04-14T23:58:25.039188+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T03:14:23.219955+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0096232650000000006,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":29,"num_read_kb":17,"num_write":152,"num_write_kb":4171,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"45'158","reported_seq":239,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:20.992696+0000","last_change":"2024-04-14T23:58:20.992726+0000","last_active":"2024-04-14T23:58:20.992696+0000","last_peered":"2024-04-14T23:58:20.992696+0000","last_clean":"2024-04-14T23:58:20.992696+0000","last_became_active":"2024-04-14T23:48:42.233354+0000","last_became_peered":"2024-04-14T23:48:42.233354+0000","last_unstale":"2024-04-14T23:58:20.992696+0000","last_undegraded":"2024-04-14T23:58:20.992696+0000","last_fullsized":"2024-04-14T23:58:20.992696+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'158","last_scrub_stamp":"2024-04-14T23:58:20.992582+0000","last_deep_scrub":"45'158","last_deep_scrub_stamp":"2024-04-14T23:58:20.992582+0000","last_clean_scrub_stamp":"2024-04-14T23:58:20.992582+0000","objects_scrubbed":2,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:40:27.818112+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":154,"num_write_kb":8284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"45'184","reported_seq":2324,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:20.049426+0000","last_change":"2024-04-14T23:58:20.049465+0000","last_active":"2024-04-14T23:58:20.049426+0000","last_peered":"2024-04-14T23:58:20.049426+0000","last_clean":"2024-04-14T23:58:20.049426+0000","last_became_active":"2024-04-14T23:48:43.232119+0000","last_became_peered":"2024-04-14T23:48:43.232119+0000","last_unstale":"2024-04-14T23:58:20.049426+0000","last_undegraded":"2024-04-14T23:58:20.049426+0000","last_fullsized":"2024-04-14T23:58:20.049426+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'184","last_scrub_stamp":"2024-04-14T23:58:20.049317+0000","last_deep_scrub":"45'184","last_deep_scrub_stamp":"2024-04-14T23:58:20.049317+0000","last_clean_scrub_stamp":"2024-04-14T23:58:20.049317+0000","objects_scrubbed":0,"log_size":184,"log_dups_size":0,"ondisk_log_size":184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:16:13.123824+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0012271420000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":38,"num_read_kb":16,"num_write":184,"num_write_kb":92,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"45'4","reported_seq":79,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:19.081165+0000","last_change":"2024-04-14T23:58:19.081223+0000","last_active":"2024-04-14T23:58:19.081165+0000","last_peered":"2024-04-14T23:58:19.081165+0000","last_clean":"2024-04-14T23:58:19.081165+0000","last_became_active":"2024-04-14T23:48:42.242039+0000","last_became_peered":"2024-04-14T23:48:42.242039+0000","last_unstale":"2024-04-14T23:58:19.081165+0000","last_undegraded":"2024-04-14T23:58:19.081165+0000","last_fullsized":"2024-04-14T23:58:19.081165+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'4","last_scrub_stamp":"2024-04-14T23:58:19.080991+0000","last_deep_scrub":"45'4","last_deep_scrub_stamp":"2024-04-14T23:58:19.080991+0000","last_clean_scrub_stamp":"2024-04-14T23:58:19.080991+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T04:44:48.257025+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"44'162","reported_seq":2300,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:29.851718+0000","last_change":"2024-04-14T23:58:29.851781+0000","last_active":"2024-04-14T23:58:29.851718+0000","last_peered":"2024-04-14T23:58:29.851718+0000","last_clean":"2024-04-14T23:58:29.851718+0000","last_became_active":"2024-04-14T23:48:43.241776+0000","last_became_peered":"2024-04-14T23:48:43.241776+0000","last_unstale":"2024-04-14T23:58:29.851718+0000","last_undegraded":"2024-04-14T23:58:29.851718+0000","last_fullsized":"2024-04-14T23:58:29.851718+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'162","last_scrub_stamp":"2024-04-14T23:58:29.851617+0000","last_deep_scrub":"44'162","last_deep_scrub_stamp":"2024-04-14T23:58:29.851617+0000","last_clean_scrub_stamp":"2024-04-14T23:58:29.851617+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:33:27.214087+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0024520940000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":36,"num_read_kb":17,"num_write":162,"num_write_kb":8271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"45'31","reported_seq":109,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:25.294143+0000","last_change":"2024-04-14T23:58:25.294189+0000","last_active":"2024-04-14T23:58:25.294143+0000","last_peered":"2024-04-14T23:58:25.294143+0000","last_clean":"2024-04-14T23:58:25.294143+0000","last_became_active":"2024-04-14T23:48:42.232325+0000","last_became_peered":"2024-04-14T23:48:42.232325+0000","last_unstale":"2024-04-14T23:58:25.294143+0000","last_undegraded":"2024-04-14T23:58:25.294143+0000","last_fullsized":"2024-04-14T23:58:25.294143+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'31","last_scrub_stamp":"2024-04-14T23:58:25.294029+0000","last_deep_scrub":"45'31","last_deep_scrub_stamp":"2024-04-14T23:58:25.294029+0000","last_clean_scrub_stamp":"2024-04-14T23:58:25.294029+0000","objects_scrubbed":1,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:26:41.380693+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":25,"num_write_kb":4108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"44'166","reported_seq":2297,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:24.054847+0000","last_change":"2024-04-14T23:58:24.054963+0000","last_active":"2024-04-14T23:58:24.054847+0000","last_peered":"2024-04-14T23:58:24.054847+0000","last_clean":"2024-04-14T23:58:24.054847+0000","last_became_active":"2024-04-14T23:48:43.234150+0000","last_became_peered":"2024-04-14T23:48:43.234150+0000","last_unstale":"2024-04-14T23:58:24.054847+0000","last_undegraded":"2024-04-14T23:58:24.054847+0000","last_fullsized":"2024-04-14T23:58:24.054847+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'166","last_scrub_stamp":"2024-04-14T23:58:24.054573+0000","last_deep_scrub":"44'166","last_deep_scrub_stamp":"2024-04-14T23:58:24.054573+0000","last_clean_scrub_stamp":"2024-04-14T23:58:24.054573+0000","objects_scrubbed":0,"log_size":166,"log_dups_size":0,"ondisk_log_size":166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:16:13.706243+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.011050757,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":29,"num_read_kb":14,"num_write":166,"num_write_kb":4178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"45'186","reported_seq":267,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:15.045894+0000","last_change":"2024-04-14T23:58:15.045926+0000","last_active":"2024-04-14T23:58:15.045894+0000","last_peered":"2024-04-14T23:58:15.045894+0000","last_clean":"2024-04-14T23:58:15.045894+0000","last_became_active":"2024-04-14T23:48:42.229712+0000","last_became_peered":"2024-04-14T23:48:42.229712+0000","last_unstale":"2024-04-14T23:58:15.045894+0000","last_undegraded":"2024-04-14T23:58:15.045894+0000","last_fullsized":"2024-04-14T23:58:15.045894+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'186","last_scrub_stamp":"2024-04-14T23:58:15.045794+0000","last_deep_scrub":"45'186","last_deep_scrub_stamp":"2024-04-14T23:58:15.045794+0000","last_clean_scrub_stamp":"2024-04-14T23:58:15.045794+0000","objects_scrubbed":1,"log_size":186,"log_dups_size":0,"ondisk_log_size":186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T04:25:47.800171+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":142,"num_write":420,"num_write_kb":618,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"44'166","reported_seq":2306,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:20.750901+0000","last_change":"2024-04-14T23:58:20.750961+0000","last_active":"2024-04-14T23:58:20.750901+0000","last_peered":"2024-04-14T23:58:20.750901+0000","last_clean":"2024-04-14T23:58:20.750901+0000","last_became_active":"2024-04-14T23:48:43.228739+0000","last_became_peered":"2024-04-14T23:48:43.228739+0000","last_unstale":"2024-04-14T23:58:20.750901+0000","last_undegraded":"2024-04-14T23:58:20.750901+0000","last_fullsized":"2024-04-14T23:58:20.750901+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'166","last_scrub_stamp":"2024-04-14T23:58:20.750720+0000","last_deep_scrub":"44'166","last_deep_scrub_stamp":"2024-04-14T23:58:20.750720+0000","last_clean_scrub_stamp":"2024-04-14T23:58:20.750720+0000","objects_scrubbed":0,"log_size":166,"log_dups_size":0,"ondisk_log_size":166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:20:10.384377+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.012444432,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":38,"num_read_kb":16,"num_write":166,"num_write_kb":4178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"45'210","reported_seq":296,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:19.894528+0000","last_change":"2024-04-14T23:58:19.894565+0000","last_active":"2024-04-14T23:58:19.894528+0000","last_peered":"2024-04-14T23:58:19.894528+0000","last_clean":"2024-04-14T23:58:19.894528+0000","last_became_active":"2024-04-14T23:48:42.219750+0000","last_became_peered":"2024-04-14T23:48:42.219750+0000","last_unstale":"2024-04-14T23:58:19.894528+0000","last_undegraded":"2024-04-14T23:58:19.894528+0000","last_fullsized":"2024-04-14T23:58:19.894528+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'210","last_scrub_stamp":"2024-04-14T23:58:19.894437+0000","last_deep_scrub":"45'210","last_deep_scrub_stamp":"2024-04-14T23:58:19.894437+0000","last_clean_scrub_stamp":"2024-04-14T23:58:19.894437+0000","objects_scrubbed":2,"log_size":210,"log_dups_size":0,"ondisk_log_size":210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:40:50.060823+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":25,"num_read_kb":6094,"num_write":202,"num_write_kb":4203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"45'178","reported_seq":2324,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:16.814667+0000","last_change":"2024-04-14T23:58:16.814712+0000","last_active":"2024-04-14T23:58:16.814667+0000","last_peered":"2024-04-14T23:58:16.814667+0000","last_clean":"2024-04-14T23:58:16.814667+0000","last_became_active":"2024-04-14T23:48:43.232730+0000","last_became_peered":"2024-04-14T23:48:43.232730+0000","last_unstale":"2024-04-14T23:58:16.814667+0000","last_undegraded":"2024-04-14T23:58:16.814667+0000","last_fullsized":"2024-04-14T23:58:16.814667+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'178","last_scrub_stamp":"2024-04-14T23:58:16.814570+0000","last_deep_scrub":"45'178","last_deep_scrub_stamp":"2024-04-14T23:58:16.814570+0000","last_clean_scrub_stamp":"2024-04-14T23:58:16.814570+0000","objects_scrubbed":0,"log_size":178,"log_dups_size":0,"ondisk_log_size":178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:52:27.267230+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.014562113999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":45,"num_read_kb":26,"num_write":173,"num_write_kb":8277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"45'29","reported_seq":107,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:24.256050+0000","last_change":"2024-04-14T23:58:24.256081+0000","last_active":"2024-04-14T23:58:24.256050+0000","last_peered":"2024-04-14T23:58:24.256050+0000","last_clean":"2024-04-14T23:58:24.256050+0000","last_became_active":"2024-04-14T23:48:42.231796+0000","last_became_peered":"2024-04-14T23:48:42.231796+0000","last_unstale":"2024-04-14T23:58:24.256050+0000","last_undegraded":"2024-04-14T23:58:24.256050+0000","last_fullsized":"2024-04-14T23:58:24.256050+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'29","last_scrub_stamp":"2024-04-14T23:58:24.255964+0000","last_deep_scrub":"45'29","last_deep_scrub_stamp":"2024-04-14T23:58:24.255964+0000","last_clean_scrub_stamp":"2024-04-14T23:58:24.255964+0000","objects_scrubbed":1,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T10:37:19.379734+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":27,"num_write_kb":4121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"43'183","reported_seq":2322,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:21.086890+0000","last_change":"2024-04-14T23:58:21.086931+0000","last_active":"2024-04-14T23:58:21.086890+0000","last_peered":"2024-04-14T23:58:21.086890+0000","last_clean":"2024-04-14T23:58:21.086890+0000","last_became_active":"2024-04-14T23:48:43.232594+0000","last_became_peered":"2024-04-14T23:48:43.232594+0000","last_unstale":"2024-04-14T23:58:21.086890+0000","last_undegraded":"2024-04-14T23:58:21.086890+0000","last_fullsized":"2024-04-14T23:58:21.086890+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'183","last_scrub_stamp":"2024-04-14T23:58:21.086776+0000","last_deep_scrub":"43'183","last_deep_scrub_stamp":"2024-04-14T23:58:21.086776+0000","last_clean_scrub_stamp":"2024-04-14T23:58:21.086776+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:27:22.447748+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001144237,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":37,"num_read_kb":20,"num_write":182,"num_write_kb":91,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"45'241","reported_seq":318,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:16.012270+0000","last_change":"2024-04-14T23:58:16.012300+0000","last_active":"2024-04-14T23:58:16.012270+0000","last_peered":"2024-04-14T23:58:16.012270+0000","last_clean":"2024-04-14T23:58:16.012270+0000","last_became_active":"2024-04-14T23:48:42.230902+0000","last_became_peered":"2024-04-14T23:48:42.230902+0000","last_unstale":"2024-04-14T23:58:16.012270+0000","last_undegraded":"2024-04-14T23:58:16.012270+0000","last_fullsized":"2024-04-14T23:58:16.012270+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'241","last_scrub_stamp":"2024-04-14T23:58:16.012202+0000","last_deep_scrub":"45'241","last_deep_scrub_stamp":"2024-04-14T23:58:16.012202+0000","last_clean_scrub_stamp":"2024-04-14T23:58:16.012202+0000","objects_scrubbed":1,"log_size":241,"log_dups_size":0,"ondisk_log_size":241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T11:54:31.822148+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":238,"num_write_kb":4263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"44'168","reported_seq":2303,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:23.790544+0000","last_change":"2024-04-14T23:58:23.790606+0000","last_active":"2024-04-14T23:58:23.790544+0000","last_peered":"2024-04-14T23:58:23.790544+0000","last_clean":"2024-04-14T23:58:23.790544+0000","last_became_active":"2024-04-14T23:48:43.232747+0000","last_became_peered":"2024-04-14T23:48:43.232747+0000","last_unstale":"2024-04-14T23:58:23.790544+0000","last_undegraded":"2024-04-14T23:58:23.790544+0000","last_fullsized":"2024-04-14T23:58:23.790544+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'168","last_scrub_stamp":"2024-04-14T23:58:23.790330+0000","last_deep_scrub":"44'168","last_deep_scrub_stamp":"2024-04-14T23:58:23.790330+0000","last_clean_scrub_stamp":"2024-04-14T23:58:23.790330+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:03:30.100154+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.014090612000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":33,"num_read_kb":16,"num_write":168,"num_write_kb":8274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"45'26","reported_seq":105,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:18.079517+0000","last_change":"2024-04-14T23:58:18.079558+0000","last_active":"2024-04-14T23:58:18.079517+0000","last_peered":"2024-04-14T23:58:18.079517+0000","last_clean":"2024-04-14T23:58:18.079517+0000","last_became_active":"2024-04-14T23:48:42.238524+0000","last_became_peered":"2024-04-14T23:48:42.238524+0000","last_unstale":"2024-04-14T23:58:18.079517+0000","last_undegraded":"2024-04-14T23:58:18.079517+0000","last_fullsized":"2024-04-14T23:58:18.079517+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'26","last_scrub_stamp":"2024-04-14T23:58:18.079359+0000","last_deep_scrub":"45'26","last_deep_scrub_stamp":"2024-04-14T23:58:18.079359+0000","last_clean_scrub_stamp":"2024-04-14T23:58:18.079359+0000","objects_scrubbed":1,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T05:31:33.651137+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":23,"num_write_kb":4102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"45'166","reported_seq":2303,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:25.859111+0000","last_change":"2024-04-14T23:58:25.859169+0000","last_active":"2024-04-14T23:58:25.859111+0000","last_peered":"2024-04-14T23:58:25.859111+0000","last_clean":"2024-04-14T23:58:25.859111+0000","last_became_active":"2024-04-14T23:48:43.233651+0000","last_became_peered":"2024-04-14T23:48:43.233651+0000","last_unstale":"2024-04-14T23:58:25.859111+0000","last_undegraded":"2024-04-14T23:58:25.859111+0000","last_fullsized":"2024-04-14T23:58:25.859111+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'166","last_scrub_stamp":"2024-04-14T23:58:25.858940+0000","last_deep_scrub":"45'166","last_deep_scrub_stamp":"2024-04-14T23:58:25.858940+0000","last_clean_scrub_stamp":"2024-04-14T23:58:25.858940+0000","objects_scrubbed":0,"log_size":166,"log_dups_size":0,"ondisk_log_size":166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T09:29:01.271732+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.015058134000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":13,"num_write":166,"num_write_kb":8274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"45'74","reported_seq":161,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:20.260342+0000","last_change":"2024-04-14T23:58:20.260380+0000","last_active":"2024-04-14T23:58:20.260342+0000","last_peered":"2024-04-14T23:58:20.260342+0000","last_clean":"2024-04-14T23:58:20.260342+0000","last_became_active":"2024-04-14T23:48:42.218580+0000","last_became_peered":"2024-04-14T23:48:42.218580+0000","last_unstale":"2024-04-14T23:58:20.260342+0000","last_undegraded":"2024-04-14T23:58:20.260342+0000","last_fullsized":"2024-04-14T23:58:20.260342+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'74","last_scrub_stamp":"2024-04-14T23:58:20.260253+0000","last_deep_scrub":"45'74","last_deep_scrub_stamp":"2024-04-14T23:58:20.260253+0000","last_clean_scrub_stamp":"2024-04-14T23:58:20.260253+0000","objects_scrubbed":3,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:33:07.813241+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":28682,"num_write":62,"num_write_kb":8222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"35'166","reported_seq":2291,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:22.941423+0000","last_change":"2024-04-14T23:58:22.941488+0000","last_active":"2024-04-14T23:58:22.941423+0000","last_peered":"2024-04-14T23:58:22.941423+0000","last_clean":"2024-04-14T23:58:22.941423+0000","last_became_active":"2024-04-14T23:48:43.219601+0000","last_became_peered":"2024-04-14T23:48:43.219601+0000","last_unstale":"2024-04-14T23:58:22.941423+0000","last_undegraded":"2024-04-14T23:58:22.941423+0000","last_fullsized":"2024-04-14T23:58:22.941423+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'166","last_scrub_stamp":"2024-04-14T23:58:22.941301+0000","last_deep_scrub":"35'166","last_deep_scrub_stamp":"2024-04-14T23:58:22.941301+0000","last_clean_scrub_stamp":"2024-04-14T23:58:22.941301+0000","objects_scrubbed":0,"log_size":166,"log_dups_size":0,"ondisk_log_size":166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T01:06:07.167981+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001540055,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":23,"num_read_kb":13,"num_write":166,"num_write_kb":83,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"44'151","reported_seq":2295,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:22.004945+0000","last_change":"2024-04-14T23:58:22.004974+0000","last_active":"2024-04-14T23:58:22.004945+0000","last_peered":"2024-04-14T23:58:22.004945+0000","last_clean":"2024-04-14T23:58:22.004945+0000","last_became_active":"2024-04-14T23:48:43.221842+0000","last_became_peered":"2024-04-14T23:48:43.221842+0000","last_unstale":"2024-04-14T23:58:22.004945+0000","last_undegraded":"2024-04-14T23:58:22.004945+0000","last_fullsized":"2024-04-14T23:58:22.004945+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'151","last_scrub_stamp":"2024-04-14T23:58:22.004855+0000","last_deep_scrub":"44'151","last_deep_scrub_stamp":"2024-04-14T23:58:22.004855+0000","last_clean_scrub_stamp":"2024-04-14T23:58:22.004855+0000","objects_scrubbed":0,"log_size":151,"log_dups_size":0,"ondisk_log_size":151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:27:14.296122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024436879999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":43,"num_read_kb":23,"num_write":150,"num_write_kb":4170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"45'64","reported_seq":148,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:22.209925+0000","last_change":"2024-04-14T23:58:22.209975+0000","last_active":"2024-04-14T23:58:22.209925+0000","last_peered":"2024-04-14T23:58:22.209925+0000","last_clean":"2024-04-14T23:58:22.209925+0000","last_became_active":"2024-04-14T23:48:42.221028+0000","last_became_peered":"2024-04-14T23:48:42.221028+0000","last_unstale":"2024-04-14T23:58:22.209925+0000","last_undegraded":"2024-04-14T23:58:22.209925+0000","last_fullsized":"2024-04-14T23:58:22.209925+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'64","last_scrub_stamp":"2024-04-14T23:58:22.209836+0000","last_deep_scrub":"45'64","last_deep_scrub_stamp":"2024-04-14T23:58:22.209836+0000","last_clean_scrub_stamp":"2024-04-14T23:58:22.209836+0000","objects_scrubbed":2,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T09:44:30.125155+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":24,"num_read_kb":12309,"num_write":62,"num_write_kb":4128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"45'166","reported_seq":2298,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:22.989177+0000","last_change":"2024-04-14T23:58:22.989204+0000","last_active":"2024-04-14T23:58:22.989177+0000","last_peered":"2024-04-14T23:58:22.989177+0000","last_clean":"2024-04-14T23:58:22.989177+0000","last_became_active":"2024-04-14T23:48:43.224312+0000","last_became_peered":"2024-04-14T23:48:43.224312+0000","last_unstale":"2024-04-14T23:58:22.989177+0000","last_undegraded":"2024-04-14T23:58:22.989177+0000","last_fullsized":"2024-04-14T23:58:22.989177+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'166","last_scrub_stamp":"2024-04-14T23:58:22.989090+0000","last_deep_scrub":"45'166","last_deep_scrub_stamp":"2024-04-14T23:58:22.989090+0000","last_clean_scrub_stamp":"2024-04-14T23:58:22.989090+0000","objects_scrubbed":0,"log_size":166,"log_dups_size":0,"ondisk_log_size":166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:13:29.117737+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00232957,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":31,"num_read_kb":13,"num_write":166,"num_write_kb":4182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"45'674","reported_seq":768,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:21.245890+0000","last_change":"2024-04-14T23:58:21.245928+0000","last_active":"2024-04-14T23:58:21.245890+0000","last_peered":"2024-04-14T23:58:21.245890+0000","last_clean":"2024-04-14T23:58:21.245890+0000","last_became_active":"2024-04-14T23:48:42.219813+0000","last_became_peered":"2024-04-14T23:48:42.219813+0000","last_unstale":"2024-04-14T23:58:21.245890+0000","last_undegraded":"2024-04-14T23:58:21.245890+0000","last_fullsized":"2024-04-14T23:58:21.245890+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'674","last_scrub_stamp":"2024-04-14T23:58:21.245801+0000","last_deep_scrub":"45'674","last_deep_scrub_stamp":"2024-04-14T23:58:21.245801+0000","last_clean_scrub_stamp":"2024-04-14T23:58:21.245801+0000","objects_scrubbed":4,"log_size":674,"log_dups_size":0,"ondisk_log_size":674,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:34:37.696248+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":44198,"num_write":659,"num_write_kb":12787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"37'165","reported_seq":2302,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:19.706014+0000","last_change":"2024-04-14T23:58:19.706132+0000","last_active":"2024-04-14T23:58:19.706014+0000","last_peered":"2024-04-14T23:58:19.706014+0000","last_clean":"2024-04-14T23:58:19.706014+0000","last_became_active":"2024-04-14T23:48:43.236595+0000","last_became_peered":"2024-04-14T23:48:43.236595+0000","last_unstale":"2024-04-14T23:58:19.706014+0000","last_undegraded":"2024-04-14T23:58:19.706014+0000","last_fullsized":"2024-04-14T23:58:19.706014+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'165","last_scrub_stamp":"2024-04-14T23:58:19.705790+0000","last_deep_scrub":"37'165","last_deep_scrub_stamp":"2024-04-14T23:58:19.705790+0000","last_clean_scrub_stamp":"2024-04-14T23:58:19.705790+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T06:38:26.930221+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.013391734000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":19,"num_write":164,"num_write_kb":82,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"45'49","reported_seq":129,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:16.791106+0000","last_change":"2024-04-14T23:58:16.791226+0000","last_active":"2024-04-14T23:58:16.791106+0000","last_peered":"2024-04-14T23:58:16.791106+0000","last_clean":"2024-04-14T23:58:16.791106+0000","last_became_active":"2024-04-14T23:48:42.238892+0000","last_became_peered":"2024-04-14T23:48:42.238892+0000","last_unstale":"2024-04-14T23:58:16.791106+0000","last_undegraded":"2024-04-14T23:58:16.791106+0000","last_fullsized":"2024-04-14T23:58:16.791106+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'49","last_scrub_stamp":"2024-04-14T23:58:16.790841+0000","last_deep_scrub":"45'49","last_deep_scrub_stamp":"2024-04-14T23:58:16.790841+0000","last_clean_scrub_stamp":"2024-04-14T23:58:16.790841+0000","objects_scrubbed":1,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T07:30:28.843688+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":16384,"num_write":44,"num_write_kb":4126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"45'176","reported_seq":2308,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:19.738755+0000","last_change":"2024-04-14T23:58:19.738820+0000","last_active":"2024-04-14T23:58:19.738755+0000","last_peered":"2024-04-14T23:58:19.738755+0000","last_clean":"2024-04-14T23:58:19.738755+0000","last_became_active":"2024-04-14T23:48:43.222707+0000","last_became_peered":"2024-04-14T23:48:43.222707+0000","last_unstale":"2024-04-14T23:58:19.738755+0000","last_undegraded":"2024-04-14T23:58:19.738755+0000","last_fullsized":"2024-04-14T23:58:19.738755+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'176","last_scrub_stamp":"2024-04-14T23:58:19.738586+0000","last_deep_scrub":"45'176","last_deep_scrub_stamp":"2024-04-14T23:58:19.738586+0000","last_clean_scrub_stamp":"2024-04-14T23:58:19.738586+0000","objects_scrubbed":0,"log_size":176,"log_dups_size":0,"ondisk_log_size":176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T08:05:49.648926+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.011942019,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":30,"num_read_kb":12,"num_write":176,"num_write_kb":4183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"45'79","reported_seq":169,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-14T23:58:12.850896+0000","last_change":"2024-04-14T23:58:12.850977+0000","last_active":"2024-04-14T23:58:12.850896+0000","last_peered":"2024-04-14T23:58:12.850896+0000","last_clean":"2024-04-14T23:58:12.850896+0000","last_became_active":"2024-04-14T23:48:42.226238+0000","last_became_peered":"2024-04-14T23:48:42.226238+0000","last_unstale":"2024-04-14T23:58:12.850896+0000","last_undegraded":"2024-04-14T23:58:12.850896+0000","last_fullsized":"2024-04-14T23:58:12.850896+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'79","last_scrub_stamp":"2024-04-14T23:58:12.850700+0000","last_deep_scrub":"45'79","last_deep_scrub_stamp":"2024-04-14T23:58:12.850700+0000","last_clean_scrub_stamp":"2024-04-14T23:58:12.850700+0000","objects_scrubbed":3,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T02:09:52.524693+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":25,"num_read_kb":20500,"num_write":71,"num_write_kb":4136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2223,"num_read_kb":65817,"num_write":10661,"num_write_kb":287920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10759,"ondisk_log_size":10759,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":152026419,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":556,"num_read_kb":479121,"num_write":3811,"num_write_kb":151495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5561,"num_omap_keys":11,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":64839680,"data_stored":304052838,"data_compressed":34430816,"data_compressed_allocated":44859392,"data_compressed_original":284131328,"omap_allocated":0,"internal_metadata":0},"log_size":3733,"ondisk_log_size":3733,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65128,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574971,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76420,"kb_used_data":13308,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94295420,"statfs":{"total":96636764160,"available":96558510080,"internally_reserved":0,"allocated":13627392,"data_stored":60044059,"data_compressed":6906850,"data_compressed_allocated":9056256,"data_compressed_original":55754752,"omap_allocated":1590,"internal_metadata":64616906},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574972,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57228,"kb_used_data":12164,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94314612,"statfs":{"total":96636764160,"available":96578162688,"internally_reserved":0,"allocated":12455936,"data_stored":50571688,"data_compressed":5492002,"data_compressed_allocated":7053312,"data_compressed_original":45449216,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574971,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73208,"kb_used_data":5936,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94298632,"statfs":{"total":96636764160,"available":96561799168,"internally_reserved":0,"allocated":6078464,"data_stored":29599586,"data_compressed":3110985,"data_compressed_allocated":4100096,"data_compressed_original":27897856,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574972,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76316,"kb_used_data":9044,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94295524,"statfs":{"total":96636764160,"available":96558616576,"internally_reserved":0,"allocated":9261056,"data_stored":46376838,"data_compressed":5258839,"data_compressed_allocated":6774784,"data_compressed_original":44175360,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574970,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":75612,"kb_used_data":8340,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94296228,"statfs":{"total":96636764160,"available":96559337472,"internally_reserved":0,"allocated":8540160,"data_stored":34330992,"data_compressed":3819253,"data_compressed_allocated":4960256,"data_compressed_original":31031296,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574971,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47116,"kb_used_data":3460,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94324724,"statfs":{"total":96636764160,"available":96588517376,"internally_reserved":0,"allocated":3543040,"data_stored":13358800,"data_compressed":1445694,"data_compressed_allocated":2031616,"data_compressed_original":12120064,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574971,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54240,"kb_used_data":9176,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94317600,"statfs":{"total":96636764160,"available":96581222400,"internally_reserved":0,"allocated":9396224,"data_stored":42676053,"data_compressed":4921274,"data_compressed_allocated":6492160,"data_compressed_original":40054784,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574970,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73292,"kb_used_data":6020,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94298548,"statfs":{"total":96636764160,"available":96561713152,"internally_reserved":0,"allocated":6164480,"data_stored":30190502,"data_compressed":3541047,"data_compressed_allocated":4481024,"data_compressed_original":28794880,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32564,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32564,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5586944,"data_stored":29360766,"data_compressed":3508483,"data_compressed_allocated":4435968,"data_compressed_original":28221440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8884224,"data_stored":42436685,"data_compressed":4921274,"data_compressed_allocated":6492160,"data_compressed_original":40054784,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3031040,"data_stored":13119432,"data_compressed":1445694,"data_compressed_allocated":2031616,"data_compressed_original":12120064,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8028160,"data_stored":34091624,"data_compressed":3819253,"data_compressed_allocated":4960256,"data_compressed_original":31031296,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8749056,"data_stored":46137470,"data_compressed":5258839,"data_compressed_allocated":6774784,"data_compressed_original":44175360,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":5566464,"data_stored":29360218,"data_compressed":3110985,"data_compressed_allocated":4100096,"data_compressed_original":27897856,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":11943936,"data_stored":50332320,"data_compressed":5492002,"data_compressed_allocated":7053312,"data_compressed_original":45449216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":13049856,"data_stored":59214323,"data_compressed":6874286,"data_compressed_allocated":9011200,"data_compressed_original":55181312,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-14T23:58:38.759 DEBUG:teuthology.orchestra.run.smithi060:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-14T23:58:39.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.000+0000 7f868a800640 1 Processor -- start 2024-04-14T23:58:39.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.001+0000 7f868a800640 1 -- start start 2024-04-14T23:58:39.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.002+0000 7f868a800640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f86840c69c0 0x7f8684071e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:39.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.002+0000 7f868a800640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8684073180 0x7f86840c99e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:39.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.002+0000 7f868a800640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8684157950 con 0x7f86840c71c0 2024-04-14T23:58:39.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.002+0000 7f868a800640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f8684151fc0 con 0x7f86840c69c0 2024-04-14T23:58:39.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.002+0000 7f868a800640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f8684158520 con 0x7f8684073180 2024-04-14T23:58:39.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.002+0000 7f8689400640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f86840c69c0 0x7f8684071e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:39.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.002+0000 7f8689e00640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f86840c71c0 0x7f868415ae30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:36742/0 (socket says 172.21.15.60:36742) 2024-04-14T23:58:39.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.002+0000 7f8689e00640 1 -- 172.21.15.60:0/3727645520 learned_addr learned my addr 172.21.15.60:0/3727645520 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:58:39.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.002+0000 7f8688a00640 1 --2- 172.21.15.60:0/3727645520 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8684073180 0x7f86840c99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:39.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.003+0000 7f8689400640 1 -- 172.21.15.60:0/3727645520 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8684073180 msgr2=0x7f86840c99e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:58:39.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.003+0000 7f8689400640 1 --2- 172.21.15.60:0/3727645520 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8684073180 0x7f86840c99e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:39.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.003+0000 7f8689400640 1 -- 172.21.15.60:0/3727645520 >> v1:172.21.15.60:6789/0 conn(0x7f86840c71c0 legacy=0x7f868415ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:39.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.003+0000 7f8689400640 1 -- 172.21.15.60:0/3727645520 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86840c6da0 con 0x7f86840c69c0 2024-04-14T23:58:39.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.004+0000 7f8689400640 1 --2- 172.21.15.60:0/3727645520 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86840c69c0 0x7f8684071e40 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f8684152140 tx=0x7f867802bf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e9671ec53cb6214 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:39.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.004+0000 7f8683e00640 1 -- 172.21.15.60:0/3727645520 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8678039070 con 0x7f86840c69c0 2024-04-14T23:58:39.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.004+0000 7f8683e00640 1 -- 172.21.15.60:0/3727645520 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f867802ccd0 con 0x7f86840c69c0 2024-04-14T23:58:39.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.004+0000 7f8683e00640 1 -- 172.21.15.60:0/3727645520 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8678034660 con 0x7f86840c69c0 2024-04-14T23:58:39.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.005+0000 7f868a800640 1 -- 172.21.15.60:0/3727645520 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86840c69c0 msgr2=0x7f8684071e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:39.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.005+0000 7f868a800640 1 --2- 172.21.15.60:0/3727645520 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86840c69c0 0x7f8684071e40 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f8684152140 tx=0x7f867802bf20 comp rx=0 tx=0).stop 2024-04-14T23:58:39.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.005+0000 7f868a800640 1 -- 172.21.15.60:0/3727645520 shutdown_connections 2024-04-14T23:58:39.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.005+0000 7f868a800640 1 --2- 172.21.15.60:0/3727645520 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8684073180 0x7f86840c99e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:39.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.005+0000 7f868a800640 1 --2- 172.21.15.60:0/3727645520 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86840c69c0 0x7f8684071e40 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:39.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.005+0000 7f868a800640 1 -- 172.21.15.60:0/3727645520 >> 172.21.15.60:0/3727645520 conn(0x7f868406f9e0 msgr2=0x7f868406fdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:39.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.005+0000 7f868a800640 1 -- 172.21.15.60:0/3727645520 shutdown_connections 2024-04-14T23:58:39.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.005+0000 7f868a800640 1 -- 172.21.15.60:0/3727645520 wait complete. 2024-04-14T23:58:39.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.007+0000 7f868a800640 1 Processor -- start 2024-04-14T23:58:39.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.007+0000 7f868a800640 1 -- start start 2024-04-14T23:58:39.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.008+0000 7f868a800640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8684073180 0x7f8684152170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:39.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.008+0000 7f868a800640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86840c69c0 0x7f868414da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:39.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.008+0000 7f868a800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86840c71c0 0x7f86841f09a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:39.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.008+0000 7f868a800640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f86841581b0 con 0x7f86840c69c0 2024-04-14T23:58:39.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.008+0000 7f868a800640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8684157ed0 con 0x7f86840c71c0 2024-04-14T23:58:39.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.008+0000 7f868a800640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f86840736c0 con 0x7f8684073180 2024-04-14T23:58:39.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.008+0000 7f8688a00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86840c69c0 0x7f868414da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:39.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.008+0000 7f8688a00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86840c69c0 0x7f868414da30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48670/0 (socket says 172.21.15.60:48670) 2024-04-14T23:58:39.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.008+0000 7f8688a00640 1 -- 172.21.15.60:0/4146919795 learned_addr learned my addr 172.21.15.60:0/4146919795 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:58:39.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.008+0000 7f8689e00640 1 --2- 172.21.15.60:0/4146919795 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86840c71c0 0x7f86841f09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:39.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.008+0000 7f8689400640 1 --2- 172.21.15.60:0/4146919795 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8684073180 0x7f8684152170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:39.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.009+0000 7f8688a00640 1 -- 172.21.15.60:0/4146919795 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8684073180 msgr2=0x7f8684152170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:39.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.009+0000 7f8688a00640 1 --2- 172.21.15.60:0/4146919795 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8684073180 0x7f8684152170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:39.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.009+0000 7f8688a00640 1 -- 172.21.15.60:0/4146919795 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86840c71c0 msgr2=0x7f86841f09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:39.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.009+0000 7f8688a00640 1 --2- 172.21.15.60:0/4146919795 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86840c71c0 0x7f86841f09a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:39.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.009+0000 7f8688a00640 1 -- 172.21.15.60:0/4146919795 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86841f1450 con 0x7f86840c69c0 2024-04-14T23:58:39.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.009+0000 7f8688a00640 1 --2- 172.21.15.60:0/4146919795 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86840c69c0 0x7f868414da30 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f867000e9b0 tx=0x7f867000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:39.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.009+0000 7f8681600640 1 -- 172.21.15.60:0/4146919795 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8670017070 con 0x7f86840c69c0 2024-04-14T23:58:39.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.009+0000 7f8681600640 1 -- 172.21.15.60:0/4146919795 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f867000c9a0 con 0x7f86840c69c0 2024-04-14T23:58:39.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.009+0000 7f8681600640 1 -- 172.21.15.60:0/4146919795 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f867000cd90 con 0x7f86840c69c0 2024-04-14T23:58:39.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.009+0000 7f868a800640 1 -- 172.21.15.60:0/4146919795 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86841f1730 con 0x7f86840c69c0 2024-04-14T23:58:39.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.010+0000 7f868a800640 1 -- 172.21.15.60:0/4146919795 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86840cb390 con 0x7f86840c69c0 2024-04-14T23:58:39.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.011+0000 7f868a800640 1 -- 172.21.15.60:0/4146919795 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f868404f540 con 0x7f86840c69c0 2024-04-14T23:58:39.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.012+0000 7f8681600640 1 -- 172.21.15.60:0/4146919795 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f867001a070 con 0x7f86840c69c0 2024-04-14T23:58:39.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.013+0000 7f8681600640 1 --2- 172.21.15.60:0/4146919795 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f866406b990 0x7f866406de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:39.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.013+0000 7f8681600640 1 -- 172.21.15.60:0/4146919795 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6255+0+0 (secure 0 0 0) 0x7f867008adf0 con 0x7f86840c69c0 2024-04-14T23:58:39.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.013+0000 7f8689400640 1 --2- 172.21.15.60:0/4146919795 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f866406b990 0x7f866406de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:39.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.014+0000 7f8689400640 1 --2- 172.21.15.60:0/4146919795 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f866406b990 0x7f866406de50 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f8678036000 tx=0x7f8678030000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:39.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.017+0000 7f8681600640 1 -- 172.21.15.60:0/4146919795 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8670059100 con 0x7f86840c69c0 2024-04-14T23:58:39.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.224+0000 7f868a800640 1 -- 172.21.15.60:0/4146919795 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7f8684068540 con 0x7f86840c69c0 2024-04-14T23:58:39.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.231+0000 7f8681600640 1 -- 172.21.15.60:0/4146919795 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v10)=0 v10) ==== 123+0+0 (secure 0 0 0) 0x7f867005d1a0 con 0x7f86840c69c0 2024-04-14T23:58:39.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.233+0000 7f8681600640 1 -- 172.21.15.60:0/4146919795 <== mon.0 v2:172.21.15.60:3300/0 8 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f8670061a60 con 0x7f86840c69c0 2024-04-14T23:58:39.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.235+0000 7f868a800640 1 -- 172.21.15.60:0/4146919795 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f866406b990 msgr2=0x7f866406de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:39.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.235+0000 7f868a800640 1 --2- 172.21.15.60:0/4146919795 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f866406b990 0x7f866406de50 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f8678036000 tx=0x7f8678030000 comp rx=0 tx=0).stop 2024-04-14T23:58:39.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.235+0000 7f868a800640 1 -- 172.21.15.60:0/4146919795 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86840c69c0 msgr2=0x7f868414da30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:39.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.235+0000 7f868a800640 1 --2- 172.21.15.60:0/4146919795 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86840c69c0 0x7f868414da30 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f867000e9b0 tx=0x7f867000ee70 comp rx=0 tx=0).stop 2024-04-14T23:58:39.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.235+0000 7f868a800640 1 -- 172.21.15.60:0/4146919795 shutdown_connections 2024-04-14T23:58:39.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.235+0000 7f868a800640 1 --2- 172.21.15.60:0/4146919795 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f866406b990 0x7f866406de50 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:39.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.235+0000 7f868a800640 1 --2- 172.21.15.60:0/4146919795 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86840c71c0 0x7f86841f09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:39.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.235+0000 7f868a800640 1 --2- 172.21.15.60:0/4146919795 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86840c69c0 0x7f868414da30 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:39.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.235+0000 7f868a800640 1 --2- 172.21.15.60:0/4146919795 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8684073180 0x7f8684152170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:39.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.236+0000 7f868a800640 1 -- 172.21.15.60:0/4146919795 >> 172.21.15.60:0/4146919795 conn(0x7f868406f9e0 msgr2=0x7f868406bb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:39.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.236+0000 7f868a800640 1 -- 172.21.15.60:0/4146919795 shutdown_connections 2024-04-14T23:58:39.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.236+0000 7f868a800640 1 -- 172.21.15.60:0/4146919795 wait complete. 2024-04-14T23:58:39.263 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-14T23:58:39.263 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:58:39.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.476+0000 7f2e9be00640 1 Processor -- start 2024-04-14T23:58:39.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.477+0000 7f2e9be00640 1 -- start start 2024-04-14T23:58:39.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.478+0000 7f2e9be00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2e9c0b1e00 0x7f2e9c0c6950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:39.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.478+0000 7f2e9be00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2e9c0c3d90 0x7f2e9c0c41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:39.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.478+0000 7f2e9be00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e9c152fb0 con 0x7f2e9c073320 2024-04-14T23:58:39.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.478+0000 7f2e9be00640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f2e9c0c6540 con 0x7f2e9c0b1e00 2024-04-14T23:58:39.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.478+0000 7f2e9be00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f2e9c0b21e0 con 0x7f2e9c0c3d90 2024-04-14T23:58:39.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.478+0000 7f2e9aa00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2e9c0b1e00 0x7f2e9c0c6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:39.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.478+0000 7f2e9aa00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2e9c0b1e00 0x7f2e9c0c6950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48678/0 (socket says 172.21.15.60:48678) 2024-04-14T23:58:39.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.478+0000 7f2e9aa00640 1 -- 172.21.15.60:0/3923314388 learned_addr learned my addr 172.21.15.60:0/3923314388 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:58:39.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.478+0000 7f2e9b400640 1 --2- 172.21.15.60:0/3923314388 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2e9c0c3d90 0x7f2e9c0c41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:39.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.479+0000 7f2e9aa00640 1 -- 172.21.15.60:0/3923314388 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2e9c0c3d90 msgr2=0x7f2e9c0c41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:58:39.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.479+0000 7f2e9aa00640 1 --2- 172.21.15.60:0/3923314388 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2e9c0c3d90 0x7f2e9c0c41c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:39.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.479+0000 7f2e9aa00640 1 -- 172.21.15.60:0/3923314388 >> v1:172.21.15.60:6789/0 conn(0x7f2e9c073320 legacy=0x7f2e9c0c7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:39.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.479+0000 7f2e9aa00640 1 -- 172.21.15.60:0/3923314388 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e9c159310 con 0x7f2e9c0b1e00 2024-04-14T23:58:39.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.479+0000 7f2e99600640 1 -- 172.21.15.60:0/3923314388 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1299522668 0 0) 0x7f2e9c152fb0 con 0x7f2e9c073320 2024-04-14T23:58:39.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.480+0000 7f2e9aa00640 1 --2- 172.21.15.60:0/3923314388 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e9c0b1e00 0x7f2e9c0c6950 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f2e8c009870 tx=0x7f2e8c02ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0d038c6c9415325 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:39.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.480+0000 7f2e99600640 1 -- 172.21.15.60:0/3923314388 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e8c02f840 con 0x7f2e9c0b1e00 2024-04-14T23:58:39.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.481+0000 7f2e99600640 1 -- 172.21.15.60:0/3923314388 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f2e8c02f9e0 con 0x7f2e9c0b1e00 2024-04-14T23:58:39.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.481+0000 7f2e99600640 1 -- 172.21.15.60:0/3923314388 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e8c02fdd0 con 0x7f2e9c0b1e00 2024-04-14T23:58:39.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.481+0000 7f2e9be00640 1 -- 172.21.15.60:0/3923314388 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e9c0b1e00 msgr2=0x7f2e9c0c6950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:39.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.481+0000 7f2e9be00640 1 --2- 172.21.15.60:0/3923314388 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e9c0b1e00 0x7f2e9c0c6950 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f2e8c009870 tx=0x7f2e8c02ba20 comp rx=0 tx=0).stop 2024-04-14T23:58:39.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.481+0000 7f2e9be00640 1 -- 172.21.15.60:0/3923314388 shutdown_connections 2024-04-14T23:58:39.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.481+0000 7f2e9be00640 1 --2- 172.21.15.60:0/3923314388 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2e9c0c3d90 0x7f2e9c0c41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:39.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.481+0000 7f2e9be00640 1 --2- 172.21.15.60:0/3923314388 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e9c0b1e00 0x7f2e9c0c6950 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:39.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.481+0000 7f2e9be00640 1 -- 172.21.15.60:0/3923314388 >> 172.21.15.60:0/3923314388 conn(0x7f2e9c0c8b50 msgr2=0x7f2e9c0c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:39.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.481+0000 7f2e9be00640 1 -- 172.21.15.60:0/3923314388 shutdown_connections 2024-04-14T23:58:39.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.481+0000 7f2e9be00640 1 -- 172.21.15.60:0/3923314388 wait complete. 2024-04-14T23:58:39.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.483+0000 7f2e9be00640 1 Processor -- start 2024-04-14T23:58:39.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.483+0000 7f2e9be00640 1 -- start start 2024-04-14T23:58:39.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.484+0000 7f2e9be00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e9c073320 0x7f2e9c1e4170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:39.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.484+0000 7f2e9be00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e9c0b1e00 0x7f2e9c072330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:39.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.484+0000 7f2e9be00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2e9c0c3d90 0x7f2e9c155240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:39.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.484+0000 7f2e9be00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f2e9c0c38b0 con 0x7f2e9c073320 2024-04-14T23:58:39.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.484+0000 7f2e9be00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2e9c159890 con 0x7f2e9c0b1e00 2024-04-14T23:58:39.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.484+0000 7f2e9be00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f2e9c1595b0 con 0x7f2e9c0c3d90 2024-04-14T23:58:39.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.485+0000 7f2e9aa00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e9c073320 0x7f2e9c1e4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:39.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.485+0000 7f2e9aa00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e9c073320 0x7f2e9c1e4170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48688/0 (socket says 172.21.15.60:48688) 2024-04-14T23:58:39.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.485+0000 7f2e9aa00640 1 -- 172.21.15.60:0/1896377892 learned_addr learned my addr 172.21.15.60:0/1896377892 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:58:39.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.485+0000 7f2e9b400640 1 --2- 172.21.15.60:0/1896377892 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2e9c0c3d90 0x7f2e9c155240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:39.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.485+0000 7f2e9a000640 1 --2- 172.21.15.60:0/1896377892 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e9c0b1e00 0x7f2e9c072330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:39.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.485+0000 7f2e9aa00640 1 -- 172.21.15.60:0/1896377892 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2e9c0c3d90 msgr2=0x7f2e9c155240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:39.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.485+0000 7f2e9aa00640 1 --2- 172.21.15.60:0/1896377892 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2e9c0c3d90 0x7f2e9c155240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:39.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.485+0000 7f2e9aa00640 1 -- 172.21.15.60:0/1896377892 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e9c0b1e00 msgr2=0x7f2e9c072330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:39.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.485+0000 7f2e9aa00640 1 --2- 172.21.15.60:0/1896377892 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e9c0b1e00 0x7f2e9c072330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:39.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.485+0000 7f2e9aa00640 1 -- 172.21.15.60:0/1896377892 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e9c1f0e20 con 0x7f2e9c073320 2024-04-14T23:58:39.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.485+0000 7f2e9b400640 1 --2- 172.21.15.60:0/1896377892 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2e9c0c3d90 0x7f2e9c155240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:58:39.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.485+0000 7f2e9aa00640 1 --2- 172.21.15.60:0/1896377892 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e9c073320 0x7f2e9c1e4170 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f2e8c0090b0 tx=0x7f2e8c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:39.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.486+0000 7f2e92a00640 1 -- 172.21.15.60:0/1896377892 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e8c047070 con 0x7f2e9c073320 2024-04-14T23:58:39.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.486+0000 7f2e92a00640 1 -- 172.21.15.60:0/1896377892 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f2e8c037bc0 con 0x7f2e9c073320 2024-04-14T23:58:39.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.486+0000 7f2e92a00640 1 -- 172.21.15.60:0/1896377892 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e8c004030 con 0x7f2e9c073320 2024-04-14T23:58:39.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.486+0000 7f2e9be00640 1 -- 172.21.15.60:0/1896377892 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e9c0c2a00 con 0x7f2e9c073320 2024-04-14T23:58:39.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.486+0000 7f2e9be00640 1 -- 172.21.15.60:0/1896377892 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e9c1f13a0 con 0x7f2e9c073320 2024-04-14T23:58:39.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.487+0000 7f2e9be00640 1 -- 172.21.15.60:0/1896377892 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e9c04f540 con 0x7f2e9c073320 2024-04-14T23:58:39.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.488+0000 7f2e92a00640 1 -- 172.21.15.60:0/1896377892 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f2e8c0041d0 con 0x7f2e9c073320 2024-04-14T23:58:39.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.489+0000 7f2e92a00640 1 --2- 172.21.15.60:0/1896377892 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f2e7806b9c0 0x7f2e7806de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:39.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.489+0000 7f2e92a00640 1 -- 172.21.15.60:0/1896377892 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6255+0+0 (secure 0 0 0) 0x7f2e8c0acc00 con 0x7f2e9c073320 2024-04-14T23:58:39.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.489+0000 7f2e9a000640 1 --2- 172.21.15.60:0/1896377892 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f2e7806b9c0 0x7f2e7806de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:39.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.490+0000 7f2e9a000640 1 --2- 172.21.15.60:0/1896377892 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f2e7806b9c0 0x7f2e7806de80 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c0c7c10 tx=0x7f2e84009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:39.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.494+0000 7f2e92a00640 1 -- 172.21.15.60:0/1896377892 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2e8c033030 con 0x7f2e9c073320 2024-04-14T23:58:39.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.733+0000 7f2e9be00640 1 -- 172.21.15.60:0/1896377892 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2e9c0c7960 con 0x7f2e9c073320 2024-04-14T23:58:39.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.734+0000 7f2e92a00640 1 -- 172.21.15.60:0/1896377892 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+6839 (secure 0 0 0) 0x7f2e8c07ae70 con 0x7f2e9c073320 2024-04-14T23:58:39.741 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:58:39.741 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5409,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/1520534122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1520534122},{"type":"v1","addr":"172.21.15.60:6835","nonce":1520534122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:57:57.529555+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":19,"addr":"172.21.15.60:6833/3281566884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3281566884},{"type":"v1","addr":"172.21.15.60:6833","nonce":3281566884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":1}]} 2024-04-14T23:58:39.741 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 36 2024-04-14T23:58:39.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.737+0000 7f2e9be00640 1 -- 172.21.15.60:0/1896377892 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f2e7806b9c0 msgr2=0x7f2e7806de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:39.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.737+0000 7f2e9be00640 1 --2- 172.21.15.60:0/1896377892 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f2e7806b9c0 0x7f2e7806de80 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c0c7c10 tx=0x7f2e84009040 comp rx=0 tx=0).stop 2024-04-14T23:58:39.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.737+0000 7f2e9be00640 1 -- 172.21.15.60:0/1896377892 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e9c073320 msgr2=0x7f2e9c1e4170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:39.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.737+0000 7f2e9be00640 1 --2- 172.21.15.60:0/1896377892 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e9c073320 0x7f2e9c1e4170 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f2e8c0090b0 tx=0x7f2e8c004730 comp rx=0 tx=0).stop 2024-04-14T23:58:39.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.738+0000 7f2e9be00640 1 -- 172.21.15.60:0/1896377892 shutdown_connections 2024-04-14T23:58:39.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.738+0000 7f2e9be00640 1 --2- 172.21.15.60:0/1896377892 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f2e7806b9c0 0x7f2e7806de80 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:39.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.738+0000 7f2e9be00640 1 --2- 172.21.15.60:0/1896377892 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2e9c0c3d90 0x7f2e9c155240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:39.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.738+0000 7f2e9be00640 1 --2- 172.21.15.60:0/1896377892 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e9c0b1e00 0x7f2e9c072330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:39.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.738+0000 7f2e9be00640 1 --2- 172.21.15.60:0/1896377892 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e9c073320 0x7f2e9c1e4170 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:39.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.738+0000 7f2e9be00640 1 -- 172.21.15.60:0/1896377892 >> 172.21.15.60:0/1896377892 conn(0x7f2e9c0c8b50 msgr2=0x7f2e9c157810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:39.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.738+0000 7f2e9be00640 1 -- 172.21.15.60:0/1896377892 shutdown_connections 2024-04-14T23:58:39.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.738+0000 7f2e9be00640 1 -- 172.21.15.60:0/1896377892 wait complete. 2024-04-14T23:58:39.765 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-14T23:58:39.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.979+0000 7f6496e00640 1 Processor -- start 2024-04-14T23:58:39.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.979+0000 7f6496e00640 1 -- start start 2024-04-14T23:58:39.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.981+0000 7f6496e00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f649006f1f0 0x7f64900c72a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:39.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.981+0000 7f6496e00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f64900731e0 0x7f64900c77e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:39.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.981+0000 7f6496e00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f649015b0d0 con 0x7f649006f5d0 2024-04-14T23:58:39.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.981+0000 7f6496e00640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f64900c6e10 con 0x7f64900731e0 2024-04-14T23:58:39.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.981+0000 7f6496e00640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f649015b370 con 0x7f649006f1f0 2024-04-14T23:58:39.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.981+0000 7f6494e00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f64900731e0 0x7f64900c77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:39.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.981+0000 7f648f400640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f649006f5d0 0x7f6490154980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:36766/0 (socket says 172.21.15.60:36766) 2024-04-14T23:58:39.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.981+0000 7f648f400640 1 -- 172.21.15.60:0/2388284387 learned_addr learned my addr 172.21.15.60:0/2388284387 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:58:39.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.981+0000 7f648fe00640 1 --2- 172.21.15.60:0/2388284387 >> v2:172.21.15.186:3300/0 conn(0x7f649006f1f0 0x7f64900c72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:39.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.982+0000 7f6494e00640 1 -- 172.21.15.60:0/2388284387 >> v2:172.21.15.186:3300/0 conn(0x7f649006f1f0 msgr2=0x7f64900c72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:58:39.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.982+0000 7f6494e00640 1 --2- 172.21.15.60:0/2388284387 >> v2:172.21.15.186:3300/0 conn(0x7f649006f1f0 0x7f64900c72a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:39.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.982+0000 7f6494e00640 1 -- 172.21.15.60:0/2388284387 >> v1:172.21.15.60:6789/0 conn(0x7f649006f5d0 legacy=0x7f6490154980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:39.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.982+0000 7f648ea00640 1 -- 172.21.15.60:0/2388284387 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2047299120 0 0) 0x7f649015b0d0 con 0x7f649006f5d0 2024-04-14T23:58:39.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.982+0000 7f6494e00640 1 -- 172.21.15.60:0/2388284387 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f649015b650 con 0x7f64900731e0 2024-04-14T23:58:39.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.983+0000 7f6494e00640 1 --2- 172.21.15.60:0/2388284387 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64900731e0 0x7f64900c77e0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f6490073940 tx=0x7f648802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39d76c5074b33f66 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:39.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.983+0000 7f648ea00640 1 -- 172.21.15.60:0/2388284387 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6488039070 con 0x7f64900731e0 2024-04-14T23:58:39.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.983+0000 7f648ea00640 1 -- 172.21.15.60:0/2388284387 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f648802cd10 con 0x7f64900731e0 2024-04-14T23:58:39.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.984+0000 7f648ea00640 1 -- 172.21.15.60:0/2388284387 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6488034680 con 0x7f64900731e0 2024-04-14T23:58:39.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.984+0000 7f6496e00640 1 -- 172.21.15.60:0/2388284387 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64900731e0 msgr2=0x7f64900c77e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:39.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.984+0000 7f6496e00640 1 --2- 172.21.15.60:0/2388284387 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64900731e0 0x7f64900c77e0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f6490073940 tx=0x7f648802bf60 comp rx=0 tx=0).stop 2024-04-14T23:58:39.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.984+0000 7f6496e00640 1 -- 172.21.15.60:0/2388284387 shutdown_connections 2024-04-14T23:58:39.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.984+0000 7f6496e00640 1 --2- 172.21.15.60:0/2388284387 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64900731e0 0x7f64900c77e0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:39.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.984+0000 7f6496e00640 1 --2- 172.21.15.60:0/2388284387 >> v2:172.21.15.186:3300/0 conn(0x7f649006f1f0 0x7f64900c72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:39.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.984+0000 7f6496e00640 1 -- 172.21.15.60:0/2388284387 >> 172.21.15.60:0/2388284387 conn(0x7f64900cc170 msgr2=0x7f64900cc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:39.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.984+0000 7f6496e00640 1 -- 172.21.15.60:0/2388284387 shutdown_connections 2024-04-14T23:58:39.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.984+0000 7f6496e00640 1 -- 172.21.15.60:0/2388284387 wait complete. 2024-04-14T23:58:39.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.986+0000 7f6496e00640 1 Processor -- start 2024-04-14T23:58:39.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.986+0000 7f6496e00640 1 -- start start 2024-04-14T23:58:39.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.987+0000 7f6496e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f649006f1f0 0x7f6490142310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:39.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.987+0000 7f6496e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f649006f5d0 0x7f64900c4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:39.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.987+0000 7f6496e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f64900731e0 0x7f64900c5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:39.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.987+0000 7f6496e00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f6490155090 con 0x7f649006f5d0 2024-04-14T23:58:39.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.987+0000 7f6496e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6490071180 con 0x7f649006f1f0 2024-04-14T23:58:39.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.987+0000 7f6496e00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f6490155210 con 0x7f64900731e0 2024-04-14T23:58:39.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.987+0000 7f648f400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f649006f5d0 0x7f64900c4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:39.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.988+0000 7f648f400640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f649006f5d0 0x7f64900c4d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48708/0 (socket says 172.21.15.60:48708) 2024-04-14T23:58:39.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.988+0000 7f648f400640 1 -- 172.21.15.60:0/2526405436 learned_addr learned my addr 172.21.15.60:0/2526405436 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:58:39.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.988+0000 7f6494e00640 1 --2- 172.21.15.60:0/2526405436 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f64900731e0 0x7f64900c5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:39.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.988+0000 7f648fe00640 1 --2- 172.21.15.60:0/2526405436 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f649006f1f0 0x7f6490142310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:39.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.988+0000 7f648f400640 1 -- 172.21.15.60:0/2526405436 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f64900731e0 msgr2=0x7f64900c5280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:39.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.988+0000 7f648f400640 1 --2- 172.21.15.60:0/2526405436 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f64900731e0 0x7f64900c5280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:39.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.988+0000 7f648f400640 1 -- 172.21.15.60:0/2526405436 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f649006f1f0 msgr2=0x7f6490142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:39.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.988+0000 7f648f400640 1 --2- 172.21.15.60:0/2526405436 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f649006f1f0 0x7f6490142310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:39.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.988+0000 7f648f400640 1 -- 172.21.15.60:0/2526405436 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64900c45b0 con 0x7f649006f5d0 2024-04-14T23:58:39.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.988+0000 7f648f400640 1 --2- 172.21.15.60:0/2526405436 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f649006f5d0 0x7f64900c4d40 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f647c00c4d0 tx=0x7f647c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:39.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.989+0000 7f6487e00640 1 -- 172.21.15.60:0/2526405436 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f647c018070 con 0x7f649006f5d0 2024-04-14T23:58:39.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.989+0000 7f6487e00640 1 -- 172.21.15.60:0/2526405436 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f647c007500 con 0x7f649006f5d0 2024-04-14T23:58:39.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.989+0000 7f6487e00640 1 -- 172.21.15.60:0/2526405436 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f647c004a40 con 0x7f649006f5d0 2024-04-14T23:58:39.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.989+0000 7f6496e00640 1 -- 172.21.15.60:0/2526405436 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6490142930 con 0x7f649006f5d0 2024-04-14T23:58:39.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.989+0000 7f6496e00640 1 -- 172.21.15.60:0/2526405436 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6490142ef0 con 0x7f649006f5d0 2024-04-14T23:58:39.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.990+0000 7f6496e00640 1 -- 172.21.15.60:0/2526405436 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6490076970 con 0x7f649006f5d0 2024-04-14T23:58:39.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.991+0000 7f6487e00640 1 -- 172.21.15.60:0/2526405436 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f647c004be0 con 0x7f649006f5d0 2024-04-14T23:58:39.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.992+0000 7f6487e00640 1 --2- 172.21.15.60:0/2526405436 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f647006b990 0x7f647006de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:40.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.992+0000 7f6487e00640 1 -- 172.21.15.60:0/2526405436 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6255+0+0 (secure 0 0 0) 0x7f647c089dd0 con 0x7f649006f5d0 2024-04-14T23:58:40.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.992+0000 7f648fe00640 1 --2- 172.21.15.60:0/2526405436 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f647006b990 0x7f647006de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:40.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.993+0000 7f648fe00640 1 --2- 172.21.15.60:0/2526405436 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f647006b990 0x7f647006de50 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6480009b40 tx=0x7f6480009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:40.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:39.997+0000 7f6487e00640 1 -- 172.21.15.60:0/2526405436 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f647c058040 con 0x7f649006f5d0 2024-04-14T23:58:40.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.243+0000 7f6496e00640 1 -- 172.21.15.60:0/2526405436 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f649006cbe0 con 0x7f649006f5d0 2024-04-14T23:58:40.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.244+0000 7f6487e00640 1 -- 172.21.15.60:0/2526405436 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+6839 (secure 0 0 0) 0x7f647c05c0e0 con 0x7f649006f5d0 2024-04-14T23:58:40.251 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:58:40.251 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4197,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/1527815634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":1527815634},{"type":"v1","addr":"172.21.15.60:6837","nonce":1527815634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6838/3958319486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3958319486},{"type":"v1","addr":"172.21.15.186:6838","nonce":3958319486}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3910951185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3910951185},{"type":"v1","addr":"172.21.15.186:6837","nonce":3910951185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2155020404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":2155020404},{"type":"v1","addr":"172.21.15.186:6839","nonce":2155020404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5409,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/1520534122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":1520534122},{"type":"v1","addr":"172.21.15.60:6835","nonce":1520534122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-14T23:48:42.700188+0000","modified":"2024-04-14T23:57:57.529555+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5286":{"gid":5286,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":19,"addr":"172.21.15.60:6833/3281566884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3281566884},{"type":"v1","addr":"172.21.15.60:6833","nonce":3281566884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5286,"qdb_cluster":[5286]},"id":1}]} 2024-04-14T23:58:40.251 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 36 2024-04-14T23:58:40.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.247+0000 7f6496e00640 1 -- 172.21.15.60:0/2526405436 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f647006b990 msgr2=0x7f647006de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:40.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.247+0000 7f6496e00640 1 --2- 172.21.15.60:0/2526405436 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f647006b990 0x7f647006de50 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6480009b40 tx=0x7f6480009190 comp rx=0 tx=0).stop 2024-04-14T23:58:40.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.247+0000 7f6496e00640 1 -- 172.21.15.60:0/2526405436 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f649006f5d0 msgr2=0x7f64900c4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:40.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.247+0000 7f6496e00640 1 --2- 172.21.15.60:0/2526405436 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f649006f5d0 0x7f64900c4d40 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f647c00c4d0 tx=0x7f647c00c990 comp rx=0 tx=0).stop 2024-04-14T23:58:40.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.247+0000 7f6496e00640 1 -- 172.21.15.60:0/2526405436 shutdown_connections 2024-04-14T23:58:40.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.247+0000 7f6496e00640 1 --2- 172.21.15.60:0/2526405436 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f647006b990 0x7f647006de50 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:40.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.247+0000 7f6496e00640 1 --2- 172.21.15.60:0/2526405436 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f64900731e0 0x7f64900c5280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:40.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.247+0000 7f6496e00640 1 --2- 172.21.15.60:0/2526405436 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f649006f5d0 0x7f64900c4d40 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:40.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.247+0000 7f6496e00640 1 --2- 172.21.15.60:0/2526405436 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f649006f1f0 0x7f6490142310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:40.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.247+0000 7f6496e00640 1 -- 172.21.15.60:0/2526405436 >> 172.21.15.60:0/2526405436 conn(0x7f64900cc170 msgr2=0x7f649014a590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:40.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.247+0000 7f6496e00640 1 -- 172.21.15.60:0/2526405436 shutdown_connections 2024-04-14T23:58:40.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.247+0000 7f6496e00640 1 -- 172.21.15.60:0/2526405436 wait complete. 2024-04-14T23:58:40.273 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-14T23:58:40.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.485+0000 7f16c0a00640 1 Processor -- start 2024-04-14T23:58:40.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.486+0000 7f16c0a00640 1 -- start start 2024-04-14T23:58:40.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.487+0000 7f16c0a00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f16bc0c6950 0x7f16bc06b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:40.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.487+0000 7f16c0a00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f16bc15ba70 0x7f16bc070d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:40.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.487+0000 7f16c0a00640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16bc06b850 con 0x7f16bc0c7080 2024-04-14T23:58:40.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.487+0000 7f16c0a00640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f16bc06c740 con 0x7f16bc15ba70 2024-04-14T23:58:40.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.487+0000 7f16c0a00640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f16bc06bb30 con 0x7f16bc0c6950 2024-04-14T23:58:40.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.488+0000 7f16ba800640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f16bc0c7080 0x7f16bc15b360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.60:53768/0 (socket says 172.21.15.60:53768) 2024-04-14T23:58:40.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.488+0000 7f16ba800640 1 -- 172.21.15.60:0/2289411871 learned_addr learned my addr 172.21.15.60:0/2289411871 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:58:40.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.488+0000 7f16b3e00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f16bc0c6950 0x7f16bc06b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:40.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.488+0000 7f16bb200640 1 --2- 172.21.15.60:0/2289411871 >> v2:172.21.15.60:3300/0 conn(0x7f16bc15ba70 0x7f16bc070d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:40.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.488+0000 7f16b3e00640 1 -- 172.21.15.60:0/2289411871 >> v2:172.21.15.60:3300/0 conn(0x7f16bc15ba70 msgr2=0x7f16bc070d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:58:40.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.488+0000 7f16b3e00640 1 --2- 172.21.15.60:0/2289411871 >> v2:172.21.15.60:3300/0 conn(0x7f16bc15ba70 0x7f16bc070d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:40.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.488+0000 7f16b3e00640 1 -- 172.21.15.60:0/2289411871 >> v1:172.21.15.186:6789/0 conn(0x7f16bc0c7080 legacy=0x7f16bc15b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:40.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.488+0000 7f16b3e00640 1 -- 172.21.15.60:0/2289411871 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16bc06be10 con 0x7f16bc0c6950 2024-04-14T23:58:40.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.489+0000 7f16b3e00640 1 --2- 172.21.15.60:0/2289411871 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16bc0c6950 0x7f16bc06b1c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f16bc0733b0 tx=0x7f16a402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6d19ee87ceca79b server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:40.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.490+0000 7f16b9e00640 1 -- 172.21.15.60:0/2289411871 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a4034070 con 0x7f16bc0c6950 2024-04-14T23:58:40.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.490+0000 7f16b9e00640 1 -- 172.21.15.60:0/2289411871 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f16a4030610 con 0x7f16bc0c6950 2024-04-14T23:58:40.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.490+0000 7f16b9e00640 1 -- 172.21.15.60:0/2289411871 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a4030a00 con 0x7f16bc0c6950 2024-04-14T23:58:40.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.490+0000 7f16c0a00640 1 -- 172.21.15.60:0/2289411871 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16bc0c6950 msgr2=0x7f16bc06b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:40.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.490+0000 7f16c0a00640 1 --2- 172.21.15.60:0/2289411871 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16bc0c6950 0x7f16bc06b1c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f16bc0733b0 tx=0x7f16a402b960 comp rx=0 tx=0).stop 2024-04-14T23:58:40.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.491+0000 7f16c0a00640 1 -- 172.21.15.60:0/2289411871 shutdown_connections 2024-04-14T23:58:40.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.491+0000 7f16c0a00640 1 --2- 172.21.15.60:0/2289411871 >> v2:172.21.15.60:3300/0 conn(0x7f16bc15ba70 0x7f16bc070d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:40.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.491+0000 7f16c0a00640 1 --2- 172.21.15.60:0/2289411871 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16bc0c6950 0x7f16bc06b1c0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:40.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.491+0000 7f16c0a00640 1 -- 172.21.15.60:0/2289411871 >> 172.21.15.60:0/2289411871 conn(0x7f16bc0c8b50 msgr2=0x7f16bc0c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:40.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.491+0000 7f16c0a00640 1 -- 172.21.15.60:0/2289411871 shutdown_connections 2024-04-14T23:58:40.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.491+0000 7f16c0a00640 1 -- 172.21.15.60:0/2289411871 wait complete. 2024-04-14T23:58:40.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.492+0000 7f16c0a00640 1 Processor -- start 2024-04-14T23:58:40.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.492+0000 7f16c0a00640 1 -- start start 2024-04-14T23:58:40.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.493+0000 7f16c0a00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f16bc0c6950 0x7f16bc0c3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:40.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.493+0000 7f16c0a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16bc0c7080 0x7f16bc152bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:40.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.493+0000 7f16c0a00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16bc15ba70 0x7f16bc151350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:40.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.493+0000 7f16c0a00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f16bc0c7460 con 0x7f16bc15ba70 2024-04-14T23:58:40.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.493+0000 7f16c0a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f16bc06c3d0 con 0x7f16bc0c7080 2024-04-14T23:58:40.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.493+0000 7f16c0a00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f16bc0c75e0 con 0x7f16bc0c6950 2024-04-14T23:58:40.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.494+0000 7f16bb200640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16bc15ba70 0x7f16bc151350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:40.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.494+0000 7f16ba800640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f16bc0c6950 0x7f16bc0c3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:40.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.494+0000 7f16bb200640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16bc15ba70 0x7f16bc151350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48734/0 (socket says 172.21.15.60:48734) 2024-04-14T23:58:40.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.494+0000 7f16bb200640 1 -- 172.21.15.60:0/193292897 learned_addr learned my addr 172.21.15.60:0/193292897 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:58:40.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.494+0000 7f16b3e00640 1 --2- 172.21.15.60:0/193292897 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16bc0c7080 0x7f16bc152bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:40.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.495+0000 7f16ba800640 1 -- 172.21.15.60:0/193292897 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16bc0c7080 msgr2=0x7f16bc152bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:40.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.495+0000 7f16ba800640 1 --2- 172.21.15.60:0/193292897 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16bc0c7080 0x7f16bc152bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:40.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.495+0000 7f16ba800640 1 -- 172.21.15.60:0/193292897 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16bc15ba70 msgr2=0x7f16bc151350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:40.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.495+0000 7f16ba800640 1 --2- 172.21.15.60:0/193292897 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16bc15ba70 0x7f16bc151350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:40.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.495+0000 7f16ba800640 1 -- 172.21.15.60:0/193292897 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16bc1f0f40 con 0x7f16bc0c6950 2024-04-14T23:58:40.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.495+0000 7f16ba800640 1 --2- 172.21.15.60:0/193292897 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f16bc0c6950 0x7f16bc0c3ec0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f16ac00c370 tx=0x7f16ac00c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:40.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.495+0000 7f16b2a00640 1 -- 172.21.15.60:0/193292897 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16ac017070 con 0x7f16bc0c6950 2024-04-14T23:58:40.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.495+0000 7f16b2a00640 1 -- 172.21.15.60:0/193292897 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f16ac005ce0 con 0x7f16bc0c6950 2024-04-14T23:58:40.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.495+0000 7f16c0a00640 1 -- 172.21.15.60:0/193292897 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16bc1f1220 con 0x7f16bc0c6950 2024-04-14T23:58:40.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.495+0000 7f16b2a00640 1 -- 172.21.15.60:0/193292897 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16ac007330 con 0x7f16bc0c6950 2024-04-14T23:58:40.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.495+0000 7f16c0a00640 1 -- 172.21.15.60:0/193292897 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f16bc1f17e0 con 0x7f16bc0c6950 2024-04-14T23:58:40.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.497+0000 7f16c0a00640 1 -- 172.21.15.60:0/193292897 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16bc04f540 con 0x7f16bc0c6950 2024-04-14T23:58:40.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.497+0000 7f16b2a00640 1 -- 172.21.15.60:0/193292897 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f16ac01a070 con 0x7f16bc0c6950 2024-04-14T23:58:40.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.498+0000 7f16b2a00640 1 --2- 172.21.15.60:0/193292897 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f169806b9c0 0x7f169806de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:40.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.499+0000 7f16b2a00640 1 -- 172.21.15.60:0/193292897 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(45..45 src has 1..45) ==== 6255+0+0 (secure 0 0 0) 0x7f16ac089de0 con 0x7f16bc0c6950 2024-04-14T23:58:40.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.499+0000 7f16b3e00640 1 --2- 172.21.15.60:0/193292897 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f169806b9c0 0x7f169806de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:40.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.499+0000 7f16b3e00640 1 --2- 172.21.15.60:0/193292897 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f169806b9c0 0x7f169806de80 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f16a4036000 tx=0x7f16a403b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:40.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.503+0000 7f16b2a00640 1 -- 172.21.15.60:0/193292897 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f16ac0580e0 con 0x7f16bc0c6950 2024-04-14T23:58:40.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.709+0000 7f16c0a00640 1 -- 172.21.15.60:0/193292897 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f16bc0c4d80 con 0x7f16bc0c6950 2024-04-14T23:58:40.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.711+0000 7f16b2a00640 1 -- 172.21.15.60:0/193292897 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) ==== 74+0+17524 (secure 0 0 0) 0x7f16ac05c180 con 0x7f16bc0c6950 2024-04-14T23:58:40.718 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-14T23:58:40.718 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":45,"fsid":"5847db56-a76d-4486-b563-d83f1186af92","created":"2024-04-14T23:48:23.900003+0000","modified":"2024-04-14T23:57:39.001988+0000","last_up_change":"2024-04-14T23:48:34.203573+0000","last_in_change":"2024-04-14T23:48:26.345499+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-14T23:48:36.247372+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-14T23:48:40.274195+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-14T23:48:41.722183+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":505,"snap_epoch":41,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b2b2b755-7f42-4f9b-af9a-a178f7cc5a52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6817","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6818","nonce":2536528895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6822","nonce":2536528895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6827","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6828","nonce":2536528895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":2536528895},{"type":"v1","addr":"172.21.15.60:6825","nonce":2536528895}]},"public_addr":"172.21.15.60:6818/2536528895","cluster_addr":"172.21.15.60:6822/2536528895","heartbeat_back_addr":"172.21.15.60:6828/2536528895","heartbeat_front_addr":"172.21.15.60:6825/2536528895","state":["exists","up"]},{"osd":1,"uuid":"321418cd-5ab7-41a5-b9c8-b3fc12f68ebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6801","nonce":3115695676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6803","nonce":3115695676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6807","nonce":3115695676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":3115695676},{"type":"v1","addr":"172.21.15.60:6805","nonce":3115695676}]},"public_addr":"172.21.15.60:6801/3115695676","cluster_addr":"172.21.15.60:6803/3115695676","heartbeat_back_addr":"172.21.15.60:6807/3115695676","heartbeat_front_addr":"172.21.15.60:6805/3115695676","state":["exists","up"]},{"osd":2,"uuid":"07874e9f-3a38-41dc-be9f-2b75e84e53d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6819","nonce":3475168698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6824","nonce":3475168698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6831","nonce":3475168698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":3475168698},{"type":"v1","addr":"172.21.15.60:6829","nonce":3475168698}]},"public_addr":"172.21.15.60:6819/3475168698","cluster_addr":"172.21.15.60:6824/3475168698","heartbeat_back_addr":"172.21.15.60:6831/3475168698","heartbeat_front_addr":"172.21.15.60:6829/3475168698","state":["exists","up"]},{"osd":3,"uuid":"7e4ef6c9-b17a-47c6-a320-f5d16a6fe05e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6809","nonce":1730813674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6811","nonce":1730813674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6815","nonce":1730813674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":1730813674},{"type":"v1","addr":"172.21.15.60:6813","nonce":1730813674}]},"public_addr":"172.21.15.60:6809/1730813674","cluster_addr":"172.21.15.60:6811/1730813674","heartbeat_back_addr":"172.21.15.60:6815/1730813674","heartbeat_front_addr":"172.21.15.60:6813/1730813674","state":["exists","up"]},{"osd":4,"uuid":"759a3efc-dade-44c5-b57a-1623499aba50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6805","nonce":1464587038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6810","nonce":1464587038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6819","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6822","nonce":1464587038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6813","nonce":1464587038},{"type":"v1","addr":"172.21.15.186:6816","nonce":1464587038}]},"public_addr":"172.21.15.186:6805/1464587038","cluster_addr":"172.21.15.186:6810/1464587038","heartbeat_back_addr":"172.21.15.186:6822/1464587038","heartbeat_front_addr":"172.21.15.186:6816/1464587038","state":["exists","up"]},{"osd":5,"uuid":"24cd58c1-0cf1-4077-ab33-84c39c6907ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6801","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6803","nonce":1697098067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6809","nonce":1697098067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6820","nonce":1697098067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1697098067},{"type":"v1","addr":"172.21.15.186:6815","nonce":1697098067}]},"public_addr":"172.21.15.186:6803/1697098067","cluster_addr":"172.21.15.186:6809/1697098067","heartbeat_back_addr":"172.21.15.186:6820/1697098067","heartbeat_front_addr":"172.21.15.186:6815/1697098067","state":["exists","up"]},{"osd":6,"uuid":"6add38f1-65e9-4b0d-927a-4ab77f5e2323","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6804","nonce":1837108559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6807","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6811","nonce":1837108559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6823","nonce":1837108559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1837108559},{"type":"v1","addr":"172.21.15.186:6817","nonce":1837108559}]},"public_addr":"172.21.15.186:6804/1837108559","cluster_addr":"172.21.15.186:6811/1837108559","heartbeat_back_addr":"172.21.15.186:6823/1837108559","heartbeat_front_addr":"172.21.15.186:6817/1837108559","state":["exists","up"]},{"osd":7,"uuid":"a2326b6f-7601-4dda-a456-67c443ea5929","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6825","nonce":842537875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6827","nonce":842537875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6831","nonce":842537875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":842537875},{"type":"v1","addr":"172.21.15.186:6829","nonce":842537875}]},"public_addr":"172.21.15.186:6825/842537875","cluster_addr":"172.21.15.186:6827/842537875","heartbeat_back_addr":"172.21.15.186:6831/842537875","heartbeat_front_addr":"172.21.15.186:6829/842537875","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.975583+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.976806+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.722725+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.763705+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.502140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.528972+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.517961+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T23:48:31.513350+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.23","mappings":[{"from":3,"to":2}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"3.3b","mappings":[{"from":7,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.60:6835/2356847578":"2024-04-15T23:57:39.001841+0000","172.21.15.60:6834/2356847578":"2024-04-15T23:57:39.001841+0000","172.21.15.60:6833/1971580590":"2024-04-15T23:56:28.972769+0000","172.21.15.60:6835/1714133698":"2024-04-15T23:54:48.942868+0000","172.21.15.60:6832/1971580590":"2024-04-15T23:56:28.972769+0000","172.21.15.60:6834/1714133698":"2024-04-15T23:54:48.942868+0000","172.21.15.60:6833/78303888":"2024-04-15T23:52:28.905933+0000","172.21.15.60:6832/78303888":"2024-04-15T23:52:28.905933+0000","172.21.15.60:6835/2677133122":"2024-04-15T23:50:58.881136+0000","172.21.15.60:6834/2677133122":"2024-04-15T23:50:58.881136+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-14T23:58:40.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.714+0000 7f16c0a00640 1 -- 172.21.15.60:0/193292897 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f169806b9c0 msgr2=0x7f169806de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:40.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.714+0000 7f16c0a00640 1 --2- 172.21.15.60:0/193292897 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f169806b9c0 0x7f169806de80 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f16a4036000 tx=0x7f16a403b040 comp rx=0 tx=0).stop 2024-04-14T23:58:40.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.714+0000 7f16c0a00640 1 -- 172.21.15.60:0/193292897 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f16bc0c6950 msgr2=0x7f16bc0c3ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:40.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.714+0000 7f16c0a00640 1 --2- 172.21.15.60:0/193292897 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f16bc0c6950 0x7f16bc0c3ec0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f16ac00c370 tx=0x7f16ac00c830 comp rx=0 tx=0).stop 2024-04-14T23:58:40.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.714+0000 7f16c0a00640 1 -- 172.21.15.60:0/193292897 shutdown_connections 2024-04-14T23:58:40.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.714+0000 7f16c0a00640 1 --2- 172.21.15.60:0/193292897 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f169806b9c0 0x7f169806de80 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:40.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.714+0000 7f16c0a00640 1 --2- 172.21.15.60:0/193292897 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16bc15ba70 0x7f16bc151350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:40.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.714+0000 7f16c0a00640 1 --2- 172.21.15.60:0/193292897 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16bc0c7080 0x7f16bc152bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:40.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.714+0000 7f16c0a00640 1 --2- 172.21.15.60:0/193292897 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f16bc0c6950 0x7f16bc0c3ec0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:40.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.714+0000 7f16c0a00640 1 -- 172.21.15.60:0/193292897 >> 172.21.15.60:0/193292897 conn(0x7f16bc0c8b50 msgr2=0x7f16bc0c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:40.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.714+0000 7f16c0a00640 1 -- 172.21.15.60:0/193292897 shutdown_connections 2024-04-14T23:58:40.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.714+0000 7f16c0a00640 1 -- 172.21.15.60:0/193292897 wait complete. 2024-04-14T23:58:40.740 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-14T23:58:40.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.953+0000 7f4eff200640 1 Processor -- start 2024-04-14T23:58:40.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.953+0000 7f4eff200640 1 -- start start 2024-04-14T23:58:40.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.955+0000 7f4eff200640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4ef80b1860 0x7f4ef80b1c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:40.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.955+0000 7f4eff200640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f4ef80c6c70 0x7f4ef806c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:40.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.955+0000 7f4eff200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4ef806b1c0 0x7f4ef806eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:40.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.955+0000 7f4eff200640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f4ef8159930 con 0x7f4ef80b1860 2024-04-14T23:58:40.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.955+0000 7f4eff200640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f4ef80c6440 con 0x7f4ef80c6c70 2024-04-14T23:58:40.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.955+0000 7f4eff200640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f4ef80c6160 con 0x7f4ef806b1c0 2024-04-14T23:58:40.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.955+0000 7f4efde00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4ef80b1860 0x7f4ef80b1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:40.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.955+0000 7f4efde00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4ef80b1860 0x7f4ef80b1c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48750/0 (socket says 172.21.15.60:48750) 2024-04-14T23:58:40.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.955+0000 7f4efde00640 1 -- 172.21.15.60:0/1834800333 learned_addr learned my addr 172.21.15.60:0/1834800333 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:58:40.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.955+0000 7f4efd400640 1 --2- 172.21.15.60:0/1834800333 >> v2:172.21.15.186:3300/0 conn(0x7f4ef80c6c70 0x7f4ef806c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:40.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.955+0000 7f4efe800640 1 --2- 172.21.15.60:0/1834800333 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4ef806b1c0 0x7f4ef806eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:40.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.956+0000 7f4efde00640 1 -- 172.21.15.60:0/1834800333 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4ef806b1c0 msgr2=0x7f4ef806eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:58:40.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.956+0000 7f4efde00640 1 --2- 172.21.15.60:0/1834800333 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4ef806b1c0 0x7f4ef806eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:40.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.956+0000 7f4efde00640 1 -- 172.21.15.60:0/1834800333 >> v2:172.21.15.186:3300/0 conn(0x7f4ef80c6c70 msgr2=0x7f4ef806c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:58:40.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.956+0000 7f4efde00640 1 --2- 172.21.15.60:0/1834800333 >> v2:172.21.15.186:3300/0 conn(0x7f4ef80c6c70 0x7f4ef806c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:40.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.956+0000 7f4efde00640 1 -- 172.21.15.60:0/1834800333 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ef8158b70 con 0x7f4ef80b1860 2024-04-14T23:58:40.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.956+0000 7f4efd400640 1 --2- 172.21.15.60:0/1834800333 >> v2:172.21.15.186:3300/0 conn(0x7f4ef80c6c70 0x7f4ef806c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:58:40.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.957+0000 7f4efde00640 1 --2- 172.21.15.60:0/1834800333 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ef80b1860 0x7f4ef80b1c40 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f4ef815cfe0 tx=0x7f4eec02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f789c0710b4f43b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:40.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.957+0000 7f4efca00640 1 -- 172.21.15.60:0/1834800333 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eec035020 con 0x7f4ef80b1860 2024-04-14T23:58:40.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.957+0000 7f4efca00640 1 -- 172.21.15.60:0/1834800333 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f4eec031870 con 0x7f4ef80b1860 2024-04-14T23:58:40.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.957+0000 7f4efca00640 1 -- 172.21.15.60:0/1834800333 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eec031c60 con 0x7f4ef80b1860 2024-04-14T23:58:40.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.958+0000 7f4eff200640 1 -- 172.21.15.60:0/1834800333 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ef80b1860 msgr2=0x7f4ef80b1c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:40.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.958+0000 7f4eff200640 1 --2- 172.21.15.60:0/1834800333 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ef80b1860 0x7f4ef80b1c40 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f4ef815cfe0 tx=0x7f4eec02b910 comp rx=0 tx=0).stop 2024-04-14T23:58:40.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.958+0000 7f4eff200640 1 -- 172.21.15.60:0/1834800333 shutdown_connections 2024-04-14T23:58:40.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.958+0000 7f4eff200640 1 --2- 172.21.15.60:0/1834800333 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4ef806b1c0 0x7f4ef806eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:40.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.958+0000 7f4eff200640 1 --2- 172.21.15.60:0/1834800333 >> v2:172.21.15.186:3300/0 conn(0x7f4ef80c6c70 0x7f4ef806c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:40.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.958+0000 7f4eff200640 1 --2- 172.21.15.60:0/1834800333 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ef80b1860 0x7f4ef80b1c40 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:40.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.958+0000 7f4eff200640 1 -- 172.21.15.60:0/1834800333 >> 172.21.15.60:0/1834800333 conn(0x7f4ef80cc2f0 msgr2=0x7f4ef80cc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:40.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.958+0000 7f4eff200640 1 -- 172.21.15.60:0/1834800333 shutdown_connections 2024-04-14T23:58:40.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.958+0000 7f4eff200640 1 -- 172.21.15.60:0/1834800333 wait complete. 2024-04-14T23:58:40.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.960+0000 7f4eff200640 1 Processor -- start 2024-04-14T23:58:40.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.960+0000 7f4eff200640 1 -- start start 2024-04-14T23:58:40.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.961+0000 7f4eff200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ef806b1c0 0x7f4ef81ef940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:40.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.961+0000 7f4eff200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4ef80b1860 0x7f4ef8071990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:40.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.961+0000 7f4eff200640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ef80c6c70 0x7f4ef8071ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:40.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.961+0000 7f4eff200640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef8159410 con 0x7f4ef80c6c70 2024-04-14T23:58:40.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.961+0000 7f4eff200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef8159130 con 0x7f4ef806b1c0 2024-04-14T23:58:40.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.961+0000 7f4eff200640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7f4ef80c4720 con 0x7f4ef80b1860 2024-04-14T23:58:40.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.961+0000 7f4efde00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ef806b1c0 0x7f4ef81ef940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:40.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.962+0000 7f4efe800640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ef80c6c70 0x7f4ef8071ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:40.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.962+0000 7f4efde00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ef806b1c0 0x7f4ef81ef940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:58010/0 (socket says 172.21.15.60:58010) 2024-04-14T23:58:40.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.962+0000 7f4efd400640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4ef80b1860 0x7f4ef8071990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:40.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.962+0000 7f4efde00640 1 -- 172.21.15.60:0/1005796972 learned_addr learned my addr 172.21.15.60:0/1005796972 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:58:40.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.962+0000 7f4efde00640 1 -- 172.21.15.60:0/1005796972 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4ef80b1860 msgr2=0x7f4ef8071990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:40.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.962+0000 7f4efde00640 1 --2- 172.21.15.60:0/1005796972 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4ef80b1860 0x7f4ef8071990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:40.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.962+0000 7f4efde00640 1 -- 172.21.15.60:0/1005796972 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ef80c6c70 msgr2=0x7f4ef8071ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:40.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.962+0000 7f4efde00640 1 --2- 172.21.15.60:0/1005796972 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ef80c6c70 0x7f4ef8071ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:40.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.962+0000 7f4efde00640 1 -- 172.21.15.60:0/1005796972 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ef814e690 con 0x7f4ef806b1c0 2024-04-14T23:58:40.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.962+0000 7f4efd400640 1 --2- 172.21.15.60:0/1005796972 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4ef80b1860 0x7f4ef8071990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:58:40.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.962+0000 7f4efde00640 1 --2- 172.21.15.60:0/1005796972 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ef806b1c0 0x7f4ef81ef940 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f4eec009960 tx=0x7f4eec0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:40.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.963+0000 7f4ef2000640 1 -- 172.21.15.60:0/1005796972 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eec035040 con 0x7f4ef806b1c0 2024-04-14T23:58:40.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.963+0000 7f4ef2000640 1 -- 172.21.15.60:0/1005796972 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f4eec030af0 con 0x7f4ef806b1c0 2024-04-14T23:58:40.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.963+0000 7f4ef2000640 1 -- 172.21.15.60:0/1005796972 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eec043430 con 0x7f4ef806b1c0 2024-04-14T23:58:40.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.963+0000 7f4eff200640 1 -- 172.21.15.60:0/1005796972 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ef81f0290 con 0x7f4ef806b1c0 2024-04-14T23:58:40.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.963+0000 7f4eff200640 1 -- 172.21.15.60:0/1005796972 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ef8151d80 con 0x7f4ef806b1c0 2024-04-14T23:58:40.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.964+0000 7f4eff200640 1 -- 172.21.15.60:0/1005796972 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ef804f540 con 0x7f4ef806b1c0 2024-04-14T23:58:40.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.965+0000 7f4ef2000640 1 -- 172.21.15.60:0/1005796972 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7f4eec038020 con 0x7f4ef806b1c0 2024-04-14T23:58:40.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.966+0000 7f4ef2000640 1 --2- 172.21.15.60:0/1005796972 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f4edc06b990 0x7f4edc06de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:40.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.966+0000 7f4ef2000640 1 -- 172.21.15.60:0/1005796972 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6255+0+0 (secure 0 0 0) 0x7f4eec0ac9a0 con 0x7f4ef806b1c0 2024-04-14T23:58:40.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.966+0000 7f4efd400640 1 --2- 172.21.15.60:0/1005796972 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f4edc06b990 0x7f4edc06de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:40.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.967+0000 7f4efd400640 1 --2- 172.21.15.60:0/1005796972 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f4edc06b990 0x7f4edc06de50 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f4ef814b400 tx=0x7f4ee8009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:40.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:40.971+0000 7f4ef2000640 1 -- 172.21.15.60:0/1005796972 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4eec07aca0 con 0x7f4ef806b1c0 2024-04-14T23:58:41.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.215+0000 7f4eff200640 1 -- 172.21.15.60:0/1005796972 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f4ef80c4150 con 0x7f4ef806b1c0 2024-04-14T23:58:41.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.227+0000 7f4ef2000640 1 -- 172.21.15.60:0/1005796972 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v37) ==== 161+0+0 (secure 0 0 0) 0x7f4eec07ed40 con 0x7f4ef806b1c0 2024-04-14T23:58:41.234 INFO:teuthology.orchestra.run.smithi060.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-14T23:58:41.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.232+0000 7f4eff200640 1 -- 172.21.15.60:0/1005796972 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f4edc06b990 msgr2=0x7f4edc06de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:41.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.232+0000 7f4eff200640 1 --2- 172.21.15.60:0/1005796972 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f4edc06b990 0x7f4edc06de50 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f4ef814b400 tx=0x7f4ee8009190 comp rx=0 tx=0).stop 2024-04-14T23:58:41.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.232+0000 7f4eff200640 1 -- 172.21.15.60:0/1005796972 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ef806b1c0 msgr2=0x7f4ef81ef940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:41.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.232+0000 7f4eff200640 1 --2- 172.21.15.60:0/1005796972 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ef806b1c0 0x7f4ef81ef940 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f4eec009960 tx=0x7f4eec0046c0 comp rx=0 tx=0).stop 2024-04-14T23:58:41.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.232+0000 7f4eff200640 1 -- 172.21.15.60:0/1005796972 shutdown_connections 2024-04-14T23:58:41.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.232+0000 7f4eff200640 1 --2- 172.21.15.60:0/1005796972 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7f4edc06b990 0x7f4edc06de50 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:41.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.232+0000 7f4eff200640 1 --2- 172.21.15.60:0/1005796972 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ef80c6c70 0x7f4ef8071ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:41.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.232+0000 7f4eff200640 1 --2- 172.21.15.60:0/1005796972 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4ef80b1860 0x7f4ef8071990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:41.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.232+0000 7f4eff200640 1 --2- 172.21.15.60:0/1005796972 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ef806b1c0 0x7f4ef81ef940 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:41.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.232+0000 7f4eff200640 1 -- 172.21.15.60:0/1005796972 >> 172.21.15.60:0/1005796972 conn(0x7f4ef80cc2f0 msgr2=0x7f4ef80c9d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:41.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.233+0000 7f4eff200640 1 -- 172.21.15.60:0/1005796972 shutdown_connections 2024-04-14T23:58:41.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.233+0000 7f4eff200640 1 -- 172.21.15.60:0/1005796972 wait complete. 2024-04-14T23:58:41.256 INFO:tasks.ceph.mds.a.smithi060.stderr: -4543> 2024-04-14T23:58:06.427+0000 7f3cf7400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-14T23:58:41.261 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-14T23:58:41.311 INFO:tasks.ceph.mds.a.smithi060.stdout:starting mds.a at 2024-04-14T23:58:41.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.447+0000 7fa2fbc00640 1 Processor -- start 2024-04-14T23:58:41.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.447+0000 7fa2fbc00640 1 -- start start 2024-04-14T23:58:41.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.449+0000 7fa2fbc00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa2f40c8360 0x7fa2f415ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:41.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.449+0000 7fa2fbc00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa2f40c75c0 0x7fa2f4158e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:41.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.449+0000 7fa2fbc00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2f415f7c0 con 0x7fa2f40c8740 2024-04-14T23:58:41.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.449+0000 7fa2fbc00640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fa2f40c6b70 con 0x7fa2f40c75c0 2024-04-14T23:58:41.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.449+0000 7fa2fbc00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fa2f406fa20 con 0x7fa2f40c8360 2024-04-14T23:58:41.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.449+0000 7fa2f3e00640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fa2f40c8740 0x7fa2f406f310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:36788/0 (socket says 172.21.15.60:36788) 2024-04-14T23:58:41.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.449+0000 7fa2f9200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa2f40c8360 0x7fa2f415ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:41.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.449+0000 7fa2f3e00640 1 -- 172.21.15.60:0/2122884813 learned_addr learned my addr 172.21.15.60:0/2122884813 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:58:41.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.449+0000 7fa2f9c00640 1 --2- 172.21.15.60:0/2122884813 >> v2:172.21.15.186:3300/0 conn(0x7fa2f40c75c0 0x7fa2f4158e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:41.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.449+0000 7fa2f9200640 1 -- 172.21.15.60:0/2122884813 >> v2:172.21.15.186:3300/0 conn(0x7fa2f40c75c0 msgr2=0x7fa2f4158e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:58:41.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.449+0000 7fa2f9200640 1 --2- 172.21.15.60:0/2122884813 >> v2:172.21.15.186:3300/0 conn(0x7fa2f40c75c0 0x7fa2f4158e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:41.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.449+0000 7fa2f9200640 1 -- 172.21.15.60:0/2122884813 >> v1:172.21.15.60:6789/0 conn(0x7fa2f40c8740 legacy=0x7fa2f406f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:41.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.449+0000 7fa2f9200640 1 -- 172.21.15.60:0/2122884813 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2f415faa0 con 0x7fa2f40c8360 2024-04-14T23:58:41.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.451+0000 7fa2f9200640 1 --2- 172.21.15.60:0/2122884813 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa2f40c8360 0x7fa2f415ef80 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa2e40097a0 tx=0x7fa2e402b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=25694c4832798e7c server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:41.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.451+0000 7fa2f3400640 1 -- 172.21.15.60:0/2122884813 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2e4034070 con 0x7fa2f40c8360 2024-04-14T23:58:41.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.451+0000 7fa2f3400640 1 -- 172.21.15.60:0/2122884813 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fa2e4030800 con 0x7fa2f40c8360 2024-04-14T23:58:41.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.451+0000 7fa2f3400640 1 -- 172.21.15.60:0/2122884813 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2e4030bf0 con 0x7fa2f40c8360 2024-04-14T23:58:41.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.452+0000 7fa2fbc00640 1 -- 172.21.15.60:0/2122884813 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa2f40c8360 msgr2=0x7fa2f415ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:41.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.452+0000 7fa2fbc00640 1 --2- 172.21.15.60:0/2122884813 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa2f40c8360 0x7fa2f415ef80 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa2e40097a0 tx=0x7fa2e402b940 comp rx=0 tx=0).stop 2024-04-14T23:58:41.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.452+0000 7fa2fbc00640 1 -- 172.21.15.60:0/2122884813 shutdown_connections 2024-04-14T23:58:41.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.452+0000 7fa2fbc00640 1 --2- 172.21.15.60:0/2122884813 >> v2:172.21.15.186:3300/0 conn(0x7fa2f40c75c0 0x7fa2f4158e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:41.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.452+0000 7fa2fbc00640 1 --2- 172.21.15.60:0/2122884813 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa2f40c8360 0x7fa2f415ef80 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:41.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.452+0000 7fa2fbc00640 1 -- 172.21.15.60:0/2122884813 >> 172.21.15.60:0/2122884813 conn(0x7fa2f40cc170 msgr2=0x7fa2f40cc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:41.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.452+0000 7fa2fbc00640 1 -- 172.21.15.60:0/2122884813 shutdown_connections 2024-04-14T23:58:41.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.452+0000 7fa2fbc00640 1 -- 172.21.15.60:0/2122884813 wait complete. 2024-04-14T23:58:41.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.453+0000 7fa2fbc00640 1 Processor -- start 2024-04-14T23:58:41.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.454+0000 7fa2fbc00640 1 -- start start 2024-04-14T23:58:41.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.455+0000 7fa2fbc00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2f40c75c0 0x7fa2f4162650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:41.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.455+0000 7fa2fbc00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2f40c8360 0x7fa2f40c5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:41.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.455+0000 7fa2fbc00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa2f40c8740 0x7fa2f40c5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:41.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.455+0000 7fa2fbc00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fa2f40c70a0 con 0x7fa2f40c75c0 2024-04-14T23:58:41.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.455+0000 7fa2fbc00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa2f40c6dc0 con 0x7fa2f40c8360 2024-04-14T23:58:41.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.455+0000 7fa2fbc00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fa2f415f4c0 con 0x7fa2f40c8740 2024-04-14T23:58:41.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.455+0000 7fa2f9200640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2f40c75c0 0x7fa2f4162650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:41.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.455+0000 7fa2f9200640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2f40c75c0 0x7fa2f4162650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48770/0 (socket says 172.21.15.60:48770) 2024-04-14T23:58:41.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.455+0000 7fa2f9200640 1 -- 172.21.15.60:0/1771029746 learned_addr learned my addr 172.21.15.60:0/1771029746 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:58:41.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.455+0000 7fa2f3e00640 1 --2- 172.21.15.60:0/1771029746 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2f40c8360 0x7fa2f40c5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:41.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.455+0000 7fa2f9c00640 1 --2- 172.21.15.60:0/1771029746 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa2f40c8740 0x7fa2f40c5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:41.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.455+0000 7fa2f9200640 1 -- 172.21.15.60:0/1771029746 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa2f40c8740 msgr2=0x7fa2f40c5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:41.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.455+0000 7fa2f9200640 1 --2- 172.21.15.60:0/1771029746 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa2f40c8740 0x7fa2f40c5790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:41.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.455+0000 7fa2f9200640 1 -- 172.21.15.60:0/1771029746 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2f40c8360 msgr2=0x7fa2f40c5250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:41.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.455+0000 7fa2f9200640 1 --2- 172.21.15.60:0/1771029746 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2f40c8360 0x7fa2f40c5250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:41.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.455+0000 7fa2f9200640 1 -- 172.21.15.60:0/1771029746 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2f414e390 con 0x7fa2f40c75c0 2024-04-14T23:58:41.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.456+0000 7fa2f9200640 1 --2- 172.21.15.60:0/1771029746 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2f40c75c0 0x7fa2f4162650 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fa2e40098c0 tx=0x7fa2e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:41.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.456+0000 7fa2f0c00640 1 -- 172.21.15.60:0/1771029746 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2e4048070 con 0x7fa2f40c75c0 2024-04-14T23:58:41.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.456+0000 7fa2f0c00640 1 -- 172.21.15.60:0/1771029746 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fa2e4030600 con 0x7fa2f40c75c0 2024-04-14T23:58:41.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.456+0000 7fa2f0c00640 1 -- 172.21.15.60:0/1771029746 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2e4034040 con 0x7fa2f40c75c0 2024-04-14T23:58:41.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.456+0000 7fa2fbc00640 1 -- 172.21.15.60:0/1771029746 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2f41f7ff0 con 0x7fa2f40c75c0 2024-04-14T23:58:41.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.456+0000 7fa2fbc00640 1 -- 172.21.15.60:0/1771029746 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2f41f85b0 con 0x7fa2f40c75c0 2024-04-14T23:58:41.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.457+0000 7fa2fbc00640 1 -- 172.21.15.60:0/1771029746 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2f41f8c60 con 0x7fa2f40c75c0 2024-04-14T23:58:41.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.458+0000 7fa2f0c00640 1 -- 172.21.15.60:0/1771029746 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7fa2e404b070 con 0x7fa2f40c75c0 2024-04-14T23:58:41.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.459+0000 7fa2f0c00640 1 --2- 172.21.15.60:0/1771029746 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fa2d006b990 0x7fa2d006de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:41.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.459+0000 7fa2f0c00640 1 -- 172.21.15.60:0/1771029746 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 6341+0+0 (secure 0 0 0) 0x7fa2e40ac540 con 0x7fa2f40c75c0 2024-04-14T23:58:41.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.460+0000 7fa2f3e00640 1 --2- 172.21.15.60:0/1771029746 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fa2d006b990 0x7fa2d006de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:41.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.460+0000 7fa2f3e00640 1 --2- 172.21.15.60:0/1771029746 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fa2d006b990 0x7fa2d006de50 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa2f40cc0b0 tx=0x7fa2dc007e80 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:41.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.463+0000 7fa2f0c00640 1 -- 172.21.15.60:0/1771029746 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa2f414e390 con 0x7fa2f40c75c0 2024-04-14T23:58:41.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:41.698+0000 7fa2fbc00640 1 -- 172.21.15.60:0/1771029746 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fa2f4076970 con 0x7fa2f40c75c0 2024-04-14T23:58:42.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.223+0000 7fa2f0c00640 1 -- 172.21.15.60:0/1771029746 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v38) ==== 104+0+0 (secure 0 0 0) 0x7fa2e407a870 con 0x7fa2f40c75c0 2024-04-14T23:58:42.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.227+0000 7fa2fbc00640 1 -- 172.21.15.60:0/1771029746 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fa2d006b990 msgr2=0x7fa2d006de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:42.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.227+0000 7fa2fbc00640 1 --2- 172.21.15.60:0/1771029746 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fa2d006b990 0x7fa2d006de50 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa2f40cc0b0 tx=0x7fa2dc007e80 comp rx=0 tx=0).stop 2024-04-14T23:58:42.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.227+0000 7fa2fbc00640 1 -- 172.21.15.60:0/1771029746 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2f40c75c0 msgr2=0x7fa2f4162650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:42.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.227+0000 7fa2fbc00640 1 --2- 172.21.15.60:0/1771029746 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2f40c75c0 0x7fa2f4162650 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fa2e40098c0 tx=0x7fa2e40046c0 comp rx=0 tx=0).stop 2024-04-14T23:58:42.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.227+0000 7fa2fbc00640 1 -- 172.21.15.60:0/1771029746 shutdown_connections 2024-04-14T23:58:42.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.227+0000 7fa2fbc00640 1 --2- 172.21.15.60:0/1771029746 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fa2d006b990 0x7fa2d006de50 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:42.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.227+0000 7fa2fbc00640 1 --2- 172.21.15.60:0/1771029746 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa2f40c8740 0x7fa2f40c5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:42.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.227+0000 7fa2fbc00640 1 --2- 172.21.15.60:0/1771029746 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2f40c8360 0x7fa2f40c5250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:42.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.227+0000 7fa2fbc00640 1 --2- 172.21.15.60:0/1771029746 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2f40c75c0 0x7fa2f4162650 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:42.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.227+0000 7fa2fbc00640 1 -- 172.21.15.60:0/1771029746 >> 172.21.15.60:0/1771029746 conn(0x7fa2f40cc170 msgr2=0x7fa2f4072490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:42.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.227+0000 7fa2fbc00640 1 -- 172.21.15.60:0/1771029746 shutdown_connections 2024-04-14T23:58:42.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.227+0000 7fa2fbc00640 1 -- 172.21.15.60:0/1771029746 wait complete. 2024-04-14T23:58:42.253 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-14T23:58:42.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.461+0000 7fd4e4e00640 1 Processor -- start 2024-04-14T23:58:42.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.462+0000 7fd4e4e00640 1 -- start start 2024-04-14T23:58:42.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.463+0000 7fd4e4e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd4e0154ca0 0x7fd4e006bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:42.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.463+0000 7fd4e4e00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd4e00ccbb0 0x7fd4e015e190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:42.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.463+0000 7fd4e4e00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd4e00cda40 0x7fd4e015e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:42.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.463+0000 7fd4e4e00640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fd4e015b790 con 0x7fd4e00ccbb0 2024-04-14T23:58:42.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.463+0000 7fd4e4e00640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fd4e0155080 con 0x7fd4e00cda40 2024-04-14T23:58:42.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.463+0000 7fd4e4e00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fd4e006c2d0 con 0x7fd4e0154ca0 2024-04-14T23:58:42.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.463+0000 7fd4dea00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd4e00ccbb0 0x7fd4e015e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:42.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.463+0000 7fd4dea00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd4e00ccbb0 0x7fd4e015e190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38228/0 (socket says 172.21.15.60:38228) 2024-04-14T23:58:42.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.463+0000 7fd4dea00640 1 -- 172.21.15.60:0/834657548 learned_addr learned my addr 172.21.15.60:0/834657548 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:58:42.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.463+0000 7fd4df400640 1 --2- 172.21.15.60:0/834657548 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd4e0154ca0 0x7fd4e006bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:42.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.463+0000 7fd4dfe00640 1 --2- 172.21.15.60:0/834657548 >> v2:172.21.15.186:3300/0 conn(0x7fd4e00cda40 0x7fd4e015e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:42.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.464+0000 7fd4dea00640 1 -- 172.21.15.60:0/834657548 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd4e0154ca0 msgr2=0x7fd4e006bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:58:42.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.464+0000 7fd4dea00640 1 --2- 172.21.15.60:0/834657548 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd4e0154ca0 0x7fd4e006bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:42.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.464+0000 7fd4dea00640 1 -- 172.21.15.60:0/834657548 >> v2:172.21.15.186:3300/0 conn(0x7fd4e00cda40 msgr2=0x7fd4e015e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:58:42.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.464+0000 7fd4dea00640 1 --2- 172.21.15.60:0/834657548 >> v2:172.21.15.186:3300/0 conn(0x7fd4e00cda40 0x7fd4e015e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:42.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.464+0000 7fd4dea00640 1 -- 172.21.15.60:0/834657548 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4e015aba0 con 0x7fd4e00ccbb0 2024-04-14T23:58:42.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.464+0000 7fd4dfe00640 1 --2- 172.21.15.60:0/834657548 >> v2:172.21.15.186:3300/0 conn(0x7fd4e00cda40 0x7fd4e015e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:58:42.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.464+0000 7fd4df400640 1 --2- 172.21.15.60:0/834657548 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd4e0154ca0 0x7fd4e006bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:58:42.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.465+0000 7fd4dea00640 1 --2- 172.21.15.60:0/834657548 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4e00ccbb0 0x7fd4e015e190 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fd4e0155430 tx=0x7fd4c802b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f960a8708c3daba server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:42.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.465+0000 7fd4de000640 1 -- 172.21.15.60:0/834657548 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4c8034070 con 0x7fd4e00ccbb0 2024-04-14T23:58:42.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.465+0000 7fd4de000640 1 -- 172.21.15.60:0/834657548 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fd4c8030850 con 0x7fd4e00ccbb0 2024-04-14T23:58:42.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.465+0000 7fd4de000640 1 -- 172.21.15.60:0/834657548 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4c8030c40 con 0x7fd4e00ccbb0 2024-04-14T23:58:42.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.466+0000 7fd4e4e00640 1 -- 172.21.15.60:0/834657548 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4e00ccbb0 msgr2=0x7fd4e015e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:42.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.466+0000 7fd4e4e00640 1 --2- 172.21.15.60:0/834657548 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4e00ccbb0 0x7fd4e015e190 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fd4e0155430 tx=0x7fd4c802b990 comp rx=0 tx=0).stop 2024-04-14T23:58:42.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.466+0000 7fd4e4e00640 1 -- 172.21.15.60:0/834657548 shutdown_connections 2024-04-14T23:58:42.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.466+0000 7fd4e4e00640 1 --2- 172.21.15.60:0/834657548 >> v2:172.21.15.186:3300/0 conn(0x7fd4e00cda40 0x7fd4e015e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:42.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.466+0000 7fd4e4e00640 1 --2- 172.21.15.60:0/834657548 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4e00ccbb0 0x7fd4e015e190 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:42.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.466+0000 7fd4e4e00640 1 --2- 172.21.15.60:0/834657548 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd4e0154ca0 0x7fd4e006bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:42.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.466+0000 7fd4e4e00640 1 -- 172.21.15.60:0/834657548 >> 172.21.15.60:0/834657548 conn(0x7fd4e006d760 msgr2=0x7fd4e006db70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:42.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.466+0000 7fd4e4e00640 1 -- 172.21.15.60:0/834657548 shutdown_connections 2024-04-14T23:58:42.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.466+0000 7fd4e4e00640 1 -- 172.21.15.60:0/834657548 wait complete. 2024-04-14T23:58:42.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.468+0000 7fd4e4e00640 1 Processor -- start 2024-04-14T23:58:42.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.468+0000 7fd4e4e00640 1 -- start start 2024-04-14T23:58:42.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.469+0000 7fd4e4e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd4e00ccbb0 0x7fd4e01f1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:42.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.469+0000 7fd4e4e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd4e00cda40 0x7fd4e00c7d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:42.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.469+0000 7fd4e4e00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4e0154ca0 0x7fd4e0158d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:42.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.469+0000 7fd4e4e00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fd4e015b420 con 0x7fd4e0154ca0 2024-04-14T23:58:42.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.469+0000 7fd4e4e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd4e015b140 con 0x7fd4e00ccbb0 2024-04-14T23:58:42.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.469+0000 7fd4e4e00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fd4e006bff0 con 0x7fd4e00cda40 2024-04-14T23:58:42.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.469+0000 7fd4dfe00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4e0154ca0 0x7fd4e0158d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:42.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.469+0000 7fd4df400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd4e00ccbb0 0x7fd4e01f1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:42.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.469+0000 7fd4dea00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd4e00cda40 0x7fd4e00c7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:42.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.469+0000 7fd4dfe00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4e0154ca0 0x7fd4e0158d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38230/0 (socket says 172.21.15.60:38230) 2024-04-14T23:58:42.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.469+0000 7fd4dfe00640 1 -- 172.21.15.60:0/407633543 learned_addr learned my addr 172.21.15.60:0/407633543 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:58:42.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.470+0000 7fd4dfe00640 1 -- 172.21.15.60:0/407633543 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd4e00cda40 msgr2=0x7fd4e00c7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:42.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.470+0000 7fd4dfe00640 1 --2- 172.21.15.60:0/407633543 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd4e00cda40 0x7fd4e00c7d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:42.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.470+0000 7fd4dfe00640 1 -- 172.21.15.60:0/407633543 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd4e00ccbb0 msgr2=0x7fd4e01f1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:42.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.470+0000 7fd4dfe00640 1 --2- 172.21.15.60:0/407633543 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd4e00ccbb0 0x7fd4e01f1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:42.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.470+0000 7fd4dfe00640 1 -- 172.21.15.60:0/407633543 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4e01f1580 con 0x7fd4e0154ca0 2024-04-14T23:58:42.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.470+0000 7fd4df400640 1 --2- 172.21.15.60:0/407633543 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd4e00ccbb0 0x7fd4e01f1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-14T23:58:42.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.470+0000 7fd4dfe00640 1 --2- 172.21.15.60:0/407633543 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4e0154ca0 0x7fd4e0158d80 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fd4d800e930 tx=0x7fd4d800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:42.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.470+0000 7fd4d7400640 1 -- 172.21.15.60:0/407633543 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4d8017070 con 0x7fd4e0154ca0 2024-04-14T23:58:42.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.470+0000 7fd4d7400640 1 -- 172.21.15.60:0/407633543 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fd4d8004020 con 0x7fd4e0154ca0 2024-04-14T23:58:42.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.471+0000 7fd4d7400640 1 -- 172.21.15.60:0/407633543 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4d8004450 con 0x7fd4e0154ca0 2024-04-14T23:58:42.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.471+0000 7fd4e4e00640 1 -- 172.21.15.60:0/407633543 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4e00c9990 con 0x7fd4e0154ca0 2024-04-14T23:58:42.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.471+0000 7fd4e4e00640 1 -- 172.21.15.60:0/407633543 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4e00c8f70 con 0x7fd4e0154ca0 2024-04-14T23:58:42.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.472+0000 7fd4e4e00640 1 -- 172.21.15.60:0/407633543 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4e004f540 con 0x7fd4e0154ca0 2024-04-14T23:58:42.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.473+0000 7fd4d7400640 1 -- 172.21.15.60:0/407633543 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7fd4d800c940 con 0x7fd4e0154ca0 2024-04-14T23:58:42.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.474+0000 7fd4d7400640 1 --2- 172.21.15.60:0/407633543 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fd4bc06b990 0x7fd4bc06de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:42.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.474+0000 7fd4d7400640 1 -- 172.21.15.60:0/407633543 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 6341+0+0 (secure 0 0 0) 0x7fd4d808a9b0 con 0x7fd4e0154ca0 2024-04-14T23:58:42.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.474+0000 7fd4df400640 1 --2- 172.21.15.60:0/407633543 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fd4bc06b990 0x7fd4bc06de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:42.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.475+0000 7fd4df400640 1 --2- 172.21.15.60:0/407633543 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fd4bc06b990 0x7fd4bc06de50 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fd4d0009fd0 tx=0x7fd4d000b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:42.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.478+0000 7fd4d7400640 1 -- 172.21.15.60:0/407633543 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd4d8058bc0 con 0x7fd4e0154ca0 2024-04-14T23:58:42.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:42.679+0000 7fd4e4e00640 1 -- 172.21.15.60:0/407633543 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fd4e00cb420 con 0x7fd4e0154ca0 2024-04-14T23:58:43.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.234+0000 7fd4d7400640 1 -- 172.21.15.60:0/407633543 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v47) ==== 181+0+0 (secure 0 0 0) 0x7fd4d805cc60 con 0x7fd4e0154ca0 2024-04-14T23:58:43.241 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' removed 2024-04-14T23:58:43.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.240+0000 7fd4e4e00640 1 -- 172.21.15.60:0/407633543 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fd4bc06b990 msgr2=0x7fd4bc06de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:43.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.240+0000 7fd4e4e00640 1 --2- 172.21.15.60:0/407633543 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fd4bc06b990 0x7fd4bc06de50 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fd4d0009fd0 tx=0x7fd4d000b040 comp rx=0 tx=0).stop 2024-04-14T23:58:43.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.240+0000 7fd4e4e00640 1 -- 172.21.15.60:0/407633543 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4e0154ca0 msgr2=0x7fd4e0158d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:43.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.240+0000 7fd4e4e00640 1 --2- 172.21.15.60:0/407633543 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4e0154ca0 0x7fd4e0158d80 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fd4d800e930 tx=0x7fd4d800edf0 comp rx=0 tx=0).stop 2024-04-14T23:58:43.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.241+0000 7fd4e4e00640 1 -- 172.21.15.60:0/407633543 shutdown_connections 2024-04-14T23:58:43.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.241+0000 7fd4e4e00640 1 --2- 172.21.15.60:0/407633543 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fd4bc06b990 0x7fd4bc06de50 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:43.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.241+0000 7fd4e4e00640 1 --2- 172.21.15.60:0/407633543 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4e0154ca0 0x7fd4e0158d80 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:43.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.241+0000 7fd4e4e00640 1 --2- 172.21.15.60:0/407633543 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd4e00cda40 0x7fd4e00c7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:43.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.241+0000 7fd4e4e00640 1 --2- 172.21.15.60:0/407633543 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd4e00ccbb0 0x7fd4e01f1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:43.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.241+0000 7fd4e4e00640 1 -- 172.21.15.60:0/407633543 >> 172.21.15.60:0/407633543 conn(0x7fd4e006d760 msgr2=0x7fd4e0159cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:43.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.241+0000 7fd4e4e00640 1 -- 172.21.15.60:0/407633543 shutdown_connections 2024-04-14T23:58:43.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.242+0000 7fd4e4e00640 1 -- 172.21.15.60:0/407633543 wait complete. 2024-04-14T23:58:43.277 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-14T23:58:43.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.480+0000 7fcc6be00640 1 Processor -- start 2024-04-14T23:58:43.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.480+0000 7fcc6be00640 1 -- start start 2024-04-14T23:58:43.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.482+0000 7fcc6be00640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fcc6c0ccbb0 0x7fcc6c15e190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:43.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.482+0000 7fcc6be00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fcc6c0cda40 0x7fcc6c15e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:43.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.482+0000 7fcc6be00640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc6c155080 con 0x7fcc6c154ca0 2024-04-14T23:58:43.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.482+0000 7fcc6be00640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fcc6c15b790 con 0x7fcc6c0ccbb0 2024-04-14T23:58:43.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.482+0000 7fcc6be00640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fcc6c06c2d0 con 0x7fcc6c0cda40 2024-04-14T23:58:43.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.482+0000 7fcc6aa00640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fcc6c154ca0 0x7fcc6c06bab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:45338/0 (socket says 172.21.15.60:45338) 2024-04-14T23:58:43.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.482+0000 7fcc6aa00640 1 -- 172.21.15.60:0/3660729465 learned_addr learned my addr 172.21.15.60:0/3660729465 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-04-14T23:58:43.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.482+0000 7fcc63e00640 1 --2- 172.21.15.60:0/3660729465 >> v2:172.21.15.60:3300/0 conn(0x7fcc6c0ccbb0 0x7fcc6c15e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:43.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.482+0000 7fcc6b400640 1 --2- 172.21.15.60:0/3660729465 >> v2:172.21.15.186:3300/0 conn(0x7fcc6c0cda40 0x7fcc6c15e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:43.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.483+0000 7fcc63e00640 1 -- 172.21.15.60:0/3660729465 >> v2:172.21.15.186:3300/0 conn(0x7fcc6c0cda40 msgr2=0x7fcc6c15e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-14T23:58:43.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.483+0000 7fcc6a000640 1 -- 172.21.15.60:0/3660729465 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 882388050 0 0) 0x7fcc5c001110 con 0x7fcc6c154ca0 2024-04-14T23:58:43.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.483+0000 7fcc63e00640 1 --2- 172.21.15.60:0/3660729465 >> v2:172.21.15.186:3300/0 conn(0x7fcc6c0cda40 0x7fcc6c15e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:43.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.483+0000 7fcc63e00640 1 -- 172.21.15.60:0/3660729465 >> v1:172.21.15.60:6789/0 conn(0x7fcc6c154ca0 legacy=0x7fcc6c06bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:43.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.483+0000 7fcc63e00640 1 -- 172.21.15.60:0/3660729465 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc6c15aba0 con 0x7fcc6c0ccbb0 2024-04-14T23:58:43.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.484+0000 7fcc63e00640 1 --2- 172.21.15.60:0/3660729465 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc6c0ccbb0 0x7fcc6c15e190 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c0cd980 tx=0x7fcc5402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=678676efbbbca22a server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:43.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.484+0000 7fcc6a000640 1 -- 172.21.15.60:0/3660729465 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc54034070 con 0x7fcc6c0ccbb0 2024-04-14T23:58:43.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.485+0000 7fcc6a000640 1 -- 172.21.15.60:0/3660729465 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fcc54030610 con 0x7fcc6c0ccbb0 2024-04-14T23:58:43.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.485+0000 7fcc6a000640 1 -- 172.21.15.60:0/3660729465 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc54030a00 con 0x7fcc6c0ccbb0 2024-04-14T23:58:43.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.485+0000 7fcc6be00640 1 -- 172.21.15.60:0/3660729465 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc6c0ccbb0 msgr2=0x7fcc6c15e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:43.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.485+0000 7fcc6be00640 1 --2- 172.21.15.60:0/3660729465 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc6c0ccbb0 0x7fcc6c15e190 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c0cd980 tx=0x7fcc5402b960 comp rx=0 tx=0).stop 2024-04-14T23:58:43.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.485+0000 7fcc6be00640 1 -- 172.21.15.60:0/3660729465 shutdown_connections 2024-04-14T23:58:43.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.485+0000 7fcc6be00640 1 --2- 172.21.15.60:0/3660729465 >> v2:172.21.15.186:3300/0 conn(0x7fcc6c0cda40 0x7fcc6c15e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:43.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.485+0000 7fcc6be00640 1 --2- 172.21.15.60:0/3660729465 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc6c0ccbb0 0x7fcc6c15e190 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:43.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.485+0000 7fcc6be00640 1 -- 172.21.15.60:0/3660729465 >> 172.21.15.60:0/3660729465 conn(0x7fcc6c06d760 msgr2=0x7fcc6c06db70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:43.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.485+0000 7fcc6be00640 1 -- 172.21.15.60:0/3660729465 shutdown_connections 2024-04-14T23:58:43.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.485+0000 7fcc6be00640 1 -- 172.21.15.60:0/3660729465 wait complete. 2024-04-14T23:58:43.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.487+0000 7fcc6be00640 1 Processor -- start 2024-04-14T23:58:43.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.487+0000 7fcc6be00640 1 -- start start 2024-04-14T23:58:43.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.488+0000 7fcc6be00640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc6c0ccbb0 0x7fcc6c0cf620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:43.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.488+0000 7fcc6be00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc6c0cda40 0x7fcc6c156950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:43.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.488+0000 7fcc6be00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcc6c154ca0 0x7fcc6c0c56b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:43.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.488+0000 7fcc6be00640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fcc6c15b420 con 0x7fcc6c0ccbb0 2024-04-14T23:58:43.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.488+0000 7fcc6be00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcc6c15b140 con 0x7fcc6c0cda40 2024-04-14T23:58:43.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.488+0000 7fcc6be00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 -- 0x7fcc6c0cb640 con 0x7fcc6c154ca0 2024-04-14T23:58:43.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.489+0000 7fcc63e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc6c0cda40 0x7fcc6c156950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:43.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.489+0000 7fcc63e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc6c0cda40 0x7fcc6c156950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.60:33940/0 (socket says 172.21.15.60:33940) 2024-04-14T23:58:43.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.489+0000 7fcc63e00640 1 -- 172.21.15.60:0/2918357194 learned_addr learned my addr 172.21.15.60:0/2918357194 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-04-14T23:58:43.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.489+0000 7fcc6aa00640 1 --2- 172.21.15.60:0/2918357194 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc6c0ccbb0 0x7fcc6c0cf620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:43.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.489+0000 7fcc6b400640 1 --2- 172.21.15.60:0/2918357194 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcc6c154ca0 0x7fcc6c0c56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:43.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.489+0000 7fcc63e00640 1 -- 172.21.15.60:0/2918357194 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcc6c154ca0 msgr2=0x7fcc6c0c56b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:43.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.489+0000 7fcc63e00640 1 --2- 172.21.15.60:0/2918357194 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcc6c154ca0 0x7fcc6c0c56b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:43.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.489+0000 7fcc63e00640 1 -- 172.21.15.60:0/2918357194 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc6c0ccbb0 msgr2=0x7fcc6c0cf620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:43.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.489+0000 7fcc63e00640 1 --2- 172.21.15.60:0/2918357194 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc6c0ccbb0 0x7fcc6c0cf620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:43.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.489+0000 7fcc63e00640 1 -- 172.21.15.60:0/2918357194 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc6c0d0510 con 0x7fcc6c0cda40 2024-04-14T23:58:43.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.489+0000 7fcc63e00640 1 --2- 172.21.15.60:0/2918357194 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc6c0cda40 0x7fcc6c156950 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fcc54000c00 tx=0x7fcc540046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:43.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.490+0000 7fcc62a00640 1 -- 172.21.15.60:0/2918357194 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc54034040 con 0x7fcc6c0cda40 2024-04-14T23:58:43.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.490+0000 7fcc62a00640 1 -- 172.21.15.60:0/2918357194 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fcc54030410 con 0x7fcc6c0cda40 2024-04-14T23:58:43.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.490+0000 7fcc62a00640 1 -- 172.21.15.60:0/2918357194 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc54039c70 con 0x7fcc6c0cda40 2024-04-14T23:58:43.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.490+0000 7fcc6be00640 1 -- 172.21.15.60:0/2918357194 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc6c0d07d0 con 0x7fcc6c0cda40 2024-04-14T23:58:43.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.490+0000 7fcc6be00640 1 -- 172.21.15.60:0/2918357194 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc6c1f3c10 con 0x7fcc6c0cda40 2024-04-14T23:58:43.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.491+0000 7fcc6be00640 1 -- 172.21.15.60:0/2918357194 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc6c1f4220 con 0x7fcc6c0cda40 2024-04-14T23:58:43.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.492+0000 7fcc62a00640 1 -- 172.21.15.60:0/2918357194 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90154+0+0 (secure 0 0 0) 0x7fcc54037070 con 0x7fcc6c0cda40 2024-04-14T23:58:43.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.493+0000 7fcc62a00640 1 --2- 172.21.15.60:0/2918357194 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fcc4806b9c0 0x7fcc4806de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-14T23:58:43.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.493+0000 7fcc62a00640 1 -- 172.21.15.60:0/2918357194 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5903+0+0 (secure 0 0 0) 0x7fcc540ac8d0 con 0x7fcc6c0cda40 2024-04-14T23:58:43.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.493+0000 7fcc6aa00640 1 --2- 172.21.15.60:0/2918357194 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fcc4806b9c0 0x7fcc4806de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-14T23:58:43.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.494+0000 7fcc6aa00640 1 --2- 172.21.15.60:0/2918357194 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fcc4806b9c0 0x7fcc4806de80 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c007fd0 tx=0x7fcc5c007de0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-14T23:58:43.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.497+0000 7fcc62a00640 1 -- 172.21.15.60:0/2918357194 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fcc5407aca0 con 0x7fcc6c0cda40 2024-04-14T23:58:43.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:43.702+0000 7fcc6be00640 1 -- 172.21.15.60:0/2918357194 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fcc6c04f540 con 0x7fcc6c0cda40 2024-04-14T23:58:44.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:44.240+0000 7fcc62a00640 1 -- 172.21.15.60:0/2918357194 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v48) ==== 169+0+0 (secure 0 0 0) 0x7fcc5407ed40 con 0x7fcc6c0cda40 2024-04-14T23:58:44.247 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' removed 2024-04-14T23:58:44.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:44.246+0000 7fcc6be00640 1 -- 172.21.15.60:0/2918357194 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fcc4806b9c0 msgr2=0x7fcc4806de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:44.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:44.246+0000 7fcc6be00640 1 --2- 172.21.15.60:0/2918357194 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fcc4806b9c0 0x7fcc4806de80 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c007fd0 tx=0x7fcc5c007de0 comp rx=0 tx=0).stop 2024-04-14T23:58:44.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:44.246+0000 7fcc6be00640 1 -- 172.21.15.60:0/2918357194 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc6c0cda40 msgr2=0x7fcc6c156950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-14T23:58:44.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:44.246+0000 7fcc6be00640 1 --2- 172.21.15.60:0/2918357194 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc6c0cda40 0x7fcc6c156950 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fcc54000c00 tx=0x7fcc540046c0 comp rx=0 tx=0).stop 2024-04-14T23:58:44.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:44.249+0000 7fcc6be00640 1 -- 172.21.15.60:0/2918357194 shutdown_connections 2024-04-14T23:58:44.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:44.249+0000 7fcc6be00640 1 --2- 172.21.15.60:0/2918357194 >> [v2:172.21.15.186:6832/714104835,v1:172.21.15.186:6833/714104835] conn(0x7fcc4806b9c0 0x7fcc4806de80 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:44.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:44.249+0000 7fcc6be00640 1 --2- 172.21.15.60:0/2918357194 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcc6c154ca0 0x7fcc6c0c56b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:44.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:44.249+0000 7fcc6be00640 1 --2- 172.21.15.60:0/2918357194 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc6c0cda40 0x7fcc6c156950 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:44.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:44.249+0000 7fcc6be00640 1 --2- 172.21.15.60:0/2918357194 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc6c0ccbb0 0x7fcc6c0cf620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-14T23:58:44.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:44.249+0000 7fcc6be00640 1 -- 172.21.15.60:0/2918357194 >> 172.21.15.60:0/2918357194 conn(0x7fcc6c06d760 msgr2=0x7fcc6c0cd080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-14T23:58:44.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:44.250+0000 7fcc6be00640 1 -- 172.21.15.60:0/2918357194 shutdown_connections 2024-04-14T23:58:44.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-04-14T23:58:44.250+0000 7fcc6be00640 1 -- 172.21.15.60:0/2918357194 wait complete. 2024-04-14T23:58:44.285 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-14T23:58:44.285 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-14T23:58:44.285 INFO:teuthology.orchestra.run:waiting for 300 2024-04-14T23:58:44.303 INFO:tasks.ceph.mds.a:Stopped 2024-04-14T23:58:44.303 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-14T23:58:44.303 INFO:teuthology.orchestra.run:waiting for 300 2024-04-14T23:58:44.367 INFO:tasks.ceph.mds.c:Stopped 2024-04-14T23:58:44.368 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2024-04-14T23:58:44.368 INFO:teuthology.orchestra.run:waiting for 300 2024-04-14T23:58:44.441 INFO:tasks.ceph.mds.e:Stopped 2024-04-14T23:58:44.442 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-14T23:58:44.442 INFO:teuthology.orchestra.run:waiting for 300 2024-04-14T23:58:44.453 INFO:tasks.ceph.mds.b:Stopped 2024-04-14T23:58:44.454 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-14T23:58:44.454 INFO:teuthology.orchestra.run:waiting for 300 2024-04-14T23:58:44.505 INFO:tasks.ceph.mds.d:Stopped 2024-04-14T23:58:44.506 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2024-04-14T23:58:44.506 INFO:teuthology.orchestra.run:waiting for 300 2024-04-14T23:58:44.557 INFO:tasks.ceph.mds.f:Stopped 2024-04-14T23:58:44.557 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-14T23:58:44.557 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-14T23:58:44.558 INFO:teuthology.orchestra.run:waiting for 300 2024-04-14T23:58:44.627 INFO:tasks.ceph.osd.0:Stopped 2024-04-14T23:58:44.627 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-14T23:58:44.628 INFO:teuthology.orchestra.run:waiting for 300 2024-04-14T23:58:46.603 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-14T23:58:46.611 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-14T23:58:48.899 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-04-14T23:58:50.729 INFO:tasks.ceph.osd.1:Stopped 2024-04-14T23:58:50.729 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-14T23:58:50.730 INFO:teuthology.orchestra.run:waiting for 300 2024-04-14T23:58:50.828 INFO:tasks.ceph.osd.2:Stopped 2024-04-14T23:58:50.829 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-14T23:58:50.829 INFO:teuthology.orchestra.run:waiting for 300 2024-04-14T23:58:50.897 INFO:tasks.ceph.osd.3:Stopped 2024-04-14T23:58:50.897 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-04-14T23:58:50.897 INFO:teuthology.orchestra.run:waiting for 300 2024-04-14T23:58:50.939 INFO:tasks.ceph.osd.4:Stopped 2024-04-14T23:58:50.939 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-04-14T23:58:50.939 INFO:teuthology.orchestra.run:waiting for 300 2024-04-14T23:58:50.975 INFO:tasks.ceph.osd.5:Stopped 2024-04-14T23:58:50.975 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-04-14T23:58:50.975 INFO:teuthology.orchestra.run:waiting for 300 2024-04-14T23:58:51.053 INFO:tasks.ceph.osd.6:Stopped 2024-04-14T23:58:51.054 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-04-14T23:58:51.054 INFO:teuthology.orchestra.run:waiting for 300 2024-04-14T23:58:51.128 INFO:tasks.ceph.osd.7:Stopped 2024-04-14T23:58:51.129 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-14T23:58:51.129 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-04-14T23:58:51.129 INFO:teuthology.orchestra.run:waiting for 300 2024-04-14T23:58:51.210 INFO:tasks.ceph.mgr.y:Stopped 2024-04-14T23:58:51.210 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-14T23:58:51.210 INFO:teuthology.orchestra.run:waiting for 300 2024-04-14T23:58:51.239 INFO:tasks.ceph.mgr.x:Stopped 2024-04-14T23:58:51.240 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-14T23:58:51.240 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-14T23:58:51.240 INFO:teuthology.orchestra.run:waiting for 300 2024-04-14T23:58:51.283 INFO:tasks.ceph.mon.a:Stopped 2024-04-14T23:58:51.283 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-14T23:58:51.283 INFO:teuthology.orchestra.run:waiting for 300 2024-04-14T23:58:51.296 INFO:tasks.ceph.mon.b:Stopped 2024-04-14T23:58:51.296 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-14T23:58:51.297 INFO:teuthology.orchestra.run:waiting for 300 2024-04-14T23:58:51.350 INFO:tasks.ceph.mon.c:Stopped 2024-04-14T23:58:51.350 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-14T23:58:51.350 DEBUG:teuthology.orchestra.run.smithi060:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'Replacing daemon mds' | head -n 1 2024-04-14T23:58:51.482 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi060.front.sepia.ceph.com 2024-04-14T23:58:51.482 DEBUG:teuthology.orchestra.run.smithi060:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-14T23:58:51.811 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi060.front.sepia.ceph.com 2024-04-14T23:58:51.811 DEBUG:teuthology.orchestra.run.smithi060:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-14T23:58:52.000 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi060.front.sepia.ceph.com 2024-04-14T23:58:52.000 DEBUG:teuthology.orchestra.run.smithi060:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-14T23:58:52.146 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi060.front.sepia.ceph.com 2024-04-14T23:58:52.147 DEBUG:teuthology.orchestra.run.smithi060:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-04-14T23:58:52.337 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi186.front.sepia.ceph.com 2024-04-14T23:58:52.337 DEBUG:teuthology.orchestra.run.smithi186:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-04-14T23:58:52.697 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi186.front.sepia.ceph.com 2024-04-14T23:58:52.698 DEBUG:teuthology.orchestra.run.smithi186:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-04-14T23:58:52.776 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi186.front.sepia.ceph.com 2024-04-14T23:58:52.776 DEBUG:teuthology.orchestra.run.smithi186:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-04-14T23:58:52.833 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi186.front.sepia.ceph.com 2024-04-14T23:58:52.833 DEBUG:teuthology.orchestra.run.smithi186:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-04-14T23:58:52.892 INFO:tasks.ceph:Archiving mon data... 2024-04-14T23:58:52.893 DEBUG:teuthology.misc:Transferring archived files from smithi060:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-04-14_21:24:03-fs-squid-distro-default-smithi/7656585/data/mon.a.tgz 2024-04-14T23:58:52.893 DEBUG:teuthology.orchestra.run.smithi060:> mktemp 2024-04-14T23:58:52.955 INFO:teuthology.orchestra.run.smithi060.stdout:/tmp/tmp.0S7P6hIh8R 2024-04-14T23:58:52.956 DEBUG:teuthology.orchestra.run.smithi060:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.0S7P6hIh8R 2024-04-14T23:58:53.161 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0666 /tmp/tmp.0S7P6hIh8R 2024-04-14T23:58:53.317 DEBUG:teuthology.orchestra.remote:smithi060:/tmp/tmp.0S7P6hIh8R is 288KB 2024-04-14T23:58:53.428 DEBUG:teuthology.orchestra.run.smithi060:> rm -fr /tmp/tmp.0S7P6hIh8R 2024-04-14T23:58:53.489 DEBUG:teuthology.misc:Transferring archived files from smithi186:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-04-14_21:24:03-fs-squid-distro-default-smithi/7656585/data/mon.b.tgz 2024-04-14T23:58:53.489 DEBUG:teuthology.orchestra.run.smithi186:> mktemp 2024-04-14T23:58:53.507 INFO:teuthology.orchestra.run.smithi186.stdout:/tmp/tmp.4R4LRJvFay 2024-04-14T23:58:53.508 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.4R4LRJvFay 2024-04-14T23:58:53.646 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 0666 /tmp/tmp.4R4LRJvFay 2024-04-14T23:58:53.734 DEBUG:teuthology.orchestra.remote:smithi186:/tmp/tmp.4R4LRJvFay is 293KB 2024-04-14T23:58:53.830 DEBUG:teuthology.orchestra.run.smithi186:> rm -fr /tmp/tmp.4R4LRJvFay 2024-04-14T23:58:53.846 DEBUG:teuthology.misc:Transferring archived files from smithi186:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2024-04-14_21:24:03-fs-squid-distro-default-smithi/7656585/data/mon.c.tgz 2024-04-14T23:58:53.846 DEBUG:teuthology.orchestra.run.smithi186:> mktemp 2024-04-14T23:58:53.900 INFO:teuthology.orchestra.run.smithi186.stdout:/tmp/tmp.w6dwJTFaIw 2024-04-14T23:58:53.901 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.w6dwJTFaIw 2024-04-14T23:58:53.901 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-04-14T23:58:54.022 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 0666 /tmp/tmp.w6dwJTFaIw 2024-04-14T23:58:54.107 DEBUG:teuthology.orchestra.remote:smithi186:/tmp/tmp.w6dwJTFaIw is 293KB 2024-04-14T23:58:54.203 DEBUG:teuthology.orchestra.run.smithi186:> rm -fr /tmp/tmp.w6dwJTFaIw 2024-04-14T23:58:54.217 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-14T23:58:54.218 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-14T23:58:54.226 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-14T23:58:54.384 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-14T23:58:54.398 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-14T23:58:54.399 INFO:tasks.ceph:Archiving crash dumps... 2024-04-14T23:58:54.401 DEBUG:teuthology.misc:Transferring archived files from smithi060:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-04-14_21:24:03-fs-squid-distro-default-smithi/7656585/remote/smithi060/crash 2024-04-14T23:58:54.401 DEBUG:teuthology.orchestra.run.smithi060:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-14T23:58:54.501 DEBUG:teuthology.misc:Transferring archived files from smithi186:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-04-14_21:24:03-fs-squid-distro-default-smithi/7656585/remote/smithi186/crash 2024-04-14T23:58:54.502 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-14T23:58:54.528 INFO:tasks.ceph:Shutting down logrotate 2024-04-14T23:58:54.529 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-14T23:58:54.537 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-14T23:58:54.687 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-04-14T23:58:54.789 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-04-14T23:58:54.816 INFO:tasks.ceph:Compressing logs... 2024-04-14T23:58:54.817 DEBUG:teuthology.orchestra.run.smithi060:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-14T23:58:54.824 DEBUG:teuthology.orchestra.run.smithi186:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-14T23:58:54.890 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-04-14T23:58:54.891 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38238.log 2024-04-14T23:58:54.891 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38158.log 2024-04-14T23:58:54.891 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.admin.38238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38038.log 2024-04-14T23:58:54.891 INFO:teuthology.orchestra.run.smithi186.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38238.log.gz 2024-04-14T23:58:54.891 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.38158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37998.log 2024-04-14T23:58:54.891 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-14T23:58:54.891 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.38038.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38158.log.gz 2024-04-14T23:58:54.891 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.37998.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-04-14T23:58:54.892 INFO:teuthology.orchestra.run.smithi186.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.38038.log.gz 2024-04-14T23:58:54.892 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph.audit.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.37998.log.gz 2024-04-14T23:58:54.892 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-04-14T23:58:54.892 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38198.log 2024-04-14T23:58:54.892 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38078.log 2024-04-14T23:58:54.892 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.38198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38118.log 2024-04-14T23:58:54.892 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.38078.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-04-14T23:58:54.892 INFO:teuthology.orchestra.run.smithi186.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.38198.log.gz 2024-04-14T23:58:54.893 INFO:teuthology.orchestra.run.smithi186.stderr: 93.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-14T23:58:54.893 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.38118.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-04-14T23:58:54.893 INFO:teuthology.orchestra.run.smithi186.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38078.log.gz 2024-04-14T23:58:54.893 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose/var/log/ceph/ceph-osd.7.log: -- /var/log/ceph/ceph-mds.f.log 2024-04-14T23:58:54.893 INFO:teuthology.orchestra.run.smithi186.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38118.log.gz 2024-04-14T23:58:54.893 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38278.log 2024-04-14T23:58:54.893 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-04-14T23:58:54.893 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.38278.log: 93.3% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-04-14T23:58:54.893 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-14T23:58:54.894 INFO:teuthology.orchestra.run.smithi186.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38278.log.gz 2024-04-14T23:58:54.894 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-04-14T23:58:54.894 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-mgr.x.log: /var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-04-14T23:58:54.894 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-mon.b.log: 93.3% -- replaced with /var/log/ceph/ceph-mds.f.log.gz 2024-04-14T23:58:54.894 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-04-14T23:58:54.894 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-mds.d.log: 92.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-14T23:58:54.895 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-mon.c.log: 93.4% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-04-14T23:58:54.916 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53222.log 2024-04-14T23:58:54.917 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43103.log 2024-04-14T23:58:54.918 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.53222.log: 87.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.53222.log.gz 2024-04-14T23:58:54.918 INFO:teuthology.orchestra.run.smithi060.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.52368.log 2024-04-14T23:58:54.919 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.43103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49735.log 2024-04-14T23:58:54.920 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43103.log.gz 2024-04-14T23:58:54.920 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.52368.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.52368.log.gz 2024-04-14T23:58:54.921 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41971.log 2024-04-14T23:58:54.922 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.49735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47044.log 2024-04-14T23:58:54.923 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49735.log.gz 2024-04-14T23:58:54.924 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51532.log 2024-04-14T23:58:54.924 INFO:teuthology.orchestra.run.smithi060.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41971.log.gz 2024-04-14T23:58:54.925 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.47044.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.47044.log.gz 2024-04-14T23:58:54.925 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-14T23:58:54.926 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.51532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51897.log 2024-04-14T23:58:54.927 INFO:teuthology.orchestra.run.smithi060.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51532.log.gz 2024-04-14T23:58:54.927 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44448.log 2024-04-14T23:58:54.929 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.51897.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51897.log.gz 2024-04-14T23:58:54.929 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47304.log 2024-04-14T23:58:54.930 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.44448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41979.log 2024-04-14T23:58:54.930 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44448.log.gz 2024-04-14T23:58:54.931 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.47304.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47304.log.gz 2024-04-14T23:58:54.931 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50715.log 2024-04-14T23:58:54.933 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52084.log 2024-04-14T23:58:54.933 INFO:teuthology.orchestra.run.smithi060.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41979.log.gz 2024-04-14T23:58:54.933 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.50715.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50715.log.gz 2024-04-14T23:58:54.934 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47212.log 2024-04-14T23:58:54.935 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.52084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45573.log 2024-04-14T23:58:54.935 INFO:teuthology.orchestra.run.smithi060.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.52084.log.gz 2024-04-14T23:58:54.936 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.47212.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.47212.log.gz 2024-04-14T23:58:54.936 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41975.log 2024-04-14T23:58:54.937 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49296.log 2024-04-14T23:58:54.938 INFO:teuthology.orchestra.run.smithi060.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.45573.log.gz 2024-04-14T23:58:54.939 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49464.log 2024-04-14T23:58:54.939 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41975.log.gz 2024-04-14T23:58:54.940 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.49296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.43937.log 2024-04-14T23:58:54.940 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49296.log.gz 2024-04-14T23:58:54.941 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.49464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51446.log 2024-04-14T23:58:54.942 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49464.log.gz 2024-04-14T23:58:54.943 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.0.43937.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.43937.log.gz 2024-04-14T23:58:54.943 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51164.log 2024-04-14T23:58:54.944 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.51446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53180.log 2024-04-14T23:58:54.944 INFO:teuthology.orchestra.run.smithi060.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51446.log.gz 2024-04-14T23:58:54.945 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.51164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52242.log 2024-04-14T23:58:54.945 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.53180.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51164.log.gz 2024-04-14T23:58:54.946 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.53180.log.gz 2024-04-14T23:58:54.946 INFO:teuthology.orchestra.run.smithi060.stderr: -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.30895.log 2024-04-14T23:58:54.947 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.52242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46172.log 2024-04-14T23:58:54.948 INFO:teuthology.orchestra.run.smithi060.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52242.log.gz 2024-04-14T23:58:54.949 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph.tmp-client.admin.30895.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.30895.log.gz 2024-04-14T23:58:54.949 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52718.log 2024-04-14T23:58:54.950 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.46172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44718.log 2024-04-14T23:58:54.950 INFO:teuthology.orchestra.run.smithi060.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46172.log.gz 2024-04-14T23:58:54.951 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.52718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51333.log 2024-04-14T23:58:54.951 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52718.log.gz 2024-04-14T23:58:54.953 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.44718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49673.log 2024-04-14T23:58:54.953 INFO:teuthology.orchestra.run.smithi060.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44718.log.gz 2024-04-14T23:58:54.954 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.51333.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51333.log.gz 2024-04-14T23:58:54.954 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47433.log 2024-04-14T23:58:54.955 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.49673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51939.log 2024-04-14T23:58:54.956 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49673.log.gz 2024-04-14T23:58:54.956 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.47433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53659.log 2024-04-14T23:58:54.957 INFO:teuthology.orchestra.run.smithi060.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.47433.log.gz 2024-04-14T23:58:54.958 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.51939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53719.log 2024-04-14T23:58:54.958 INFO:teuthology.orchestra.run.smithi060.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51939.log.gz 2024-04-14T23:58:54.959 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.53659.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.53659.log.gz 2024-04-14T23:58:54.959 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41222.log 2024-04-14T23:58:54.961 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.53719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45447.log 2024-04-14T23:58:54.961 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53719.log.gz 2024-04-14T23:58:54.962 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44070.log 2024-04-14T23:58:54.962 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41222.log.gz 2024-04-14T23:58:54.963 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42977.log 2024-04-14T23:58:54.963 INFO:teuthology.orchestra.run.smithi060.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45447.log.gz 2024-04-14T23:58:54.964 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.44070.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44070.log.gz 2024-04-14T23:58:54.964 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53762.log 2024-04-14T23:58:54.966 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42935.log 2024-04-14T23:58:54.966 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42977.log.gz 2024-04-14T23:58:54.967 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.53762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49117.log 2024-04-14T23:58:54.967 INFO:teuthology.orchestra.run.smithi060.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.53762.log.gz 2024-04-14T23:58:54.968 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45531.log 2024-04-14T23:58:54.968 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.42935.log.gz 2024-04-14T23:58:54.969 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.49117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42607.log 2024-04-14T23:58:54.969 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49117.log.gz 2024-04-14T23:58:54.970 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45531.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41128.log 2024-04-14T23:58:54.970 INFO:teuthology.orchestra.run.smithi060.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45531.log.gz 2024-04-14T23:58:54.971 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50799.log 2024-04-14T23:58:54.971 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42607.log.gz 2024-04-14T23:58:54.972 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41460.log 2024-04-14T23:58:54.972 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41128.log.gz 2024-04-14T23:58:54.973 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.50799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50578.log 2024-04-14T23:58:54.974 INFO:teuthology.orchestra.run.smithi060.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50799.log.gz 2024-04-14T23:58:54.975 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41460.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41460.log.gz 2024-04-14T23:58:54.975 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41360.log 2024-04-14T23:58:54.976 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.50578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47703.log 2024-04-14T23:58:54.977 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50578.log.gz 2024-04-14T23:58:54.977 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44574.log 2024-04-14T23:58:54.977 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41360.log.gz 2024-04-14T23:58:54.978 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.47703.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49060.log 2024-04-14T23:58:54.979 INFO:teuthology.orchestra.run.smithi060.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47703.log.gz 2024-04-14T23:58:54.980 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.44574.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44574.log.gz 2024-04-14T23:58:54.980 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37277.log 2024-04-14T23:58:54.981 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.49060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41977.log 2024-04-14T23:58:54.981 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49060.log.gz 2024-04-14T23:58:54.982 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.37277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41713.log 2024-04-14T23:58:54.982 INFO:teuthology.orchestra.run.smithi060.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.37277.log.gz 2024-04-14T23:58:54.983 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52023.log 2024-04-14T23:58:54.983 INFO:teuthology.orchestra.run.smithi060.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41977.log.gz 2024-04-14T23:58:54.984 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45489.log 2024-04-14T23:58:54.985 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41713.log.gz 2024-04-14T23:58:54.986 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.52023.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52023.log.gzgzip 2024-04-14T23:58:54.986 INFO:teuthology.orchestra.run.smithi060.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.45794.log 2024-04-14T23:58:54.986 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47346.log 2024-04-14T23:58:54.987 INFO:teuthology.orchestra.run.smithi060.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.45489.log.gz 2024-04-14T23:58:54.988 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45363.log 2024-04-14T23:58:54.988 INFO:teuthology.orchestra.run.smithi060.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45794.log.gz 2024-04-14T23:58:54.989 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.47346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46106.log 2024-04-14T23:58:54.989 INFO:teuthology.orchestra.run.smithi060.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47346.log.gz 2024-04-14T23:58:54.990 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45363.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45363.log.gz 2024-04-14T23:58:54.990 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43271.log 2024-04-14T23:58:54.991 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.46106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43187.log 2024-04-14T23:58:54.992 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46106.log.gz 2024-04-14T23:58:54.992 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.43271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45836.log 2024-04-14T23:58:54.993 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43271.log.gz 2024-04-14T23:58:54.994 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.43187.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43187.log.gz 2024-04-14T23:58:54.994 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51062.log 2024-04-14T23:58:54.995 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45707.log 2024-04-14T23:58:54.996 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45836.log.gz 2024-04-14T23:58:54.996 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.51062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42847.log 2024-04-14T23:58:54.996 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51062.log.gz 2024-04-14T23:58:54.998 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44760.log 2024-04-14T23:58:54.998 INFO:teuthology.orchestra.run.smithi060.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45707.log.gz 2024-04-14T23:58:54.999 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42847.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.42847.log.gz 2024-04-14T23:58:54.999 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44280.log 2024-04-14T23:58:55.000 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.44760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48889.log 2024-04-14T23:58:55.001 INFO:teuthology.orchestra.run.smithi186.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-14T23:58:55.001 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44760.log.gz 2024-04-14T23:58:55.001 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.44280.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51206.log 2024-04-14T23:58:55.001 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44280.log.gz 2024-04-14T23:58:55.003 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.48889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41966.log 2024-04-14T23:58:55.003 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48889.log.gz 2024-04-14T23:58:55.004 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.51206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42893.log 2024-04-14T23:58:55.004 INFO:teuthology.orchestra.run.smithi060.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51206.log.gz 2024-04-14T23:58:55.005 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41966.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53533.log 2024-04-14T23:58:55.005 INFO:teuthology.orchestra.run.smithi060.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41966.log.gz 2024-04-14T23:58:55.006 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51855.log 2024-04-14T23:58:55.007 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.42893.log.gz 2024-04-14T23:58:55.008 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.53533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53138.log 2024-04-14T23:58:55.008 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53533.log.gz 2024-04-14T23:58:55.009 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.51855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52634.log 2024-04-14T23:58:55.009 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51855.log.gz 2024-04-14T23:58:55.010 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.53138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41671.log 2024-04-14T23:58:55.010 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.53138.log.gz 2024-04-14T23:58:55.011 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.52634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43355.log 2024-04-14T23:58:55.011 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.52634.log.gz 2024-04-14T23:58:55.012 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50891.log 2024-04-14T23:58:55.013 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41671.log.gz 2024-04-14T23:58:55.013 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.43355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41962.log 2024-04-14T23:58:55.014 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43355.log.gz 2024-04-14T23:58:55.015 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.50891.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.43918.log 2024-04-14T23:58:55.015 INFO:teuthology.orchestra.run.smithi060.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50891.log.gz 2024-04-14T23:58:55.016 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41962.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-14T23:58:55.016 INFO:teuthology.orchestra.run.smithi060.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41962.log.gz 2024-04-14T23:58:55.017 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.0.43918.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.43918.log.gz 2024-04-14T23:58:55.017 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51488.log 2024-04-14T23:58:55.018 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49548.log 2024-04-14T23:58:55.019 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.51488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46960.log 2024-04-14T23:58:55.020 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51488.log.gz 2024-04-14T23:58:55.020 INFO:teuthology.orchestra.run.smithi060.stderr: 89.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-14T23:58:55.021 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.49548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44322.log 2024-04-14T23:58:55.021 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49548.log.gz 2024-04-14T23:58:55.022 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.46960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41544.log 2024-04-14T23:58:55.023 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46960.log.gz 2024-04-14T23:58:55.023 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.44322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43397.log 2024-04-14T23:58:55.024 INFO:teuthology.orchestra.run.smithi060.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44322.log.gz 2024-04-14T23:58:55.025 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41544.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47661.log 2024-04-14T23:58:55.025 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41544.log.gz 2024-04-14T23:58:55.026 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.43397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51981.log 2024-04-14T23:58:55.026 INFO:teuthology.orchestra.run.smithi060.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.43397.log.gz 2024-04-14T23:58:55.027 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.47661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50935.log 2024-04-14T23:58:55.027 INFO:teuthology.orchestra.run.smithi060.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.47661.log.gz 2024-04-14T23:58:55.028 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.51981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43313.log 2024-04-14T23:58:55.029 INFO:teuthology.orchestra.run.smithi060.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51981.log.gz 2024-04-14T23:58:55.030 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.50935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49015.log 2024-04-14T23:58:55.030 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50935.log.gz 2024-04-14T23:58:55.031 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.43313.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43313.log.gz 2024-04-14T23:58:55.032 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46064.log 2024-04-14T23:58:55.032 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.49015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51708.log 2024-04-14T23:58:55.033 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49015.log.gz 2024-04-14T23:58:55.034 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.46064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53575.log 2024-04-14T23:58:55.034 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46064.log.gz 2024-04-14T23:58:55.035 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.51708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44490.log 2024-04-14T23:58:55.035 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51708.log.gz 2024-04-14T23:58:55.036 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.53575.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53575.log.gz 2024-04-14T23:58:55.036 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52592.log 2024-04-14T23:58:55.037 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.44490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41418.log 2024-04-14T23:58:55.038 INFO:teuthology.orchestra.run.smithi060.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44490.log.gz 2024-04-14T23:58:55.039 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.52592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37174.log 2024-04-14T23:58:55.039 INFO:teuthology.orchestra.run.smithi060.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52592.log.gz 2024-04-14T23:58:55.040 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41418.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.41418.log.gz 2024-04-14T23:58:55.040 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42309.log 2024-04-14T23:58:55.041 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.37174.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-14T23:58:55.042 INFO:teuthology.orchestra.run.smithi060.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.37174.log.gz 2024-04-14T23:58:55.042 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43019.log 2024-04-14T23:58:55.043 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.42309.log.gz 2024-04-14T23:58:55.044 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41172.log 2024-04-14T23:58:55.045 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.43019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44676.log 2024-04-14T23:58:55.045 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43019.log.gz 2024-04-14T23:58:55.046 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47577.log 2024-04-14T23:58:55.047 INFO:teuthology.orchestra.run.smithi060.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41172.log.gz 2024-04-14T23:58:55.048 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.44676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48931.log 2024-04-14T23:58:55.048 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44676.log.gz 2024-04-14T23:58:55.049 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.47577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47619.log 2024-04-14T23:58:55.049 INFO:teuthology.orchestra.run.smithi060.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.47577.log.gz 2024-04-14T23:58:55.050 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.48931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49506.log 2024-04-14T23:58:55.051 INFO:teuthology.orchestra.run.smithi060.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48931.log.gz 2024-04-14T23:58:55.052 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.47619.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47619.log.gz 2024-04-14T23:58:55.052 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52550.log 2024-04-14T23:58:55.053 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.49506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53804.log 2024-04-14T23:58:55.054 INFO:teuthology.orchestra.run.smithi060.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49506.log.gz 2024-04-14T23:58:55.055 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.52550.log: 88.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.52550.log.gz 2024-04-14T23:58:55.055 INFO:teuthology.orchestra.run.smithi060.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41967.log 2024-04-14T23:58:55.056 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.53804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50757.log 2024-04-14T23:58:55.056 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.53804.log.gz 2024-04-14T23:58:55.057 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45259.log 2024-04-14T23:58:55.057 INFO:teuthology.orchestra.run.smithi060.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.41967.log.gz 2024-04-14T23:58:55.059 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.50757.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50757.log.gz 2024-04-14T23:58:55.059 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53366.log 2024-04-14T23:58:55.060 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45259.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45259.log.gz 2024-04-14T23:58:55.060 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40970.log 2024-04-14T23:58:55.062 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.53366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44196.log 2024-04-14T23:58:55.062 INFO:teuthology.orchestra.run.smithi060.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.53366.log.gz 2024-04-14T23:58:55.063 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44238.log 2024-04-14T23:58:55.063 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40970.log.gz 2024-04-14T23:58:55.064 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.44196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41266.log 2024-04-14T23:58:55.064 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44196.log.gz 2024-04-14T23:58:55.065 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.44238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47815.log 2024-04-14T23:58:55.065 INFO:teuthology.orchestra.run.smithi060.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44238.log.gz 2024-04-14T23:58:55.071 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48763.log 2024-04-14T23:58:55.072 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41266.log.gz 2024-04-14T23:58:55.072 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.47815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43583.log 2024-04-14T23:58:55.072 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47815.log.gz 2024-04-14T23:58:55.072 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.48763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53012.log 2024-04-14T23:58:55.072 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48763.log.gz 2024-04-14T23:58:55.072 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.43583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46273.log 2024-04-14T23:58:55.072 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43583.log.gz 2024-04-14T23:58:55.073 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.53012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50849.log 2024-04-14T23:58:55.073 INFO:teuthology.orchestra.run.smithi060.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53012.log.gz 2024-04-14T23:58:55.074 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.46273.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43145.log 2024-04-14T23:58:55.074 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46273.log.gz 2024-04-14T23:58:55.074 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.50849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41755.log 2024-04-14T23:58:55.075 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50849.log.gz 2024-04-14T23:58:55.076 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.43145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45878.log 2024-04-14T23:58:55.076 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43145.log.gz 2024-04-14T23:58:55.077 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41755.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52928.log 2024-04-14T23:58:55.077 INFO:teuthology.orchestra.run.smithi060.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41755.log.gz 2024-04-14T23:58:55.079 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40923.log 2024-04-14T23:58:55.079 INFO:teuthology.orchestra.run.smithi060.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45878.log.gz 2024-04-14T23:58:55.080 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.52928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50978.log 2024-04-14T23:58:55.080 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52928.log.gz 2024-04-14T23:58:55.081 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40923.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-04-14T23:58:55.081 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.40923.log.gz 2024-04-14T23:58:55.082 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.50978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45303.log 2024-04-14T23:58:55.083 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.50978.log.gz 2024-04-14T23:58:55.084 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53324.log 2024-04-14T23:58:55.085 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45303.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45303.log.gz 2024-04-14T23:58:55.086 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50631.log 2024-04-14T23:58:55.087 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.53324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42304.log 2024-04-14T23:58:55.088 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.53324.log.gz 2024-04-14T23:58:55.088 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.50631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41978.log 2024-04-14T23:58:55.089 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50631.log.gz 2024-04-14T23:58:55.090 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45665.log 2024-04-14T23:58:55.090 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42304.log.gz 2024-04-14T23:58:55.091 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52326.log 2024-04-14T23:58:55.092 INFO:teuthology.orchestra.run.smithi060.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41978.log.gz 2024-04-14T23:58:55.093 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47517.log 2024-04-14T23:58:55.093 INFO:teuthology.orchestra.run.smithi060.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.45665.log.gz 2024-04-14T23:58:55.094 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.52326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52410.log 2024-04-14T23:58:55.095 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52326.log.gz 2024-04-14T23:58:55.096 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.47517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44112.log 2024-04-14T23:58:55.096 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47517.log.gz 2024-04-14T23:58:55.097 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.52410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45980.log 2024-04-14T23:58:55.098 INFO:teuthology.orchestra.run.smithi060.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.52410.log.gz 2024-04-14T23:58:55.099 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.44112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41586.log 2024-04-14T23:58:55.099 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44112.log.gz 2024-04-14T23:58:55.100 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51020.log 2024-04-14T23:58:55.100 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45980.log.gz 2024-04-14T23:58:55.102 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45405.log 2024-04-14T23:58:55.102 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41586.log.gz 2024-04-14T23:58:55.103 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.51020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40824.log 2024-04-14T23:58:55.104 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51020.log.gz 2024-04-14T23:58:55.105 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44802.log 2024-04-14T23:58:55.105 INFO:teuthology.orchestra.run.smithi060.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45405.log.gz 2024-04-14T23:58:55.106 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53282.log 2024-04-14T23:58:55.107 INFO:teuthology.orchestra.run.smithi060.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40824.log.gz 2024-04-14T23:58:55.108 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.44802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44364.log 2024-04-14T23:58:55.108 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44802.log.gz 2024-04-14T23:58:55.109 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.53282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43061.log 2024-04-14T23:58:55.109 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.53282.log.gz 2024-04-14T23:58:55.111 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.44364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52146.log 2024-04-14T23:58:55.111 INFO:teuthology.orchestra.run.smithi060.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.44364.log.gz 2024-04-14T23:58:55.112 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.43061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45752.log 2024-04-14T23:58:55.112 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43061.log.gz 2024-04-14T23:58:55.113 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.52146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46022.log 2024-04-14T23:58:55.114 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52146.log.gz 2024-04-14T23:58:55.115 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53451.log 2024-04-14T23:58:55.115 INFO:teuthology.orchestra.run.smithi060.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45752.log.gz 2024-04-14T23:58:55.116 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.46022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49338.log 2024-04-14T23:58:55.117 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46022.log.gz 2024-04-14T23:58:55.117 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.53451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52886.log 2024-04-14T23:58:55.118 INFO:teuthology.orchestra.run.smithi060.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.53451.log.gz 2024-04-14T23:58:55.119 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.49338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53096.log 2024-04-14T23:58:55.119 INFO:teuthology.orchestra.run.smithi060.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.49338.log.gz 2024-04-14T23:58:55.120 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.52886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51616.log 2024-04-14T23:58:55.121 INFO:teuthology.orchestra.run.smithi060.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52886.log.gz 2024-04-14T23:58:55.122 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.53096.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.e.log 2024-04-14T23:58:55.122 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53096.log.gz 2024-04-14T23:58:55.123 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.51616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44154.log 2024-04-14T23:58:55.124 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51616.log.gz 2024-04-14T23:58:55.125 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53054.log 2024-04-14T23:58:55.126 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.44154.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-04-14T23:58:55.127 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44154.log.gz 2024-04-14T23:58:55.128 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.53054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42646.log 2024-04-14T23:58:55.128 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53054.log.gz 2024-04-14T23:58:55.129 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-mgr.y.log: 93.4%gzip -- replaced with /var/log/ceph/ceph-mds.e.log.gz -5 2024-04-14T23:58:55.129 INFO:teuthology.orchestra.run.smithi060.stderr: --verbose -- /var/log/ceph/ceph-client.admin.47170.log 2024-04-14T23:58:55.130 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36919.log 2024-04-14T23:58:55.130 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42646.log.gz 2024-04-14T23:58:55.131 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.47170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44406.log 2024-04-14T23:58:55.131 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47170.log.gz 2024-04-14T23:58:55.133 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.36919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43229.log 2024-04-14T23:58:55.133 INFO:teuthology.orchestra.run.smithi060.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.36919.log.gz 2024-04-14T23:58:55.134 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.44406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47475.log 2024-04-14T23:58:55.134 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44406.log.gz 2024-04-14T23:58:55.135 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.43229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51104.log 2024-04-14T23:58:55.135 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43229.log.gz 2024-04-14T23:58:55.136 INFO:teuthology.orchestra.run.smithi186.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-04-14T23:58:55.137 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.47475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40709.log 2024-04-14T23:58:55.137 INFO:teuthology.orchestra.run.smithi060.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.47475.log.gz 2024-04-14T23:58:55.137 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.51104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41316.log 2024-04-14T23:58:55.138 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51104.log.gz 2024-04-14T23:58:55.139 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49380.log 2024-04-14T23:58:55.139 INFO:teuthology.orchestra.run.smithi060.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40709.log.gz 2024-04-14T23:58:55.140 INFO:teuthology.orchestra.run.smithi060.stderr: 94.2% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-04-14T23:58:55.140 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51750.log 2024-04-14T23:58:55.141 INFO:teuthology.orchestra.run.smithi060.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.41316.log.gz 2024-04-14T23:58:55.141 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.49380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41074.log 2024-04-14T23:58:55.142 INFO:teuthology.orchestra.run.smithi060.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49380.log.gz 2024-04-14T23:58:55.142 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.51750.log: 88.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.51750.log.gz 2024-04-14T23:58:55.143 INFO:teuthology.orchestra.run.smithi060.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.44632.log 2024-04-14T23:58:55.143 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41629.log 2024-04-14T23:58:55.144 INFO:teuthology.orchestra.run.smithi060.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41074.log.gz 2024-04-14T23:58:55.145 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.44632.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44632.log.gz 2024-04-14T23:58:55.145 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51574.log 2024-04-14T23:58:55.146 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41629.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51795.log 2024-04-14T23:58:55.147 INFO:teuthology.orchestra.run.smithi060.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.41629.log.gz 2024-04-14T23:58:55.147 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.51574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49209.log 87.7% 2024-04-14T23:58:55.148 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51574.log.gz 2024-04-14T23:58:55.149 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.51795.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51795.log.gz 2024-04-14T23:58:55.149 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-04-14T23:58:55.150 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.49209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43481.log 2024-04-14T23:58:55.150 INFO:teuthology.orchestra.run.smithi060.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49209.log.gz 2024-04-14T23:58:55.151 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41020.log 2024-04-14T23:58:55.152 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.43481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42588.log 2024-04-14T23:58:55.153 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41020.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.43481.log.gz 2024-04-14T23:58:55.154 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42314.log 2024-04-14T23:58:55.154 INFO:teuthology.orchestra.run.smithi060.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41020.log.gz 2024-04-14T23:58:55.155 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47128.log 2024-04-14T23:58:55.155 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.42588.log.gz 2024-04-14T23:58:55.157 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42314.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47086.log 2024-04-14T23:58:55.157 INFO:teuthology.orchestra.run.smithi060.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.42314.log.gz 2024-04-14T23:58:55.158 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.47128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45623.log 2024-04-14T23:58:55.158 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47128.log.gz 2024-04-14T23:58:55.159 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.47086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42299.log 2024-04-14T23:58:55.159 INFO:teuthology.orchestra.run.smithi060.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47086.log.gz 2024-04-14T23:58:55.161 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48847.log 2024-04-14T23:58:55.161 INFO:teuthology.orchestra.run.smithi060.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45623.log.gz 2024-04-14T23:58:55.162 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42299.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-14T23:58:55.162 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42299.log.gz 2024-04-14T23:58:55.163 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.48847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53408.log 2024-04-14T23:58:55.164 INFO:teuthology.orchestra.run.smithi060.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.48847.log.gz 2024-04-14T23:58:55.164 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-14T23:58:55.166 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.53408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51666.log 2024-04-14T23:58:55.166 INFO:teuthology.orchestra.run.smithi060.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53408.log.gz 2024-04-14T23:58:55.167 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-osd.2.log: 86.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53491.log 2024-04-14T23:58:55.167 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/ceph.log.gz 2024-04-14T23:58:55.169 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.51666.log: gzip 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51666.log.gz -5 2024-04-14T23:58:55.169 INFO:teuthology.orchestra.run.smithi060.stderr: --verbose -- /var/log/ceph/ceph-mds.c.log 2024-04-14T23:58:55.170 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.53491.log: gzip 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53491.log.gz -5 2024-04-14T23:58:55.170 INFO:teuthology.orchestra.run.smithi060.stderr: --verbose -- /var/log/ceph/ceph-client.admin.49251.log 2024-04-14T23:58:55.172 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51404.log 2024-04-14T23:58:55.173 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.49251.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49251.log.gz 2024-04-14T23:58:55.174 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.51404.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51404.log.gz 2024-04-14T23:58:55.175 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43439.log 2024-04-14T23:58:55.176 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52284.log 2024-04-14T23:58:55.178 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.43439.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43439.log.gz 2024-04-14T23:58:55.179 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.52284.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52284.log.gz 2024-04-14T23:58:55.180 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52802.log 2024-04-14T23:58:55.181 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42297.log 2024-04-14T23:58:55.183 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.52802.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52802.log.gz 2024-04-14T23:58:55.184 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42297.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.42297.log.gz 2024-04-14T23:58:55.185 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52844.log 2024-04-14T23:58:55.186 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42312.log 2024-04-14T23:58:55.188 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.52844.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.52844.log.gz 2024-04-14T23:58:55.189 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42312.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42312.log.gz 2024-04-14T23:58:55.190 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53617.log 2024-04-14T23:58:55.191 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49590.log 2024-04-14T23:58:55.193 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.53617.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53617.log.gz 2024-04-14T23:58:55.194 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.49590.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.49590.log.gz 2024-04-14T23:58:55.195 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42315.log 2024-04-14T23:58:55.196 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41502.log 2024-04-14T23:58:55.197 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42315.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42315.log.gz 2024-04-14T23:58:55.199 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41502.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41502.log.gz 2024-04-14T23:58:55.200 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52760.log 2024-04-14T23:58:55.201 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42644.log 2024-04-14T23:58:55.203 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.52760.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.52760.log.gz 2024-04-14T23:58:55.204 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42644.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42644.log.gz 2024-04-14T23:58:55.204 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48805.log 2024-04-14T23:58:55.206 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50673.log 2024-04-14T23:58:55.207 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.48805.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48805.log.gz 2024-04-14T23:58:55.209 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.50673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42593.log 2024-04-14T23:58:55.210 INFO:teuthology.orchestra.run.smithi060.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50673.log.gz 2024-04-14T23:58:55.210 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42649.log 2024-04-14T23:58:55.212 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42593.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42593.log.gz 2024-04-14T23:58:55.212 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51267.log 2024-04-14T23:58:55.212 INFO:teuthology.orchestra.run.smithi186.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-04-14T23:58:55.215 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42649.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42649.log.gz 2024-04-14T23:58:55.215 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.51267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49422.log 2024-04-14T23:58:55.215 INFO:teuthology.orchestra.run.smithi060.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51267.log.gz 2024-04-14T23:58:55.217 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.49422.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49422.log.gz 2024-04-14T23:58:55.218 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42650.log 2024-04-14T23:58:55.219 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44532.log 2024-04-14T23:58:55.220 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42650.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42650.log.gz 2024-04-14T23:58:55.222 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.44532.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44532.log.gz 2024-04-14T23:58:55.222 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47002.log 2024-04-14T23:58:55.224 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42313.log 2024-04-14T23:58:55.225 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.47002.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47002.log.gz 2024-04-14T23:58:55.227 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42313.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42313.log.gz 2024-04-14T23:58:55.228 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42651.log 2024-04-14T23:58:55.229 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48973.log 2024-04-14T23:58:55.230 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42651.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.42651.log.gz 2024-04-14T23:58:55.232 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.48973.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48973.log.gz 2024-04-14T23:58:55.232 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47391.log 2024-04-14T23:58:55.234 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49167.log 2024-04-14T23:58:55.235 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.47391.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47391.log.gz 2024-04-14T23:58:55.236 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.49167.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49167.log.gz 2024-04-14T23:58:55.237 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41797.log 2024-04-14T23:58:55.238 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-04-14T23:58:55.240 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41797.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41797.log.gz 2024-04-14T23:58:55.242 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50518.log 2024-04-14T23:58:55.246 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50476.log 2024-04-14T23:58:55.246 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.50518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52970.log 2024-04-14T23:58:55.246 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50518.log.gz 2024-04-14T23:58:55.247 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.50476.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50476.log.gz 2024-04-14T23:58:55.247 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52676.log 2024-04-14T23:58:55.249 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.52970.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.52970.log.gz 2024-04-14T23:58:55.250 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45938.log 2024-04-14T23:58:55.252 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.52676.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52676.log.gz 2024-04-14T23:58:55.255 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47262.log 2024-04-14T23:58:55.256 INFO:teuthology.orchestra.run.smithi060.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.45938.log.gz 2024-04-14T23:58:55.259 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.47262.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47262.log.gz 2024-04-14T23:58:55.503 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-04-14T23:58:57.340 INFO:teuthology.orchestra.run.smithi186.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-04-14T23:58:57.405 INFO:teuthology.orchestra.run.smithi060.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-14T23:58:57.757 INFO:teuthology.orchestra.run.smithi186.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-04-14T23:58:57.888 INFO:teuthology.orchestra.run.smithi186.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-04-14T23:58:58.326 INFO:teuthology.orchestra.run.smithi060.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-04-14T23:58:58.485 INFO:teuthology.orchestra.run.smithi186.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-04-14T23:58:58.488 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-14T23:58:58.488 INFO:teuthology.orchestra.run.smithi186.stderr:real 0m3.617s 2024-04-14T23:58:58.488 INFO:teuthology.orchestra.run.smithi186.stderr:user 0m12.168s 2024-04-14T23:58:58.488 INFO:teuthology.orchestra.run.smithi186.stderr:sys 0m0.475s 2024-04-14T23:58:58.820 INFO:teuthology.orchestra.run.smithi060.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-14T23:58:59.087 INFO:teuthology.orchestra.run.smithi060.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-14T23:59:06.147 INFO:teuthology.orchestra.run.smithi060.stderr: 93.9% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-04-14T23:59:09.835 INFO:teuthology.orchestra.run.smithi060.stderr: 93.5% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-04-14T23:59:09.840 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-04-14T23:59:09.840 INFO:teuthology.orchestra.run.smithi060.stderr:real 0m14.966s 2024-04-14T23:59:09.841 INFO:teuthology.orchestra.run.smithi060.stderr:user 0m36.836s 2024-04-14T23:59:09.841 INFO:teuthology.orchestra.run.smithi060.stderr:sys 0m3.328s 2024-04-14T23:59:09.842 INFO:tasks.ceph:Archiving logs... 2024-04-14T23:59:09.842 DEBUG:teuthology.misc:Transferring archived files from smithi060:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-14_21:24:03-fs-squid-distro-default-smithi/7656585/remote/smithi060/log 2024-04-14T23:59:09.843 DEBUG:teuthology.orchestra.run.smithi060:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-14T23:59:15.039 DEBUG:teuthology.misc:Transferring archived files from smithi186:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-14_21:24:03-fs-squid-distro-default-smithi/7656585/remote/smithi186/log 2024-04-14T23:59:15.040 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-14T23:59:16.951 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-14T23:59:16.961 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-14T23:59:16.961 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-14T23:59:16.969 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-14T23:59:17.119 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-14T23:59:17.129 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-14T23:59:17.129 DEBUG:teuthology.orchestra.run.smithi060:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-14T23:59:17.137 DEBUG:teuthology.orchestra.run.smithi186:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-14T23:59:17.154 INFO:teuthology.orchestra.run.smithi186.stderr:bash: line 1: ntpq: command not found 2024-04-14T23:59:17.159 INFO:teuthology.orchestra.run.smithi186.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-14T23:59:17.159 INFO:teuthology.orchestra.run.smithi186.stdout:=============================================================================== 2024-04-14T23:59:17.160 INFO:teuthology.orchestra.run.smithi186.stdout:^* hv01.front.sepia.ceph.com 3 6 377 56 +778us[ +936us] +/- 42ms 2024-04-14T23:59:17.160 INFO:teuthology.orchestra.run.smithi186.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 54 -929us[ -929us] +/- 72ms 2024-04-14T23:59:17.160 INFO:teuthology.orchestra.run.smithi186.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 56 -288us[ -130us] +/- 69ms 2024-04-14T23:59:17.160 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-14T23:59:17.190 INFO:teuthology.orchestra.run.smithi060.stderr:bash: line 1: ntpq: command not found 2024-04-14T23:59:17.201 INFO:teuthology.orchestra.run.smithi060.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-14T23:59:17.202 INFO:teuthology.orchestra.run.smithi060.stdout:=============================================================================== 2024-04-14T23:59:17.202 INFO:teuthology.orchestra.run.smithi060.stdout:^* hv01.front.sepia.ceph.com 3 6 377 55 +850us[ +965us] +/- 42ms 2024-04-14T23:59:17.202 INFO:teuthology.orchestra.run.smithi060.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 54 -873us[ -873us] +/- 72ms 2024-04-14T23:59:17.202 INFO:teuthology.orchestra.run.smithi060.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 56 -215us[ -100us] +/- 70ms 2024-04-14T23:59:17.202 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-14T23:59:17.203 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-14T23:59:17.213 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-14T23:59:17.213 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-14T23:59:17.235 DEBUG:teuthology.orchestra.run.smithi060:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-14T23:59:17.477 DEBUG:teuthology.orchestra.run.smithi186:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-14T23:59:17.550 DEBUG:teuthology.orchestra.run.smithi060:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-14T23:59:17.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T23:59:17.664 DEBUG:teuthology.orchestra.run.smithi186:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-14T23:59:17.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T23:59:17.697 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-14T23:59:17.709 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-14T23:59:17.751 INFO:teuthology.task.internal:Duration was 1213.053452 seconds 2024-04-14T23:59:17.752 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-14T23:59:17.785 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-14T23:59:17.785 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-14T23:59:17.791 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-14T23:59:17.838 INFO:teuthology.orchestra.run.smithi186.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-14T23:59:17.966 INFO:teuthology.orchestra.run.smithi060.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-14T23:59:18.447 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-14T23:59:18.448 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi060.front.sepia.ceph.com 2024-04-14T23:59:18.448 DEBUG:teuthology.orchestra.run.smithi060:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-14T23:59:18.517 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi186.front.sepia.ceph.com 2024-04-14T23:59:18.517 DEBUG:teuthology.orchestra.run.smithi186:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-14T23:59:18.562 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-14T23:59:18.562 DEBUG:teuthology.orchestra.run.smithi060:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-14T23:59:18.570 DEBUG:teuthology.orchestra.run.smithi186:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-14T23:59:18.673 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-14T23:59:18.673 DEBUG:teuthology.orchestra.run.smithi060:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-14T23:59:18.680 DEBUG:teuthology.orchestra.run.smithi186:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-14T23:59:18.881 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-14T23:59:18.894 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-14T23:59:18.894 DEBUG:teuthology.orchestra.run.smithi060:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-14T23:59:18.990 DEBUG:teuthology.orchestra.run.smithi186:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-14T23:59:19.018 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-14T23:59:19.033 DEBUG:teuthology.orchestra.run.smithi060:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-14T23:59:19.041 DEBUG:teuthology.orchestra.run.smithi186:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-14T23:59:19.084 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern = core 2024-04-14T23:59:19.152 INFO:teuthology.orchestra.run.smithi060.stdout:kernel.core_pattern = core 2024-04-14T23:59:19.207 DEBUG:teuthology.orchestra.run.smithi060:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-14T23:59:19.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T23:59:19.261 DEBUG:teuthology.orchestra.run.smithi186:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-14T23:59:19.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T23:59:19.275 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-14T23:59:19.289 INFO:teuthology.task.internal:Transferring archived files... 2024-04-14T23:59:19.290 DEBUG:teuthology.misc:Transferring archived files from smithi060:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-14_21:24:03-fs-squid-distro-default-smithi/7656585/remote/smithi060 2024-04-14T23:59:19.290 DEBUG:teuthology.orchestra.run.smithi060:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-14T23:59:19.401 DEBUG:teuthology.misc:Transferring archived files from smithi186:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-14_21:24:03-fs-squid-distro-default-smithi/7656585/remote/smithi186 2024-04-14T23:59:19.401 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-14T23:59:19.437 INFO:teuthology.task.internal:Removing archive directory... 2024-04-14T23:59:19.438 DEBUG:teuthology.orchestra.run.smithi060:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-14T23:59:19.449 DEBUG:teuthology.orchestra.run.smithi186:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-14T23:59:19.500 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-14T23:59:19.513 INFO:teuthology.task.internal:Not uploading archives. 2024-04-14T23:59:19.513 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-14T23:59:19.587 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-14T23:59:19.587 DEBUG:teuthology.orchestra.run.smithi060:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-14T23:59:19.596 DEBUG:teuthology.orchestra.run.smithi186:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-14T23:59:19.611 INFO:teuthology.orchestra.run.smithi186.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 14 23:59 /home/ubuntu/cephtest 2024-04-14T23:59:19.647 INFO:teuthology.orchestra.run.smithi060.stdout: 393229 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 14 23:59 /home/ubuntu/cephtest 2024-04-14T23:59:19.649 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-14T23:59:19.661 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-14T23:59:19.727 INFO:teuthology.nuke:Checking targets against current locks 2024-04-14T23:59:19.754 DEBUG:teuthology.nuke:shortname: smithi060 2024-04-14T23:59:19.755 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-14T23:59:19.780 DEBUG:teuthology.nuke:shortname: smithi186 2024-04-14T23:59:19.780 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-14T23:59:19.787 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi060.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-14_21:24:03-fs-squid-distro-default-smithi/7656585', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-14 23:29:25.731194', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFjV/pqyaEiSS62epvH8tgHmy+F/PJpE6WM74CJKJ2mX8YiPrJ7j+BVMPex/LpLCEZB9HKVb/OF6pAfvG9ajCuE='} 2024-04-14T23:59:19.799 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi186.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-14_21:24:03-fs-squid-distro-default-smithi/7656585', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-14 23:29:25.733090', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLQ6oA9Y2BUpihFO/opwukGeh9/GE1Y47E4ZKJRZd++cP5yuL+qzcFkgQ6J4vfQSLvhNz3d0ALuw0/VE76drTXE='} 2024-04-14T23:59:19.835 INFO:teuthology.orchestra.console.smithi060:Power off 2024-04-14T23:59:19.836 DEBUG:teuthology.orchestra.console.smithi060:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-14T23:59:19.855 INFO:teuthology.orchestra.console.smithi186:Power off 2024-04-14T23:59:19.856 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-14T23:59:19.869 DEBUG:teuthology.orchestra.console.smithi060:power off output: Chassis Power Control: Down/Off 2024-04-14T23:59:19.869 DEBUG:teuthology.orchestra.console.smithi060:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-14T23:59:19.882 DEBUG:teuthology.orchestra.console.smithi186:power off output: Chassis Power Control: Down/Off 2024-04-14T23:59:19.882 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-14T23:59:19.894 DEBUG:teuthology.orchestra.console.smithi060:check power output: Chassis Power is on 2024-04-14T23:59:19.905 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is on 2024-04-14T23:59:23.895 DEBUG:teuthology.orchestra.console.smithi060:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-14T23:59:23.906 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-14T23:59:24.006 DEBUG:teuthology.orchestra.console.smithi060:check power output: Chassis Power is on 2024-04-14T23:59:24.017 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is on 2024-04-14T23:59:28.010 DEBUG:teuthology.orchestra.console.smithi060:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-14T23:59:28.019 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-14T23:59:28.122 DEBUG:teuthology.orchestra.console.smithi060:check power output: Chassis Power is off 2024-04-14T23:59:28.131 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is on 2024-04-14T23:59:28.224 INFO:teuthology.orchestra.console.smithi060:Power off completed 2024-04-14T23:59:32.134 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-14T23:59:32.247 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is off 2024-04-14T23:59:32.348 INFO:teuthology.orchestra.console.smithi186:Power off completed 2024-04-14T23:59:32.455 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 description: fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} duration: 1213.0534522533417 failure_reason: 'Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi060 with status 128: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a9a752df26c63acad72e1b3569fd79a515ca0765 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh''' flavor: default owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=85164b7883984711a8e74c5d09decf90 status: fail success: false 2024-04-14T23:59:32.455 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-14T23:59:32.553 INFO:teuthology.run:FAIL